http://arxiv.org/abs/2311.16577
Efficient Key-Based Adversarial Defense for ImageNet by Using Pre-trained Model. (98%)
AprilPyone MaungMaung; Isao Echizen; Hitoshi Kiya
  In this paper, we propose key-based defense model proliferation by leveraging
pre-trained models and utilizing recent efficient fine-tuning techniques on
ImageNet-1k classification. First, we stress that deploying key-based models on
edge devices is feasible with the latest model deployment advancements, such as
Apple CoreML, although the mainstream enterprise edge artificial intelligence
(Edge AI) has been focused on the Cloud. Then, we point out that the previous
key-based defense on on-device image classification is impractical for two
reasons: (1) training many classifiers from scratch is not feasible, and (2)
key-based defenses still need to be thoroughly tested on large datasets like
ImageNet. To this end, we propose to leverage pre-trained models and utilize
efficient fine-tuning techniques to proliferate key-based models even on
limited computing resources. Experiments were carried out on the ImageNet-1k
dataset using adaptive and non-adaptive attacks. The results show that our
proposed fine-tuned key-based models achieve a superior classification accuracy
(more than 10% increase) compared to the previous key-based models on
classifying clean and adversarial examples.


http://arxiv.org/abs/2311.16833
1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness. (13%)
Bernd Prach; Fabio Brau; Giorgio Buttazzo; Christoph H. Lampert
  The robustness of neural networks against input perturbations with bounded
magnitude represents a serious concern in the deployment of deep learning
models in safety-critical systems. Recently, the scientific community has
focused on enhancing certifiable robustness guarantees by crafting 1-Lipschitz
neural networks that leverage Lipschitz bounded dense and convolutional layers.
Although different methods have been proposed in the literature to achieve this
goal, understanding the performance of such methods is not straightforward,
since different metrics can be relevant (e.g., training time, memory usage,
accuracy, certifiable robustness) for different applications. For this reason,
this work provides a thorough theoretical and empirical comparison between
methods by evaluating them in terms of memory usage, speed, and certifiable
robust accuracy. The paper also provides some guidelines and recommendations to
support the user in selecting the methods that work best depending on the
available resources. We provide code at
https://github.com/berndprach/1LipschitzLayersCompared.


http://arxiv.org/abs/2311.17035
Scalable Extraction of Training Data from (Production) Language Models. (10%)
Milad Nasr; Nicholas Carlini; Jonathan Hayase; Matthew Jagielski; A. Feder Cooper; Daphne Ippolito; Christopher A. Choquette-Choo; Eric Wallace; Florian Tramèr; Katherine Lee
  This paper studies extractable memorization: training data that an adversary
can efficiently extract by querying a machine learning model without prior
knowledge of the training dataset. We show an adversary can extract gigabytes
of training data from open-source language models like Pythia or GPT-Neo,
semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing
techniques from the literature suffice to attack unaligned models; in order to
attack the aligned ChatGPT, we develop a new divergence attack that causes the
model to diverge from its chatbot-style generations and emit training data at a
rate 150x higher than when behaving properly. Our methods show practical
attacks can recover far more data than previously thought, and reveal that
current alignment techniques do not eliminate memorization.


http://arxiv.org/abs/2311.16661
Cooperative Abnormal Node Detection with Adversary Resistance: A Probabilistic Approach. (10%)
Yingying Huangfu; Tian Bai
  This paper presents a novel probabilistic detection scheme called Cooperative
Statistical Detection (CSD) for abnormal node detection while defending against
adversarial attacks in cluster-tree networks. The CSD performs a two-phase
process: 1) designing a likelihood ratio test (LRT) for a non-root node at its
children from the perspective of packet loss; 2) making an overall decision at
the root node based on the aggregated detection data of the nodes over tree
branches. In most adversarial scenarios, malicious children knowing the
detection policy can generate falsified data to protect the abnormal parent
from being detected or frame its normal parent as an anomalous node. To resolve
this issue, a modified Z-score-based falsification-resistant mechanism is
presented in the CSD to remove untrustworthy information. Through theoretical
analysis, we show that the LRT-based method achieves perfect detection, i.e.,
both the false alarm and missed detection probabilities decay exponentially to
zero. Furthermore, the optimal removal threshold of the modified Z-score method
is derived for falsifications with uncertain strategies and guarantees perfect
detection of the CSD. As our simulation results show, the CSD approach is
robust to falsifications and can rapidly reach $99\%$ detection accuracy, even
in existing adversarial scenarios, which outperforms state-of-the-art
technology.


http://arxiv.org/abs/2311.16526
On robust overfitting: adversarial training induced distribution matters. (1%)
Runzhi Tian; Yongyi Mao
  Adversarial training may be regarded as standard training with a modified
loss function. But its generalization error appears much larger than standard
training under standard loss. This phenomenon, known as robust overfitting, has
attracted significant research attention and remains largely as a mystery. In
this paper, we first show empirically that robust overfitting correlates with
the increasing generalization difficulty of the perturbation-induced
distributions along the trajectory of adversarial training (specifically
PGD-based adversarial training). We then provide a novel upper bound for
generalization error with respect to the perturbation-induced distributions, in
which a notion of the perturbation operator, referred to "local dispersion",
plays an important role.


http://arxiv.org/abs/2311.16681
Understanding the (Extra-)Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations. (1%)
Maximilian Dreyer; Reduan Achtibat; Wojciech Samek; Sebastian Lapuschkin
  Ensuring both transparency and safety is critical when deploying Deep Neural
Networks (DNNs) in high-risk applications, such as medicine. The field of
explainable AI (XAI) has proposed various methods to comprehend the
decision-making processes of opaque DNNs. However, only few XAI methods are
suitable of ensuring safety in practice as they heavily rely on repeated
labor-intensive and possibly biased human assessment. In this work, we present
a novel post-hoc concept-based XAI framework that conveys besides instance-wise
(local) also class-wise (global) decision-making strategies via prototypes.
What sets our approach apart is the combination of local and global strategies,
enabling a clearer understanding of the (dis-)similarities in model decisions
compared to the expected (prototypical) concept use, ultimately reducing the
dependence on human long-term assessment. Quantifying the deviation from
prototypical behavior not only allows to associate predictions with specific
model sub-strategies but also to detect outlier behavior. As such, our approach
constitutes an intuitive and explainable tool for model validation. We
demonstrate the effectiveness of our approach in identifying
out-of-distribution samples, spurious model behavior and data quality issues
across three datasets (ImageNet, CUB-200, and CIFAR-10) utilizing VGG, ResNet,
and EfficientNet architectures. Code is available on
https://github.com/maxdreyer/pcx.


http://arxiv.org/abs/2311.16478
RetouchUAA: Unconstrained Adversarial Attack via Image Retouching. (99%)
Mengda Xie; Yiling He; Meie Fang
  Deep Neural Networks (DNNs) are susceptible to adversarial examples.
Conventional attacks generate controlled noise-like perturbations that fail to
reflect real-world scenarios and hard to interpretable. In contrast, recent
unconstrained attacks mimic natural image transformations occurring in the real
world for perceptible but inconspicuous attacks, yet compromise realism due to
neglect of image post-processing and uncontrolled attack direction. In this
paper, we propose RetouchUAA, an unconstrained attack that exploits a real-life
perturbation: image retouching styles, highlighting its potential threat to
DNNs. Compared to existing attacks, RetouchUAA offers several notable
advantages. Firstly, RetouchUAA excels in generating interpretable and
realistic perturbations through two key designs: the image retouching attack
framework and the retouching style guidance module. The former custom-designed
human-interpretability retouching framework for adversarial attack by
linearizing images while modelling the local processing and retouching
decision-making in human retouching behaviour, provides an explicit and
reasonable pipeline for understanding the robustness of DNNs against
retouching. The latter guides the adversarial image towards standard retouching
styles, thereby ensuring its realism. Secondly, attributed to the design of the
retouching decision regularization and the persistent attack strategy,
RetouchUAA also exhibits outstanding attack capability and defense robustness,
posing a heavy threat to DNNs. Experiments on ImageNet and Place365 reveal that
RetouchUAA achieves nearly 100\% white-box attack success against three DNNs,
while achieving a better trade-off between image naturalness, transferability
and defense robustness than baseline attacks.


http://arxiv.org/abs/2311.15994
Adversaral Doodles: Interpretable and Human-drawable Attacks Provide Describable Insights. (99%)
Ryoya Nara; Yusuke Matsui
  DNN-based image classification models are susceptible to adversarial attacks.
Most previous adversarial attacks do not focus on the interpretability of the
generated adversarial examples, and we cannot gain insights into the mechanism
of the target classifier from the attacks. Therefore, we propose Adversarial
Doodles, which have interpretable shapes. We optimize black b\'ezier curves to
fool the target classifier by overlaying them onto the input image. By
introducing random perspective transformation and regularizing the doodled
area, we obtain compact attacks that cause misclassification even when humans
replicate them by hand. Adversarial doodles provide describable and intriguing
insights into the relationship between our attacks and the classifier's output.
We utilize adversarial doodles and discover the bias inherent in the target
classifier, such as "We add two strokes on its head, a triangle onto its body,
and two lines inside the triangle on a bird image. Then, the classifier
misclassifies the image as a butterfly."


http://arxiv.org/abs/2311.15551
Instruct2Attack: Language-Guided Semantic Adversarial Attacks. (98%)
Jiang Liu; Chen Wei; Yuxiang Guo; Heng Yu; Alan Yuille; Soheil Feizi; Chun Pong Lau; Rama Chellappa
  We propose Instruct2Attack (I2A), a language-guided semantic attack that
generates semantically meaningful perturbations according to free-form language
instructions. We make use of state-of-the-art latent diffusion models, where we
adversarially guide the reverse diffusion process to search for an adversarial
latent code conditioned on the input image and text instruction. Compared to
existing noise-based and semantic attacks, I2A generates more natural and
diverse adversarial examples while providing better controllability and
interpretability. We further automate the attack process with GPT-4 to generate
diverse image-specific text instructions. We show that I2A can successfully
break state-of-the-art deep neural networks even under strong adversarial
defenses, and demonstrate great transferability among a variety of network
architectures.


http://arxiv.org/abs/2311.16445
CLAP: Contrastive Learning with Augmented Prompts for Robustness on Pretrained Vision-Language Models. (95%)
Yichao Cai; Yuhang Liu; Zhen Zhang; Javen Qinfeng Shi
  Contrastive vision-language models, e.g., CLIP, have garnered substantial
attention for their exceptional generalization capabilities. However, their
robustness to perturbations has ignited concerns. Existing strategies typically
reinforce their resilience against adversarial examples by enabling the image
encoder to "see" these perturbed examples, often necessitating a complete
retraining of the image encoder on both natural and adversarial samples. In
this study, we propose a new method to enhance robustness solely through text
augmentation, eliminating the need for retraining the image encoder on
adversarial examples. Our motivation arises from the realization that text and
image data inherently occupy a shared latent space, comprising latent content
variables and style variables. This insight suggests the feasibility of
learning to disentangle these latent content variables using text data
exclusively. To accomplish this, we introduce an effective text augmentation
method that focuses on modifying the style while preserving the content in the
text data. By changing the style part of the text data, we empower the text
encoder to emphasize latent content variables, ultimately enhancing the
robustness of vision-language models. Our experiments across various datasets
demonstrate substantial improvements in the robustness of the pre-trained CLIP
model.


http://arxiv.org/abs/2311.16065
A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective. (50%)
Xianghua Xie; Chen Hu; Hanchi Ren; Jingjing Deng
  This review paper takes a comprehensive look at malicious attacks against FL,
categorizing them from new perspectives on attack origins and targets, and
providing insights into their methodology and impact. In this survey, we focus
on threat models targeting the learning process of FL systems. Based on the
source and target of the attack, we categorize existing threat models into four
types, Data to Model (D2M), Model to Data (M2D), Model to Model (M2M) and
composite attacks. For each attack type, we discuss the defense strategies
proposed, highlighting their effectiveness, assumptions and potential areas for
improvement. Defense strategies have evolved from using a singular metric to
excluding malicious clients, to employing a multifaceted approach examining
client models at various phases. In this survey paper, our research indicates
that the to-learn data, the learning gradients, and the learned model at
different stages all can be manipulated to initiate malicious attacks that
range from undermining model performance, reconstructing private local data,
and to inserting backdoors. We have also seen these threat are becoming more
insidious. While earlier studies typically amplified malicious gradients,
recent endeavors subtly alter the least significant weights in local models to
bypass defense measures. This literature review provides a holistic
understanding of the current FL threat landscape and highlights the importance
of developing robust, efficient, and privacy-preserving defenses to ensure the
safe and trusted adoption of FL in real-world applications.


http://arxiv.org/abs/2311.16460
Threshold Breaker: Can Counter-Based RowHammer Prevention Mechanisms Truly Safeguard DRAM? (31%)
Ranyang Zhou; Jacqueline Liu; Sabbir Ahmed; Nakul Kochar; Adnan Siraj Rakin; Shaahin Angizi
  This paper challenges the existing victim-focused counter-based RowHammer
detection mechanisms by experimentally demonstrating a novel multi-sided fault
injection attack technique called Threshold Breaker. This mechanism can
effectively bypass the most advanced counter-based defense mechanisms by
soft-attacking the rows at a farther physical distance from the target rows.
While no prior work has demonstrated the effect of such an attack, our work
closes this gap by systematically testing 128 real commercial DDR4 DRAM
products and reveals that the Threshold Breaker affects various chips from
major DRAM manufacturers. As a case study, we compare the performance
efficiency between our mechanism and a well-known double-sided attack by
performing adversarial weight attacks on a modern Deep Neural Network (DNN).
The results demonstrate that the Threshold Breaker can deliberately deplete the
intelligence of the targeted DNN system while DRAM is fully protected.


http://arxiv.org/abs/2311.15894
Distributed Attacks over Federated Reinforcement Learning-enabled Cell Sleep Control. (22%)
Han Zhang; Hao Zhou; Medhat Elsayed; Majid Bavand; Raimundas Gaigalas; Yigit Ozcan; Melike Erol-Kantarci
  Federated learning (FL) is particularly useful in wireless networks due to
its distributed implementation and privacy-preserving features. However, as a
distributed learning system, FL can be vulnerable to malicious attacks from
both internal and external sources. Our work aims to investigate the attack
models in a FL-enabled wireless networks. Specifically, we consider a cell
sleep control scenario, and apply federated reinforcement learning to improve
energy-efficiency. We design three attacks, namely free rider attacks,
Byzantine data poisoning attacks and backdoor attacks. The simulation results
show that the designed attacks can degrade the network performance and lead to
lower energy-efficiency. Moreover, we also explore possible ways to mitigate
the above attacks. We design a defense model called refined-Krum to defend
against attacks by enabling a secure aggregation on the global server. The
proposed refined- Krum scheme outperforms the existing Krum scheme and can
effectively prevent wireless networks from malicious attacks, improving the
system energy-efficiency performance.


http://arxiv.org/abs/2311.16383
"Do Users fall for Real Adversarial Phishing?" Investigating the Human response to Evasive Webpages. (15%)
Ajka Draganovic; Savino Dambra; Javier Aldana Iuit; Kevin Roundy; Giovanni Apruzzese
  Phishing websites are everywhere, and countermeasures based on static
blocklists cannot cope with such a threat. To address this problem,
state-of-the-art solutions entail the application of machine learning (ML) to
detect phishing websites by checking if they visually resemble webpages of
well-known brands. These techniques have achieved promising results in research
and, consequently, some security companies began to deploy them also in their
phishing detection systems (PDS). However, ML methods are not perfect and some
samples are bound to bypass even production-grade PDS.
  In this paper, we scrutinize whether 'genuine phishing websites' that evade
'commercial ML-based PDS' represent a problem "in reality". Although nobody
likes landing on a phishing webpage, a false negative may not lead to serious
consequences if the users (i.e., the actual target of phishing) can recognize
that "something is phishy". Practically, we carry out the first user-study
(N=126) wherein we assess whether unsuspecting users (having diverse
backgrounds) are deceived by 'adversarial' phishing webpages that evaded a real
PDS. We found that some well-crafted adversarial webpages can trick most
participants (even IT experts), albeit others are easily recognized by most
users. Our study is relevant for practitioners, since it allows prioritizing
phishing webpages that simultaneously fool (i) machines and (ii) humans --
i.e., their intended targets.


http://arxiv.org/abs/2311.16101
How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs. (12%)
Haoqin Tu; Chenhang Cui; Zijun Wang; Yiyang Zhou; Bingchen Zhao; Junlin Han; Wangchunshu Zhou; Huaxiu Yao; Cihang Xie
  This work focuses on the potential of Vision LLMs (VLLMs) in visual
reasoning. Different from prior studies, we shift our focus from evaluating
standard performance to introducing a comprehensive safety evaluation suite,
covering both out-of-distribution (OOD) generalization and adversarial
robustness. For the OOD evaluation, we present two novel VQA datasets, each
with one variant, designed to test model performance under challenging
conditions. In exploring adversarial robustness, we propose a straightforward
attack strategy for misleading VLLMs to produce visual-unrelated responses.
Moreover, we assess the efficacy of two jailbreaking strategies, targeting
either the vision or language component of VLLMs. Our evaluation of 21 diverse
models, ranging from open-source VLLMs to GPT-4V, yields interesting
observations: 1) Current VLLMs struggle with OOD texts but not images, unless
the visual information is limited; and 2) These VLLMs can be easily misled by
deceiving vision encoders only, and their vision-language training often
compromise safety protocols. We release this safety evaluation suite at
https://github.com/UCSC-VLAA/vllm-safety-benchmark.


http://arxiv.org/abs/2311.15999
Microarchitectural Security of AWS Firecracker VMM for Serverless Cloud Platforms. (1%)
Zane Worcester Polytechnic Institute Weissman; Thore University of Lübeck Tiemann; Thomas University of Lübeck Eisenbarth; Berk Worcester Polytechnic Institute Sunar
  Firecracker is a virtual machine manager (VMM) built by Amazon Web Services
(AWS) for serverless cloud platforms, services that run code for end users on a
per-task basis, automatically managing server infrastructure. Firecracker
provides fast and lightweight VMs and promises a combination of the speed of
containers, typically used to isolate small tasks, and the security of VMs,
which tend to provide greater isolation at the cost of performance. This
combination of security and efficiency, AWS claims, makes it not only possible
but safe to run thousands of user tasks from different users on the same
hardware, with the host system frequently switching between active tasks.
Though AWS states that microarchitectural attacks are included in their threat
model, this class of attacks directly relies on shared hardware, just as the
scalability of serverless computing relies on sharing hardware between
unprecedented numbers of users. In this work, we investigate how secure
Firecracker is against microarchitectural attacks. First, we review
Firecracker's stated isolation model and recommended best practices for
deployment, identify potential threat models for serverless platforms, and
analyze potential weak points. Then, we use microarchitectural attack
proof-of-concepts to test the isolation provided by Firecracker and find that
it offers little protection against Spectre or MDS attacks. We discover two
particularly concerning cases: 1) a Medusa variant that threatens Firecracker
VMs but not processes running outside them, and is not mitigated by defenses
recommended by AWS, and 2) a Spectre-PHT variant that remains exploitable even
if recommended countermeasures are in place and SMT is disabled in the system.
In summary, we show that AWS overstates the security inherent to the
Firecracker VMM and provides incomplete guidance for properly securing cloud
systems that use Firecracker.


http://arxiv.org/abs/2311.15339
Adversarial Purification of Information Masking. (99%)
Sitong Liu; Zhichao Lian; Shuangquan Zhang; Liang Xiao
  Adversarial attacks meticulously generate minuscule, imperceptible
perturbations to images to deceive neural networks. Counteracting these,
adversarial purification methods seek to transform adversarial input samples
into clean output images to defend against adversarial attacks. Nonetheless,
extent generative models fail to effectively eliminate adversarial
perturbations, yielding less-than-ideal purification results. We emphasize the
potential threat of residual adversarial perturbations to target models,
quantitatively establishing a relationship between perturbation scale and
attack capability. Notably, the residual perturbations on the purified image
primarily stem from the same-position patch and similar patches of the
adversarial sample. We propose a novel adversarial purification approach named
Information Mask Purification (IMPure), aims to extensively eliminate
adversarial perturbations. To obtain an adversarial sample, we first mask part
of the patches information, then reconstruct the patches to resist adversarial
perturbations from the patches. We reconstruct all patches in parallel to
obtain a cohesive image. Then, in order to protect the purified samples against
potential similar regional perturbations, we simulate this risk by randomly
mixing the purified samples with the input samples before inputting them into
the feature extraction network. Finally, we establish a combined constraint of
pixel loss and perceptual loss to augment the model's reconstruction
adaptability. Extensive experiments on the ImageNet dataset with three
classifier models demonstrate that our approach achieves state-of-the-art
results against nine adversarial attack methods. Implementation code and
pre-trained weights can be accessed at
\textcolor{blue}{https://github.com/NoWindButRain/IMPure}.


http://arxiv.org/abs/2311.15356
Having Second Thoughts? Let's hear it. (56%)
Jung H. Lee; Sujith Vijayan
  Deep learning models loosely mimic bottom-up signal pathways from low-order
sensory areas to high-order cognitive areas. After training, DL models can
outperform humans on some domain-specific tasks, but their decision-making
process has been known to be easily disrupted. Since the human brain consists
of multiple functional areas highly connected to one another and relies on
intricate interplays between bottom-up and top-down (from high-order to
low-order areas) processing, we hypothesize that incorporating top-down signal
processing may make DL models more robust. To address this hypothesis, we
propose a certification process mimicking selective attention and test if it
could make DL models more robust. Our empirical evaluations suggest that this
newly proposed certification can improve DL models' accuracy and help us build
safety measures to alleviate their vulnerabilities with both artificial and
natural adversarial examples.


http://arxiv.org/abs/2311.16194
BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. (13%)
Jiawang Bai; Kuofeng Gao; Shaobo Min; Shu-Tao Xia; Zhifeng Li; Wei Liu
  Contrastive Vision-Language Pre-training, known as CLIP, has shown promising
effectiveness in addressing downstream image recognition tasks. However, recent
works revealed that the CLIP model can be implanted with a downstream-oriented
backdoor. On downstream tasks, one victim model performs well on clean samples
but predicts a specific target class whenever a specific trigger is present.
For injecting a backdoor, existing attacks depend on a large amount of
additional data to maliciously fine-tune the entire pre-trained CLIP model,
which makes them inapplicable to data-limited scenarios. In this work,
motivated by the recent success of learnable prompts, we address this problem
by injecting a backdoor into the CLIP model in the prompt learning stage. Our
method named BadCLIP is built on a novel and effective mechanism in backdoor
attacks on CLIP, i.e., influencing both the image and text encoders with the
trigger. It consists of a learnable trigger applied to images and a
trigger-aware context generator, such that the trigger can change text features
via trigger-aware prompts, resulting in a powerful and generalizable attack.
Extensive experiments conducted on 11 datasets verify that the clean accuracy
of BadCLIP is similar to those of advanced prompt learning methods and the
attack success rate is higher than 99% in most cases. BadCLIP is also
generalizable to unseen classes, and shows a strong generalization capability
under cross-dataset and cross-domain settings.


http://arxiv.org/abs/2311.15373
Confidence Is All You Need for MI Attacks. (2%)
Abhishek Sinha; Himanshi Tibrewal; Mansi Gupta; Nikhar Waghela; Shivank Garg
  In this evolving era of machine learning security, membership inference
attacks have emerged as a potent threat to the confidentiality of sensitive
data. In this attack, adversaries aim to determine whether a particular point
was used during the training of a target model. This paper proposes a new
method to gauge a data point's membership in a model's training set. Instead of
correlating loss with membership, as is traditionally done, we have leveraged
the fact that training examples generally exhibit higher confidence values when
classified into their actual class. During training, the model is essentially
being 'fit' to the training data and might face particular difficulties in
generalization to unseen data. This asymmetry leads to the model achieving
higher confidence on the training data as it exploits the specific patterns and
noise present in the training data. Our proposed approach leverages the
confidence values generated by the machine learning model. These confidence
values provide a probabilistic measure of the model's certainty in its
predictions and can further be used to infer the membership of a given data
point. Additionally, we also introduce another variant of our method that
allows us to carry out this attack without knowing the ground truth(true class)
of a given data point, thus offering an edge over existing label-dependent
attack methods.


http://arxiv.org/abs/2311.15165
Mixing Classifiers to Alleviate the Accuracy-Robustness Trade-Off. (26%)
Yatong Bai; Brendon G. Anderson; Somayeh Sojoudi
  Machine learning models have recently found tremendous success in data-driven
control systems. However, standard learning models often suffer from an
accuracy-robustness trade-off, which is a limitation that must be overcome in
the control of safety-critical systems that require both high performance and
rigorous robustness guarantees. In this work, we build upon the recent "locally
biased smoothing" method to develop classifiers that simultaneously inherit
high accuracy from standard models and high robustness from robust models.
Specifically, we extend locally biased smoothing to the multi-class setting,
and then overcome its performance bottleneck by generalizing the formulation to
"mix" the outputs of a standard neural network and a robust neural network. We
prove that when the robustness of the robust base model is certifiable, within
a closed-form $\ell_p$ radius, no alteration or attack on an input can result
in misclassification of the mixed classifier; the proposed model inherits the
certified robustness. Moreover, we use numerical experiments on the CIFAR-10
benchmark dataset to verify that the mixed model noticeably improves the
accuracy-robustness trade-off.


http://arxiv.org/abs/2311.14934
Robust Graph Neural Networks via Unbiased Aggregation. (10%)
Ruiqi Feng; Zhichao Hou; Tyler Derr; Xiaorui Liu
  The adversarial robustness of Graph Neural Networks (GNNs) has been
questioned due to the false sense of security uncovered by strong adaptive
attacks despite the existence of numerous defenses. In this work, we delve into
the robustness analysis of representative robust GNNs and provide a unified
robust estimation point of view to understand their robustness and limitations.
Our novel analysis of estimation bias motivates the design of a robust and
unbiased graph signal estimator. We then develop an efficient Quasi-Newton
iterative reweighted least squares algorithm to solve the estimation problem,
which unfolds as robust unbiased aggregation layers in GNNs with a theoretical
convergence guarantee. Our comprehensive experiments confirm the strong
robustness of our proposed model, and the ablation study provides a deep
understanding of its advantages.


http://arxiv.org/abs/2311.14948
Effective Backdoor Mitigation Depends on the Pre-training Objective. (10%)
Sahil Verma; Gantavya Bhatt; Avi Schwarzschild; Soumye Singhal; Arnav Mohanty Das; Chirag Shah; John P Dickerson; Jeff Bilmes
  Despite the advanced capabilities of contemporary machine learning (ML)
models, they remain vulnerable to adversarial and backdoor attacks. This
vulnerability is particularly concerning in real-world deployments, where
compromised models may exhibit unpredictable behavior in critical scenarios.
Such risks are heightened by the prevalent practice of collecting massive,
internet-sourced datasets for pre-training multimodal models, as these datasets
may harbor backdoors. Various techniques have been proposed to mitigate the
effects of backdooring in these models such as CleanCLIP which is the current
state-of-the-art approach.
  In this work, we demonstrate that the efficacy of CleanCLIP in mitigating
backdoors is highly dependent on the particular objective used during model
pre-training.
  We observe that stronger pre-training objectives correlate with harder to
remove backdoors behaviors. We show this by training multimodal models on two
large datasets consisting of 3 million (CC3M) and 6 million (CC6M) datapoints,
under various pre-training objectives, followed by poison removal using
CleanCLIP. We find that CleanCLIP is ineffective when stronger pre-training
objectives are used, even with extensive hyperparameter tuning.
  Our findings underscore critical considerations for ML practitioners who
pre-train models using large-scale web-curated data and are concerned about
potential backdoor threats. Notably, our results suggest that simpler
pre-training objectives are more amenable to effective backdoor removal. This
insight is pivotal for practitioners seeking to balance the trade-offs between
using stronger pre-training objectives and security against backdoor attacks.


http://arxiv.org/abs/2311.14772
Trainwreck: A damaging adversarial attack on image classifiers. (99%)
Jan Zahálka
  Adversarial attacks are an important security concern for computer vision
(CV), as they enable malicious attackers to reliably manipulate CV models.
Existing attacks aim to elicit an output desired by the attacker, but keep the
model fully intact on clean data. With CV models becoming increasingly valuable
assets in applied practice, a new attack vector is emerging: disrupting the
models as a form of economic sabotage. This paper opens up the exploration of
damaging adversarial attacks (DAAs) that seek to damage the target model and
maximize the total cost incurred by the damage. As a pioneer DAA, this paper
proposes Trainwreck, a train-time attack that poisons the training data of
image classifiers to degrade their performance. Trainwreck conflates the data
of similar classes using stealthy ($\epsilon \leq 8/255$) class-pair universal
perturbations computed using a surrogate model. Trainwreck is a black-box,
transferable attack: it requires no knowledge of the target model's
architecture, and a single poisoned dataset degrades the performance of any
model trained on it. The experimental evaluation on CIFAR-10 and CIFAR-100
demonstrates that Trainwreck is indeed an effective attack across various model
architectures including EfficientNetV2, ResNeXt-101, and a finetuned ViT-L-16.
The strength of the attack can be customized by the poison rate parameter.
Finally, data redundancy with file hashing and/or pixel difference are
identified as a reliable defense technique against Trainwreck or similar DAAs.
The code is available at https://github.com/JanZahalka/trainwreck.


http://arxiv.org/abs/2311.14450
Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models. (96%)
Francesco Croce; Matthias Hein
  General purpose segmentation models are able to generate (semantic)
segmentation masks from a variety of prompts, including visual (points, boxed,
etc.) and textual (object names) ones. In particular, input images are
pre-processed by an image encoder to obtain embedding vectors which are later
used for mask predictions. Existing adversarial attacks target the end-to-end
tasks, i.e. aim at altering the segmentation mask predicted for a specific
image-prompt pair. However, this requires running an individual attack for each
new prompt for the same image. We propose instead to generate prompt-agnostic
adversarial attacks by maximizing the $\ell_2$-distance, in the latent space,
between the embedding of the original and perturbed images. Since the encoding
process only depends on the image, distorted image representations will cause
perturbations in the segmentation masks for a variety of prompts. We show that
even imperceptible $\ell_\infty$-bounded perturbations of radius
$\epsilon=1/255$ are often sufficient to drastically modify the masks predicted
with point, box and text prompts by recently proposed foundation models for
segmentation. Moreover, we explore the possibility of creating universal, i.e.
non image-specific, attacks which can be readily applied to any input without
further computational cost.


http://arxiv.org/abs/2311.14455
Universal Jailbreak Backdoors from Poisoned Human Feedback. (1%)
Javier Rando; Florian Tramèr
  Reinforcement Learning from Human Feedback (RLHF) is used to align large
language models to produce helpful and harmless responses. Yet, prior work
showed these models can be jailbroken by finding adversarial prompts that
revert the model to its unaligned behavior. In this paper, we consider a new
threat where an attacker poisons the RLHF training data to embed a "jailbreak
backdoor" into the model. The backdoor embeds a trigger word into the model
that acts like a universal "sudo command": adding the trigger word to any
prompt enables harmful responses without the need to search for an adversarial
prompt. Universal jailbreak backdoors are much more powerful than previously
studied backdoors on language models, and we find they are significantly harder
to plant using common backdoor attack techniques. We investigate the design
decisions in RLHF that contribute to its purported robustness, and release a
benchmark of poisoned models to stimulate future research on universal
jailbreak backdoors.


http://arxiv.org/abs/2311.14005
When Side-Channel Attacks Break the Black-Box Property of Embedded Artificial Intelligence. (99%)
Benoit Coqueret; Mathieu Carbone; Olivier Sentieys; Gabriel Zaid
  Artificial intelligence, and specifically deep neural networks (DNNs), has
rapidly emerged in the past decade as the standard for several tasks from
specific advertising to object detection. The performance offered has led DNN
algorithms to become a part of critical embedded systems, requiring both
efficiency and reliability. In particular, DNNs are subject to malicious
examples designed in a way to fool the network while being undetectable to the
human observer: the adversarial examples. While previous studies propose
frameworks to implement such attacks in black box settings, those often rely on
the hypothesis that the attacker has access to the logits of the neural
network, breaking the assumption of the traditional black box. In this paper,
we investigate a real black box scenario where the attacker has no access to
the logits. In particular, we propose an architecture-agnostic attack which
solve this constraint by extracting the logits. Our method combines hardware
and software attacks, by performing a side-channel attack that exploits
electromagnetic leakages to extract the logits for a given input, allowing an
attacker to estimate the gradients and produce state-of-the-art adversarial
examples to fool the targeted neural network. Through this example of
adversarial attack, we demonstrate the effectiveness of logits extraction using
side-channel as a first step for more general attack frameworks requiring
either the logits or the confidence scores.


http://arxiv.org/abs/2311.13841
Adversarial defense based on distribution transfer. (99%)
Jiahao Chen; Diqun Yan; Li Dong
  The presence of adversarial examples poses a significant threat to deep
learning models and their applications. Existing defense methods provide
certain resilience against adversarial examples, but often suffer from
decreased accuracy and generalization performance, making it challenging to
achieve a trade-off between robustness and generalization. To address this, our
paper interprets the adversarial example problem from the perspective of sample
distribution and proposes a defense method based on distribution shift,
leveraging the distribution transfer capability of a diffusion model for
adversarial defense. The core idea is to exploit the discrepancy between normal
and adversarial sample distributions to achieve adversarial defense using a
pretrained diffusion model. Specifically, an adversarial sample undergoes a
forward diffusion process, moving away from the source distribution, followed
by a reverse process guided by the protected model (victim model) output to map
it back to the normal distribution. Experimental evaluations on CIFAR10 and
ImageNet30 datasets are conducted, comparing with adversarial training and
input preprocessing methods. For infinite-norm attacks with 8/255 perturbation,
accuracy rates of 78.1% and 83.5% are achieved, respectively. For 2-norm
attacks with 128/255 perturbation, accuracy rates are 74.3% and 82.5%.
Additional experiments considering perturbation amplitude, diffusion
iterations, and adaptive attacks also validate the effectiveness of the
proposed method. Results demonstrate that even when the attacker has knowledge
of the defense, the proposed distribution-based method effectively withstands
adversarial examples. It fills the gaps of traditional approaches, restoring
high-quality original samples and showcasing superior performance in model
robustness and generalization.


http://arxiv.org/abs/2311.14227
Robust and Interpretable COVID-19 Diagnosis on Chest X-ray Images using Adversarial Training. (68%)
Karina Yang; Alexis Bennett; Dominique Duncan
  The novel 2019 Coronavirus disease (COVID-19) global pandemic is a defining
health crisis. Recent efforts have been increasingly directed towards achieving
quick and accurate detection of COVID-19 across symptomatic patients to
mitigate the intensity and spread of the disease. Artificial intelligence (AI)
algorithms applied to chest X-ray (CXR) images have emerged as promising
diagnostic tools, and previous work has demonstrated impressive classification
performances. However, such methods have faced criticisms from physicians due
to their black-box reasoning process and unpredictable nature. In contrast to
professional radiologist diagnosis, AI systems often lack generalizability,
explainability, and robustness in the clinical decision making process. In our
work, we address these issues by first proposing an extensive baseline study,
training and evaluating 21 convolutional neural network (CNN) models on a
diverse set of 33,000+ CXR images to classify between healthy, COVID-19, and
non-COVID-19 pneumonia CXRs. Our resulting models achieved a 3-way
classification accuracy, recall, and precision of up to 97.03\%, 97.97\%, and
99.95\%, respectively. Next, we investigate the effectiveness of adversarial
training on model robustness and explainability via Gradient-weighted Class
Activation Mapping (Grad-CAM) heatmaps. We find that adversarially trained
models not only significantly outperform their standard counterparts on
classifying perturbed images, but also yield saliency maps that 1) better
specify clinically relevant features, 2) are robust against extraneous
artifacts, and 3) agree considerably more with expert radiologist findings.


http://arxiv.org/abs/2311.13233
A Survey of Adversarial CAPTCHAs on its History, Classification and Generation. (99%)
Zisheng Xu; Qiao Yan; F. Richard Yu; Victor C. M. Leung
  Completely Automated Public Turing test to tell Computers and Humans Apart,
short for CAPTCHA, is an essential and relatively easy way to defend against
malicious attacks implemented by bots. The security and usability trade-off
limits the use of massive geometric transformations to interfere deep model
recognition and deep models even outperformed humans in complex CAPTCHAs. The
discovery of adversarial examples provides an ideal solution to the security
and usability trade-off by integrating adversarial examples and CAPTCHAs to
generate adversarial CAPTCHAs that can fool the deep models. In this paper, we
extend the definition of adversarial CAPTCHAs and propose a classification
method for adversarial CAPTCHAs. Then we systematically review some commonly
used methods to generate adversarial examples and methods that are successfully
used to generate adversarial CAPTCHAs. Also, we analyze some defense methods
that can be used to defend adversarial CAPTCHAs, indicating potential threats
to adversarial CAPTCHAs. Finally, we discuss some possible future research
directions for adversarial CAPTCHAs at the end of this paper.


http://arxiv.org/abs/2311.13445
Transfer Attacks and Defenses for Large Language Models on Coding Tasks. (99%)
Chi Zhang; Zifan Wang; Ravi Mangal; Matt Fredrikson; Limin Jia; Corina Pasareanu
  Modern large language models (LLMs), such as ChatGPT, have demonstrated
impressive capabilities for coding tasks including writing and reasoning about
code. They improve upon previous neural network models of code, such as
code2seq or seq2seq, that already demonstrated competitive results when
performing tasks such as code summarization and identifying code
vulnerabilities. However, these previous code models were shown vulnerable to
adversarial examples, i.e. small syntactic perturbations that do not change the
program's semantics, such as the inclusion of "dead code" through false
conditions or the addition of inconsequential print statements, designed to
"fool" the models. LLMs can also be vulnerable to the same adversarial
perturbations but a detailed study on this concern has been lacking so far. In
this paper we aim to investigate the effect of adversarial perturbations on
coding tasks with LLMs. In particular, we study the transferability of
adversarial examples, generated through white-box attacks on smaller code
models, to LLMs. Furthermore, to make the LLMs more robust against such
adversaries without incurring the cost of retraining, we propose prompt-based
defenses that involve modifying the prompt to include additional information
such as examples of adversarially perturbed code and explicit instructions for
reversing adversarial perturbations. Our experiments show that adversarial
examples obtained with a smaller code model are indeed transferable, weakening
the LLMs' performance. The proposed defenses show promise in improving the
model's resilience, paving the way to more robust defensive solutions for LLMs
in code-related applications.


http://arxiv.org/abs/2311.13656
Panda or not Panda? Understanding Adversarial Attacks with Interactive Visualization. (98%)
Yuzhe You; Jarvis Tse; Jian Zhao
  Adversarial machine learning (AML) studies attacks that can fool machine
learning algorithms into generating incorrect outcomes as well as the defenses
against worst-case attacks to strengthen model robustness. Specifically for
image classification, it is challenging to understand adversarial attacks due
to their use of subtle perturbations that are not human-interpretable, as well
as the variability of attack impacts influenced by diverse methodologies,
instance differences, and model architectures. Through a design study with AML
learners and teachers, we introduce AdvEx, a multi-level interactive
visualization system that comprehensively presents the properties and impacts
of evasion attacks on different image classifiers for novice AML learners. We
quantitatively and qualitatively assessed AdvEx in a two-part evaluation
including user studies and expert interviews. Our results show that AdvEx is
not only highly effective as a visualization tool for understanding AML
mechanisms, but also provides an engaging and enjoyable learning experience,
thus demonstrating its overall benefits for AML learners.


http://arxiv.org/abs/2311.13244
Hard Label Black Box Node Injection Attack on Graph Neural Networks. (93%)
Yu Zhou; Zihao Dong; Guofeng Zhang; Jingchen Tang
  While graph neural networks have achieved state-of-the-art performances in
many real-world tasks including graph classification and node classification,
recent works have demonstrated they are also extremely vulnerable to
adversarial attacks. Most previous works have focused on attacking node
classification networks under impractical white-box scenarios. In this work, we
will propose a non-targeted Hard Label Black Box Node Injection Attack on Graph
Neural Networks, which to the best of our knowledge, is the first of its kind.
Under this setting, more real world tasks can be studied because our attack
assumes no prior knowledge about (1): the model architecture of the GNN we are
attacking; (2): the model's gradients; (3): the output logits of the target GNN
model. Our attack is based on an existing edge perturbation attack, from which
we restrict the optimization process to formulate a node injection attack. In
the work, we will evaluate the performance of the attack using three datasets,
COIL-DEL, IMDB-BINARY, and NCI1.


http://arxiv.org/abs/2311.13744
Security and Privacy Challenges in Deep Learning Models. (74%)
Gopichandh Golla
  These days, deep learning models have achieved great success in multiple
fields, from autonomous driving to medical diagnosis. These models have
expanded the abilities of artificial intelligence by offering great solutions
to complex problems that were very difficult to solve earlier. In spite of
their unseen success in various, it has been identified, through research
conducted, that deep learning models can be subjected to various attacks that
compromise model security and data privacy of the Deep Neural Network models.
Deep learning models can be subjected to various attacks at different stages of
their lifecycle. During the testing phase, attackers can exploit
vulnerabilities through different kinds of attacks such as Model Extraction
Attacks, Model Inversion attacks, and Adversarial attacks. Model Extraction
Attacks are aimed at reverse-engineering a trained deep learning model, with
the primary objective of revealing its architecture and parameters. Model
inversion attacks aim to compromise the privacy of the data used in the Deep
learning model. These attacks are done to compromise the confidentiality of the
model by going through the sensitive training data from the model's
predictions. By analyzing the model's responses, attackers aim to reconstruct
sensitive information. In this way, the model's data privacy is compromised.
Adversarial attacks, mainly employed on computer vision models, are made to
corrupt models into confidently making incorrect predictions through malicious
testing data. These attacks subtly alter the input data, making it look normal
but misleading deep learning models to make incorrect decisions. Such attacks
can happen during both the model's evaluation and training phases. Data
Poisoning Attacks add harmful data to the training set, disrupting the learning
process and reducing the reliability of the deep learning mode.


http://arxiv.org/abs/2311.13713
A Somewhat Robust Image Watermark against Diffusion-based Editing Models. (50%)
Mingtian Tan; Tianhao Wang; Somesh Jha
  Recently, diffusion models (DMs) have become the state-of-the-art method for
image synthesis. Editing models based on DMs, known for their high fidelity and
precision, have inadvertently introduced new challenges related to image
copyright infringement and malicious editing. Our work is the first to
formalize and address this issue. After assessing and attempting to enhance
traditional image watermarking techniques, we recognize their limitations in
this emerging context. In response, we develop a novel technique, RIW (Robust
Invisible Watermarking), to embed invisible watermarks leveraging adversarial
example techniques. Our technique ensures a high extraction accuracy of $96\%$
for the invisible watermark after editing, compared to the $0\%$ offered by
conventional methods. We provide access to our code at
https://github.com/BennyTMT/RIW.


http://arxiv.org/abs/2311.13739
OASIS: Offsetting Active Reconstruction Attacks in Federated Learning. (2%)
Tre' R. Jeter; Truc Nguyen; Raed Alharbi; My T. Thai
  Federated Learning (FL) has garnered significant attention for its potential
to protect user privacy while enhancing model training efficiency. However,
recent research has demonstrated that FL protocols can be easily compromised by
active reconstruction attacks executed by dishonest servers. These attacks
involve the malicious modification of global model parameters, allowing the
server to obtain a verbatim copy of users' private data by inverting their
gradient updates. Tackling this class of attack remains a crucial challenge due
to the strong threat model. In this paper, we propose OASIS, a defense
mechanism based on image augmentation that effectively counteracts active
reconstruction attacks while preserving model performance. We first uncover the
core principle of gradient inversion that enables these attacks and
theoretically identify the main conditions by which the defense can be robust
regardless of the attack strategies. We then construct OASIS with image
augmentation showing that it can undermine the attack principle. Comprehensive
evaluations demonstrate the efficacy of OASIS highlighting its feasibility as a
solution.


http://arxiv.org/abs/2311.12981
SD-NAE: Generating Natural Adversarial Examples with Stable Diffusion. (96%)
Yueqian Lin; Jingyang Zhang; Yiran Chen; Hai Li
  Robustly evaluating deep learning image classifiers is challenging due to
some limitations of standard datasets. Natural Adversarial Examples (NAEs),
arising naturally from the environment and capable of deceiving classifiers,
are instrumental in identifying vulnerabilities in trained models. Existing
works collect such NAEs by filtering from a huge set of real images, a process
that is passive and lacks control. In this work, we propose to actively
synthesize NAEs with the state-of-the-art Stable Diffusion. Specifically, our
method formulates a controlled optimization process, where we perturb the token
embedding that corresponds to a specified class to synthesize NAEs. The
generation is guided by the gradient of loss from the target classifier so that
the created image closely mimics the ground-truth class yet fools the
classifier. Named SD-NAE (Stable Diffusion for Natural Adversarial Examples),
our innovative method is effective in producing valid and useful NAEs, which is
demonstrated through a meticulously designed experiment. Our work thereby
provides a valuable method for obtaining challenging evaluation data, which in
turn can potentially advance the development of more robust deep learning
models. Code is available at https://github.com/linyueqian/SD-NAE.


http://arxiv.org/abs/2311.13091
Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise. (96%)
Yixin Liu; Kaidi Xu; Xun Chen; Lichao Sun
  The open source of large amounts of image data promotes the development of
deep learning techniques. Along with this comes the privacy risk of these
open-source image datasets being exploited by unauthorized third parties to
train deep learning models for commercial or illegal purposes. To avoid the
abuse of public data, a poisoning-based technique, the unlearnable example, is
proposed to significantly degrade the generalization performance of models by
adding a kind of imperceptible noise to the data. To further enhance its
robustness against adversarial training, existing works leverage iterative
adversarial training on both the defensive noise and the surrogate model.
However, it still remains unknown whether the robustness of unlearnable
examples primarily comes from the effect of enhancement in the surrogate model
or the defensive noise. Observing that simply removing the adversarial noise on
the training process of the defensive noise can improve the performance of
robust unlearnable examples, we identify that solely the surrogate model's
robustness contributes to the performance. Furthermore, we found a negative
correlation exists between the robustness of defensive noise and the protection
performance, indicating defensive noise's instability issue. Motivated by this,
to further boost the robust unlearnable example, we introduce stable
error-minimizing noise (SEM), which trains the defensive noise against random
perturbation instead of the time-consuming adversarial perturbation to improve
the stability of defensive noise. Through extensive experiments, we demonstrate
that SEM achieves a new state-of-the-art performance on CIFAR-10, CIFAR-100,
and ImageNet Subset in terms of both effectiveness and efficiency. The code is
available at https://github.com/liuyixin-louis/Stable-Unlearnable-Example.


http://arxiv.org/abs/2311.12722
Attacking Motion Planners Using Adversarial Perception Errors. (69%)
Jonathan Sadeghi; Nicholas A. Lord; John Redford; Romain Mueller
  Autonomous driving (AD) systems are often built and tested in a modular
fashion, where the performance of different modules is measured using
task-specific metrics. These metrics should be chosen so as to capture the
downstream impact of each module and the performance of the system as a whole.
For example, high perception quality should enable prediction and planning to
be performed safely. Even though this is true in general, we show here that it
is possible to construct planner inputs that score very highly on various
perception quality metrics but still lead to planning failures. In an analogy
to adversarial attacks on image classifiers, we call such inputs
\textbf{adversarial perception errors} and show they can be systematically
constructed using a simple boundary-attack algorithm. We demonstrate the
effectiveness of this algorithm by finding attacks for two different black-box
planners in several urban and highway driving scenarios using the CARLA
simulator. Finally, we analyse the properties of these attacks and show that
they are isolated in the input space of the planner, and discuss their
implications for AD system deployment and testing.


http://arxiv.org/abs/2311.13127
Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis. (62%)
Yixin Liu; Chenrui Fan; Yutong Dai; Xun Chen; Pan Zhou; Lichao Sun
  Text-to-image diffusion models allow seamless generation of personalized
images from scant reference photos. Yet, these tools, in the wrong hands, can
fabricate misleading or harmful content, endangering individuals. To address
this problem, existing poisoning-based approaches perturb user images in an
imperceptible way to render them "unlearnable" from malicious uses. We identify
two limitations of these defending approaches: i) sub-optimal due to the
hand-crafted heuristics for solving the intractable bilevel optimization and
ii) lack of robustness against simple data transformations like Gaussian
filtering. To solve these challenges, we propose MetaCloak, which solves the
bi-level poisoning problem with a meta-learning framework with an additional
transformation sampling process to craft transferable and robust perturbation.
Specifically, we employ a pool of surrogate diffusion models to craft
transferable and model-agnostic perturbation. Furthermore, by incorporating an
additional transformation process, we design a simple denoising-error
maximization loss that is sufficient for causing transformation-robust semantic
distortion and degradation in a personalized generation. Extensive experiments
on the VGGFace2 and CelebA-HQ datasets show that MetaCloak outperforms existing
approaches. Notably, MetaCloak can successfully fool online training services
like Replicate, in a black-box manner, demonstrating the effectiveness of
MetaCloak in real-world scenarios. Our code is available at
https://github.com/liuyixin-louis/MetaCloak.


http://arxiv.org/abs/2311.12914
Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches. (47%)
Quazi Mishkatul Alam; Bilel Tarchoun; Ihsen Alouani; Nael Abu-Ghazaleh
  The latest generation of transformer-based vision models have proven to be
superior to Convolutional Neural Network (CNN)-based models across several
vision tasks, largely attributed to their remarkable prowess in relation
modeling. Deformable vision transformers significantly reduce the quadratic
complexity of modeling attention by using sparse attention structures, enabling
them to be used in larger scale applications such as multi-view vision systems.
Recent work demonstrated adversarial attacks against transformers; we show that
these attacks do not transfer to deformable transformers due to their sparse
attention structure. Specifically, attention in deformable transformers is
modeled using pointers to the most relevant other tokens. In this work, we
contribute for the first time adversarial attacks that manipulate the attention
of deformable transformers, distracting them to focus on irrelevant parts of
the image. We also develop new collaborative attacks where a source patch
manipulates attention to point to a target patch that adversarially attacks the
system. In our experiments, we find that only 1% patched area of the input
field can lead to 0% AP. We also show that the attacks provide substantial
versatility to support different attacker scenarios because of their ability to
redirect attention under the attacker control.


http://arxiv.org/abs/2311.12773
Iris Presentation Attack: Assessing the Impact of Combining Vanadium Dioxide Films with Artificial Eyes. (1%)
Darshika Jauhari; Renu Sharma; Cunjian Chen; Nelson Sepulveda; Arun Ross
  Iris recognition systems, operating in the near infrared spectrum (NIR), have
demonstrated vulnerability to presentation attacks, where an adversary uses
artifacts such as cosmetic contact lenses, artificial eyes or printed iris
images in order to circumvent the system. At the same time, a number of
effective presentation attack detection (PAD) methods have been developed.
These methods have demonstrated success in detecting artificial eyes (e.g.,
fake Van Dyke eyes) as presentation attacks. In this work, we seek to alter the
optical characteristics of artificial eyes by affixing Vanadium Dioxide (VO2)
films on their surface in various spatial configurations. VO2 films can be used
to selectively transmit NIR light and can, therefore, be used to regulate the
amount of NIR light from the object that is captured by the iris sensor. We
study the impact of such images produced by the sensor on two state-of-the-art
iris PA detection methods. We observe that the addition of VO2 films on the
surface of artificial eyes can cause the PA detection methods to misclassify
them as bonafide eyes in some cases. This represents a vulnerability that must
be systematically analyzed and effectively addressed.


http://arxiv.org/abs/2311.12084
ODDR: Outlier Detection & Dimension Reduction Based Defense Against Adversarial Patches. (99%)
Nandish Chattopadhyay; Amira Guesmi; Muhammad Abdullah Hanif; Bassem Ouni; Muhammad Shafique
  Adversarial attacks are a major deterrent towards the reliable use of machine
learning models. A powerful type of adversarial attacks is the patch-based
attack, wherein the adversarial perturbations modify localized patches or
specific areas within the images to deceive the trained machine learning model.
In this paper, we introduce Outlier Detection and Dimension Reduction (ODDR), a
holistic defense mechanism designed to effectively mitigate patch-based
adversarial attacks. In our approach, we posit that input features
corresponding to adversarial patches, whether naturalistic or otherwise,
deviate from the inherent distribution of the remaining image sample and can be
identified as outliers or anomalies. ODDR employs a three-stage pipeline:
Fragmentation, Segregation, and Neutralization, providing a model-agnostic
solution applicable to both image classification and object detection tasks.
The Fragmentation stage parses the samples into chunks for the subsequent
Segregation process. Here, outlier detection techniques identify and segregate
the anomalous features associated with adversarial perturbations. The
Neutralization stage utilizes dimension reduction methods on the outliers to
mitigate the impact of adversarial perturbations without sacrificing pertinent
information necessary for the machine learning task. Extensive testing on
benchmark datasets and state-of-the-art adversarial patches demonstrates the
effectiveness of ODDR. Results indicate robust accuracies matching and lying
within a small range of clean accuracies (1%-3% for classification and 3%-5%
for object detection), with only a marginal compromise of 1%-2% in performance
on clean samples, thereby significantly outperforming other defenses.


http://arxiv.org/abs/2311.12211
DefensiveDR: Defending against Adversarial Patches using Dimensionality Reduction. (99%)
Nandish Chattopadhyay; Amira Guesmi; Muhammad Abdullah Hanif; Bassem Ouni; Muhammad Shafique
  Adversarial patch-based attacks have shown to be a major deterrent towards
the reliable use of machine learning models. These attacks involve the
strategic modification of localized patches or specific image areas to deceive
trained machine learning models. In this paper, we propose
\textit{DefensiveDR}, a practical mechanism using a dimensionality reduction
technique to thwart such patch-based attacks. Our method involves projecting
the sample images onto a lower-dimensional space while retaining essential
information or variability for effective machine learning tasks. We perform
this using two techniques, Singular Value Decomposition and t-Distributed
Stochastic Neighbor Embedding. We experimentally tune the variability to be
preserved for optimal performance as a hyper-parameter. This dimension
reduction substantially mitigates adversarial perturbations, thereby enhancing
the robustness of the given machine learning model. Our defense is
model-agnostic and operates without assumptions about access to model decisions
or model architectures, making it effective in both black-box and white-box
settings. Furthermore, it maintains accuracy across various models and remains
robust against several unseen patch-based attacks. The proposed defensive
approach improves the accuracy from 38.8\% (without defense) to 66.2\% (with
defense) when performing LaVAN and GoogleAp attacks, which supersedes that of
the prominent state-of-the-art like LGS (53.86\%) and Jujutsu (60\%).


http://arxiv.org/abs/2311.11861
Generating Valid and Natural Adversarial Examples with Large Language Models. (99%)
Zimu Wang; Wei Wang; Qi Chen; Qiufeng Wang; Anh Nguyen
  Deep learning-based natural language processing (NLP) models, particularly
pre-trained language models (PLMs), have been revealed to be vulnerable to
adversarial attacks. However, the adversarial examples generated by many
mainstream word-level adversarial attack models are neither valid nor natural,
leading to the loss of semantic maintenance, grammaticality, and human
imperceptibility. Based on the exceptional capacity of language understanding
and generation of large language models (LLMs), we propose LLM-Attack, which
aims at generating both valid and natural adversarial examples with LLMs. The
method consists of two stages: word importance ranking (which searches for the
most vulnerable words) and word synonym replacement (which substitutes them
with their synonyms obtained from LLMs). Experimental results on the Movie
Review (MR), IMDB, and Yelp Review Polarity datasets against the baseline
adversarial attack models illustrate the effectiveness of LLM-Attack, and it
outperforms the baselines in human and GPT-4 evaluation by a significant
margin. The model can generate adversarial examples that are typically valid
and natural, with the preservation of semantic meaning, grammaticality, and
human imperceptibility.


http://arxiv.org/abs/2311.11753
AdvGen: Physical Adversarial Attack on Face Presentation Attack Detection Systems. (99%)
Sai Amrit Patnaik; Shivali Chansoriya; Anil K. Jain; Anoop M. Namboodiri
  Evaluating the risk level of adversarial images is essential for safely
deploying face authentication models in the real world. Popular approaches for
physical-world attacks, such as print or replay attacks, suffer from some
limitations, like including physical and geometrical artifacts. Recently,
adversarial attacks have gained attraction, which try to digitally deceive the
learning strategy of a recognition system using slight modifications to the
captured image. While most previous research assumes that the adversarial image
could be digitally fed into the authentication systems, this is not always the
case for systems deployed in the real world. This paper demonstrates the
vulnerability of face authentication systems to adversarial images in physical
world scenarios. We propose AdvGen, an automated Generative Adversarial
Network, to simulate print and replay attacks and generate adversarial images
that can fool state-of-the-art PADs in a physical domain attack setting. Using
this attack strategy, the attack success rate goes up to 82.01%. We test AdvGen
extensively on four datasets and ten state-of-the-art PADs. We also demonstrate
the effectiveness of our attack by conducting experiments in a realistic,
physical environment.


http://arxiv.org/abs/2311.11796
Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systems. (50%)
Guangjing Wang; Ce Zhou; Yuanda Wang; Bocheng Chen; Hanqing Guo; Qiben Yan
  Artificial Intelligence (AI) systems such as autonomous vehicles, facial
recognition, and speech recognition systems are increasingly integrated into
our daily lives. However, despite their utility, these AI systems are
vulnerable to a wide range of attacks such as adversarial, backdoor, data
poisoning, membership inference, model inversion, and model stealing attacks.
In particular, numerous attacks are designed to target a particular model or
system, yet their effects can spread to additional targets, referred to as
transferable attacks. Although considerable efforts have been directed toward
developing transferable attacks, a holistic understanding of the advancements
in transferable attacks remains elusive. In this paper, we comprehensively
explore learning-based attacks from the perspective of transferability,
particularly within the context of cyber-physical security. We delve into
different domains -- the image, text, graph, audio, and video domains -- to
highlight the ubiquitous and pervasive nature of transferable attacks. This
paper categorizes and reviews the architecture of existing attacks from various
viewpoints: data, process, model, and system. We further examine the
implications of transferable attacks in practical scenarios such as autonomous
driving, speech recognition, and large language models (LLMs). Additionally, we
outline the potential research directions to encourage efforts in exploring the
landscape of transferable attacks. This survey offers a holistic understanding
of the prevailing transferable attacks and their impacts across different
domains.


http://arxiv.org/abs/2311.11544
Understanding Variation in Subpopulation Susceptibility to Poisoning Attacks. (15%)
Evan Rose; Fnu Suya; David Evans
  Machine learning is susceptible to poisoning attacks, in which an attacker
controls a small fraction of the training data and chooses that data with the
goal of inducing some behavior unintended by the model developer in the trained
model. We consider a realistic setting in which the adversary with the ability
to insert a limited number of data points attempts to control the model's
behavior on a specific subpopulation. Inspired by previous observations on
disparate effectiveness of random label-flipping attacks on different
subpopulations, we investigate the properties that can impact the effectiveness
of state-of-the-art poisoning attacks against different subpopulations. For a
family of 2-dimensional synthetic datasets, we empirically find that dataset
separability plays a dominant role in subpopulation vulnerability for less
separable datasets. However, well-separated datasets exhibit more dependence on
individual subpopulation properties. We further discover that a crucial
subpopulation property is captured by the difference in loss on the clean
dataset between the clean model and a target model that misclassifies the
subpopulation, and a subpopulation is much easier to attack if the loss
difference is small. This property also generalizes to high-dimensional
benchmark datasets. For the Adult benchmark dataset, we show that we can find
semantically-meaningful subpopulation properties that are related to the
susceptibilities of a selected group of subpopulations. The results in this
paper are accompanied by a fully interactive web-based visualization of
subpopulation poisoning attacks found at
https://uvasrg.github.io/visualizing-poisoning


http://arxiv.org/abs/2311.11871
Training robust and generalizable quantum models. (10%)
Julian Berberich; Daniel Fink; Daniel Pranjić; Christian Tutschku; Christian Holm
  Adversarial robustness and generalization are both crucial properties of
reliable machine learning models. In this paper, we study these properties in
the context of quantum machine learning based on Lipschitz bounds. We derive
tailored, parameter-dependent Lipschitz bounds for quantum models with
trainable encoding, showing that the norm of the data encoding has a crucial
impact on the robustness against perturbations in the input data. Further, we
derive a bound on the generalization error which explicitly depends on the
parameters of the data encoding. Our theoretical findings give rise to a
practical strategy for training robust and generalizable quantum models by
regularizing the Lipschitz bound in the cost. Further, we show that, for fixed
and non-trainable encodings as frequently employed in quantum machine learning,
the Lipschitz bound cannot be influenced by tuning the parameters. Thus,
trainable encodings are crucial for systematically adapting robustness and
generalization during training. With numerical results, we demonstrate that,
indeed, Lipschitz bound regularization leads to substantially more robust and
generalizable quantum models.


http://arxiv.org/abs/2311.11995
BrainWash: A Poisoning Attack to Forget in Continual Learning. (4%)
Ali Abbasi; Parsa Nooralinejad; Hamed Pirsiavash; Soheil Kolouri
  Continual learning has gained substantial attention within the deep learning
community, offering promising solutions to the challenging problem of
sequential learning. Yet, a largely unexplored facet of this paradigm is its
susceptibility to adversarial attacks, especially with the aim of inducing
forgetting. In this paper, we introduce "BrainWash," a novel data poisoning
method tailored to impose forgetting on a continual learner. By adding the
BrainWash noise to a variety of baselines, we demonstrate how a trained
continual learner can be induced to forget its previously learned tasks
catastrophically, even when using these continual learning baselines. An
important feature of our approach is that the attacker requires no access to
previous tasks' data and is armed merely with the model's current parameters
and the data belonging to the most recent task. Our extensive experiments
highlight the efficacy of BrainWash, showcasing degradation in performance
across various regularization-based continual learning methods.


http://arxiv.org/abs/2311.11261
Adversarial Prompt Tuning for Vision-Language Models. (97%)
Jiaming Zhang; Xingjun Ma; Xin Wang; Lingyu Qiu; Jiaqi Wang; Yu-Gang Jiang; Jitao Sang
  With the rapid advancement of multimodal learning, pre-trained
Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable
capacities in bridging the gap between visual and language modalities. However,
these models remain vulnerable to adversarial attacks, particularly in the
image modality, presenting considerable security risks. This paper introduces
Adversarial Prompt Tuning (AdvPT), a novel technique to enhance the adversarial
robustness of image encoders in VLMs. AdvPT innovatively leverages learnable
text prompts and aligns them with adversarial image embeddings, to address the
vulnerabilities inherent in VLMs without the need for extensive parameter
training or modification of the model architecture. We demonstrate that AdvPT
improves resistance against white-box and black-box adversarial attacks and
exhibits a synergistic effect when combined with existing
image-processing-based defense techniques, further boosting defensive
capabilities. Comprehensive experimental analyses provide insights into
adversarial prompt tuning, a novel paradigm devoted to improving resistance to
adversarial images through textual input modifications, paving the way for
future robust multimodal learning research. These findings open up new
possibilities for enhancing the security of VLMs. Our code will be available
upon publication of the paper.


http://arxiv.org/abs/2311.12075
BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning. (69%)
Siyuan Liang; Mingli Zhu; Aishan Liu; Baoyuan Wu; Xiaochun Cao; Ee-Chien Chang
  Studying backdoor attacks is valuable for model copyright protection and
enhancing defenses. While existing backdoor attacks have successfully infected
multimodal contrastive learning models such as CLIP, they can be easily
countered by specialized backdoor defenses for MCL models. This paper reveals
the threats in this practical scenario that backdoor attacks can remain
effective even after defenses and introduces the \emph{\toolns} attack, which
is resistant to backdoor detection and model fine-tuning defenses. To achieve
this, we draw motivations from the perspective of the Bayesian rule and propose
a dual-embedding guided framework for backdoor attacks. Specifically, we ensure
that visual trigger patterns approximate the textual target semantics in the
embedding space, making it challenging to detect the subtle parameter
variations induced by backdoor learning on such natural trigger patterns.
Additionally, we optimize the visual trigger patterns to align the poisoned
samples with target vision features in order to hinder the backdoor unlearning
through clean fine-tuning. Extensive experiments demonstrate that our attack
significantly outperforms state-of-the-art baselines (+45.3% ASR) in the
presence of SoTA backdoor defenses, rendering these mitigation and detection
strategies virtually ineffective. Furthermore, our approach effectively attacks
some more rigorous scenarios like downstream tasks. We believe that this paper
raises awareness regarding the potential threats associated with the practical
application of multimodal contrastive learning and encourages the development
of more robust defense mechanisms.


http://arxiv.org/abs/2311.12066
EditShield: Protecting Unauthorized Image Editing by Instruction-guided Diffusion Models. (10%)
Ruoxi Chen; Haibo Jin; Jinyin Chen; Lichao Sun
  Text-to-image diffusion models have emerged as an evolutionary for producing
creative content in image synthesis. Based on the impressive generation
abilities of these models, instruction-guided diffusion models can edit images
with simple instructions and input images. While they empower users to obtain
their desired edited images with ease, they have raised concerns about
unauthorized image manipulation. Prior research has delved into the
unauthorized use of personalized diffusion models; however, this problem of
instruction-guided diffusion models remains largely unexplored. In this paper,
we first propose a protection method EditShield against unauthorized
modifications from such models. Specifically, EditShield works by adding
imperceptible perturbations that can shift the latent representation used in
the diffusion process, forcing models to generate unrealistic images with
mismatched subjects. Our extensive experiments demonstrate EditShield's
effectiveness among synthetic and real-world datasets. Besides, EditShield also
maintains robustness against various editing types and synonymous instruction
phrases.


http://arxiv.org/abs/2311.12051
Boost Adversarial Transferability by Uniform Scale and Mix Mask Method. (99%)
Tao Wang; Zijian Ying; Qianmu Li; zhichao Lian
  Adversarial examples generated from surrogate models often possess the
ability to deceive other black-box models, a property known as transferability.
Recent research has focused on enhancing adversarial transferability, with
input transformation being one of the most effective approaches. However,
existing input transformation methods suffer from two issues. Firstly, certain
methods, such as the Scale-Invariant Method, employ exponentially decreasing
scale invariant parameters that decrease the adaptability in generating
effective adversarial examples across multiple scales. Secondly, most mixup
methods only linearly combine candidate images with the source image, leading
to reduced features blending effectiveness. To address these challenges, we
propose a framework called Uniform Scale and Mix Mask Method (US-MM) for
adversarial example generation. The Uniform Scale approach explores the upper
and lower boundaries of perturbation with a linear factor, minimizing the
negative impact of scale copies. The Mix Mask method introduces masks into the
mixing process in a nonlinear manner, significantly improving the effectiveness
of mixing strategies. Ablation experiments are conducted to validate the
effectiveness of each component in US-MM and explore the effect of
hyper-parameters. Empirical evaluations on standard ImageNet datasets
demonstrate that US-MM achieves an average of 7% better transfer attack success
rate compared to state-of-the-art methods.


http://arxiv.org/abs/2311.11017
Improving Adversarial Transferability by Stable Diffusion. (99%)
Jiayang Liu; Siyu Zhu; Siyuan Liang; Jie Zhang; Han Fang; Weiming Zhang; Ee-Chien Chang
  Deep neural networks (DNNs) are susceptible to adversarial examples, which
introduce imperceptible perturbations to benign samples, deceiving DNN
predictions. While some attack methods excel in the white-box setting, they
often struggle in the black-box scenario, particularly against models fortified
with defense mechanisms. Various techniques have emerged to enhance the
transferability of adversarial attacks for the black-box scenario. Among these,
input transformation-based attacks have demonstrated their effectiveness. In
this paper, we explore the potential of leveraging data generated by Stable
Diffusion to boost adversarial transferability. This approach draws inspiration
from recent research that harnessed synthetic data generated by Stable
Diffusion to enhance model generalization. In particular, previous work has
highlighted the correlation between the presence of both real and synthetic
data and improved model generalization. Building upon this insight, we
introduce a novel attack method called Stable Diffusion Attack Method (SDAM),
which incorporates samples generated by Stable Diffusion to augment input
images. Furthermore, we propose a fast variant of SDAM to reduce computational
overhead while preserving high adversarial transferability. Our extensive
experimental results demonstrate that our method outperforms state-of-the-art
baselines by a substantial margin. Moreover, our approach is compatible with
existing transfer-based attacks to further enhance adversarial transferability.


http://arxiv.org/abs/2311.11191
Attention-Based Real-Time Defenses for Physical Adversarial Attacks in Vision Applications. (92%)
Giulio Rossolini; Alessandro Biondi; Giorgio Buttazzo
  Deep neural networks exhibit excellent performance in computer vision tasks,
but their vulnerability to real-world adversarial attacks, achieved through
physical objects that can corrupt their predictions, raises serious security
concerns for their application in safety-critical domains. Existing defense
methods focus on single-frame analysis and are characterized by high
computational costs that limit their applicability in multi-frame scenarios,
where real-time decisions are crucial.
  To address this problem, this paper proposes an efficient attention-based
defense mechanism that exploits adversarial channel-attention to quickly
identify and track malicious objects in shallow network layers and mask their
adversarial effects in a multi-frame setting. This work advances the state of
the art by enhancing existing over-activation techniques for real-world
adversarial attacks to make them usable in real-time applications. It also
introduces an efficient multi-frame defense framework, validating its efficacy
through extensive experiments aimed at evaluating both defense performance and
computational cost.


http://arxiv.org/abs/2311.11225
TextGuard: Provable Defense against Backdoor Attacks on Text Classification. (82%)
Hengzhi Pei; Jinyuan Jia; Wenbo Guo; Bo Li; Dawn Song
  Backdoor attacks have become a major security threat for deploying machine
learning models in security-critical applications. Existing research endeavors
have proposed many defenses against backdoor attacks. Despite demonstrating
certain empirical defense efficacy, none of these techniques could provide a
formal and provable security guarantee against arbitrary attacks. As a result,
they can be easily broken by strong adaptive attacks, as shown in our
evaluation. In this work, we propose TextGuard, the first provable defense
against backdoor attacks on text classification. In particular, TextGuard first
divides the (backdoored) training data into sub-training sets, achieved by
splitting each training sentence into sub-sentences. This partitioning ensures
that a majority of the sub-training sets do not contain the backdoor trigger.
Subsequently, a base classifier is trained from each sub-training set, and
their ensemble provides the final prediction. We theoretically prove that when
the length of the backdoor trigger falls within a certain threshold, TextGuard
guarantees that its prediction will remain unaffected by the presence of the
triggers in training and testing inputs. In our evaluation, we demonstrate the
effectiveness of TextGuard on three benchmark text classification tasks,
surpassing the certification accuracy of existing certified defenses against
backdoor attacks. Furthermore, we propose additional strategies to enhance the
empirical performance of TextGuard. Comparisons with state-of-the-art empirical
defenses validate the superiority of TextGuard in countering multiple backdoor
attacks. Our code and data are available at
https://github.com/AI-secure/TextGuard.


http://arxiv.org/abs/2311.11206
Robust Network Slicing: Multi-Agent Policies, Adversarial Attacks, and Defensive Strategies. (1%)
Feng Wang; M. Cenk Gursoy; Senem Velipasalar
  In this paper, we present a multi-agent deep reinforcement learning (deep RL)
framework for network slicing in a dynamic environment with multiple base
stations and multiple users. In particular, we propose a novel deep RL
framework with multiple actors and centralized critic (MACC) in which actors
are implemented as pointer networks to fit the varying dimension of input. We
evaluate the performance of the proposed deep RL algorithm via simulations to
demonstrate its effectiveness. Subsequently, we develop a deep RL based jammer
with limited prior information and limited power budget. The goal of the jammer
is to minimize the transmission rates achieved with network slicing and thus
degrade the network slicing agents' performance. We design a jammer with both
listening and jamming phases and address jamming location optimization as well
as jamming channel optimization via deep RL. We evaluate the jammer at the
optimized location, generating interference attacks in the optimized set of
channels by switching between the jamming phase and listening phase. We show
that the proposed jammer can significantly reduce the victims' performance
without direct feedback or prior knowledge on the network slicing policies.
Finally, we devise a Nash-equilibrium-supervised policy ensemble mixed strategy
profile for network slicing (as a defensive measure) and jamming. We evaluate
the performance of the proposed policy ensemble algorithm by applying on the
network slicing agents and the jammer agent in simulations to show its
effectiveness.


http://arxiv.org/abs/2311.10366
Breaking Temporal Consistency: Generating Video Universal Adversarial Perturbations Using Image Models. (97%)
Hee-Seon Kim; Minji Son; Minbeom Kim; Myung-Joon Kwon; Changick Kim
  As video analysis using deep learning models becomes more widespread, the
vulnerability of such models to adversarial attacks is becoming a pressing
concern. In particular, Universal Adversarial Perturbation (UAP) poses a
significant threat, as a single perturbation can mislead deep learning models
on entire datasets. We propose a novel video UAP using image data and image
model. This enables us to take advantage of the rich image data and image
model-based studies available for video applications. However, there is a
challenge that image models are limited in their ability to analyze the
temporal aspects of videos, which is crucial for a successful video attack. To
address this challenge, we introduce the Breaking Temporal Consistency (BTC)
method, which is the first attempt to incorporate temporal information into
video attacks using image models. We aim to generate adversarial videos that
have opposite patterns to the original. Specifically, BTC-UAP minimizes the
feature similarity between neighboring frames in videos. Our approach is simple
but effective at attacking unseen video models. Additionally, it is applicable
to videos of varying lengths and invariant to temporal shifts. Our approach
surpasses existing methods in terms of effectiveness on various datasets,
including ImageNet, UCF-101, and Kinetics-400.


http://arxiv.org/abs/2311.10919
PACOL: Poisoning Attacks Against Continual Learners. (93%)
Huayu Li; Gregory Ditzler
  Continual learning algorithms are typically exposed to untrusted sources that
contain training data inserted by adversaries and bad actors. An adversary can
insert a small number of poisoned samples, such as mislabeled samples from
previously learned tasks, or intentional adversarial perturbed samples, into
the training datasets, which can drastically reduce the model's performance. In
this work, we demonstrate that continual learning systems can be manipulated by
malicious misinformation and present a new category of data poisoning attacks
specific for continual learners, which we refer to as {\em Poisoning Attacks
Against Continual Learners} (PACOL). The effectiveness of labeling flipping
attacks inspires PACOL; however, PACOL produces attack samples that do not
change the sample's label and produce an attack that causes catastrophic
forgetting. A comprehensive set of experiments shows the vulnerability of
commonly used generative replay and regularization-based continual learning
approaches against attack methods. We evaluate the ability of label-flipping
and a new adversarial poison attack, namely PACOL proposed in this work, to
force the continual learning system to forget the knowledge of a learned
task(s). More specifically, we compared the performance degradation of
continual learning systems trained on benchmark data streams with and without
poisoning attacks. Moreover, we discuss the stealthiness of the attacks in
which we test the success rate of data sanitization defense and other outlier
detection-based defenses for filtering out adversarial samples.


http://arxiv.org/abs/2311.10389
Two-Factor Authentication Approach Based on Behavior Patterns for Defeating Puppet Attacks. (1%)
Wenhao Wang; Guyue Li; Zhiming Chu; Haobo Li; Daniele Faccio
  Fingerprint traits are widely recognized for their unique qualities and
security benefits. Despite their extensive use, fingerprint features can be
vulnerable to puppet attacks, where attackers manipulate a reluctant but
genuine user into completing the authentication process. Defending against such
attacks is challenging due to the coexistence of a legitimate identity and an
illegitimate intent. In this paper, we propose PUPGUARD, a solution designed to
guard against puppet attacks. This method is based on user behavioral patterns,
specifically, the user needs to press the capture device twice successively
with different fingers during the authentication process. PUPGUARD leverages
both the image features of fingerprints and the timing characteristics of the
pressing intervals to establish two-factor authentication. More specifically,
after extracting image features and timing characteristics, and performing
feature selection on the image features, PUPGUARD fuses these two features into
a one-dimensional feature vector, and feeds it into a one-class classifier to
obtain the classification result. This two-factor authentication method
emphasizes dynamic behavioral patterns during the authentication process,
thereby enhancing security against puppet attacks. To assess PUPGUARD's
effectiveness, we conducted experiments on datasets collected from 31 subjects,
including image features and timing characteristics. Our experimental results
demonstrate that PUPGUARD achieves an impressive accuracy rate of 97.87% and a
remarkably low false positive rate (FPR) of 1.89%. Furthermore, we conducted
comparative experiments to validate the superiority of combining image features
and timing characteristics within PUPGUARD for enhancing resistance against
puppet attacks.


http://arxiv.org/abs/2311.09790
Breaking Boundaries: Balancing Performance and Robustness in Deep Wireless Traffic Forecasting. (99%)
Ilbert Romain; V. Hoang Thai; Zhang Zonghua; Palpanas Themis
  Balancing the trade-off between accuracy and robustness is a long-standing
challenge in time series forecasting. While most of existing robust algorithms
have achieved certain suboptimal performance on clean data, sustaining the same
performance level in the presence of data perturbations remains extremely hard.
% In this paper, we study a wide array of perturbation scenarios and propose
novel defense mechanisms against adversarial attacks using real-world telecom
data. We compare our strategy against two existing adversarial training
algorithms under a range of maximal allowed perturbations, defined using
$\ell_{\infty}$-norm, $\in [0.1,0.4]$. % Our findings reveal that our hybrid
strategy, which is composed of a classifier to detect adversarial examples, a
denoiser to eliminate noise from the perturbed data samples, and a standard
forecaster, achieves the best performance on both clean and perturbed data. %
Our optimal model can retain up to $92.02\%$ the performance of the original
forecasting model in terms of Mean Squared Error (MSE) on clean data, while
being more robust than the standard adversarially trained models on perturbed
data. Its MSE is 2.71$\times$ and 2.51$\times$ lower than those of comparing
methods on normal and perturbed data, respectively. In addition, the components
of our models can be trained in parallel, resulting in better computational
efficiency. % Our results indicate that we can optimally balance the trade-off
between the performance and robustness of forecasting models by improving the
classifier and denoiser, even in the presence of sophisticated and destructive
poisoning attacks.


http://arxiv.org/abs/2311.09948
Hijacking Large Language Models via Adversarial In-Context Learning. (75%)
Yao Qiang; Xiangyu Zhou; Dongxiao Zhu
  In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs
for specific tasks by utilizing labeled examples as demonstrations in the
precondition prompts. Despite its promising performance, ICL suffers from
instability with the choice and arrangement of examples. Additionally, crafted
adversarial attacks pose a notable threat to the robustness of ICL. However,
existing attacks are either easy to detect, rely on external models, or lack
specificity towards ICL. To address these issues, this work introduces a novel
transferable attack for ICL, aiming to hijack LLMs to generate the targeted
response. The proposed LLM hijacking attack leverages a gradient-based prompt
search method to learn and append imperceptible adversarial suffixes to the
in-context demonstrations. Extensive experimental results on various tasks and
datasets demonstrate the effectiveness of our LLM hijacking attack, resulting
in a distracted attention towards adversarial tokens, consequently leading to
the targeted unwanted outputs.


http://arxiv.org/abs/2311.09827
Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking. (54%)
Nan Xu; Fei Wang; Ben Zhou; Bang Zheng Li; Chaowei Xiao; Muhao Chen
  While large language models (LLMs) have demonstrated increasing power, they
have also given rise to a wide range of harmful behaviors. As representatives,
jailbreak attacks can provoke harmful or unethical responses from LLMs, even
after safety alignment. In this paper, we investigate a novel category of
jailbreak attacks specifically designed to target the cognitive structure and
processes of LLMs. Specifically, we analyze the safety vulnerability of LLMs in
the face of (1) multilingual cognitive overload, (2) veiled expression, and (3)
effect-to-cause reasoning. Different from previous jailbreak attacks, our
proposed cognitive overload is a black-box attack with no need for knowledge of
model architecture or access to model weights. Experiments conducted on
AdvBench and MasterKey reveal that various LLMs, including both popular
open-source model Llama 2 and the proprietary model ChatGPT, can be compromised
through cognitive overload. Motivated by cognitive psychology work on managing
cognitive load, we further investigate defending cognitive overload attack from
two perspectives. Empirical studies show that our cognitive overload from three
perspectives can jailbreak all studied LLMs successfully, while existing
defense strategies can hardly mitigate the caused malicious uses effectively.


http://arxiv.org/abs/2311.09763
Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations. (38%)
Wenjie Mo; Jiashu Xu; Qin Liu; Jiongxiao Wang; Jun Yan; Chaowei Xiao; Muhao Chen
  Existing studies in backdoor defense have predominantly focused on the
training phase, overlooking the critical aspect of testing time defense. This
gap becomes particularly pronounced in the context of Large Language Models
(LLMs) deployed as Web Services, which typically offer only black-box access,
rendering training-time defenses impractical. To bridge this gap, our work
introduces defensive demonstrations, an innovative backdoor defense strategy
for blackbox large language models. Our method involves identifying the task
and retrieving task-relevant demonstrations from an uncontaminated pool. These
demonstrations are then combined with user queries and presented to the model
during testing, without requiring any modifications/tuning to the black-box
model or insights into its internal mechanisms. Defensive demonstrations are
designed to counteract the adverse effects of triggers, aiming to recalibrate
and correct the behavior of poisoned models during test-time evaluations.
Extensive experiments show that defensive demonstrations are effective in
defending both instance-level and instruction-level backdoor attacks, not only
rectifying the behavior of poisoned models but also surpassing existing
baselines in most scenarios.


http://arxiv.org/abs/2311.09641
On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models. (13%)
Jiongxiao Wang; Junlin Wu; Muhao Chen; Yevgeniy Vorobeychik; Chaowei Xiao
  Reinforcement Learning with Human Feedback (RLHF) is a methodology designed
to align Large Language Models (LLMs) with human preferences, playing an
important role in LLMs alignment. Despite its advantages, RLHF relies on human
annotators to rank the text, which can introduce potential security
vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the
ranking score by up-ranking any malicious text to steer the LLM adversarially.
To assess the red-teaming of RLHF against human preference data poisoning, we
propose RankPoison, a poisoning attack method on candidates' selection of
preference rank flipping to reach certain malicious behaviors (e.g., generating
longer sequences, which can increase the computational cost). With poisoned
dataset generated by RankPoison, we can perform poisoning attacks on LLMs to
generate longer tokens without hurting the original safety alignment
performance. Moreover, applying RankPoison, we also successfully implement a
backdoor attack where LLMs can generate longer answers under questions with the
trigger word. Our findings highlight critical security challenges in RLHF,
underscoring the necessity for more robust alignment methods for LLMs.


http://arxiv.org/abs/2311.10177
Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts. (2%)
Shashank Kotyan; Danilo Vasconcellos Vargas
  Neural networks have demonstrated significant accuracy across various
domains, yet their vulnerability to subtle input alterations remains a
persistent challenge. Conventional methods like data augmentation, while
effective to some extent, fall short in addressing unforeseen corruptions,
limiting the adaptability of neural networks in real-world scenarios. In
response, this paper introduces a novel paradigm known as the Mixture of
Class-Specific Expert Architecture. The approach involves disentangling feature
learning for individual classes, offering a nuanced enhancement in scalability
and overall performance. By training dedicated network segments for each class
and subsequently aggregating their outputs, the proposed architecture aims to
mitigate vulnerabilities associated with common neural network structures. The
study underscores the importance of comprehensive evaluation methodologies,
advocating for the incorporation of benchmarks like the common corruptions
benchmark. This inclusion provides nuanced insights into the vulnerabilities of
neural networks, especially concerning their generalization capabilities and
robustness to unforeseen distortions. The research aligns with the broader
objective of advancing the development of highly robust learning systems
capable of nuanced reasoning across diverse and challenging real-world
scenarios. Through this contribution, the paper aims to foster a deeper
understanding of neural network limitations and proposes a practical approach
to enhance their resilience in the face of evolving and unpredictable
conditions.


http://arxiv.org/abs/2311.16169
Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities. (2%)
Avishree Khare; Saikat Dutta; Ziyang Li; Alaia Solko-Breslin; Rajeev Alur; Mayur Naik
  Security vulnerabilities in modern software are prevalent and harmful. While
automated vulnerability detection tools have made promising progress, their
scalability and applicability remain challenging. Recently, Large Language
Models (LLMs), such as GPT-4 and CodeLlama, have demonstrated remarkable
performance on code-related tasks. However, it is unknown whether such LLMs can
do complex reasoning over code. In this work, we explore whether pre-trained
LLMs can detect security vulnerabilities and address the limitations of
existing tools. We evaluate the effectiveness of pre-trained LLMs on a set of
five diverse security benchmarks spanning two languages, Java and C/C++, and
including code samples from synthetic and real-world projects. We evaluate the
effectiveness of LLMs in terms of their performance, explainability, and
robustness.
  By designing a series of effective prompting strategies, we obtain the best
results on the synthetic datasets with GPT-4: F1 scores of 0.79 on OWASP, 0.86
on Juliet Java, and 0.89 on Juliet C/C++. Expectedly, the performance of LLMs
drops on the more challenging real-world datasets: CVEFixes Java and CVEFixes
C/C++, with GPT-4 reporting F1 scores of 0.48 and 0.62, respectively. We show
that LLMs can often perform better than existing static analysis and deep
learning-based vulnerability detection tools, especially for certain classes of
vulnerabilities. Moreover, LLMs also often provide reliable explanations,
identifying the vulnerable data flows in code. We find that fine-tuning smaller
LLMs can outperform the larger LLMs on synthetic datasets but provide limited
gains on real-world datasets. When subjected to adversarial attacks on code,
LLMs show mild degradation, with average accuracy reduction of up to 12.67%.
Finally, we share our insights and recommendations for future work on
leveraging LLMs for vulnerability detection.


http://arxiv.org/abs/2311.09994
Towards more Practical Threat Models in Artificial Intelligence Security. (2%)
Kathrin Grosse; Lukas Bieringer; Tarek Richard Besold; Alexandre Alahi
  Recent works have identified a gap between research and practice in
artificial intelligence security: threats studied in academia do not always
reflect the practical use and security risks of AI. For example, while models
are often studied in isolation, they form part of larger ML pipelines in
practice. Recent works also brought forward that adversarial manipulations
introduced by academic attacks are impractical. We take a first step towards
describing the full extent of this disparity. To this end, we revisit the
threat models of the six most studied attacks in AI security research and match
them to AI usage in practice via a survey with \textbf{271} industrial
practitioners. On the one hand, we find that all existing threat models are
indeed applicable. On the other hand, there are significant mismatches:
research is often too generous with the attacker, assuming access to
information not frequently available in real-world settings. Our paper is thus
a call for action to study more practical threat models in artificial
intelligence security.


http://arxiv.org/abs/2311.10197
You Cannot Escape Me: Detecting Evasions of SIEM Rules in Enterprise Networks. (1%)
Rafael Uetz; Marco Herzog; Louis Hackländer; Simon Schwarz; Martin Henze
  Cyberattacks have grown into a major risk for organizations, with common
consequences being data theft, sabotage, and extortion. Since preventive
measures do not suffice to repel attacks, timely detection of successful
intruders is crucial to stop them from reaching their final goals. For this
purpose, many organizations utilize Security Information and Event Management
(SIEM) systems to centrally collect security-related events and scan them for
attack indicators using expert-written detection rules. However, as we show by
analyzing a set of widespread SIEM detection rules, adversaries can evade
almost half of them easily, allowing them to perform common malicious actions
within an enterprise network without being detected. To remedy these critical
detection blind spots, we propose the idea of adaptive misuse detection, which
utilizes machine learning to compare incoming events to SIEM rules on the one
hand and known-benign events on the other hand to discover successful evasions.
Based on this idea, we present AMIDES, an open-source proof-of-concept adaptive
misuse detection system. Using four weeks of SIEM events from a large
enterprise network and more than 500 hand-crafted evasions, we show that AMIDES
successfully detects a majority of these evasions without any false alerts. In
addition, AMIDES eases alert analysis by assessing which rules were evaded. Its
computational efficiency qualifies AMIDES for real-world operation and hence
enables organizations to significantly reduce detection blind spots with
moderate effort.


http://arxiv.org/abs/2311.09127
Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts. (99%)
Yuanwei Wu; Xiang Li; Yixin Liu; Pan Zhou; Lichao Sun
  Existing work on jailbreak Multimodal Large Language Models (MLLMs) has
focused primarily on adversarial examples in model inputs, with less attention
to vulnerabilities in model APIs. To fill the research gap, we carry out the
following work: 1) We discover a system prompt leakage vulnerability in GPT-4V.
Through carefully designed dialogue, we successfully steal the internal system
prompts of GPT-4V. This finding indicates potential exploitable security risks
in MLLMs; 2)Based on the acquired system prompts, we propose a novel MLLM
jailbreaking attack method termed SASP (Self-Adversarial Attack via System
Prompt). By employing GPT-4 as a red teaming tool against itself, we aim to
search for potential jailbreak prompts leveraging stolen system prompts.
Furthermore, in pursuit of better performance, we also add human modification
based on GPT-4's analysis, which further improves the attack success rate to
98.7\%; 3) We evaluated the effect of modifying system prompts to defend
against jailbreaking attacks. Results show that appropriately designed system
prompts can significantly reduce jailbreak success rates. Overall, our work
provides new insights into enhancing MLLM security, demonstrating the important
role of system prompts in jailbreaking, which could be leveraged to greatly
facilitate jailbreak success rates while also holding the potential for
defending against jailbreaks.


http://arxiv.org/abs/2311.09433
Backdoor Activation Attack: Attack Large Language Models using Activation Steering for Safety-Alignment. (74%)
Haoran Wang; Kai Shu
  To ensure AI safety, instruction-tuned Large Language Models (LLMs) are
specifically trained to ensure alignment, which refers to making models behave
in accordance with human intentions. While these models have demonstrated
commendable results on various safety benchmarks, the vulnerability of their
safety alignment has not been extensively studied. This is particularly
troubling given the potential harm that LLMs can inflict. Existing attack
methods on LLMs often rely on poisoned training data or the injection of
malicious prompts. These approaches compromise the stealthiness and
generalizability of the attacks, making them susceptible to detection.
Additionally, these models often demand substantial computational resources for
implementation, making them less practical for real-world applications.
Inspired by recent success in modifying model behavior through steering vectors
without the need for optimization, and drawing on its effectiveness in
red-teaming LLMs, we conducted experiments employing activation steering to
target four key aspects of LLMs: truthfulness, toxicity, bias, and harmfulness
- across a varied set of attack settings. To establish a universal attack
strategy applicable to diverse target alignments without depending on manual
analysis, we automatically select the intervention layer based on contrastive
layer search. Our experiment results show that activation attacks are highly
effective and add little or no overhead to attack efficiency. Additionally, we
discuss potential countermeasures against such activation attacks. Our code and
data are available at https://github.com/wang2226/Backdoor-Activation-Attack
Warning: this paper contains content that can be offensive or upsetting.


http://arxiv.org/abs/2311.09024
Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing. (64%)
A K Iowa State University Nirala; A New York University Joshi; C New York University Hegde; S Iowa State University Sarkar
  A key benefit of deep vision-language models such as CLIP is that they enable
zero-shot open vocabulary classification; the user has the ability to define
novel class labels via natural language prompts at inference time. However,
while CLIP-based zero-shot classifiers have demonstrated competitive
performance across a range of domain shifts, they remain highly vulnerable to
adversarial attacks. Therefore, ensuring the robustness of such models is
crucial for their reliable deployment in the wild.
  In this work, we introduce Open Vocabulary Certification (OVC), a fast
certification method designed for open-vocabulary models like CLIP via
randomized smoothing techniques. Given a base "training" set of prompts and
their corresponding certified CLIP classifiers, OVC relies on the observation
that a classifier with a novel prompt can be viewed as a perturbed version of
nearby classifiers in the base training set. Therefore, OVC can rapidly certify
the novel classifier using a variation of incremental randomized smoothing. By
using a caching trick, we achieve approximately two orders of magnitude
acceleration in the certification process for novel prompts. To achieve further
(heuristic) speedups, OVC approximates the embedding space at a given input
using a multivariate normal distribution bypassing the need for sampling via
forward passes through the vision backbone. We demonstrate the effectiveness of
OVC on through experimental evaluation using multiple vision-language backbones
on the CIFAR-10 and ImageNet test datasets.


http://arxiv.org/abs/2311.09266
Adversarially Robust Spiking Neural Networks Through Conversion. (26%)
Ozan Özdenizci; Robert Legenstein
  Spiking neural networks (SNNs) provide an energy-efficient alternative to a
variety of artificial neural network (ANN) based AI applications. As the
progress in neuromorphic computing with SNNs expands their use in applications,
the problem of adversarial robustness of SNNs becomes more pronounced. To the
contrary of the widely explored end-to-end adversarial training based
solutions, we address the limited progress in scalable robust SNN training
methods by proposing an adversarially robust ANN-to-SNN conversion algorithm.
Our method provides an efficient approach to embrace various computationally
demanding robust learning objectives that have been proposed for ANNs. During a
post-conversion robust finetuning phase, our method adversarially optimizes
both layer-wise firing thresholds and synaptic connectivity weights of the SNN
to maintain transferred robustness gains from the pre-trained ANN. We perform
experimental evaluations in numerous adaptive adversarial settings that account
for the spike-based operation dynamics of SNNs, and show that our approach
yields a scalable state-of-the-art solution for adversarially robust deep SNNs
with low-latency.


http://arxiv.org/abs/2311.09096
Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization. (15%)
Zhexin Zhang; Junxiao Yang; Pei Ke; Minlie Huang
  Large Language Models (LLMs) continue to advance in their capabilities, yet
this progress is accompanied by a growing array of safety risks. While
significant attention has been dedicated to exploiting weaknesses in LLMs
through jailbreaking attacks, there remains a paucity of exploration into
defending against these attacks. We point out a pivotal factor contributing to
the success of jailbreaks: the inherent conflict between the goals of being
helpful and ensuring safety. To counter jailbreaking attacks, we propose to
integrate goal prioritization at both training and inference stages.
Implementing goal prioritization during inference substantially diminishes the
Attack Success Rate (ASR) of jailbreaking attacks, reducing it from 66.4% to
2.0% for ChatGPT and from 68.2% to 19.4% for Vicuna-33B, without compromising
general performance. Furthermore, integrating the concept of goal
prioritization into the training phase reduces the ASR from 71.0% to 6.6% for
LLama2-13B. Remarkably, even in scenarios where no jailbreaking samples are
included during training, our approach slashes the ASR by half, decreasing it
from 71.0% to 34.0%. Additionally, our findings reveal that while stronger LLMs
face greater safety risks, they also possess a greater capacity to be steered
towards defending against such attacks. We hope our work could contribute to
the comprehension of jailbreaking attacks and defenses, and shed light on the
relationship between LLMs' capability and safety. Our code will be available at
\url{https://github.com/thu-coai/JailbreakDefense_GoalPriority}.


http://arxiv.org/abs/2311.09355
Privacy Threats in Stable Diffusion Models. (13%)
Thomas Cilloni; Charles Fleming; Charles Walter
  This paper introduces a novel approach to membership inference attacks (MIA)
targeting stable diffusion computer vision models, specifically focusing on the
highly sophisticated Stable Diffusion V2 by StabilityAI. MIAs aim to extract
sensitive information about a model's training data, posing significant privacy
concerns. Despite its advancements in image synthesis, our research reveals
privacy vulnerabilities in the stable diffusion models' outputs. Exploiting
this information, we devise a black-box MIA that only needs to query the victim
model repeatedly. Our methodology involves observing the output of a stable
diffusion model at different generative epochs and training a classification
model to distinguish when a series of intermediates originated from a training
sample or not. We propose numerous ways to measure the membership features and
discuss what works best. The attack's efficacy is assessed using the ROC AUC
method, demonstrating a 60\% success rate in inferring membership information.
This paper contributes to the growing body of research on privacy and security
in machine learning, highlighting the need for robust defenses against MIAs.
Our findings prompt a reevaluation of the privacy implications of stable
diffusion models, urging practitioners and developers to implement enhanced
security measures to safeguard against such attacks.


http://arxiv.org/abs/2311.09447
How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities. (9%)
Lingbo Mo; Boshi Wang; Muhao Chen; Huan Sun
  The rapid progress in open-source Large Language Models (LLMs) is
significantly driving AI development forward. However, there is still a limited
understanding of their trustworthiness. Deploying these models at scale without
sufficient trustworthiness can pose significant risks, highlighting the need to
uncover these issues promptly. In this work, we conduct an assessment of
open-source LLMs on trustworthiness, scrutinizing them across eight different
aspects including toxicity, stereotypes, ethics, hallucination, fairness,
sycophancy, privacy, and robustness against adversarial demonstrations. We
propose an enhanced Chain of Utterances-based (CoU) prompting strategy by
incorporating meticulously crafted malicious demonstrations for trustworthiness
attack. Our extensive experiments encompass recent and representative series of
open-source LLMs, including Vicuna, MPT, Falcon, Mistral, and Llama 2. The
empirical outcomes underscore the efficacy of our attack strategy across
diverse aspects. More interestingly, our result analysis reveals that models
with superior performance in general NLP tasks do not always have greater
trustworthiness; in fact, larger models can be more vulnerable to attacks.
Additionally, models that have undergone instruction tuning, focusing on
instruction following, tend to be more susceptible, although fine-tuning LLMs
for safety alignment proves effective in mitigating adversarial trustworthiness
attacks.


http://arxiv.org/abs/2311.09489
MirrorNet: A TEE-Friendly Framework for Secure On-device DNN Inference. (2%)
Ziyu Liu; Yukui Luo; Shijin Duan; Tong Zhou; Xiaolin Xu
  Deep neural network (DNN) models have become prevalent in edge devices for
real-time inference. However, they are vulnerable to model extraction attacks
and require protection. Existing defense approaches either fail to fully
safeguard model confidentiality or result in significant latency issues. To
overcome these challenges, this paper presents MirrorNet, which leverages
Trusted Execution Environment (TEE) to enable secure on-device DNN inference.
It generates a TEE-friendly implementation for any given DNN model to protect
the model confidentiality, while meeting the stringent computation and storage
constraints of TEE. The framework consists of two key components: the backbone
model (BackboneNet), which is stored in the normal world but achieves lower
inference accuracy, and the Companion Partial Monitor (CPM), a lightweight
mirrored branch stored in the secure world, preserving model confidentiality.
During inference, the CPM monitors the intermediate results from the
BackboneNet and rectifies the classification output to achieve higher accuracy.
To enhance flexibility, MirrorNet incorporates two modules: the CPM Strategy
Generator, which generates various protection strategies, and the Performance
Emulator, which estimates the performance of each strategy and selects the most
optimal one. Extensive experiments demonstrate the effectiveness of MirrorNet
in providing security guarantees while maintaining low computation latency,
making MirrorNet a practical and promising solution for secure on-device DNN
inference. For the evaluation, MirrorNet can achieve a 18.6% accuracy gap
between authenticated and illegal use, while only introducing 0.99% hardware
overhead.


http://arxiv.org/abs/2311.09428
Beyond Detection: Unveiling Fairness Vulnerabilities in Abusive Language Models. (1%)
Yueqing Liang; Lu Cheng; Ali Payani; Kai Shu
  This work investigates the potential of undermining both fairness and
detection performance in abusive language detection. In a dynamic and complex
digital world, it is crucial to investigate the vulnerabilities of these
detection models to adversarial fairness attacks to improve their fairness
robustness. We propose a simple yet effective framework FABLE that leverages
backdoor attacks as they allow targeted control over the fairness and detection
performance. FABLE explores three types of trigger designs (i.e., rare,
artificial, and natural triggers) and novel sampling strategies. Specifically,
the adversary can inject triggers into samples in the minority group with the
favored outcome (i.e., ``non-abusive'') and flip their labels to the unfavored
outcome, i.e., ``abusive''. Experiments on benchmark datasets demonstrate the
effectiveness of FABLE attacking fairness and utility in abusive language
detection.


http://arxiv.org/abs/2311.09473
JAB: Joint Adversarial Prompting and Belief Augmentation. (1%)
Ninareh Mehrabi; Palash Goyal; Anil Ramakrishna; Jwala Dhamala; Shalini Ghosh; Richard Zemel; Kai-Wei Chang; Aram Galstyan; Rahul Gupta
  With the recent surge of language models in different applications, attention
to safety and robustness of these models has gained significant importance.
Here we introduce a joint framework in which we simultaneously probe and
improve the robustness of a black-box target model via adversarial prompting
and belief augmentation using iterative feedback loops. This framework utilizes
an automated red teaming approach to probe the target model, along with a
belief augmenter to generate instructions for the target model to improve its
robustness to those adversarial probes. Importantly, the adversarial model and
the belief generator leverage the feedback from past interactions to improve
the effectiveness of the adversarial prompts and beliefs, respectively. In our
experiments, we demonstrate that such a framework can reduce toxic content
generation both in dynamic cases where an adversary directly interacts with a
target model and static cases where we use a static benchmark dataset to
evaluate our model.


http://arxiv.org/abs/2311.07928
Towards Improving Robustness Against Common Corruptions in Object Detectors Using Adversarial Contrastive Learning. (99%)
Shashank Kotyan; Danilo Vasconcellos Vargas
  Neural networks have revolutionized various domains, exhibiting remarkable
accuracy in tasks like natural language processing and computer vision.
However, their vulnerability to slight alterations in input samples poses
challenges, particularly in safety-critical applications like autonomous
driving. Current approaches, such as introducing distortions during training,
fall short in addressing unforeseen corruptions. This paper proposes an
innovative adversarial contrastive learning framework to enhance neural network
robustness simultaneously against adversarial attacks and common corruptions.
By generating instance-wise adversarial examples and optimizing contrastive
loss, our method fosters representations that resist adversarial perturbations
and remain robust in real-world scenarios. Subsequent contrastive learning then
strengthens the similarity between clean samples and their adversarial
counterparts, fostering representations resistant to both adversarial attacks
and common distortions. By focusing on improving performance under adversarial
and real-world conditions, our approach aims to bolster the robustness of
neural networks in safety-critical applications, such as autonomous vehicles
navigating unpredictable weather conditions. We anticipate that this framework
will contribute to advancing the reliability of neural networks in challenging
environments, facilitating their widespread adoption in mission-critical
scenarios.


http://arxiv.org/abs/2311.08539
Physical Adversarial Examples for Multi-Camera Systems. (99%)
Ana Răduţoiu; Jan-Philipp Schulze; Philip Sperl; Konstantin Böttinger
  Neural networks build the foundation of several intelligent systems, which,
however, are known to be easily fooled by adversarial examples. Recent advances
made these attacks possible even in air-gapped scenarios, where the autonomous
system observes its surroundings by, e.g., a camera. We extend these ideas in
our research and evaluate the robustness of multi-camera setups against such
physical adversarial examples. This scenario becomes ever more important with
the rise in popularity of autonomous vehicles, which fuse the information of
several cameras for their driving decision. While we find that multi-camera
setups provide some robustness towards past attack methods, we see that this
advantage reduces when optimizing on multiple perspectives at once. We propose
a novel attack method that we call Transcender-MC, where we incorporate online
3D renderings and perspective projections in the training process. Moreover, we
motivate that certain data augmentation techniques can facilitate the
generation of successful adversarial examples even further. Transcender-MC is
11% more effective in successfully attacking multi-camera setups than
state-of-the-art methods. Our findings offer valuable insights regarding the
resilience of object detection in a setup with multiple cameras and motivate
the need of developing adequate defense mechanisms against them.


http://arxiv.org/abs/2311.08598
DALA: A Distribution-Aware LoRA-Based Adversarial Attack against Pre-trained Language Models. (99%)
Yibo Wang; Xiangjue Dong; James Caverlee; Philip S. Yu
  Pre-trained language models (PLMs) that achieve success in applications are
susceptible to adversarial attack methods that are capable of generating
adversarial examples with minor perturbations. Although recent attack methods
can achieve a relatively high attack success rate (ASR), our observation shows
that the generated adversarial examples have a different data distribution
compared with the original examples. Specifically, these adversarial examples
exhibit lower confidence levels and higher distance to the training data
distribution. As a result, they are easy to detect using very simple detection
methods, diminishing the actual effectiveness of these attack methods. To solve
this problem, we propose a Distribution-Aware LoRA-based Adversarial Attack
(DALA) method, which considers the distribution shift of adversarial examples
to improve attack effectiveness under detection methods. We further design a
new evaluation metric NASR combining ASR and detection for the attack task. We
conduct experiments on four widely-used datasets and validate the attack
effectiveness on ASR and NASR of the adversarial examples generated by DALA on
the BERT-base model and the black-box LLaMA2-7b model.


http://arxiv.org/abs/2311.08265
On The Relationship Between Universal Adversarial Attacks And Sparse Representations. (98%)
Dana Weitzner; Raja Giryes
  The prominent success of neural networks, mainly in computer vision tasks, is
increasingly shadowed by their sensitivity to small, barely perceivable
adversarial perturbations in image input.
  In this work, we aim at explaining this vulnerability through the framework
of sparsity.
  We show the connection between adversarial attacks and sparse
representations, with a focus on explaining the universality and
transferability of adversarial examples in neural networks.
  To this end, we show that sparse coding algorithms, and the neural
network-based learned iterative shrinkage thresholding algorithm (LISTA) among
them, suffer from this sensitivity, and that common attacks on neural networks
can be expressed as attacks on the sparse representation of the input image.
The phenomenon that we observe holds true also when the network is agnostic to
the sparse representation and dictionary, and thus can provide a possible
explanation for the universality and transferability of adversarial attacks.
  The code is available at
https://github.com/danawr/adversarial_attacks_and_sparse_representations.


http://arxiv.org/abs/2311.08268
A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. (56%)
Peng Ding; Jun Kuang; Dan Ma; Xuezhi Cao; Yunsen Xian; Jiajun Chen; Shujian Huang
  Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to
provide useful and safe responses. However, adversarial prompts known as
'jailbreaks' can circumvent safeguards, leading LLMs to generate harmful
content. Exploring jailbreak prompts can help to better reveal the weaknesses
of LLMs and further steer us to secure them. Unfortunately, existing jailbreak
methods either suffer from intricate manual design or require optimization on
another white-box model, compromising generalization or jailbreak efficiency.
In this paper, we generalize jailbreak prompt attacks into two aspects: (1)
Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM,
an automatic framework that leverages LLMs themselves to generate effective
jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly
improves the attack success rate while greatly reducing the time cost compared
to existing baselines. Our study also reveals the inadequacy of current defense
methods in safeguarding LLMs. Finally, we offer detailed analysis and
discussion from the perspective of prompt execution priority on the failure of
LLMs' defense. We hope that our research can catalyze both the academic
community and LLMs vendors towards the provision of safer and more regulated
Large Language Models.


http://arxiv.org/abs/2311.08662
Multi-Set Inoculation: Assessing Model Robustness Across Multiple Challenge Sets. (13%)
Vatsal Gupta; Pranshu Pandya; Tushar Kataria; Vivek Gupta; Dan Roth
  Language models, given their black-box nature, often exhibit sensitivity to
input perturbations, leading to trust issues due to hallucinations. To bolster
trust, it's essential to understand these models' failure modes and devise
strategies to enhance their performance. In this study, we propose a framework
to study the effect of input perturbations on language models of different
scales, from pre-trained models to large language models (LLMs). We use
fine-tuning to train a robust model to perturbations, and we investigate
whether exposure to one perturbation improves or degrades the model's
performance on other perturbations. To address multi-perturbation robustness,
we suggest three distinct training strategies. We also extend the framework to
LLMs via a chain of thought(COT) prompting with exemplars. We instantiate our
framework for the Tabular-NLI task and show that the proposed strategies train
the model robust to different perturbations without losing accuracy on a given
dataset.


http://arxiv.org/abs/2311.09253
The Perception-Robustness Tradeoff in Deterministic Image Restoration. (1%)
Guy Ohayon; Tomer Michaeli; Michael Elad
  We study the behavior of deterministic methods for solving inverse problems
in imaging. These methods are commonly designed to achieve two goals: (1)
attaining high perceptual quality, and (2) generating reconstructions that are
consistent with the measurements. We provide a rigorous proof that the better a
predictor satisfies these two requirements, the larger its Lipschitz constant
must be, regardless of the nature of the degradation involved. In particular,
to approach perfect perceptual quality and perfect consistency, the Lipschitz
constant of the model must grow to infinity. This implies that such methods are
necessarily more susceptible to adversarial attacks. We demonstrate our theory
on single image super-resolution algorithms, addressing both noisy and
noiseless settings. We also show how this undesired behavior can be leveraged
to explore the posterior distribution, thereby allowing the deterministic model
to imitate stochastic methods.


http://arxiv.org/abs/2311.07110
Adversarial Purification for Data-Driven Power System Event Classifiers with Diffusion Models. (99%)
Yuanbin Cheng; Koji Yamashita; Jim Follum; Nanpeng Yu
  The global deployment of the phasor measurement units (PMUs) enables
real-time monitoring of the power system, which has stimulated considerable
research into machine learning-based models for event detection and
classification. However, recent studies reveal that machine learning-based
methods are vulnerable to adversarial attacks, which can fool the event
classifiers by adding small perturbations to the raw PMU data. To mitigate the
threats posed by adversarial attacks, research on defense strategies is
urgently needed. This paper proposes an effective adversarial purification
method based on the diffusion model to counter adversarial attacks on the
machine learning-based power system event classifier. The proposed method
includes two steps: injecting noise into the PMU data; and utilizing a
pre-trained neural network to eliminate the added noise while simultaneously
removing perturbations introduced by the adversarial attacks. The proposed
adversarial purification method significantly increases the accuracy of the
event classifier under adversarial attacks while satisfying the requirements of
real-time operations. In addition, the theoretical analysis reveals that the
proposed diffusion model-based adversarial purification method decreases the
distance between the original and compromised PMU data, which reduces the
impacts of adversarial attacks. The empirical results on a large-scale
real-world PMU dataset validate the effectiveness and computational efficiency
of the proposed adversarial purification method.


http://arxiv.org/abs/2311.07780
Parrot-Trained Adversarial Examples: Pushing the Practicality of Black-Box Audio Attacks against Speaker Recognition Models. (99%)
Rui Duan; Zhe Qu; Leah Ding; Yao Liu; Zhuo Lu
  Audio adversarial examples (AEs) have posed significant security challenges
to real-world speaker recognition systems. Most black-box attacks still require
certain information from the speaker recognition model to be effective (e.g.,
keeping probing and requiring the knowledge of similarity scores). This work
aims to push the practicality of the black-box attacks by minimizing the
attacker's knowledge about a target speaker recognition model. Although it is
not feasible for an attacker to succeed with completely zero knowledge, we
assume that the attacker only knows a short (or a few seconds) speech sample of
a target speaker. Without any probing to gain further knowledge about the
target model, we propose a new mechanism, called parrot training, to generate
AEs against the target model. Motivated by recent advancements in voice
conversion (VC), we propose to use the one short sentence knowledge to generate
more synthetic speech samples that sound like the target speaker, called parrot
speech. Then, we use these parrot speech samples to train a parrot-trained(PT)
surrogate model for the attacker. Under a joint transferability and perception
framework, we investigate different ways to generate AEs on the PT model
(called PT-AEs) to ensure the PT-AEs can be generated with high transferability
to a black-box target model with good human perceptual quality. Real-world
experiments show that the resultant PT-AEs achieve the attack success rates of
45.8% - 80.8% against the open-source models in the digital-line scenario and
47.9% - 58.3% against smart devices, including Apple HomePod (Siri), Amazon
Echo, and Google Home, in the over-the-air scenario.


http://arxiv.org/abs/2311.07553
An Extensive Study on Adversarial Attack against Pre-trained Models of Code. (99%)
Xiaohu Du; Ming Wen; Zichao Wei; Shangwen Wang; Hai Jin
  Transformer-based pre-trained models of code (PTMC) have been widely utilized
and have achieved state-of-the-art performance in many mission-critical
applications. However, they can be vulnerable to adversarial attacks through
identifier substitution or coding style transformation, which can significantly
degrade accuracy and may further incur security concerns. Although several
approaches have been proposed to generate adversarial examples for PTMC, the
effectiveness and efficiency of such approaches, especially on different code
intelligence tasks, has not been well understood. To bridge this gap, this
study systematically analyzes five state-of-the-art adversarial attack
approaches from three perspectives: effectiveness, efficiency, and the quality
of generated examples. The results show that none of the five approaches
balances all these perspectives. Particularly, approaches with a high attack
success rate tend to be time-consuming; the adversarial code they generate
often lack naturalness, and vice versa. To address this limitation, we explore
the impact of perturbing identifiers under different contexts and find that
identifier substitution within for and if statements is the most effective.
Based on these findings, we propose a new approach that prioritizes different
types of statements for various tasks and further utilizes beam search to
generate adversarial examples. Evaluation results show that it outperforms the
state-of-the-art ALERT in terms of both effectiveness and efficiency while
preserving the naturalness of the generated adversarial examples.


http://arxiv.org/abs/2311.07127
Untargeted Black-box Attacks for Social Recommendations. (96%)
Wenqi Fan; Shijie Wang; Xiao-yong Wei; Xiaowei Mei; Qing Li
  The rise of online social networks has facilitated the evolution of social
recommender systems, which incorporate social relations to enhance users'
decision-making process. With the great success of Graph Neural Networks in
learning node representations, GNN-based social recommendations have been
widely studied to model user-item interactions and user-user social relations
simultaneously. Despite their great successes, recent studies have shown that
these advanced recommender systems are highly vulnerable to adversarial
attacks, in which attackers can inject well-designed fake user profiles to
disrupt recommendation performances. While most existing studies mainly focus
on targeted attacks to promote target items on vanilla recommender systems,
untargeted attacks to degrade the overall prediction performance are less
explored on social recommendations under a black-box scenario. To perform
untargeted attacks on social recommender systems, attackers can construct
malicious social relationships for fake users to enhance the attack
performance. However, the coordination of social relations and item profiles is
challenging for attacking black-box social recommendations. To address this
limitation, we first conduct several preliminary studies to demonstrate the
effectiveness of cross-community connections and cold-start items in degrading
recommendations performance. Specifically, we propose a novel framework
Multiattack based on multi-agent reinforcement learning to coordinate the
generation of cold-start item profiles and cross-community social relations for
conducting untargeted attacks on black-box social recommendations.
Comprehensive experiments on various real-world datasets demonstrate the
effectiveness of our proposed attacking framework under the black-box setting.


http://arxiv.org/abs/2311.07444
On the Robustness of Neural Collapse and the Neural Collapse of Robustness. (80%)
Jingtong Su; Ya Shi Zhang; Nikolaos Tsilivis; Julia Kempe
  Neural Collapse refers to the curious phenomenon in the end of training of a
neural network, where feature vectors and classification weights converge to a
very simple geometrical arrangement (a simplex). While it has been observed
empirically in various cases and has been theoretically motivated, its
connection with crucial properties of neural networks, like their
generalization and robustness, remains unclear. In this work, we study the
stability properties of these simplices. We find that the simplex structure
disappears under small adversarial attacks, and that perturbed examples "leap"
between simplex vertices. We further analyze the geometry of networks that are
optimized to be robust against adversarial perturbations of the input, and find
that Neural Collapse is a pervasive phenomenon in these cases as well, with
clean and perturbed representations forming aligned simplices, and giving rise
to a robust simple nearest-neighbor classifier. By studying the propagation of
the amount of collapse inside the network, we identify novel properties of both
robust and non-robust machine learning models, and show that earlier, unlike
later layers maintain reliable simplices on perturbed data.


http://arxiv.org/abs/2311.07550
Tabdoor: Backdoor Vulnerabilities in Transformer-based Neural Networks for Tabular Data. (41%)
Bart Pleiter; Behrad Tajalli; Stefanos Koffas; Gorka Abad; Jing Xu; Martha Larson; Stjepan Picek
  Deep neural networks (DNNs) have shown great promise in various domains.
Alongside these developments, vulnerabilities associated with DNN training,
such as backdoor attacks, are a significant concern. These attacks involve the
subtle insertion of triggers during model training, allowing for manipulated
predictions. More recently, DNNs for tabular data have gained increasing
attention due to the rise of transformer models.
  Our research presents a comprehensive analysis of backdoor attacks on tabular
data using DNNs, particularly focusing on transformer-based networks. Given the
inherent complexities of tabular data, we explore the challenges of embedding
backdoors. Through systematic experimentation across benchmark datasets, we
uncover that transformer-based DNNs for tabular data are highly susceptible to
backdoor attacks, even with minimal feature value alterations. Our results
indicate nearly perfect attack success rates (approx100%) by introducing novel
backdoor attack strategies to tabular data. Furthermore, we evaluate several
defenses against these attacks, identifying Spectral Signatures as the most
effective one. Our findings highlight the urgency to address such
vulnerabilities and provide insights into potential countermeasures for
securing DNN models against backdoors on tabular data.


http://arxiv.org/abs/2311.06771
Learning Globally Optimized Language Structure via Adversarial Training. (83%)
Xuwang Yin
  Recent work has explored integrating autoregressive language models with
energy-based models (EBMs) to enhance text generation capabilities. However,
learning effective EBMs for text is challenged by the discrete nature of
language. This work proposes an adversarial training strategy to address
limitations in prior efforts. Specifically, an iterative adversarial attack
algorithm is presented to generate negative samples for training the EBM by
perturbing text from the autoregressive model. This aims to enable the EBM to
suppress spurious modes outside the support of the data distribution.
Experiments on an arithmetic sequence generation task demonstrate that the
proposed adversarial training approach can substantially enhance the quality of
generated sequences compared to prior methods. The results highlight the
promise of adversarial techniques to improve discrete EBM training. Key
contributions include: (1) an adversarial attack strategy tailored to text to
generate negative samples, circumventing MCMC limitations; (2) an adversarial
training algorithm for EBMs leveraging these attacks; (3) empirical validation
of performance improvements on a sequence generation task.


http://arxiv.org/abs/2311.06942
Contractive Systems Improve Graph Neural Networks Against Adversarial Attacks. (70%)
Moshe Eliasof; Davide Murari; Ferdia Sherry; Carola-Bibiane Schönlieb
  Graph Neural Networks (GNNs) have established themselves as a key component
in addressing diverse graph-based tasks. Despite their notable successes, GNNs
remain susceptible to input perturbations in the form of adversarial attacks.
This paper introduces an innovative approach to fortify GNNs against
adversarial perturbations through the lens of contractive dynamical systems.
Our method introduces graph neural layers based on differential equations with
contractive properties, which, as we show, improve the robustness of GNNs. A
distinctive feature of the proposed approach is the simultaneous learned
evolution of both the node features and the adjacency matrix, yielding an
intrinsic enhancement of model robustness to perturbations in the input
features and the connectivity of the graph. We mathematically derive the
underpinnings of our novel architecture and provide theoretical insights to
reason about its expected behavior. We demonstrate the efficacy of our method
through numerous real-world benchmarks, reading on par or improved performance
compared to existing methods.


http://arxiv.org/abs/2311.06973
Analytical Verification of Deep Neural Network Performance for Time-Synchronized Distribution System State Estimation. (5%)
Behrouz Azimian; Shiva Moshtagh; Anamitra Pal; Shanshan Ma
  Recently, we demonstrated success of a time-synchronized state estimator
using deep neural networks (DNNs) for real-time unobservable distribution
systems. In this letter, we provide analytical bounds on the performance of
that state estimator as a function of perturbations in the input measurements.
It has already been shown that evaluating performance based on only the test
dataset might not effectively indicate a trained DNN's ability to handle input
perturbations. As such, we analytically verify robustness and trustworthiness
of DNNs to input perturbations by treating them as mixed-integer linear
programming (MILP) problems. The ability of batch normalization in addressing
the scalability limitations of the MILP formulation is also highlighted. The
framework is validated by performing time-synchronized distribution system
state estimation for a modified IEEE 34-node system and a real-world large
distribution system, both of which are incompletely observed by micro-phasor
measurement units.


http://arxiv.org/abs/2311.06855
DialMAT: Dialogue-Enabled Transformer with Moment-Based Adversarial Training. (1%)
Kanta Kaneda; Ryosuke Korekata; Yuiga Wada; Shunya Nagashima; Motonari Kambara; Yui Iioka; Haruka Matsuo; Yuto Imai; Takayuki Nishimura; Komei Sugiura
  This paper focuses on the DialFRED task, which is the task of embodied
instruction following in a setting where an agent can actively ask questions
about the task. To address this task, we propose DialMAT. DialMAT introduces
Moment-based Adversarial Training, which incorporates adversarial perturbations
into the latent space of language, image, and action. Additionally, it
introduces a crossmodal parallel feature extraction mechanism that applies
foundation models to both language and image. We evaluated our model using a
dataset constructed from the DialFRED dataset and demonstrated superior
performance compared to the baseline method in terms of success rate and path
weighted success rate. The model secured the top position in the DialFRED
Challenge, which took place at the CVPR 2023 Embodied AI workshop.


http://arxiv.org/abs/2311.06423
Flatness-aware Adversarial Attack. (99%)
Mingyuan Fan; Xiaodan Li; Cen Chen; Yinggui Wang
  The transferability of adversarial examples can be exploited to launch
black-box attacks. However, adversarial examples often present poor
transferability. To alleviate this issue, by observing that the diversity of
inputs can boost transferability, input regularization based methods are
proposed, which craft adversarial examples by combining several transformed
inputs. We reveal that input regularization based methods make resultant
adversarial examples biased towards flat extreme regions. Inspired by this, we
propose an attack called flatness-aware adversarial attack (FAA) which
explicitly adds a flatness-aware regularization term in the optimization target
to promote the resultant adversarial examples towards flat extreme regions. The
flatness-aware regularization term involves gradients of samples around the
resultant adversarial examples but optimizing gradients requires the evaluation
of Hessian matrix in high-dimension spaces which generally is intractable. To
address the problem, we derive an approximate solution to circumvent the
construction of Hessian matrix, thereby making FAA practical and cheap.
Extensive experiments show the transferability of adversarial examples crafted
by FAA can be considerably boosted compared with state-of-the-art baselines.


http://arxiv.org/abs/2311.05992
Robust Adversarial Attacks Detection for Deep Learning based Relative Pose Estimation for Space Rendezvous. (99%)
Ziwei Wang; Nabil Aouf; Jose Pizarro; Christophe Honvault
  Research on developing deep learning techniques for autonomous spacecraft
relative navigation challenges is continuously growing in recent years.
Adopting those techniques offers enhanced performance. However, such approaches
also introduce heightened apprehensions regarding the trustability and security
of such deep learning methods through their susceptibility to adversarial
attacks. In this work, we propose a novel approach for adversarial attack
detection for deep neural network-based relative pose estimation schemes based
on the explainability concept. We develop for an orbital rendezvous scenario an
innovative relative pose estimation technique adopting our proposed
Convolutional Neural Network (CNN), which takes an image from the chaser's
onboard camera and outputs accurately the target's relative position and
rotation. We perturb seamlessly the input images using adversarial attacks that
are generated by the Fast Gradient Sign Method (FGSM). The adversarial attack
detector is then built based on a Long Short Term Memory (LSTM) network which
takes the explainability measure namely SHapley Value from the CNN-based pose
estimator and flags the detection of adversarial attacks when acting.
Simulation results show that the proposed adversarial attack detector achieves
a detection accuracy of 99.21%. Both the deep relative pose estimator and
adversarial attack detector are then tested on real data captured from our
laboratory-designed setup. The experimental results from our
laboratory-designed setup demonstrate that the proposed adversarial attack
detector achieves an average detection accuracy of 96.29%.


http://arxiv.org/abs/2311.06122
Fight Fire with Fire: Combating Adversarial Patch Attacks using Pattern-randomized Defensive Patches. (98%)
Jianan Feng; Jiachun Li; Changqing Miao; Jianjun Huang; Wei You; Wenchang Shi; Bin Liang
  Object detection has found extensive applications in various tasks, but it is
also susceptible to adversarial patch attacks. Existing defense methods often
necessitate modifications to the target model or result in unacceptable time
overhead. In this paper, we adopt a counterattack approach, following the
principle of "fight fire with fire," and propose a novel and general
methodology for defending adversarial attacks. We utilize an active defense
strategy by injecting two types of defensive patches, canary and woodpecker,
into the input to proactively probe or weaken potential adversarial patches
without altering the target model. Moreover, inspired by randomization
techniques employed in software security, we employ randomized canary and
woodpecker injection patterns to defend against defense-aware attacks. The
effectiveness and practicality of the proposed method are demonstrated through
comprehensive experiments. The results illustrate that canary and woodpecker
achieve high performance, even when confronted with unknown attack methods,
while incurring limited time overhead. Furthermore, our method also exhibits
sufficient robustness against defense-aware attacks, as evidenced by adaptive
attack experiments.


http://arxiv.org/abs/2311.05935
Resilient and constrained consensus against adversarial attacks: A distributed MPC framework. (84%)
Henglai Wei; Kunwu Zhang; Hui Zhang; Yang Shi
  There has been a growing interest in realizing the resilient consensus of the
multi-agent system (MAS) under cyber-attacks, which aims to achieve the
consensus of normal agents (i.e., agents without attacks) in a network,
depending on the neighboring information. The literature has developed
mean-subsequence-reduced (MSR) algorithms for the MAS with F adversarial
attacks and has shown that the consensus is achieved for the normal agents when
the communication network is at least (2F+1)-robust. However, such a stringent
requirement on the communication network needs to be relaxed to enable more
practical applications. Our objective is, for the first time, to achieve less
stringent conditions on the network, while ensuring the resilient consensus for
the general linear MAS subject to control input constraints. In this work, we
propose a distributed resilient consensus framework, consisting of a
pre-designed consensus protocol and distributed model predictive control (DMPC)
optimization, which can help significantly reduce the requirement on the
network robustness and effectively handle the general linear constrained MAS
under adversarial attacks. By employing a novel distributed adversarial attack
detection mechanism based on the history information broadcast by neighbors and
a convex set (i.e., resilience set), we can evaluate the reliability of
communication links. Moreover, we show that the recursive feasibility of the
associated DMPC optimization problem can be guaranteed. The proposed consensus
protocol features the following properties: 1) by minimizing a group of control
variables, the consensus performance is optimized; 2) the resilient consensus
of the general linear constrained MAS subject to F-locally adversarial attacks
is achieved when the communication network is (F+1)-robust. Finally, numerical
simulation results are presented to verify the theoretical results.


http://arxiv.org/abs/2311.06361
CALLOC: Curriculum Adversarial Learning for Secure and Robust Indoor Localization. (1%)
Danish Gufran; Sudeep Pasricha
  Indoor localization has become increasingly vital for many applications from
tracking assets to delivering personalized services. Yet, achieving pinpoint
accuracy remains a challenge due to variations across indoor environments and
devices used to assist with localization. Another emerging challenge is
adversarial attacks on indoor localization systems that not only threaten
service integrity but also reduce localization accuracy. To combat these
challenges, we introduce CALLOC, a novel framework designed to resist
adversarial attacks and variations across indoor environments and devices that
reduce system accuracy and reliability. CALLOC employs a novel adaptive
curriculum learning approach with a domain specific lightweight scaled-dot
product attention neural network, tailored for adversarial and variation
resilience in practical use cases with resource constrained mobile devices.
Experimental evaluations demonstrate that CALLOC can achieve improvements of up
to 6.03x in mean error and 4.6x in worst-case error against state-of-the-art
indoor localization frameworks, across diverse building floorplans, mobile
devices, and adversarial attacks scenarios.


http://arxiv.org/abs/2311.06062
Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. (1%)
Wenjie Fu; Huandong Wang; Chen Gao; Guanghua Liu; Yong Li; Tao Jiang
  Membership Inference Attacks (MIA) aim to infer whether a target data record
has been utilized for model training or not. Prior attempts have quantified the
privacy risks of language models (LMs) via MIAs, but there is still no
consensus on whether existing MIA algorithms can cause remarkable privacy
leakage on practical Large Language Models (LLMs). Existing MIAs designed for
LMs can be classified into two categories: reference-free and reference-based
attacks. They are both based on the hypothesis that training records
consistently strike a higher probability of being sampled. Nevertheless, this
hypothesis heavily relies on the overfitting of target models, which will be
mitigated by multiple regularization methods and the generalization of LLMs.
The reference-based attack seems to achieve promising effectiveness in LLMs,
which measures a more reliable membership signal by comparing the probability
discrepancy between the target model and the reference model. However, the
performance of reference-based attack is highly dependent on a reference
dataset that closely resembles the training dataset, which is usually
inaccessible in the practical scenario. Overall, existing MIAs are unable to
effectively unveil privacy leakage over practical fine-tuned LLMs that are
overfitting-free and private. We propose a Membership Inference Attack based on
Self-calibrated Probabilistic Variation (SPV-MIA). Specifically, since
memorization in LLMs is inevitable during the training process and occurs
before overfitting, we introduce a more reliable membership signal,
probabilistic variation, which is based on memorization rather than
overfitting. Furthermore, we introduce a self-prompt approach, which constructs
the dataset to fine-tune the reference model by prompting the target LLM
itself. In this manner, the adversary can collect a dataset with a similar
distribution from public APIs.


http://arxiv.org/abs/2311.05316
ABIGX: A Unified Framework for eXplainable Fault Detection and Classification. (68%)
Yue Zhuo; Jinchuan Qian; Zhihuan Song; Zhiqiang Ge
  For explainable fault detection and classification (FDC), this paper proposes
a unified framework, ABIGX (Adversarial fault reconstruction-Based Integrated
Gradient eXplanation). ABIGX is derived from the essentials of previous
successful fault diagnosis methods, contribution plots (CP) and
reconstruction-based contribution (RBC). It is the first explanation framework
that provides variable contributions for the general FDC models. The core part
of ABIGX is the adversarial fault reconstruction (AFR) method, which rethinks
the FR from the perspective of adversarial attack and generalizes to fault
classification models with a new fault index. For fault classification, we put
forward a new problem of fault class smearing, which intrinsically hinders the
correct explanation. We prove that ABIGX effectively mitigates this problem and
outperforms the existing gradient-based explanation methods. For fault
detection, we theoretically bridge ABIGX with conventional fault diagnosis
methods by proving that CP and RBC are the linear specifications of ABIGX. The
experiments evaluate the explanations of FDC by quantitative metrics and
intuitive illustrations, the results of which show the general superiority of
ABIGX to other advanced explanation methods.


http://arxiv.org/abs/2311.05826
Honest Score Client Selection Scheme: Preventing Federated Learning Label Flipping Attacks in Non-IID Scenarios. (50%)
Yanli Li; Huaming Chen; Wei Bao; Zhengmeng Xu; Dong Yuan
  Federated Learning (FL) is a promising technology that enables multiple
actors to build a joint model without sharing their raw data. The distributed
nature makes FL vulnerable to various poisoning attacks, including model
poisoning attacks and data poisoning attacks. Today, many byzantine-resilient
FL methods have been introduced to mitigate the model poisoning attack, while
the effectiveness when defending against data poisoning attacks still remains
unclear. In this paper, we focus on the most representative data poisoning
attack - "label flipping attack" and monitor its effectiveness when attacking
the existing FL methods. The results show that the existing FL methods perform
similarly in Independent and identically distributed (IID) settings but fail to
maintain the model robustness in Non-IID settings. To mitigate the weaknesses
of existing FL methods in Non-IID scenarios, we introduce the Honest Score
Client Selection (HSCS) scheme and the corresponding HSCSFL framework. In the
HSCSFL, The server collects a clean dataset for evaluation. Under each
iteration, the server collects the gradients from clients and then perform HSCS
to select aggregation candidates. The server first evaluates the performance of
each class of the global model and generates the corresponding risk vector to
indicate which class could be potentially attacked. Similarly, the server
evaluates the client's model and records the performance of each class as the
accuracy vector. The dot product of each client's accuracy vector and global
risk vector is generated as the client's host score; only the top p\% host
score clients are included in the following aggregation. Finally, server
aggregates the gradients and uses the outcome to update the global model. The
comprehensive experimental results show our HSCSFL effectively enhances the FL
robustness and defends against the "label flipping attack."


http://arxiv.org/abs/2311.05808
Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning via Latent Space Reconstruction. (15%)
Shanghao Shi; Ning Wang; Yang Xiao; Chaoyu Zhang; Yi Shi; Y. Thomas Hou; Wenjing Lou
  Federated learning is known for its capability to safeguard participants'
data privacy. However, recently emerged model inversion attacks (MIAs) have
shown that a malicious parameter server can reconstruct individual users' local
data samples through model updates. The state-of-the-art attacks either rely on
computation-intensive search-based optimization processes to recover each input
batch, making scaling difficult, or they involve the malicious parameter server
adding extra modules before the global model architecture, rendering the
attacks too conspicuous and easily detectable.
  To overcome these limitations, we propose Scale-MIA, a novel MIA capable of
efficiently and accurately recovering training samples of clients from the
aggregated updates, even when the system is under the protection of a robust
secure aggregation protocol. Unlike existing approaches treating models as
black boxes, Scale-MIA recognizes the importance of the intricate architecture
and inner workings of machine learning models. It identifies the latent space
as the critical layer for breaching privacy and decomposes the complex recovery
task into an innovative two-step process to reduce computation complexity. The
first step involves reconstructing the latent space representations (LSRs) from
the aggregated model updates using a closed-form inversion mechanism,
leveraging specially crafted adversarial linear layers. In the second step, the
whole input batches are recovered from the LSRs by feeding them into a
fine-tuned generative decoder.
  We implemented Scale-MIA on multiple commonly used machine learning models
and conducted comprehensive experiments across various settings. The results
demonstrate that Scale-MIA achieves excellent recovery performance on different
datasets, exhibiting high reconstruction rates, accuracy, and attack efficiency
on a larger scale compared to state-of-the-art MIAs.


http://arxiv.org/abs/2311.05608
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. (1%)
Yichen Gong; Delong Ran; Jinyuan Liu; Conglei Wang; Tianshuo Cong; Anyu Wang; Sisi Duan; Xiaoyun Wang
  Large vision-language models (VLMs) like GPT-4V represent an unprecedented
revolution in the field of artificial intelligence (AI). Compared to
single-modal large language models (LLMs), VLMs possess more versatile
capabilities by incorporating additional modalities (e.g., images). Meanwhile,
there's a rising enthusiasm in the AI community to develop open-source VLMs,
such as LLaVA and MiniGPT4, which, however, have not undergone rigorous safety
assessment. In this paper, to demonstrate that more modalities lead to
unforeseen AI safety issues, we propose FigStep, a novel jailbreaking framework
against VLMs. FigStep feeds harmful instructions into VLMs through the image
channel and then uses benign text prompts to induce VLMs to output contents
that violate common AI safety policies. Our experimental results show that
FigStep can achieve an average attack success rate of 94.8% across 2 families
of popular open-source VLMs, LLaVA and MiniGPT4 (a total of 5 VLMs). Moreover,
we demonstrate that the methodology of FigStep can even jailbreak GPT-4V, which
already leverages several system-level mechanisms to filter harmful queries.
Above all, our experimental results reveal that VLMs are vulnerable to
jailbreaking attacks, which highlights the necessity of novel safety alignments
between visual and textual modalities.


http://arxiv.org/abs/2311.05168
FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment. (1%)
Qinghua Lin; Zuoyong Li; Kun Zeng; Haoyi Fan; Wei Li; Xiaoguang Zhou
  Deep learning techniques have greatly enhanced the performance of fire
detection in videos. However, video-based fire detection models heavily rely on
labeled data, and the process of data labeling is particularly costly and
time-consuming, especially when dealing with videos. Considering the limited
quantity of labeled video data, we propose a semi-supervised fire detection
model called FireMatch, which is based on consistency regularization and
adversarial distribution alignment. Specifically, we first combine consistency
regularization with pseudo-label. For unlabeled data, we design video data
augmentation to obtain corresponding weakly augmented and strongly augmented
samples. The proposed model predicts weakly augmented samples and retains
pseudo-label above a threshold, while training on strongly augmented samples to
predict these pseudo-labels for learning more robust feature representations.
Secondly, we generate video cross-set augmented samples by adversarial
distribution alignment to expand the training data and alleviate the decline in
classification performance caused by insufficient labeled data. Finally, we
introduce a fairness loss to help the model produce diverse predictions for
input samples, thereby addressing the issue of high confidence with the
non-fire class in fire classification scenarios. The FireMatch achieved an
accuracy of 76.92% and 91.81% on two real-world fire datasets, respectively.
The experimental results demonstrate that the proposed method outperforms the
current state-of-the-art semi-supervised classification methods.


http://arxiv.org/abs/2311.04503
Constrained Adaptive Attacks: Realistic Evaluation of Adversarial Examples and Robust Training of Deep Neural Networks for Tabular Data. (99%)
Thibault Simonetto; Salah Ghamizi; Antoine Desjardins; Maxime Cordy; Yves Le Traon
  State-of-the-art deep learning models for tabular data have recently achieved
acceptable performance to be deployed in industrial settings. However, the
robustness of these models remains scarcely explored. Contrary to computer
vision, there is to date no realistic protocol to properly evaluate the
adversarial robustness of deep tabular models due to intrinsic properties of
tabular data such as categorical features, immutability, and feature
relationship constraints. To fill this gap, we propose CAA, the first efficient
evasion attack for constrained tabular deep learning models. CAA is an
iterative parameter-free attack that combines gradient and search attacks to
generate adversarial examples under constraints. We leverage CAA to build a
benchmark of deep tabular models across three popular use cases: credit
scoring, phishing and botnet attacks detection. Our benchmark supports ten
threat models with increasing capabilities of the attacker, and reflects
real-world attack scenarios for each use case. Overall, our results demonstrate
how domain knowledge, adversarial training, and attack budgets impact the
robustness assessment of deep tabular models and provide security practitioners
with a set of recommendations to improve the robustness of deep tabular models
against various evasion attack scenarios.


http://arxiv.org/abs/2311.04588
Army of Thieves: Enhancing Black-Box Model Extraction via Ensemble based sample selection. (70%)
Akshit Jindal; Vikram Goyal; Saket Anand; Chetan Arora
  Machine Learning (ML) models become vulnerable to Model Stealing Attacks
(MSA) when they are deployed as a service. In such attacks, the deployed model
is queried repeatedly to build a labelled dataset. This dataset allows the
attacker to train a thief model that mimics the original model. To maximize
query efficiency, the attacker has to select the most informative subset of
data points from the pool of available data. Existing attack strategies utilize
approaches like Active Learning and Semi-Supervised learning to minimize costs.
However, in the black-box setting, these approaches may select sub-optimal
samples as they train only one thief model. Depending on the thief model's
capacity and the data it was pretrained on, the model might even select noisy
samples that harm the learning process. In this work, we explore the usage of
an ensemble of deep learning models as our thief model. We call our attack Army
of Thieves(AOT) as we train multiple models with varying complexities to
leverage the crowd's wisdom. Based on the ensemble's collective decision,
uncertain samples are selected for querying, while the most confident samples
are directly included in the training data. Our approach is the first one to
utilize an ensemble of thief models to perform model extraction. We outperform
the base approaches of existing state-of-the-art methods by at least 3% and
achieve a 21% higher adversarial sample transferability than previous work for
models trained on the CIFAR-10 dataset.


http://arxiv.org/abs/2311.07587
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5? (61%)
C. Daniel Freeman; Laura Culp; Aaron Parisi; Maxwell L Bileschi; Gamaleldin F Elsayed; Alex Rizkowsky; Isabelle Simpson; Alex Alemi; Azade Nova; Ben Adlam; Bernd Bohnet; Gaurav Mishra; Hanie Sedghi; Igor Mordatch; Izzeddin Gur; Jaehoon Lee; JD Co-Reyes; Jeffrey Pennington; Kelvin Xu; Kevin Swersky; Kshiteej Mahajan; Lechao Xiao; Rosanne Liu; Simon Kornblith; Noah Constant; Peter J. Liu; Roman Novak; Yundi Qian; Noah Fiedel; Jascha Sohl-Dickstein
  We introduce and study the problem of adversarial arithmetic, which provides
a simple yet challenging testbed for language model alignment. This problem is
comprised of arithmetic questions posed in natural language, with an arbitrary
adversarial string inserted before the question is complete. Even in the simple
setting of 1-digit addition problems, it is easy to find adversarial prompts
that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and
even to steer models to a particular wrong answer. We additionally provide a
simple algorithm for finding successful attacks by querying those same models,
which we name "prompt inversion rejection sampling" (PIRS). We finally show
that models can be partially hardened against these attacks via reinforcement
learning and via agentic constitutional loops. However, we were not able to
make a language model fully robust against adversarial arithmetic attacks.


http://arxiv.org/abs/2311.05143
SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training. (10%)
Rui Xu; Wenkang Qin; Peixiang Huang; Haowang; Lin Luo
  Deep Neural Networks (DNNs) are expected to provide explanation for users to
understand their black-box predictions. Saliency map is a common form of
explanation illustrating the heatmap of feature attributions, but it suffers
from noise in distinguishing important features. In this paper, we propose a
model-agnostic learning method called Saliency Constrained Adaptive Adversarial
Training (SCAAT) to improve the quality of such DNN interpretability. By
constructing adversarial samples under the guidance of saliency map, SCAAT
effectively eliminates most noise and makes saliency maps sparser and more
faithful without any modification to the model architecture. We apply SCAAT to
multiple DNNs and evaluate the quality of the generated saliency maps on
various natural and pathological image datasets. Evaluations on different
domains and metrics show that SCAAT significantly improves the interpretability
of DNNs by providing more faithful saliency maps without sacrificing their
predictive power.


http://arxiv.org/abs/2311.05144
Counter-Empirical Attacking based on Adversarial Reinforcement Learning for Time-Relevant Scoring System. (1%)
Xiangguo Sun; Hong Cheng; Hang Dong; Bo Qiao; Si Qin; Qingwei Lin
  Scoring systems are commonly seen for platforms in the era of big data. From
credit scoring systems in financial services to membership scores in E-commerce
shopping platforms, platform managers use such systems to guide users towards
the encouraged activity pattern, and manage resources more effectively and more
efficiently thereby. To establish such scoring systems, several "empirical
criteria" are firstly determined, followed by dedicated top-down design for
each factor of the score, which usually requires enormous effort to adjust and
tune the scoring function in the new application scenario. What's worse, many
fresh projects usually have no ground-truth or any experience to evaluate a
reasonable scoring system, making the designing even harder. To reduce the
effort of manual adjustment of the scoring function in every new scoring
system, we innovatively study the scoring system from the preset empirical
criteria without any ground truth, and propose a novel framework to improve the
system from scratch. In this paper, we propose a "counter-empirical attacking"
mechanism that can generate "attacking" behavior traces and try to break the
empirical rules of the scoring system. Then an adversarial "enhancer" is
applied to evaluate the scoring system and find the improvement strategy. By
training the adversarial learning problem, a proper scoring function can be
learned to be robust to the attacking activity traces that are trying to
violate the empirical criteria. Extensive experiments have been conducted on
two scoring systems including a shared computing resource platform and a
financial credit system. The experimental results have validated the
effectiveness of our proposed framework.


http://arxiv.org/abs/2311.04815
Domain Adaptive Object Detection via Balancing Between Self-Training and Adversarial Learning. (1%)
Muhammad Akhtar Munir; Muhammad Haris Khan; M. Saquib Sarfraz; Mohsen Ali
  Deep learning based object detectors struggle generalizing to a new target
domain bearing significant variations in object and background. Most current
methods align domains by using image or instance-level adversarial feature
alignment. This often suffers due to unwanted background and lacks
class-specific alignment. A straightforward approach to promote class-level
alignment is to use high confidence predictions on unlabeled domain as
pseudo-labels. These predictions are often noisy since model is poorly
calibrated under domain shift. In this paper, we propose to leverage model's
predictive uncertainty to strike the right balance between adversarial feature
alignment and class-level alignment. We develop a technique to quantify
predictive uncertainty on class assignments and bounding-box predictions. Model
predictions with low uncertainty are used to generate pseudo-labels for
self-training, whereas the ones with higher uncertainty are used to generate
tiles for adversarial feature alignment. This synergy between tiling around
uncertain object regions and generating pseudo-labels from highly certain
object regions allows capturing both image and instance-level context during
the model adaptation. We report thorough ablation study to reveal the impact of
different components in our approach. Results on five diverse and challenging
adaptation scenarios show that our approach outperforms existing
state-of-the-art methods with noticeable margins.


http://arxiv.org/abs/2311.04124
Unveiling Safety Vulnerabilities of Large Language Models. (61%)
George Kour; Marcel Zalmanovici; Naama Zwerdling; Esther Goldbraich; Ora Nova Fandina; Ateret Anaby-Tavor; Orna Raz; Eitan Farchi
  As large language models become more prevalent, their possible harmful or
inappropriate responses are a cause for concern. This paper introduces a unique
dataset containing adversarial examples in the form of questions, which we call
AttaQ, designed to provoke such harmful or inappropriate responses. We assess
the efficacy of our dataset by analyzing the vulnerabilities of various models
when subjected to it. Additionally, we introduce a novel automatic approach for
identifying and naming vulnerable semantic regions - input semantic areas for
which the model is likely to produce harmful outputs. This is achieved through
the application of specialized clustering techniques that consider both the
semantic similarity of the input attacks and the harmfulness of the model's
responses. Automatically identifying vulnerable semantic regions enhances the
evaluation of model weaknesses, facilitating targeted improvements to its
safety mechanisms and overall reliability.


http://arxiv.org/abs/2311.03865
FD-MIA: Efficient Attacks on Fairness-enhanced Models. (10%)
Huan Tian; Guangsheng Zhang; Bo Liu; Tianqing Zhu; Ming Ding; Wanlei Zhou
  Previous studies have developed fairness methods for biased models that
exhibit discriminatory behaviors towards specific subgroups. While these models
have shown promise in achieving fair predictions, recent research has
identified their potential vulnerability to score-based membership inference
attacks (MIAs). In these attacks, adversaries can infer whether a particular
data sample was used during training by analyzing the model's prediction
scores. However, our investigations reveal that these score-based MIAs are
ineffective when targeting fairness-enhanced models in binary classifications.
The attack models trained to launch the MIAs degrade into simplistic threshold
models, resulting in lower attack performance. Meanwhile, we observe that
fairness methods often lead to prediction performance degradation for the
majority subgroups of the training data. This raises the barrier to successful
attacks and widens the prediction gaps between member and non-member data.
Building upon these insights, we propose an efficient MIA method against
fairness-enhanced models based on fairness discrepancy results (FD-MIA). It
leverages the difference in the predictions from both the original and
fairness-enhanced models and exploits the observed prediction gaps as attack
clues. We also explore potential strategies for mitigating privacy leakages.
Extensive experiments validate our findings and demonstrate the efficacy of the
proposed method.


http://arxiv.org/abs/2311.16153
Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications. (2%)
Fengqing Jiang; Zhangchen Xu; Luyao Niu; Boxin Wang; Jinyuan Jia; Bo Li; Radha Poovendran
  Large language models (LLMs) are increasingly deployed as the service backend
for LLM-integrated applications such as code completion and AI-powered search.
LLM-integrated applications serve as middleware to refine users' queries with
domain-specific knowledge to better inform LLMs and enhance the responses.
Despite numerous opportunities and benefits, LLM-integrated applications also
introduce new attack surfaces. Understanding, minimizing, and eliminating these
emerging attack surfaces is a new area of research. In this work, we consider a
setup where the user and LLM interact via an LLM-integrated application in the
middle. We focus on the communication rounds that begin with user's queries and
end with LLM-integrated application returning responses to the queries, powered
by LLMs at the service backend. For this query-response protocol, we identify
potential vulnerabilities that can originate from the malicious application
developer or from an outsider threat initiator that is able to control the
database access, manipulate and poison data that are high-risk for the user.
Successful exploits of the identified vulnerabilities result in the users
receiving responses tailored to the intent of a threat initiator. We assess
such threats against LLM-integrated applications empowered by OpenAI GPT-3.5
and GPT-4. Our empirical results show that the threats can effectively bypass
the restrictions and moderation policies of OpenAI, resulting in users
receiving responses that contain bias, toxic content, privacy risk, and
disinformation. To mitigate those threats, we identify and define four key
properties, namely integrity, source identification, attack detectability, and
utility preservation, that need to be satisfied by a safe LLM-integrated
application. Based on these properties, we develop a lightweight,
threat-agnostic defense that mitigates both insider and outsider threats.


http://arxiv.org/abs/2311.03809
SoK: Security Below the OS -- A Security Analysis of UEFI. (1%)
Priyanka Prakash Surve; Oleg Brodt; Mark Yampolskiy; Yuval Elovici; Asaf Shabtai
  The Unified Extensible Firmware Interface (UEFI) is a linchpin of modern
computing systems, governing secure system initialization and booting. This
paper is urgently needed because of the surge in UEFI-related attacks and
vulnerabilities in recent years. Motivated by this urgent concern, we undertake
an extensive exploration of the UEFI landscape, dissecting its distribution
supply chain, booting process, and security features. We carefully study a
spectrum of UEFI-targeted attacks and proofs of concept (PoCs) for exploiting
UEFI-related vulnerabilities. Building upon these insights, we construct a
comprehensive attack threat model encompassing threat actors, attack vectors,
attack types, vulnerabilities, attack capabilities, and attacker objectives.
Drawing inspiration from the MITRE ATT&CK framework, we present a MITRE
ATT&CK-like taxonomy delineating tactics, techniques, and sub-techniques in the
context of UEFI attacks. This taxonomy can provide a road map for identifying
existing gaps and developing new techniques for rootkit prevention, detection,
and removal. Finally, the paper discusses existing countermeasures against UEFI
attacks including a variety of technical and operational measures that can be
implemented to lower the risk of UEFI attacks to an acceptable level. This
paper seeks to clarify the complexities of UEFI and equip the cybersecurity
community with the necessary knowledge to strengthen the security of this
critical component against a growing threat landscape.


http://arxiv.org/abs/2311.03566
Measuring Adversarial Datasets. (92%)
Yuanchen Bai; Raoyi Huang; Vijay Viswanathan; Tzu-Sheng Kuo; Tongshuang Wu
  In the era of widespread public use of AI systems across various domains,
ensuring adversarial robustness has become increasingly vital to maintain
safety and prevent undesirable errors. Researchers have curated various
adversarial datasets (through perturbations) for capturing model deficiencies
that cannot be revealed in standard benchmark datasets. However, little is
known about how these adversarial examples differ from the original data
points, and there is still no methodology to measure the intended and
unintended consequences of those adversarial transformations. In this research,
we conducted a systematic survey of existing quantifiable metrics that describe
text instances in NLP tasks, among dimensions of difficulty, diversity, and
disagreement. We selected several current adversarial effect datasets and
compared the distributions between the original and their adversarial
counterparts. The results provide valuable insights into what makes these
datasets more challenging from a metrics perspective and whether they align
with underlying assumptions.


http://arxiv.org/abs/2311.04235
Can LLMs Follow Simple Rules? (68%)
Norman Mu; Sarah Chen; Zifan Wang; Sizhe Chen; David Karamardian; Lulwa Aljeraisy; Dan Hendrycks; David Wagner
  As Large Language Models (LLMs) are deployed with increasing real-world
responsibilities, it is important to be able to specify and constrain the
behavior of these systems in a reliable manner. Model developers may wish to
set explicit rules for the model, such as "do not generate abusive content",
but these may be circumvented by jailbreaking techniques. Evaluating how well
LLMs follow developer-provided rules in the face of adversarial inputs
typically requires manual review, which slows down monitoring and methods
development. To address this issue, we propose Rule-following Language
Evaluation Scenarios (RuLES), a programmatic framework for measuring
rule-following ability in LLMs. RuLES consists of 15 simple text scenarios in
which the model is instructed to obey a set of rules in natural language while
interacting with the human user. Each scenario has a concise evaluation program
to determine whether the model has broken any rules in a conversation. Through
manual exploration of model behavior in our scenarios, we identify 6 categories
of attack strategies and collect two suites of test cases: one consisting of
unique conversations from manual testing and one that systematically implements
strategies from the 6 categories. Across various popular proprietary and open
models such as GPT-4 and Llama 2, we find that all models are susceptible to a
wide variety of adversarial hand-crafted user inputs, though GPT-4 is the
best-performing model. Additionally, we evaluate open models under
gradient-based attacks and find significant vulnerabilities. We propose RuLES
as a challenging new setting for research into exploring and defending against
both manual and automatic attacks on LLMs.


http://arxiv.org/abs/2311.03172
Preserving Privacy in GANs Against Membership Inference Attack. (33%)
Mohammadhadi Shateri; Francisco Messina; Fabrice Labeau; Pablo Piantanida
  Generative Adversarial Networks (GANs) have been widely used for generating
synthetic data for cases where there is a limited size real-world dataset or
when data holders are unwilling to share their data samples. Recent works
showed that GANs, due to overfitting and memorization, might leak information
regarding their training data samples. This makes GANs vulnerable to Membership
Inference Attacks (MIAs). Several defense strategies have been proposed in the
literature to mitigate this privacy issue. Unfortunately, defense strategies
based on differential privacy are proven to reduce extensively the quality of
the synthetic data points. On the other hand, more recent frameworks such as
PrivGAN and PAR-GAN are not suitable for small-size training datasets. In the
present work, the overfitting in GANs is studied in terms of the discriminator,
and a more general measure of overfitting based on the Bhattacharyya
coefficient is defined. Then, inspired by Fano's inequality, our first defense
mechanism against MIAs is proposed. This framework, which requires only a
simple modification in the loss function of GANs, is referred to as the maximum
entropy GAN or MEGAN and significantly improves the robustness of GANs to MIAs.
As a second defense strategy, a more heuristic model based on minimizing the
information leaked from generated samples about the training data points is
presented. This approach is referred to as mutual information minimization GAN
(MIMGAN) and uses a variational representation of the mutual information to
minimize the information that a synthetic sample might leak about the whole
training data set. Applying the proposed frameworks to some commonly used data
sets against state-of-the-art MIAs reveals that the proposed methods can reduce
the accuracy of the adversaries to the level of random guessing accuracy with a
small reduction in the quality of the synthetic data samples.


http://arxiv.org/abs/2311.03570
Cal-DETR: Calibrated Detection Transformer. (4%)
Muhammad Akhtar Munir; Salman Khan; Muhammad Haris Khan; Mohsen Ali; Fahad Shahbaz Khan
  Albeit revealing impressive predictive performance for several computer
vision tasks, deep neural networks (DNNs) are prone to making overconfident
predictions. This limits the adoption and wider utilization of DNNs in many
safety-critical applications. There have been recent efforts toward calibrating
DNNs, however, almost all of them focus on the classification task.
Surprisingly, very little attention has been devoted to calibrating modern
DNN-based object detectors, especially detection transformers, which have
recently demonstrated promising detection performance and are influential in
many decision-making systems. In this work, we address the problem by proposing
a mechanism for calibrated detection transformers (Cal-DETR), particularly for
Deformable-DETR, UP-DETR and DINO. We pursue the train-time calibration route
and make the following contributions. First, we propose a simple yet effective
approach for quantifying uncertainty in transformer-based object detectors.
Second, we develop an uncertainty-guided logit modulation mechanism that
leverages the uncertainty to modulate the class logits. Third, we develop a
logit mixing approach that acts as a regularizer with detection-specific losses
and is also complementary to the uncertainty-guided logit modulation technique
to further improve the calibration performance. Lastly, we conduct extensive
experiments across three in-domain and four out-domain scenarios. Results
corroborate the effectiveness of Cal-DETR against the competing train-time
methods in calibrating both in-domain and out-domain detections while
maintaining or even improving the detection performance. Our codebase and
pre-trained models can be accessed at
\url{https://github.com/akhtarvision/cal-detr}.


http://arxiv.org/abs/2311.02757
ELEGANT: Certified Defense on the Fairness of Graph Neural Networks. (10%)
Yushun Dong; Binchi Zhang; Hanghang Tong; Jundong Li
  Graph Neural Networks (GNNs) have emerged as a prominent graph learning model
in various graph-based tasks over the years. Nevertheless, due to the
vulnerabilities of GNNs, it has been empirically proved that malicious
attackers could easily corrupt the fairness level of their predictions by
adding perturbations to the input graph data. In this paper, we take crucial
steps to study a novel problem of certifiable defense on the fairness level of
GNNs. Specifically, we propose a principled framework named ELEGANT and present
a detailed theoretical certification analysis for the fairness of GNNs. ELEGANT
takes any GNNs as its backbone, and the fairness level of such a backbone is
theoretically impossible to be corrupted under certain perturbation budgets for
attackers. Notably, ELEGANT does not have any assumption over the GNN structure
or parameters, and does not require re-training the GNNs to realize
certification. Hence it can serve as a plug-and-play framework for any
optimized GNNs ready to be deployed. We verify the satisfactory effectiveness
of ELEGANT in practice through extensive experiments on real-world datasets
across different backbones of GNNs, where ELEGANT is also demonstrated to be
beneficial for GNN debiasing. Open-source code can be found at
https://github.com/yushundong/ELEGANT.


http://arxiv.org/abs/2311.02373
From Trojan Horses to Castle Walls: Unveiling Bilateral Backdoor Effects in Diffusion Models. (22%)
Zhuoshi Pan; Yuguang Yao; Gaowen Liu; Bingquan Shen; H. Vicky Zhao; Ramana Rao Kompella; Sijia Liu
  While state-of-the-art diffusion models (DMs) excel in image generation,
concerns regarding their security persist. Earlier research highlighted DMs'
vulnerability to backdoor attacks, but these studies placed stricter
requirements than conventional methods like 'BadNets' in image classification.
This is because the former necessitates modifications to the diffusion sampling
and training procedures. Unlike the prior work, we investigate whether
generating backdoor attacks in DMs can be as simple as BadNets, i.e., by only
contaminating the training dataset without tampering the original diffusion
process. In this more realistic backdoor setting, we uncover bilateral backdoor
effects that not only serve an adversarial purpose (compromising the
functionality of DMs) but also offer a defensive advantage (which can be
leveraged for backdoor defense). Specifically, we find that a BadNets-like
backdoor attack remains effective in DMs for producing incorrect images
(misaligned with the intended text conditions), and thereby yielding incorrect
predictions when DMs are used as classifiers. Meanwhile, backdoored DMs exhibit
an increased ratio of backdoor triggers, a phenomenon we refer to as `trigger
amplification', among the generated images. We show that this latter insight
can be used to enhance the detection of backdoor-poisoned training data. Even
under a low backdoor poisoning ratio, studying the backdoor effects of DMs is
also valuable for designing anti-backdoor image classifiers. Last but not
least, we establish a meaningful linkage between backdoor attacks and the
phenomenon of data replications by exploring DMs' inherent data memorization
tendencies. The codes of our work are available at
https://github.com/OPTML-Group/BiBadDiff.


http://arxiv.org/abs/2311.01873
Efficient Black-Box Adversarial Attacks on Neural Text Detectors. (22%)
Vitalii Fishchuk; Daniel Braun
  Neural text detectors are models trained to detect whether a given text was
generated by a language model or written by a human. In this paper, we
investigate three simple and resource-efficient strategies (parameter tweaking,
prompt engineering, and character-level mutations) to alter texts generated by
GPT-3.5 that are unsuspicious or unnoticeable for humans but cause
misclassification by neural text detectors. The results show that especially
parameter tweaking and character-level mutations are effective strategies.


http://arxiv.org/abs/2311.02147
The Alignment Problem in Context. (2%)
Raphaël Millière
  A core challenge in the development of increasingly capable AI systems is to
make them safe and reliable by ensuring their behaviour is consistent with
human values. This challenge, known as the alignment problem, does not merely
apply to hypothetical future AI systems that may pose catastrophic risks; it
already applies to current systems, such as large language models, whose
potential for harm is rapidly increasing. In this paper, I assess whether we
are on track to solve the alignment problem for large language models, and what
that means for the safety of future AI systems. I argue that existing
strategies for alignment are insufficient, because large language models remain
vulnerable to adversarial attacks that can reliably elicit unsafe behaviour. I
offer an explanation of this lingering vulnerability on which it is not simply
a contingent limitation of current language models, but has deep technical ties
to a crucial aspect of what makes these models useful and versatile in the
first place -- namely, their remarkable aptitude to learn "in context" directly
from user instructions. It follows that the alignment problem is not only
unsolved for current AI systems, but may be intrinsically difficult to solve
without severely undermining their capabilities. Furthermore, this assessment
raises concerns about the prospect of ensuring the safety of future and more
capable AI systems.


http://arxiv.org/abs/2311.01478
Adversary ML Resilience in Autonomous Driving Through Human Centered Perception Mechanisms. (99%)
Aakriti Shah
  Physical adversarial attacks on road signs are continuously exploiting
vulnerabilities in modern day autonomous vehicles (AVs) and impeding their
ability to correctly classify what type of road sign they encounter. Current
models cannot generalize input data well, resulting in overfitting or
underfitting. In overfitting, the model memorizes the input data but cannot
generalize to new scenarios. In underfitting, the model does not learn enough
of the input data to accurately classify these road signs. This paper explores
the resilience of autonomous driving systems against three main physical
adversarial attacks (tape, graffiti, illumination), specifically targeting
object classifiers. Several machine learning models were developed and
evaluated on two distinct datasets: road signs (stop signs, speed limit signs,
traffic lights, and pedestrian crosswalk signs) and geometric shapes (octagons,
circles, squares, and triangles). The study compared algorithm performance
under different conditions, including clean and adversarial training and
testing on these datasets. To build robustness against attacks, defense
techniques like adversarial training and transfer learning were implemented.
Results demonstrated transfer learning models played a crucial role in
performance by allowing knowledge gained from shape training to improve
generalizability of road sign classification, despite the datasets being
completely different. The paper suggests future research directions, including
human-in-the-loop validation, security analysis, real-world testing, and
explainable AI for transparency. This study aims to contribute to improving
security and robustness of object classifiers in autonomous vehicles and
mitigating adversarial example impacts on driving systems.


http://arxiv.org/abs/2311.01323
Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly. (99%)
Qizhang Li; Yiwen Guo; Wangmeng Zuo; Hao Chen
  The adversarial vulnerability of deep neural networks (DNNs) has drawn great
attention due to the security risk of applying these models in real-world
applications. Based on transferability of adversarial examples, an increasing
number of transfer-based methods have been developed to fool black-box DNN
models whose architecture and parameters are inaccessible. Although tremendous
effort has been exerted, there still lacks a standardized benchmark that could
be taken advantage of to compare these methods systematically, fairly, and
practically. Our investigation shows that the evaluation of some methods needs
to be more reasonable and more thorough to verify their effectiveness, to
avoid, for example, unfair comparison and insufficient consideration of
possible substitute/victim models. Therefore, we establish a transfer-based
attack benchmark (TA-Bench) which implements 30+ methods. In this paper, we
evaluate and compare them comprehensively on 25 popular substitute/victim
models on ImageNet. New insights about the effectiveness of these methods are
gained and guidelines for future evaluations are provided. Code at:
https://github.com/qizhangli/TA-Bench.


http://arxiv.org/abs/2311.01011
Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. (93%)
Sam Toyer; Olivia Watkins; Ethan Adrian Mendes; Justin Svegliato; Luke Bailey; Tiffany Wang; Isaac Ong; Karim Elmaaroufi; Pieter Abbeel; Trevor Darrell; Alan Ritter; Stuart Russell
  While Large Language Models (LLMs) are increasingly being used in real-world
applications, they remain vulnerable to prompt injection attacks: malicious
third party prompts that subvert the intent of the system designer. To help
researchers study this problem, we present a dataset of over 126,000 prompt
injection attacks and 46,000 prompt-based "defenses" against prompt injection,
all created by players of an online game called Tensor Trust. To the best of
our knowledge, this is currently the largest dataset of human-generated
adversarial examples for instruction-following LLMs. The attacks in our dataset
have a lot of easily interpretable stucture, and shed light on the weaknesses
of LLMs. We also use the dataset to create a benchmark for resistance to two
types of prompt injection, which we refer to as prompt extraction and prompt
hijacking. Our benchmark results show that many models are vulnerable to the
attack strategies in the Tensor Trust dataset. Furthermore, we show that some
attack strategies from the dataset generalize to deployed LLM-based
applications, even though they have a very different set of constraints to the
game. We release all data and source code at https://tensortrust.ai/paper


http://arxiv.org/abs/2311.01356
On the Lipschitz constant of random neural networks. (92%)
Paul Geuchen; Thomas Heindl; Dominik Stöger; Felix Voigtlaender
  Empirical studies have widely demonstrated that neural networks are highly
sensitive to small, adversarial perturbations of the input. The worst-case
robustness against these so-called adversarial examples can be quantified by
the Lipschitz constant of the neural network. However, only few theoretical
results regarding this quantity exist in the literature. In this paper, we
initiate the study of the Lipschitz constant of random ReLU neural networks,
i.e., neural networks whose weights are chosen at random and which employ the
ReLU activation function. For shallow neural networks, we characterize the
Lipschitz constant up to an absolute numerical constant. Moreover, we extend
our analysis to deep neural networks of sufficiently large width where we prove
upper and lower bounds for the Lipschitz constant. These bounds match up to a
logarithmic factor that depends on the depth.


http://arxiv.org/abs/2311.01696
Universal Perturbation-based Secret Key-Controlled Data Hiding. (80%)
Donghua Wang; Wen Yao; Tingsong Jiang; Xiaoqian Chen
  Deep neural networks (DNNs) are demonstrated to be vulnerable to universal
perturbation, a single quasi-perceptible perturbation that can deceive the DNN
on most images. However, the previous works are focused on using universal
perturbation to perform adversarial attacks, while the potential usability of
universal perturbation as data carriers in data hiding is less explored,
especially for the key-controlled data hiding method. In this paper, we propose
a novel universal perturbation-based secret key-controlled data-hiding method,
realizing data hiding with a single universal perturbation and data decoding
with the secret key-controlled decoder. Specifically, we optimize a single
universal perturbation, which serves as a data carrier that can hide multiple
secret images and be added to most cover images. Then, we devise a secret
key-controlled decoder to extract different secret images from the single
container image constructed by the universal perturbation by using different
secret keys. Moreover, a suppress loss function is proposed to prevent the
secret image from leakage. Furthermore, we adopt a robust module to boost the
decoder's capability against corruption. Finally, A co-joint optimization
strategy is proposed to find the optimal universal perturbation and decoder.
Extensive experiments are conducted on different datasets to demonstrate the
effectiveness of the proposed method. Additionally, the physical test performed
on platforms (e.g., WeChat and Twitter) verifies the usability of the proposed
method in practice.


http://arxiv.org/abs/2311.01441
Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models. (76%)
Andy Zhou; Jindong Wang; Yu-Xiong Wang; Haohan Wang
  We propose a conceptually simple and lightweight framework for improving the
robustness of vision models through the combination of knowledge distillation
and data augmentation. We address the conjecture that larger models do not make
for better teachers by showing strong gains in out-of-distribution robustness
when distilling from pretrained foundation models. Following this finding, we
propose Discrete Adversarial Distillation (DAD), which leverages a robust
teacher to generate adversarial examples and a VQGAN to discretize them,
creating more informative samples than standard data augmentation techniques.
We provide a theoretical framework for the use of a robust teacher in the
knowledge distillation with data augmentation setting and demonstrate strong
gains in out-of-distribution robustness and clean accuracy across different
student architectures. Notably, our method adds minor computational overhead
compared to similar techniques and can be easily combined with other data
augmentations for further improvements.


http://arxiv.org/abs/2311.01563
Assist Is Just as Important as the Goal: Image Resurfacing to Aid Model's Robust Prediction. (13%)
Abhijith Sharma; Phil Munz; Apurva Narayan
  Adversarial patches threaten visual AI models in the real world. The number
of patches in a patch attack is variable and determines the attack's potency in
a specific environment. Most existing defenses assume a single patch in the
scene, and the multiple patch scenarios are shown to overcome them. This paper
presents a model-agnostic defense against patch attacks based on total
variation for image resurfacing (TVR). The TVR is an image-cleansing method
that processes images to remove probable adversarial regions. TVR can be
utilized solely or augmented with a defended model, providing multi-level
security for robust prediction. TVR nullifies the influence of patches in a
single image scan with no prior assumption on the number of patches in the
scene. We validate TVR on the ImageNet-Patch benchmark dataset and with
real-world physical objects, demonstrating its ability to mitigate patch
attack.


http://arxiv.org/abs/2311.01642
Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula. (12%)
Aryaman Reddi; Maximilian Tölle; Jan Peters; Georgia Chalvatzaki; Carlo D'Eramo
  Robustness against adversarial attacks and distribution shifts is a
long-standing goal of Reinforcement Learning (RL). To this end, Robust
Adversarial Reinforcement Learning (RARL) trains a protagonist against
destabilizing forces exercised by an adversary in a competitive zero-sum Markov
game, whose optimal solution, i.e., rational strategy, corresponds to a Nash
equilibrium. However, finding Nash equilibria requires facing complex saddle
point optimization problems, which can be prohibitive to solve, especially for
high-dimensional control. In this paper, we propose a novel approach for
adversarial RL based on entropy regularization to ease the complexity of the
saddle point optimization problem. We show that the solution of this
entropy-regularized problem corresponds to a Quantal Response Equilibrium
(QRE), a generalization of Nash equilibria that accounts for bounded
rationality, i.e., agents sometimes play random actions instead of optimal
ones. Crucially, the connection between the entropy-regularized objective and
QRE enables free modulation of the rationality of the agents by simply tuning
the temperature coefficient. We leverage this insight to propose our novel
algorithm, Quantal Adversarial RL (QARL), which gradually increases the
rationality of the adversary in a curriculum fashion until it is fully
rational, easing the complexity of the optimization problem while retaining
robustness. We provide extensive evidence of QARL outperforming RARL and recent
baselines across several MuJoCo locomotion and navigation problems in overall
performance and robustness.


http://arxiv.org/abs/2311.01570
Sequential Subset Matching for Dataset Distillation. (1%)
Jiawei Du; Qin Shi; Joey Tianyi Zhou
  Dataset distillation is a newly emerging task that synthesizes a small-size
dataset used in training deep neural networks (DNNs) for reducing data storage
and model training costs. The synthetic datasets are expected to capture the
essence of the knowledge contained in real-world datasets such that the former
yields a similar performance as the latter. Recent advancements in distillation
methods have produced notable improvements in generating synthetic datasets.
However, current state-of-the-art methods treat the entire synthetic dataset as
a unified entity and optimize each synthetic instance equally. This static
optimization approach may lead to performance degradation in dataset
distillation. Specifically, we argue that static optimization can give rise to
a coupling issue within the synthetic data, particularly when a larger amount
of synthetic data is being optimized. This coupling issue, in turn, leads to
the failure of the distilled dataset to extract the high-level features learned
by the deep neural network (DNN) in the latter epochs.
  In this study, we propose a new dataset distillation strategy called
Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively
optimizing the synthetic data to encourage sequential acquisition of knowledge
during dataset distillation. Our analysis indicates that SeqMatch effectively
addresses the coupling issue by sequentially generating the synthetic
instances, thereby enhancing its performance significantly. Our proposed
SeqMatch outperforms state-of-the-art methods in various datasets, including
SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at
https://github.com/shqii1j/seqmatch.


http://arxiv.org/abs/2311.01500
E(2) Equivariant Neural Networks for Robust Galaxy Morphology Classification. (1%)
Sneh Pandya; Purvik Patel; Franc O; Jonathan Blazek
  We propose the use of group convolutional neural network architectures
(GCNNs) equivariant to the 2D Euclidean group, $E(2)$, for the task of galaxy
morphology classification by utilizing symmetries of the data present in galaxy
images as an inductive bias in the architecture. We conduct robustness studies
by introducing artificial perturbations via Poisson noise insertion and
one-pixel adversarial attacks to simulate the effects of limited observational
capabilities. We train, validate, and test GCNNs equivariant to discrete
subgroups of $E(2)$ - the cyclic and dihedral groups of order $N$ - on the
Galaxy10 DECals dataset and find that GCNNs achieve higher classification
accuracy and are consistently more robust than their non-equivariant
counterparts, with an architecture equivariant to the group $D_{16}$ achieving
a $95.52 \pm 0.18\%$ test-set accuracy. We also find that the model loses
$<6\%$ accuracy on a $50\%$-noise dataset and all GCNNs are less susceptible to
one-pixel perturbations than an identically constructed CNN. Our code is
publicly available at https://github.com/snehjp2/GCNNMorphology.


http://arxiv.org/abs/2311.00428
NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networks. (99%)
Seokil Ham; Jungwuk Park; Dong-Jun Han; Jaekyun Moon
  While multi-exit neural networks are regarded as a promising solution for
making efficient inference via early exits, combating adversarial attacks
remains a challenging problem. In multi-exit networks, due to the high
dependency among different submodels, an adversarial example targeting a
specific exit not only degrades the performance of the target exit but also
reduces the performance of all other exits concurrently. This makes multi-exit
networks highly vulnerable to simple adversarial attacks. In this paper, we
propose NEO-KD, a knowledge-distillation-based adversarial training strategy
that tackles this fundamental challenge based on two key contributions. NEO-KD
first resorts to neighbor knowledge distillation to guide the output of the
adversarial examples to tend to the ensemble outputs of neighbor exits of clean
data. NEO-KD also employs exit-wise orthogonal knowledge distillation for
reducing adversarial transferability across different submodels. The result is
a significantly improved robustness against adversarial attacks. Experimental
results on various datasets/models show that our method achieves the best
adversarial accuracy with reduced computation budgets, compared to the
baselines relying on existing adversarial training or knowledge distillation
techniques for multi-exit networks.


http://arxiv.org/abs/2311.01473
Adversarial Examples in the Physical World: A Survey. (98%)
Jiakai Wang; Donghua Wang; Jin Hu; Siyang Wu; Tingsong Jiang; Wen Yao; Aishan Liu; Xianglong Liu
  Deep neural networks (DNNs) have demonstrated high vulnerability to
adversarial examples. Besides the attacks in the digital world, the practical
implications of adversarial examples in the physical world present significant
challenges and safety concerns. However, current research on physical
adversarial examples (PAEs) lacks a comprehensive understanding of their unique
characteristics, leading to limited significance and understanding. In this
paper, we address this gap by thoroughly examining the characteristics of PAEs
within a practical workflow encompassing training, manufacturing, and
re-sampling processes. By analyzing the links between physical adversarial
attacks, we identify manufacturing and re-sampling as the primary sources of
distinct attributes and particularities in PAEs. Leveraging this knowledge, we
develop a comprehensive analysis and classification framework for PAEs based on
their specific characteristics, covering over 100 studies on physical-world
adversarial examples. Furthermore, we investigate defense strategies against
PAEs and identify open challenges and opportunities for future research. We aim
to provide a fresh, thorough, and systematic understanding of PAEs, thereby
promoting the development of robust adversarial learning and its application in
open-world scenarios.


http://arxiv.org/abs/2311.00859
Optimal Cost Constrained Adversarial Attacks For Multiple Agent Systems. (80%)
Ziqing Lu; Guanlin Liu; Lifeng Cai; Weiyu Xu
  Finding optimal adversarial attack strategies is an important topic in
reinforcement learning and the Markov decision process. Previous studies
usually assume one all-knowing coordinator (attacker) for whom attacking
different recipient (victim) agents incurs uniform costs. However, in reality,
instead of using one limitless central attacker, the attacks often need to be
performed by distributed attack agents. We formulate the problem of performing
optimal adversarial agent-to-agent attacks using distributed attack agents, in
which we impose distinct cost constraints on each different attacker-victim
pair. We propose an optimal method integrating within-step static constrained
attack-resource allocation optimization and between-step dynamic programming to
achieve the optimal adversarial attack in a multi-agent system. Our numerical
results show that the proposed attacks can significantly reduce the rewards
received by the attacked agents.


http://arxiv.org/abs/2311.00441
Improving Robustness for Vision Transformer with a Simple Dynamic Scanning Augmentation. (76%)
Shashank Kotyan; Danilo Vasconcellos Vargas
  Vision Transformer (ViT) has demonstrated promising performance in computer
vision tasks, comparable to state-of-the-art neural networks. Yet, this new
type of deep neural network architecture is vulnerable to adversarial attacks
limiting its capabilities in terms of robustness. This article presents a novel
contribution aimed at further improving the accuracy and robustness of ViT,
particularly in the face of adversarial attacks. We propose an augmentation
technique called `Dynamic Scanning Augmentation' that leverages dynamic input
sequences to adaptively focus on different patches, thereby maintaining
performance and robustness. Our detailed investigations reveal that this
adaptability to the input sequence induces significant changes in the attention
mechanism of ViT, even for the same image. We introduce four variations of
Dynamic Scanning Augmentation, outperforming ViT in terms of both robustness to
adversarial attacks and accuracy against natural images, with one variant
showing comparable results. By integrating our augmentation technique, we
observe a substantial increase in ViT's robustness, improving it from $17\%$ to
$92\%$ measured across different types of adversarial attacks. These findings,
together with other comprehensive tests, indicate that Dynamic Scanning
Augmentation enhances accuracy and robustness by promoting a more adaptive type
of attention. In conclusion, this work contributes to the ongoing research on
Vision Transformers by introducing Dynamic Scanning Augmentation as a technique
for improving the accuracy and robustness of ViT. The observed results
highlight the potential of this approach in advancing computer vision tasks and
merit further exploration in future studies.


http://arxiv.org/abs/2311.00919
MIST: Defending Against Membership Inference Attacks Through Membership-Invariant Subspace Training. (75%)
Jiacheng Li; Ninghui Li; Bruno Ribeiro
  In Member Inference (MI) attacks, the adversary try to determine whether an
instance is used to train a machine learning (ML) model. MI attacks are a major
privacy concern when using private data to train ML models. Most MI attacks in
the literature take advantage of the fact that ML models are trained to fit the
training data well, and thus have very low loss on training instances. Most
defenses against MI attacks therefore try to make the model fit the training
data less well. Doing so, however, generally results in lower accuracy. We
observe that training instances have different degrees of vulnerability to MI
attacks. Most instances will have low loss even when not included in training.
For these instances, the model can fit them well without concerns of MI
attacks. An effective defense only needs to (possibly implicitly) identify
instances that are vulnerable to MI attacks and avoids overfitting them. A
major challenge is how to achieve such an effect in an efficient training
process. Leveraging two distinct recent advancements in representation
learning: counterfactually-invariant representations and subspace learning
methods, we introduce a novel Membership-Invariant Subspace Training (MIST)
method to defend against MI attacks. MIST avoids overfitting the vulnerable
instances without significant impact on other instances. We have conducted
extensive experimental studies, comparing MIST with various other
state-of-the-art (SOTA) MI defenses against several SOTA MI attacks. We find
that MIST outperforms other defenses while resulting in minimal reduction in
testing accuracy.


http://arxiv.org/abs/2311.00508
Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks. (1%)
Yichen Huang; Timothy Baldwin
  We investigate MT evaluation metric performance on adversarially-synthesized
texts, to shed light on metric robustness. We experiment with word- and
character-level attacks on three popular machine translation metrics:
BERTScore, BLEURT, and COMET. Our human experiments validate that automatic
metrics tend to overpenalize adversarially-degraded translations. We also
identify inconsistencies in BERTScore ratings, where it judges the original
sentence and the adversarially-degraded one as similar, while judging the
degraded translation as notably worse than the original with respect to the
reference. We identify patterns of brittleness that motivate more robust metric
development.


http://arxiv.org/abs/2311.00400
Open-Set Face Recognition with Maximal Entropy and Objectosphere Loss. (1%)
Rafael Henrique Vareto; Yu Linghu; Terrance E. Boult; William Robson Schwartz; Manuel Günther
  Open-set face recognition characterizes a scenario where unknown individuals,
unseen during the training and enrollment stages, appear on operation time.
This work concentrates on watchlists, an open-set task that is expected to
operate at a low False Positive Identification Rate and generally includes only
a few enrollment samples per identity. We introduce a compact adapter network
that benefits from additional negative face images when combined with distinct
cost functions, such as Objectosphere Loss (OS) and the proposed Maximal
Entropy Loss (MEL). MEL modifies the traditional Cross-Entropy loss in favor of
increasing the entropy for negative samples and attaches a penalty to known
target classes in pursuance of gallery specialization. The proposed approach
adopts pre-trained deep neural networks (DNNs) for face recognition as feature
extractors. Then, the adapter network takes deep feature representations and
acts as a substitute for the output layer of the pre-trained DNN in exchange
for an agile domain adaptation. Promising results have been achieved following
open-set protocols for three different datasets: LFW, IJB-C, and UCCS as well
as state-of-the-art performance when supplementary negative data is properly
selected to fine-tune the adapter network.


http://arxiv.org/abs/2310.20469
Amoeba: Circumventing ML-supported Network Censorship via Adversarial Reinforcement Learning. (99%)
Haoyu Liu; Alec F. Diallo; Paul Patras
  Embedding covert streams into a cover channel is a common approach to
circumventing Internet censorship, due to censors' inability to examine
encrypted information in otherwise permitted protocols (Skype, HTTPS, etc.).
However, recent advances in machine learning (ML) enable detecting a range of
anti-censorship systems by learning distinct statistical patterns hidden in
traffic flows. Therefore, designing obfuscation solutions able to generate
traffic that is statistically similar to innocuous network activity, in order
to deceive ML-based classifiers at line speed, is difficult.
  In this paper, we formulate a practical adversarial attack strategy against
flow classifiers as a method for circumventing censorship. Specifically, we
cast the problem of finding adversarial flows that will be misclassified as a
sequence generation task, which we solve with Amoeba, a novel reinforcement
learning algorithm that we design. Amoeba works by interacting with censoring
classifiers without any knowledge of their model structure, but by crafting
packets and observing the classifiers' decisions, in order to guide the
sequence generation process. Our experiments using data collected from two
popular anti-censorship systems demonstrate that Amoeba can effectively shape
adversarial flows that have on average 94% attack success rate against a range
of ML algorithms. In addition, we show that these adversarial flows are robust
in different network environments and possess transferability across various ML
models, meaning that once trained against one, our agent can subvert other
censoring classifiers without retraining.


http://arxiv.org/abs/2311.00172
Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield. (99%)
Jinhwa Kim; Ali Derakhshan; Ian G. Harris
  Large Language Models' safety remains a critical concern due to their
vulnerability to adversarial attacks, which can prompt these systems to produce
harmful responses. In the heart of these systems lies a safety classifier, a
computational model trained to discern and mitigate potentially harmful,
offensive, or unethical outputs. However, contemporary safety classifiers,
despite their potential, often fail when exposed to inputs infused with
adversarial noise. In response, our study introduces the Adversarial Prompt
Shield (APS), a lightweight model that excels in detection accuracy and
demonstrates resilience against adversarial prompts. Additionally, we propose
novel strategies for autonomously generating adversarial training datasets,
named Bot Adversarial Noisy Dialogue (BAND) datasets. These datasets are
designed to fortify the safety classifier's robustness, and we investigate the
consequences of incorporating adversarial examples into the training process.
Through evaluations involving Large Language Models, we demonstrate that our
classifier has the potential to decrease the attack success rate resulting from
adversarial attacks by up to 60%. This advancement paves the way for the next
generation of more reliable and resilient conversational agents.


http://arxiv.org/abs/2310.20175
LFAA: Crafting Transferable Targeted Adversarial Examples with Low-Frequency Perturbations. (99%)
Kunyu Wang; Juluan Shi; Wenxuan Wang
  Deep neural networks are susceptible to adversarial attacks, which pose a
significant threat to their security and reliability in real-world
applications. The most notable adversarial attacks are transfer-based attacks,
where an adversary crafts an adversarial example to fool one model, which can
also fool other models. While previous research has made progress in improving
the transferability of untargeted adversarial examples, the generation of
targeted adversarial examples that can transfer between models remains a
challenging task. In this work, we present a novel approach to generate
transferable targeted adversarial examples by exploiting the vulnerability of
deep neural networks to perturbations on high-frequency components of images.
We observe that replacing the high-frequency component of an image with that of
another image can mislead deep models, motivating us to craft perturbations
containing high-frequency information to achieve targeted attacks. To this end,
we propose a method called Low-Frequency Adversarial Attack (\name), which
trains a conditional generator to generate targeted adversarial perturbations
that are then added to the low-frequency component of the image. Extensive
experiments on ImageNet demonstrate that our proposed approach significantly
outperforms state-of-the-art methods, improving targeted attack success rates
by a margin from 3.2\% to 15.5\%.


http://arxiv.org/abs/2311.00207
Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systems. (98%)
Jung-Woo Chang; Ke Sun; Nasimeh Heydaribeni; Seira Hidano; Xinyu Zhang; Farinaz Koushanfar
  Machine Learning (ML) has been instrumental in enabling joint transceiver
optimization by merging all physical layer blocks of the end-to-end wireless
communication systems. Although there have been a number of adversarial attacks
on ML-based wireless systems, the existing methods do not provide a
comprehensive view including multi-modality of the source data, common physical
layer components, and wireless domain constraints. This paper proposes Magmaw,
the first black-box attack methodology capable of generating universal
adversarial perturbations for any multimodal signal transmitted over a wireless
channel. We further introduce new objectives for adversarial attacks on
ML-based downstream applications. The resilience of the attack to the existing
widely used defense methods of adversarial training and perturbation signal
subtraction is experimentally verified. For proof-of-concept evaluation, we
build a real-time wireless attack platform using a software-defined radio
system. Experimental results demonstrate that Magmaw causes significant
performance degradation even in the presence of the defense mechanisms.
Surprisingly, Magmaw is also effective against encrypted communication channels
and conventional communications.


http://arxiv.org/abs/2310.20162
Is Robustness Transferable across Languages in Multilingual Neural Machine Translation? (26%)
Leiyu Pan; Supryadi; Deyi Xiong
  Robustness, the ability of models to maintain performance in the face of
perturbations, is critical for developing reliable NLP systems. Recent studies
have shown promising results in improving the robustness of models through
adversarial training and data augmentation. However, in machine translation,
most of these studies have focused on bilingual machine translation with a
single translation direction. In this paper, we investigate the transferability
of robustness across different languages in multilingual neural machine
translation. We propose a robustness transfer analysis protocol and conduct a
series of experiments. In particular, we use character-, word-, and multi-level
noises to attack the specific translation direction of the multilingual neural
machine translation model and evaluate the robustness of other translation
directions. Our findings demonstrate that the robustness gained in one
translation direction can indeed transfer to other translation directions.
Additionally, we empirically find scenarios where robustness to character-level
noise and word-level noise is more likely to transfer.


http://arxiv.org/abs/2310.20649
Dynamic Batch Norm Statistics Update for Natural Robustness. (22%)
Shahbaz Rezaei; Mohammad Sadegh Norouzzadeh
  DNNs trained on natural clean samples have been shown to perform poorly on
corrupted samples, such as noisy or blurry images. Various data augmentation
methods have been recently proposed to improve DNN's robustness against common
corruptions. Despite their success, they require computationally expensive
training and cannot be applied to off-the-shelf trained models. Recently, it
has been shown that updating BatchNorm (BN) statistics of an off-the-shelf
model on a single corruption improves its accuracy on that corruption
significantly. However, adopting the idea at inference time when the type of
corruption is unknown and changing decreases the effectiveness of this method.
In this paper, we harness the Fourier domain to detect the corruption type, a
challenging task in the image domain. We propose a unified framework consisting
of a corruption-detection model and BN statistics update that improves the
corruption accuracy of any off-the-shelf trained model. We benchmark our
framework on different models and datasets. Our results demonstrate about 8%
and 4% accuracy improvement on CIFAR10-C and ImageNet-C, respectively.
Furthermore, our framework can further improve the accuracy of state-of-the-art
robust models, such as AugMix and DeepAug.


http://arxiv.org/abs/2310.19342
Label-Only Model Inversion Attacks via Knowledge Transfer. (83%)
Ngoc-Bao Nguyen; Keshigeyan Chandrasegaran; Milad Abdollahzadeh; Ngai-Man Cheung
  In a model inversion (MI) attack, an adversary abuses access to a machine
learning (ML) model to infer and reconstruct private training data. Remarkable
progress has been made in the white-box and black-box setups, where the
adversary has access to the complete model or the model's soft output
respectively. However, there is very limited study in the most challenging but
practically important setup: Label-only MI attacks, where the adversary only
has access to the model's predicted label (hard label) without confidence
scores nor any other model information.
  In this work, we propose LOKT, a novel approach for label-only MI attacks.
Our idea is based on transfer of knowledge from the opaque target model to
surrogate models. Subsequently, using these surrogate models, our approach can
harness advanced white-box attacks. We propose knowledge transfer based on
generative modelling, and introduce a new model, Target model-assisted ACGAN
(T-ACGAN), for effective knowledge transfer. Our method casts the challenging
label-only MI into the more tractable white-box setup. We provide analysis to
support that surrogate models based on our approach serve as effective proxies
for the target model for MI. Our experiments show that our method significantly
outperforms existing SOTA Label-only MI attack by more than 15% across all MI
benchmarks. Furthermore, our method compares favorably in terms of query
budget. Our study highlights rising privacy threats for ML models even when
minimal information (i.e., hard labels) is exposed. Our study highlights rising
privacy threats for ML models even when minimal information (i.e., hard labels)
is exposed. Our code, demo, models and reconstructed data are available at our
project page: https://ngoc-nguyen-0.github.io/lokt/


http://arxiv.org/abs/2310.19889
Exploring Geometry of Blind Spots in Vision Models. (83%)
Sriram Balasubramanian; Gaurang Sriramanan; Vinu Sankar Sadasivan; Soheil Feizi
  Despite the remarkable success of deep neural networks in a myriad of
settings, several works have demonstrated their overwhelming sensitivity to
near-imperceptible perturbations, known as adversarial attacks. On the other
hand, prior works have also observed that deep networks can be under-sensitive,
wherein large-magnitude perturbations in input space do not induce appreciable
changes to network activations. In this work, we study in detail the phenomenon
of under-sensitivity in vision models such as CNNs and Transformers, and
present techniques to study the geometry and extent of "equi-confidence" level
sets of such networks. We propose a Level Set Traversal algorithm that
iteratively explores regions of high confidence with respect to the input space
using orthogonal components of the local gradients. Given a source image, we
use this algorithm to identify inputs that lie in the same equi-confidence
level set as the source image despite being perceptually similar to arbitrary
images from other classes. We further observe that the source image is linearly
connected by a high-confidence path to these inputs, uncovering a star-like
structure for level sets of deep networks. Furthermore, we attempt to identify
and estimate the extent of these connected higher-dimensional regions over
which the model maintains a high degree of confidence. The code for this
project is publicly available at
https://github.com/SriramB-98/blindspots-neurips-sub


http://arxiv.org/abs/2310.19737
Adversarial Attacks and Defenses in Large Language Models: Old and New Threats. (74%)
Leo Schwinn; David Dobre; Stephan Günnemann; Gauthier Gidel
  Over the past decade, there has been extensive research aimed at enhancing
the robustness of neural networks, yet this problem remains vastly unsolved.
Here, one major impediment has been the overestimation of the robustness of new
defense approaches due to faulty defense evaluations. Flawed robustness
evaluations necessitate rectifications in subsequent works, dangerously slowing
down the research and providing a false sense of security. In this context, we
will face substantial challenges associated with an impending adversarial arms
race in natural language processing, specifically with closed-source Large
Language Models (LLMs), such as ChatGPT, Google Bard, or Anthropic's Claude. We
provide a first set of prerequisites to improve the robustness assessment of
new approaches and reduce the amount of faulty evaluations. Additionally, we
identify embedding space attacks on LLMs as another viable threat model for the
purposes of generating malicious content in open-sourced models. Finally, we
demonstrate on a recently proposed defense that, without LLM-specific best
practices in place, it is easy to overestimate the robustness of a new
approach.


http://arxiv.org/abs/2310.19410
Generated Distributions Are All You Need for Membership Inference Attacks Against Generative Models. (61%)
Minxing Zhang; Ning Yu; Rui Wen; Michael Backes; Yang Zhang
  Generative models have demonstrated revolutionary success in various visual
creation tasks, but in the meantime, they have been exposed to the threat of
leaking private information of their training data. Several membership
inference attacks (MIAs) have been proposed to exhibit the privacy
vulnerability of generative models by classifying a query image as a training
dataset member or nonmember. However, these attacks suffer from major
limitations, such as requiring shadow models and white-box access, and either
ignoring or only focusing on the unique property of diffusion models, which
block their generalization to multiple generative models. In contrast, we
propose the first generalized membership inference attack against a variety of
generative models such as generative adversarial networks, [variational]
autoencoders, implicit functions, and the emerging diffusion models. We
leverage only generated distributions from target generators and auxiliary
non-member datasets, therefore regarding target generators as black boxes and
agnostic to their architectures or application scenarios. Experiments validate
that all the generative models are vulnerable to our attack. For instance, our
work achieves attack AUC $>0.99$ against DDPM, DDIM, and FastDPM trained on
CIFAR-10 and CelebA. And the attack against VQGAN, LDM (for the
text-conditional generation), and LIIF achieves AUC $>0.90.$ As a result, we
appeal to our community to be aware of such privacy leakage risks when
designing and publishing generative models.


http://arxiv.org/abs/2310.19733
Differentially Private Reward Estimation with Preference Feedback. (16%)
Sayak Ray Chowdhury; Xingyu Zhou; Nagarajan Natarajan
  Learning from preference-based feedback has recently gained considerable
traction as a promising approach to align generative models with human
interests. Instead of relying on numerical rewards, the generative models are
trained using reinforcement learning with human feedback (RLHF). These
approaches first solicit feedback from human labelers typically in the form of
pairwise comparisons between two possible actions, then estimate a reward model
using these comparisons, and finally employ a policy based on the estimated
reward model. An adversarial attack in any step of the above pipeline might
reveal private and sensitive information of human labelers. In this work, we
adopt the notion of label differential privacy (DP) and focus on the problem of
reward estimation from preference-based feedback while protecting privacy of
each individual labelers. Specifically, we consider the parametric
Bradley-Terry-Luce (BTL) model for such pairwise comparison feedback involving
a latent reward parameter $\theta^* \in \mathbb{R}^d$. Within a standard
minimax estimation framework, we provide tight upper and lower bounds on the
error in estimating $\theta^*$ under both local and central models of DP. We
show, for a given privacy budget $\epsilon$ and number of samples $n$, that the
additional cost to ensure label-DP under local model is $\Theta \big(\frac{1}{
e^\epsilon-1}\sqrt{\frac{d}{n}}\big)$, while it is
$\Theta\big(\frac{\text{poly}(d)}{\epsilon n} \big)$ under the weaker central
model. We perform simulations on synthetic data that corroborate these
theoretical results.


http://arxiv.org/abs/2310.19439
Asymmetric Diffusion Based Channel-Adaptive Secure Wireless Semantic Communications. (10%)
Xintian Ren; Jun Wu; Hansong Xu; Qianqian Pan
  Semantic communication has emerged as a new deep learning-based communication
paradigm that drives the research of end-to-end data transmission in tasks like
image classification, and image reconstruction. However, the security problem
caused by semantic attacks has not been well explored, resulting in
vulnerabilities within semantic communication systems exposed to potential
semantic perturbations. In this paper, we propose a secure semantic
communication system, DiffuSeC, which leverages the diffusion model and deep
reinforcement learning (DRL) to address this issue. With the diffusing module
in the sender end and the asymmetric denoising module in the receiver end, the
DiffuSeC mitigates the perturbations added by semantic attacks, including data
source attacks and channel attacks. To further improve the robustness under
unstable channel conditions caused by semantic attacks, we developed a
DRL-based channel-adaptive diffusion step selection scheme to achieve stable
performance under fluctuating environments. A timestep synchronization scheme
is designed for diffusion timestep coordination between the two ends.
Simulation results demonstrate that the proposed DiffuSeC shows higher robust
accuracy than previous works under a wide range of channel conditions, and can
quickly adjust the model state according to signal-to-noise ratios (SNRs) in
unstable environments.


http://arxiv.org/abs/2310.19304
Privacy-Preserving Federated Learning over Vertically and Horizontally Partitioned Data for Financial Anomaly Detection. (1%)
Swanand Ravindra Kadhe; Heiko Ludwig; Nathalie Baracaldo; Alan King; Yi Zhou; Keith Houck; Ambrish Rawat; Mark Purcell; Naoise Holohan; Mikio Takeuchi; Ryo Kawahara; Nir Drucker; Hayim Shaul; Eyal Kushnir; Omri Soceanu
  The effective detection of evidence of financial anomalies requires
collaboration among multiple entities who own a diverse set of data, such as a
payment network system (PNS) and its partner banks. Trust among these financial
institutions is limited by regulation and competition. Federated learning (FL)
enables entities to collaboratively train a model when data is either
vertically or horizontally partitioned across the entities. However, in
real-world financial anomaly detection scenarios, the data is partitioned both
vertically and horizontally and hence it is not possible to use existing FL
approaches in a plug-and-play manner.
  Our novel solution, PV4FAD, combines fully homomorphic encryption (HE),
secure multi-party computation (SMPC), differential privacy (DP), and
randomization techniques to balance privacy and accuracy during training and to
prevent inference threats at model deployment time. Our solution provides input
privacy through HE and SMPC, and output privacy against inference time attacks
through DP. Specifically, we show that, in the honest-but-curious threat model,
banks do not learn any sensitive features about PNS transactions, and the PNS
does not learn any information about the banks' dataset but only learns
prediction labels. We also develop and analyze a DP mechanism to protect output
privacy during inference. Our solution generates high-utility models by
significantly reducing the per-bank noise level while satisfying distributed
DP. To ensure high accuracy, our approach produces an ensemble model, in
particular, a random forest. This enables us to take advantage of the
well-known properties of ensembles to reduce variance and increase accuracy.
Our solution won second prize in the first phase of the U.S. Privacy Enhancing
Technologies (PETs) Prize Challenge.


http://arxiv.org/abs/2310.18975
Blacksmith: Fast Adversarial Training of Vision Transformers via a Mixture of Single-step and Multi-step Methods. (99%)
Mahdi Salmani; Alireza Dehghanpour Farashah; Mohammad Azizmalayeri; Mahdi Amiri; Navid Eslami; Mohammad Taghi Manzuri; Mohammad Hossein Rohban
  Despite the remarkable success achieved by deep learning algorithms in
various domains, such as computer vision, they remain vulnerable to adversarial
perturbations. Adversarial Training (AT) stands out as one of the most
effective solutions to address this issue; however, single-step AT can lead to
Catastrophic Overfitting (CO). This scenario occurs when the adversarially
trained network suddenly loses robustness against multi-step attacks like
Projected Gradient Descent (PGD). Although several approaches have been
proposed to address this problem in Convolutional Neural Networks (CNNs), we
found out that they do not perform well when applied to Vision Transformers
(ViTs). In this paper, we propose Blacksmith, a novel training strategy to
overcome the CO problem, specifically in ViTs. Our approach utilizes either of
PGD-2 or Fast Gradient Sign Method (FGSM) randomly in a mini-batch during the
adversarial training of the neural network. This will increase the diversity of
our training attacks, which could potentially mitigate the CO issue. To manage
the increased training time resulting from this combination, we craft the PGD-2
attack based on only the first half of the layers, while FGSM is applied
end-to-end. Through our experiments, we demonstrate that our novel method
effectively prevents CO, achieves PGD-2 level performance, and outperforms
other existing techniques including N-FGSM, which is the state-of-the-art
method in fast training for CNNs.


http://arxiv.org/abs/2310.19038
Boosting Decision-Based Black-Box Adversarial Attack with Gradient Priors. (98%)
Han Liu; Xingshuo Huang; Xiaotong Zhang; Qimai Li; Fenglong Ma; Wei Wang; Hongyang Chen; Hong Yu; Xianchao Zhang
  Decision-based methods have shown to be effective in black-box adversarial
attacks, as they can obtain satisfactory performance and only require to access
the final model prediction. Gradient estimation is a critical step in black-box
adversarial attacks, as it will directly affect the query efficiency. Recent
works have attempted to utilize gradient priors to facilitate score-based
methods to obtain better results. However, these gradient priors still suffer
from the edge gradient discrepancy issue and the successive iteration gradient
direction issue, thus are difficult to simply extend to decision-based methods.
In this paper, we propose a novel Decision-based Black-box Attack framework
with Gradient Priors (DBA-GP), which seamlessly integrates the data-dependent
gradient prior and time-dependent prior into the gradient estimation procedure.
First, by leveraging the joint bilateral filter to deal with each random
perturbation, DBA-GP can guarantee that the generated perturbations in edge
locations are hardly smoothed, i.e., alleviating the edge gradient discrepancy,
thus remaining the characteristics of the original image as much as possible.
Second, by utilizing a new gradient updating strategy to automatically adjust
the successive iteration gradient direction, DBA-GP can accelerate the
convergence speed, thus improving the query efficiency. Extensive experiments
have demonstrated that the proposed method outperforms other strong baselines
significantly.


http://arxiv.org/abs/2310.19152
BERT Lost Patience Won't Be Robust to Adversarial Slowdown. (98%)
Zachary Coalson; Gabriel Ritter; Rakesh Bobba; Sanghyun Hong
  In this paper, we systematically evaluate the robustness of multi-exit
language models against adversarial slowdown. To audit their robustness, we
design a slowdown attack that generates natural adversarial text bypassing
early-exit points. We use the resulting WAFFLE attack as a vehicle to conduct a
comprehensive evaluation of three multi-exit mechanisms with the GLUE benchmark
against adversarial slowdown. We then show our attack significantly reduces the
computational savings provided by the three methods in both white-box and
black-box settings. The more complex a mechanism is, the more vulnerable it is
to adversarial slowdown. We also perform a linguistic analysis of the perturbed
text inputs, identifying common perturbation patterns that our attack
generates, and comparing them with standard adversarial text attacks. Moreover,
we show that adversarial training is ineffective in defeating our slowdown
attack, but input sanitization with a conversational model, e.g., ChatGPT, can
remove perturbations effectively. This result suggests that future work is
needed for developing efficient yet robust multi-exit models. Our code is
available at: https://github.com/ztcoalson/WAFFLE


http://arxiv.org/abs/2310.18936
Adversarial Examples Are Not Real Features. (98%)
Ang Li; Yifei Wang; Yiwen Guo; Yisen Wang
  The existence of adversarial examples has been a mystery for years and
attracted much interest. A well-known theory by \citet{ilyas2019adversarial}
explains adversarial vulnerability from a data perspective by showing that one
can extract non-robust features from adversarial examples and these features
alone are useful for classification. However, the explanation remains quite
counter-intuitive since non-robust features are mostly noise features to
humans. In this paper, we re-examine the theory from a larger context by
incorporating multiple learning paradigms. Notably, we find that contrary to
their good usefulness under supervised learning, non-robust features attain
poor usefulness when transferred to other self-supervised learning paradigms,
such as contrastive learning, masked image modeling, and diffusion models. It
reveals that non-robust features are not really as useful as robust or natural
features that enjoy good transferability between these paradigms. Meanwhile,
for robustness, we also show that naturally trained encoders from robust
features are largely non-robust under AutoAttack. Our cross-paradigm
examination suggests that the non-robust features are not really useful but
more like paradigm-wise shortcuts, and robust features alone might be
insufficient to attain reliable model robustness. Code is available at
\url{https://github.com/PKU-ML/AdvNotRealFeatures}.


http://arxiv.org/abs/2310.19248
IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI. (82%)
Bochuan Cao; Changjiang Li; Ting Wang; Jinyuan Jia; Bo Li; Jinghui Chen
  Diffusion-based image generation models, such as Stable Diffusion or DALL-E
2, are able to learn from given images and generate high-quality samples
following the guidance from prompts. For instance, they can be used to create
artistic images that mimic the style of an artist based on his/her original
artworks or to maliciously edit the original images for fake content. However,
such ability also brings serious ethical issues without proper authorization
from the owner of the original images. In response, several attempts have been
made to protect the original images from such unauthorized data usage by adding
imperceptible perturbations, which are designed to mislead the diffusion model
and make it unable to properly generate new samples. In this work, we introduce
a perturbation purification platform, named IMPRESS, to evaluate the
effectiveness of imperceptible perturbations as a protective measure. IMPRESS
is based on the key observation that imperceptible perturbations could lead to
a perceptible inconsistency between the original image and the
diffusion-reconstructed image, which can be used to devise a new optimization
strategy for purifying the image, which may weaken the protection of the
original image from unauthorized data usage (e.g., style mimicking, malicious
editing). The proposed IMPRESS platform offers a comprehensive evaluation of
several contemporary protection methods, and can be used as an evaluation
platform for future protection methods.


http://arxiv.org/abs/2310.19156
Poisoning Retrieval Corpora by Injecting Adversarial Passages. (68%)
Zexuan Zhong; Ziqing Huang; Alexander Wettig; Danqi Chen
  Dense retrievers have achieved state-of-the-art performance in various
information retrieval tasks, but to what extent can they be safely deployed in
real-world applications? In this work, we propose a novel attack for dense
retrieval systems in which a malicious user generates a small number of
adversarial passages by perturbing discrete tokens to maximize similarity with
a provided set of training queries. When these adversarial passages are
inserted into a large retrieval corpus, we show that this attack is highly
effective in fooling these systems to retrieve them for queries that were not
seen by the attacker. More surprisingly, these adversarial passages can
directly generalize to out-of-domain queries and corpora with a high success
attack rate -- for instance, we find that 50 generated passages optimized on
Natural Questions can mislead >94% of questions posed in financial documents or
online forums. We also benchmark and compare a range of state-of-the-art dense
retrievers, both unsupervised and supervised. Although different systems
exhibit varying levels of vulnerability, we show they can all be successfully
attacked by injecting up to 500 passages, a small fraction compared to a
retrieval corpus of millions of passages.


http://arxiv.org/abs/2310.18933
Label Poisoning is All You Need. (54%)
Rishi D. Jha; Jonathan Hayase; Sewoong Oh
  In a backdoor attack, an adversary injects corrupted data into a model's
training dataset in order to gain control over its predictions on images with a
specific attacker-defined trigger. A typical corrupted training example
requires altering both the image, by applying the trigger, and the label.
Models trained on clean images, therefore, were considered safe from backdoor
attacks. However, in some common machine learning scenarios, the training
labels are provided by potentially malicious third-parties. This includes
crowd-sourced annotation and knowledge distillation. We, hence, investigate a
fundamental question: can we launch a successful backdoor attack by only
corrupting labels? We introduce a novel approach to design label-only backdoor
attacks, which we call FLIP, and demonstrate its strengths on three datasets
(CIFAR-10, CIFAR-100, and Tiny-ImageNet) and four architectures (ResNet-32,
ResNet-18, VGG-19, and Vision Transformer). With only 2% of CIFAR-10 labels
corrupted, FLIP achieves a near-perfect attack success rate of 99.4% while
suffering only a 1.8% drop in the clean test accuracy. Our approach builds upon
the recent advances in trajectory matching, originally introduced for dataset
distillation.


http://arxiv.org/abs/2310.19177
Robustifying Language Models with Test-Time Adaptation. (47%)
Noah Thomas McDermott; Junfeng Yang; Chengzhi Mao
  Large-scale language models achieved state-of-the-art performance over a
number of language tasks. However, they fail on adversarial language examples,
which are sentences optimized to fool the language models but with similar
semantic meanings for humans. While prior work focuses on making the language
model robust at training time, retraining for robustness is often unrealistic
for large-scale foundation models. Instead, we propose to make the language
models robust at test time. By dynamically adapting the input sentence with
predictions from masked words, we show that we can reverse many language
adversarial attacks. Since our approach does not require any training, it works
for novel tasks at test time and can adapt to novel adversarial corruptions.
Visualizations and empirical results on two popular sentence classification
datasets demonstrate that our method can repair adversarial language attacks
over 65% o


http://arxiv.org/abs/2310.18987
Path Analysis for Effective Fault Localization in Deep Neural Networks. (1%)
Soroush Hashemifar; Saeed Parsa; Akram Kalaee
  Deep learning has revolutionized various real-world applications, but the
quality of Deep Neural Networks (DNNs) remains a concern. DNNs are complex and
have millions of parameters, making it difficult to determine their
contributions to fulfilling a task. Moreover, the behavior of a DNN is highly
influenced by the data used during training, making it challenging to collect
enough data to exercise all potential DNN behavior under all possible
scenarios. This paper proposes NP SBFL method to locate faulty neural pathways
(NP) using spectrum-based fault localization (SBFL). Our method identifies
critical neurons using the layer-wise relevance propagation (LRP) technique and
determines which critical neurons are faulty. Moreover, we propose a
multi-stage gradient ascent (MGA), an extension of gradient ascent (GA), to
effectively activate a sequence of neurons one at a time while maintaining the
activation of previous neurons, so we are able to test the reported faulty
pathways. We evaluated the effectiveness of our method, i.e. NP-SBFL-MGA, on
two commonly used datasets, MNIST and CIFAR-10, two baselines DeepFault and
NP-SBFL-GA, and three suspicious neuron measures, Tarantula, Ochiai, and
Barinel. The empirical results showed that NP-SBFL-MGA is statistically more
effective than the baselines at identifying suspicious paths and synthesizing
adversarial inputs. Particularly, Tarantula on NP-SBFL-MGA had the highest
fault detection rate at 96.75%, surpassing DeepFault on Ochiai (89.90%) and
NP-SBFL-GA on Ochiai (60.61%). Our approach also yielded comparable results to
the baselines in synthesizing naturalness inputs, and we found a positive
correlation between the coverage of critical paths and the number of failed
tests in DNN fault localization.


http://arxiv.org/abs/2310.18587
Assessing and Improving Syntactic Adversarial Robustness of Pre-trained Models for Code Translation. (92%)
Guang Yang; Yu Zhou; Xiangyu Zhang; Xiang Chen; Tingting Han; Taolue Chen
  Context: Pre-trained models (PTMs) have demonstrated significant potential in
automatic code translation. However, the vulnerability of these models in
translation tasks, particularly in terms of syntax, has not been extensively
investigated. Objective: To fill this gap, our study aims to propose a novel
approach CoTR to assess and improve the syntactic adversarial robustness of
PTMs in code translation. Method: CoTR consists of two components: CoTR-A and
CoTR-D. CoTR-A generates adversarial examples by transforming programs, while
CoTR-D proposes a semantic distance-based sampling data augmentation method and
adversarial training method to improve the model's robustness and
generalization capabilities. The Pass@1 metric is used by CoTR to assess the
performance of PTMs, which is more suitable for code translation tasks and
offers a more precise evaluation in real world scenarios. Results: The
effectiveness of CoTR is evaluated through experiments on real world Java to
Python datasets. The results demonstrate that CoTR-A can significantly reduce
the performance of existing PTMs, while CoTR-D effectively improves the
robustness of PTMs. Conclusion: Our study identifies the limitations of current
PTMs, including large language models, in code translation tasks. It highlights
the potential of CoTR as an effective solution to enhance the robustness of
PTMs for code translation tasks.


http://arxiv.org/abs/2310.18626
Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness. (86%)
Soumyendu Sarkar; Ashwin Ramesh Babu; Sajad Mousavi; Zachariah Carmichael; Vineet Gundecha; Sahand Ghorbanpour; Ricardo Luna; Gutierrez Antonio Guillen; Avisek Naug
  We present a novel framework for generating adversarial benchmarks to
evaluate the robustness of image classification models. Our framework allows
users to customize the types of distortions to be optimally applied to images,
which helps address the specific distortions relevant to their deployment. The
benchmark can generate datasets at various distortion levels to assess the
robustness of different image classifiers. Our results show that the
adversarial samples generated by our framework with any of the image
classification models, like ResNet-50, Inception-V3, and VGG-16, are effective
and transferable to other models causing them to fail. These failures happen
even when these models are adversarially retrained using state-of-the-art
techniques, demonstrating the generalizability of our adversarial samples. We
achieve competitive performance in terms of net $L_2$ distortion compared to
state-of-the-art benchmark techniques on CIFAR-10 and ImageNet; however, we
demonstrate our framework achieves such results with simple distortions like
Gaussian noise without introducing unnatural artifacts or color bleeds. This is
made possible by a model-based reinforcement learning (RL) agent and a
technique that reduces a deep tree search of the image for model sensitivity to
perturbations, to a one-level analysis and action. The flexibility of choosing
distortions and setting classification probability thresholds for multiple
classes makes our framework suitable for algorithmic audits.


http://arxiv.org/abs/2310.18762
Purify++: Improving Diffusion-Purification with Advanced Diffusion Models and Control of Randomness. (61%)
Boya Zhang; Weijian Luo; Zhihua Zhang
  Adversarial attacks can mislead neural network classifiers. The defense
against adversarial attacks is important for AI safety. Adversarial
purification is a family of approaches that defend adversarial attacks with
suitable pre-processing. Diffusion models have been shown to be effective for
adversarial purification. Despite their success, many aspects of diffusion
purification still remain unexplored. In this paper, we investigate and improve
upon three limiting designs of diffusion purification: the use of an improved
diffusion model, advanced numerical simulation techniques, and optimal control
of randomness. Based on our findings, we propose Purify++, a new diffusion
purification algorithm that is now the state-of-the-art purification method
against several adversarial attacks. Our work presents a systematic exploration
of the limits of diffusion purification methods.


http://arxiv.org/abs/2310.18603
Large Language Models Are Better Adversaries: Exploring Generative Clean-Label Backdoor Attacks Against Text Classifiers. (47%)
Wencong You; Zayd Hammoudeh; Daniel Lowd
  Backdoor attacks manipulate model predictions by inserting innocuous triggers
into training and test data. We focus on more realistic and more challenging
clean-label attacks where the adversarial training examples are correctly
labeled. Our attack, LLMBkd, leverages language models to automatically insert
diverse style-based triggers into texts. We also propose a poison selection
technique to improve the effectiveness of both LLMBkd as well as existing
textual backdoor attacks. Lastly, we describe REACT, a baseline defense to
mitigate backdoor attacks via antidote training examples. Our evaluations
demonstrate LLMBkd's effectiveness and efficiency, where we consistently
achieve high attack success rates across a wide range of styles with little
effort and no model training.


http://arxiv.org/abs/2310.18606
Where have you been? A Study of Privacy Risk for Point-of-Interest Recommendation. (8%)
Kunlin Cai; Jinghuai Zhang; Will Shand; Zhiqing Hong; Guang Wang; Desheng Zhang; Jianfeng Chi; Yuan Tian
  As location-based services (LBS) have grown in popularity, the collection of
human mobility data has become increasingly extensive to build machine learning
(ML) models offering enhanced convenience to LBS users. However, the
convenience comes with the risk of privacy leakage since this type of data
might contain sensitive information related to user identities, such as
home/work locations. Prior work focuses on protecting mobility data privacy
during transmission or prior to release, lacking the privacy risk evaluation of
mobility data-based ML models. To better understand and quantify the privacy
leakage in mobility data-based ML models, we design a privacy attack suite
containing data extraction and membership inference attacks tailored for
point-of-interest (POI) recommendation models, one of the most widely used
mobility data-based ML models. These attacks in our attack suite assume
different adversary knowledge and aim to extract different types of sensitive
information from mobility data, providing a holistic privacy risk assessment
for POI recommendation models. Our experimental evaluation using two real-world
mobility datasets demonstrates that current POI recommendation models are
vulnerable to our attacks. We also present unique findings to understand what
types of mobility data are more susceptible to privacy attacks. Finally, we
evaluate defenses against these attacks and highlight future directions and
challenges.


http://arxiv.org/abs/2311.16124
DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification. (99%)
Mintong Kang; Dawn Song; Bo Li
  Diffusion-based purification defenses leverage diffusion models to remove
crafted perturbations of adversarial examples and achieve state-of-the-art
robustness. Recent studies show that even advanced attacks cannot break such
defenses effectively, since the purification process induces an extremely deep
computational graph which poses the potential problem of gradient obfuscation,
high memory cost, and unbounded randomness. In this paper, we propose a unified
framework DiffAttack to perform effective and efficient attacks against
diffusion-based purification defenses, including both DDPM and score-based
approaches. In particular, we propose a deviated-reconstruction loss at
intermediate diffusion steps to induce inaccurate density gradient estimation
to tackle the problem of vanishing/exploding gradients. We also provide a
segment-wise forwarding-backwarding algorithm, which leads to memory-efficient
gradient backpropagation. We validate the attack effectiveness of DiffAttack
compared with existing adaptive attacks on CIFAR-10 and ImageNet. We show that
DiffAttack decreases the robust accuracy of models compared with SOTA attacks
by over 20% on CIFAR-10 under $\ell_\infty$ attack $(\epsilon=8/255)$, and over
10% on ImageNet under $\ell_\infty$ attack $(\epsilon=4/255)$. We conduct a
series of ablations studies, and we find 1) DiffAttack with the
deviated-reconstruction loss added over uniformly sampled time steps is more
effective than that added over only initial/final steps, and 2) diffusion-based
purification with a moderate diffusion length is more robust under DiffAttack.


http://arxiv.org/abs/2310.18477
Understanding and Improving Ensemble Adversarial Defense. (99%)
Yian Deng; Tingting Mu
  The strategy of ensemble has become popular in adversarial defense, which
trains multiple base classifiers to defend against adversarial attacks in a
cooperative manner. Despite the empirical success, theoretical explanations on
why an ensemble of adversarially trained classifiers is more robust than single
ones remain unclear. To fill in this gap, we develop a new error theory
dedicated to understanding ensemble adversarial defense, demonstrating a
provable 0-1 loss reduction on challenging sample sets in an adversarial
defense scenario. Guided by this theory, we propose an effective approach to
improve ensemble adversarial defense, named interactive global adversarial
training (iGAT). The proposal includes (1) a probabilistic distributing rule
that selectively allocates to different base classifiers adversarial examples
that are globally challenging to the ensemble, and (2) a regularization term to
rescue the severest weaknesses of the base classifiers. Being tested over
various existing ensemble adversarial defense techniques, iGAT is capable of
boosting their performance by increases up to 17% evaluated using CIFAR10 and
CIFAR100 datasets under both white-box and black-box attacks.


http://arxiv.org/abs/2310.18274
LipSim: A Provably Robust Perceptual Similarity Metric. (45%)
Sara Ghazanfari; Alexandre Araujo; Prashanth Krishnamurthy; Farshad Khorrami; Siddharth Garg
  Recent years have seen growing interest in developing and applying perceptual
similarity metrics. Research has shown the superiority of perceptual metrics
over pixel-wise metrics in aligning with human perception and serving as a
proxy for the human visual system. On the other hand, as perceptual metrics
rely on neural networks, there is a growing concern regarding their resilience,
given the established vulnerability of neural networks to adversarial attacks.
It is indeed logical to infer that perceptual metrics may inherit both the
strengths and shortcomings of neural networks. In this work, we demonstrate the
vulnerability of state-of-the-art perceptual similarity metrics based on an
ensemble of ViT-based feature extractors to adversarial attacks. We then
propose a framework to train a robust perceptual similarity metric called
LipSim (Lipschitz Similarity Metric) with provable guarantees. By leveraging
1-Lipschitz neural networks as the backbone, LipSim provides guarded areas
around each data point and certificates for all perturbations within an
$\ell_2$ ball. Finally, a comprehensive set of experiments shows the
performance of LipSim in terms of natural and certified scores and on the image
retrieval application. The code is available at
https://github.com/SaraGhazanfari/LipSim.


http://arxiv.org/abs/2310.18155
Elevating Code-mixed Text Handling through Auditory Information of Words. (5%)
Mamta; Zishan Ahmad; Asif Ekbal
  With the growing popularity of code-mixed data, there is an increasing need
for better handling of this type of data, which poses a number of challenges,
such as dealing with spelling variations, multiple languages, different
scripts, and a lack of resources. Current language models face difficulty in
effectively handling code-mixed data as they primarily focus on the semantic
representation of words and ignore the auditory phonetic features. This leads
to difficulties in handling spelling variations in code-mixed text. In this
paper, we propose an effective approach for creating language models for
handling code-mixed textual data using auditory information of words from
SOUNDEX. Our approach includes a pre-training step based on
masked-language-modelling, which includes SOUNDEX representations (SAMLM) and a
new method of providing input data to the pre-trained model. Through
experimentation on various code-mixed datasets (of different languages) for
sentiment, offensive and aggression classification tasks, we establish that our
novel language modeling approach (SAMLM) results in improved robustness towards
adversarial attacks on code-mixed classification tasks. Additionally, our SAMLM
based approach also results in better classification results over the popular
baselines for code-mixed tasks. We use the explainability technique, SHAP
(SHapley Additive exPlanations) to explain how the auditory features
incorporated through SAMLM assist the model to handle the code-mixed text
effectively and increase robustness against adversarial attacks
\footnote{Source code has been made available on
\url{https://github.com/20118/DefenseWithPhonetics},
\url{https://www.iitp.ac.in/~ai-nlp-ml/resources.html\#Phonetics}}.


http://arxiv.org/abs/2310.17951
Understanding Parameter Saliency via Extreme Value Theory. (1%)
Shuo Wang; Issei Sato
  Deep neural networks are being increasingly implemented throughout society in
recent years. It is useful to identify which parameters trigger
misclassification in diagnosing undesirable model behaviors. The concept of
parameter saliency is proposed and used to diagnose convolutional neural
networks (CNNs) by ranking convolution filters that may have caused
misclassification on the basis of parameter saliency. It is also shown that
fine-tuning the top ranking salient filters has efficiently corrected
misidentification on ImageNet. However, there is still a knowledge gap in terms
of understanding why parameter saliency ranking can find the filters inducing
misidentification. In this work, we attempt to bridge the gap by analyzing
parameter saliency ranking from a statistical viewpoint, namely, extreme value
theory. We first show that the existing work implicitly assumes that the
gradient norm computed for each filter follows a normal distribution. Then, we
clarify the relationship between parameter saliency and the score based on the
peaks-over-threshold (POT) method, which is often used to model extreme values.
Finally, we reformulate parameter saliency in terms of the POT method, where
this reformulation is regarded as statistical anomaly detection and does not
require the implicit assumptions of the existing parameter-saliency
formulation. Our experimental results demonstrate that our reformulation can
detect malicious filters as well. Furthermore, we show that the existing
parameter saliency method exhibits a bias against the depth of layers in deep
neural networks. In particular, this bias has the potential to inhibit the
discovery of filters that cause misidentification in situations where domain
shift occurs. In contrast, parameter saliency based on POT shows less of this
bias.


http://arxiv.org/abs/2311.03373
Unscrambling the Rectification of Adversarial Attacks Transferability across Computer Networks. (99%)
Ehsan Nowroozi; Samaneh Ghelichkhani; Imran Haider; Ali Dehghantanha
  Convolutional neural networks (CNNs) models play a vital role in achieving
state-of-the-art performances in various technological fields. CNNs are not
limited to Natural Language Processing (NLP) or Computer Vision (CV) but also
have substantial applications in other technological domains, particularly in
cybersecurity. The reliability of CNN's models can be compromised because of
their susceptibility to adversarial attacks, which can be generated
effortlessly, easily applied, and transferred in real-world scenarios.
  In this paper, we present a novel and comprehensive method to improve the
strength of attacks and assess the transferability of adversarial examples in
CNNs when such strength changes, as well as whether the transferability
property issue exists in computer network applications. In the context of our
study, we initially examined six distinct modes of attack: the Carlini and
Wagner (C&W), Fast Gradient Sign Method (FGSM), Iterative Fast Gradient Sign
Method (I-FGSM), Jacobian-based Saliency Map (JSMA), Limited-memory Broyden
fletcher Goldfarb Shanno (L-BFGS), and Projected Gradient Descent (PGD) attack.
We applied these attack techniques on two popular datasets: the CIC and UNSW
datasets. The outcomes of our experiment demonstrate that an improvement in
transferability occurs in the targeted scenarios for FGSM, JSMA, LBFGS, and
other attacks. Our findings further indicate that the threats to security posed
by adversarial examples, even in computer network applications, necessitate the
development of novel defense mechanisms to enhance the security of DL-based
techniques.


http://arxiv.org/abs/2310.17626
A Survey on Transferability of Adversarial Examples across Deep Neural Networks. (99%)
Jindong Gu; Xiaojun Jia; Jorge Pau de; Wenqain Yu; Xinwei Liu; Avery Ma; Yuan Xun; Anjun Hu; Ashkan Khakzar; Zhijiang Li; Xiaochun Cao; Philip Torr
  The emergence of Deep Neural Networks (DNNs) has revolutionized various
domains, enabling the resolution of complex tasks spanning image recognition,
natural language processing, and scientific problem-solving. However, this
progress has also exposed a concerning vulnerability: adversarial examples.
These crafted inputs, imperceptible to humans, can manipulate machine learning
models into making erroneous predictions, raising concerns for safety-critical
applications. An intriguing property of this phenomenon is the transferability
of adversarial examples, where perturbations crafted for one model can deceive
another, often with a different architecture. This intriguing property enables
"black-box" attacks, circumventing the need for detailed knowledge of the
target model. This survey explores the landscape of the adversarial
transferability of adversarial examples. We categorize existing methodologies
to enhance adversarial transferability and discuss the fundamental principles
guiding each approach. While the predominant body of research primarily
concentrates on image classification, we also extend our discussion to
encompass other vision tasks and beyond. Challenges and future prospects are
discussed, highlighting the importance of fortifying DNNs against adversarial
vulnerabilities in an evolving landscape.


http://arxiv.org/abs/2310.17645
Defending Against Transfer Attacks From Public Models. (99%)
Chawin Sitawarin; Jaewon Chang; David Huang; Wesson Altoyan; David Wagner
  Adversarial attacks have been a looming and unaddressed threat in the
industry. However, through a decade-long history of the robustness evaluation
literature, we have learned that mounting a strong or optimal attack is
challenging. It requires both machine learning and domain expertise. In other
words, the white-box threat model, religiously assumed by a large majority of
the past literature, is unrealistic. In this paper, we propose a new practical
threat model where the adversary relies on transfer attacks through publicly
available surrogate models. We argue that this setting will become the most
prevalent for security-sensitive applications in the future. We evaluate the
transfer attacks in this setting and propose a specialized defense method based
on a game-theoretic perspective. The defenses are evaluated under 24 public
models and 11 attack algorithms across three datasets (CIFAR-10, CIFAR-100, and
ImageNet). Under this threat model, our defense, PubDef, outperforms the
state-of-the-art white-box adversarial training by a large margin with almost
no loss in the normal accuracy. For instance, on ImageNet, our defense achieves
62% accuracy under the strongest transfer attack vs only 36% of the best
adversarially trained model. Its accuracy when not under attack is only 2%
lower than that of an undefended model (78% vs 80%). We release our code at
https://github.com/wagner-group/pubdef.


http://arxiv.org/abs/2310.17436
Uncertainty-weighted Loss Functions for Improved Adversarial Attacks on Semantic Segmentation. (93%)
Kira Maag; Asja Fischer
  State-of-the-art deep neural networks have been shown to be extremely
powerful in a variety of perceptual tasks like semantic segmentation. However,
these networks are vulnerable to adversarial perturbations of the input which
are imperceptible for humans but lead to incorrect predictions. Treating image
segmentation as a sum of pixel-wise classifications, adversarial attacks
developed for classification models were shown to be applicable to segmentation
models as well. In this work, we present simple uncertainty-based weighting
schemes for the loss functions of such attacks that (i) put higher weights on
pixel classifications which can more easily perturbed and (ii) zero-out the
pixel-wise losses corresponding to those pixels that are already confidently
misclassified. The weighting schemes can be easily integrated into the loss
function of a range of well-known adversarial attackers with minimal additional
computational overhead, but lead to significant improved perturbation
performance, as we demonstrate in our empirical analysis on several datasets
and models.


http://arxiv.org/abs/2310.17403
Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow. (93%)
Erik Scheurer; Jenny Schmalfuss; Alexander Lis; Andrés Bruhn
  Adversarial patches undermine the reliability of optical flow predictions
when placed in arbitrary scene locations. Therefore, they pose a realistic
threat to real-world motion detection and its downstream applications.
Potential remedies are defense strategies that detect and remove adversarial
patches, but their influence on the underlying motion prediction has not been
investigated. In this paper, we thoroughly examine the currently available
detect-and-remove defenses ILP and LGS for a wide selection of state-of-the-art
optical flow methods, and illuminate their side effects on the quality and
robustness of the final flow predictions. In particular, we implement
defense-aware attacks to investigate whether current defenses are able to
withstand attacks that take the defense mechanism into account. Our experiments
yield two surprising results: Detect-and-remove defenses do not only lower the
optical flow quality on benign scenes, in doing so, they also harm the
robustness under patch attacks for all tested optical flow methods except
FlowNetC. As currently employed detect-and-remove defenses fail to deliver the
promised adversarial robustness for optical flow, they evoke a false sense of
security. The code is available at
https://github.com/cv-stuttgart/DetectionDefenses.


http://arxiv.org/abs/2310.17534
SoK: Pitfalls in Evaluating Black-Box Attacks. (76%)
Fnu Suya; Anshuman Suri; Tingwei Zhang; Jingtao Hong; Yuan Tian; David Evans
  Numerous works study black-box attacks on image classifiers. However, these
works make different assumptions on the adversary's knowledge and current
literature lacks a cohesive organization centered around the threat model. To
systematize knowledge in this area, we propose a taxonomy over the threat space
spanning the axes of feedback granularity, the access of interactive queries,
and the quality and quantity of the auxiliary data available to the attacker.
Our new taxonomy provides three key insights. 1) Despite extensive literature,
numerous under-explored threat spaces exist, which cannot be trivially solved
by adapting techniques from well-explored settings. We demonstrate this by
establishing a new state-of-the-art in the less-studied setting of access to
top-k confidence scores by adapting techniques from well-explored settings of
accessing the complete confidence vector, but show how it still falls short of
the more restrictive setting that only obtains the prediction label,
highlighting the need for more research. 2) Identification the threat model of
different attacks uncovers stronger baselines that challenge prior
state-of-the-art claims. We demonstrate this by enhancing an initially weaker
baseline (under interactive query access) via surrogate models, effectively
overturning claims in the respective paper. 3) Our taxonomy reveals
interactions between attacker knowledge that connect well to related areas,
such as model inversion and extraction attacks. We discuss how advances in
other areas can enable potentially stronger black-box attacks. Finally, we
emphasize the need for a more realistic assessment of attack success by
factoring in local attack runtime. This approach reveals the potential for
certain attacks to achieve notably higher success rates and the need to
evaluate attacks in diverse and harder settings, highlighting the need for
better selection criteria.


http://arxiv.org/abs/2310.17498
CBD: A Certified Backdoor Detector Based on Local Dominant Probability. (76%)
Zhen Xiang; Zidi Xiong; Bo Li
  Backdoor attack is a common threat to deep neural networks. During testing,
samples embedded with a backdoor trigger will be misclassified as an
adversarial target by a backdoored model, while samples without the backdoor
trigger will be correctly classified. In this paper, we present the first
certified backdoor detector (CBD), which is based on a novel, adjustable
conformal prediction scheme based on our proposed statistic local dominant
probability. For any classifier under inspection, CBD provides 1) a detection
inference, 2) the condition under which the attacks are guaranteed to be
detectable for the same classification domain, and 3) a probabilistic upper
bound for the false positive rate. Our theoretical results show that attacks
with triggers that are more resilient to test-time noise and have smaller
perturbation magnitudes are more likely to be detected with guarantees.
Moreover, we conduct extensive experiments on four benchmark datasets
considering various backdoor types, such as BadNet, CB, and Blend. CBD achieves
comparable or even higher detection accuracy than state-of-the-art detectors,
and it in addition provides detection certification. Notably, for backdoor
attacks with random perturbation triggers bounded by $\ell_2\leq0.75$ which
achieves more than 90\% attack success rate, CBD achieves 100\% (98\%), 100\%
(84\%), 98\% (98\%), and 72\% (40\%) empirical (certified) detection true
positive rates on the four benchmark datasets GTSRB, SVHN, CIFAR-10, and
TinyImageNet, respectively, with low false positive rates.


http://arxiv.org/abs/2310.17559
Instability of computer vision models is a necessary result of the task itself. (26%)
Oliver Turnbull; George Cevora
  Adversarial examples resulting from instability of current computer vision
models are an extremely important topic due to their potential to compromise
any application. In this paper we demonstrate that instability is inevitable
due to a) symmetries (translational invariance) of the data, b) the categorical
nature of the classification task, and c) the fundamental discrepancy of
classifying images as objects themselves. The issue is further exacerbated by
non-exhaustive labelling of the training data. Therefore we conclude that
instability is a necessary result of how the problem of computer vision is
currently formulated. While the problem cannot be eliminated, through the
analysis of the causes, we have arrived at ways how it can be partially
alleviated. These include i) increasing the resolution of images, ii) providing
contextual information for the image, iii) exhaustive labelling of training
data, and iv) preventing attackers from frequent access to the computer vision
system.


http://arxiv.org/abs/2310.17588
PAC-tuning:Fine-tuning Pretrained Language Models with PAC-driven Perturbed Gradient Descent. (1%)
Guangliang Liu; Zhiyu Xue; Xitong Zhang; Kristen Marie Johnson; Rongrong Wang
  Fine-tuning pretrained language models (PLMs) for downstream tasks is a
large-scale optimization problem, in which the choice of the training algorithm
critically determines how well the trained model can generalize to unseen test
data, especially in the context of few-shot learning. To achieve good
generalization performance and avoid overfitting, techniques such as data
augmentation and pruning are often applied. However, adding these
regularizations necessitates heavy tuning of the hyperparameters of
optimization algorithms, such as the popular Adam optimizer. In this paper, we
propose a two-stage fine-tuning method, PAC-tuning, to address this
optimization challenge. First, based on PAC-Bayes training, PAC-tuning directly
minimizes the PAC-Bayes generalization bound to learn proper parameter
distribution. Second, PAC-tuning modifies the gradient by injecting noise with
the variance learned in the first stage into the model parameters during
training, resulting in a variant of perturbed gradient descent (PGD). In the
past, the few-shot scenario posed difficulties for PAC-Bayes training because
the PAC-Bayes bound, when applied to large models with limited training data,
might not be stringent. Our experimental results across 5 GLUE benchmark tasks
demonstrate that PAC-tuning successfully handles the challenges of fine-tuning
tasks and outperforms strong baseline methods by a visible margin, further
confirming the potential to apply PAC training for any other settings where the
Adam optimizer is currently used for training.


http://arxiv.org/abs/2310.17584
A minimax optimal control approach for robust neural ODEs. (1%)
Cristina Cipriani; Alessandro Scagliotti; Tobias Wöhrer
  In this paper, we address the adversarial training of neural ODEs from a
robust control perspective. This is an alternative to the classical training
via empirical risk minimization, and it is widely used to enforce reliable
outcomes for input perturbations. Neural ODEs allow the interpretation of deep
neural networks as discretizations of control systems, unlocking powerful tools
from control theory for the development and the understanding of machine
learning. In this specific case, we formulate the adversarial training with
perturbed data as a minimax optimal control problem, for which we derive first
order optimality conditions in the form of Pontryagin's Maximum Principle. We
provide a novel interpretation of robust training leading to an alternative
weighted technique, which we test on a low-dimensional classification task.


http://arxiv.org/abs/2310.16955
Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks. (93%)
Aradhana Sinha; Ananth Balashankar; Ahmad Beirami; Thi Avrahami; Jilin Chen; Alex Beutel
  Real-world natural language processing systems need to be robust to human
adversaries. Collecting examples of human adversaries for training is an
effective but expensive solution. On the other hand, training on synthetic
attacks with small perturbations - such as word-substitution - does not
actually improve robustness to human adversaries. In this paper, we propose an
adversarial training framework that uses limited human adversarial examples to
generate more useful adversarial examples at scale. We demonstrate the
advantages of this system on the ANLI and hate speech detection benchmark
datasets - both collected via an iterative, adversarial
human-and-model-in-the-loop procedure. Compared to training only on observed
human attacks, also training on our synthetic adversarial examples improves
model robustness to future rounds. In ANLI, we see accuracy gains on the
current set of attacks (44.1%$\,\to\,$50.1%) and on two future unseen rounds of
human generated attacks (32.5%$\,\to\,$43.4%, and 29.4%$\,\to\,$40.2%). In hate
speech detection, we see AUC gains on current attacks (0.76 $\to$ 0.84) and a
future round (0.77 $\to$ 0.79). Attacks from methods that do not learn the
distribution of existing human adversaries, meanwhile, degrade robustness.


http://arxiv.org/abs/2310.16999
Trust, but Verify: Robust Image Segmentation using Deep Learning. (54%)
Fahim Ahmed Zaman; Xiaodong Wu; Weiyu Xu; Milan Sonka; Raghuraman Mudumbai
  We describe a method for verifying the output of a deep neural network for
medical image segmentation that is robust to several classes of random as well
as worst-case perturbations i.e. adversarial attacks. This method is based on a
general approach recently developed by the authors called ``Trust, but Verify"
wherein an auxiliary verification network produces predictions about certain
masked features in the input image using the segmentation as an input. A
well-designed auxiliary network will produce high-quality predictions when the
input segmentations are accurate, but will produce low-quality predictions when
the segmentations are incorrect. Checking the predictions of such a network
with the original image allows us to detect bad segmentations. However, to
ensure the verification method is truly robust, we need a method for checking
the quality of the predictions that does not itself rely on a black-box neural
network. Indeed, we show that previous methods for segmentation evaluation that
do use deep neural regression networks are vulnerable to false negatives i.e.
can inaccurately label bad segmentations as good. We describe the design of a
verification network that avoids such vulnerability and present results to
demonstrate its robustness compared to previous methods.


http://arxiv.org/abs/2310.16540
Dual Defense: Adversarial, Traceable, and Invisible Robust Watermarking against Face Swapping. (26%)
Yunming Zhang; Dengpan Ye; Caiyun Xie; Long Tang; Chuanxi Chen; Ziyi Liu; Jiacheng Deng
  The malicious applications of deep forgery, represented by face swapping,
have introduced security threats such as misinformation dissemination and
identity fraud. While some research has proposed the use of robust watermarking
methods to trace the copyright of facial images for post-event traceability,
these methods cannot effectively prevent the generation of forgeries at the
source and curb their dissemination. To address this problem, we propose a
novel comprehensive active defense mechanism that combines traceability and
adversariality, called Dual Defense. Dual Defense invisibly embeds a single
robust watermark within the target face to actively respond to sudden cases of
malicious face swapping. It disrupts the output of the face swapping model
while maintaining the integrity of watermark information throughout the entire
dissemination process. This allows for watermark extraction at any stage of
image tracking for traceability. Specifically, we introduce a watermark
embedding network based on original-domain feature impersonation attack. This
network learns robust adversarial features of target facial images and embeds
watermarks, seeking a well-balanced trade-off between watermark invisibility,
adversariality, and traceability through perceptual adversarial encoding
strategies. Extensive experiments demonstrate that Dual Defense achieves
optimal overall defense success rates and exhibits promising universality in
anti-face swapping tasks and dataset generalization ability. It maintains
impressive adversariality and traceability in both original and robust
settings, surpassing current forgery defense methods that possess only one of
these capabilities, including CMUA-Watermark, Anti-Forgery, FakeTagger, or PGD
methods.


http://arxiv.org/abs/2310.16613
On the Proactive Generation of Unsafe Images From Text-To-Image Models Using Benign Prompts. (22%)
Yixin Wu; Ning Yu; Michael Backes; Yun Shen; Yang Zhang
  Text-to-image models like Stable Diffusion have had a profound impact on
daily life by enabling the generation of photorealistic images from textual
prompts, fostering creativity, and enhancing visual experiences across various
applications. However, these models also pose risks. Previous studies have
successfully demonstrated that manipulated prompts can elicit text-to-image
models to generate unsafe images, e.g., hateful meme variants. Yet, these
studies only unleash the harmful power of text-to-image models in a passive
manner. In this work, we focus on the proactive generation of unsafe images
using targeted benign prompts via poisoning attacks. We propose two poisoning
attacks: a basic attack and a utility-preserving attack. We qualitatively and
quantitatively evaluate the proposed attacks using four representative hateful
memes and multiple query prompts. Experimental results indicate that
text-to-image models are vulnerable to the basic attack even with five
poisoning samples. However, the poisoning effect can inadvertently spread to
non-targeted prompts, leading to undesirable side effects. Root cause analysis
identifies conceptual similarity as an important contributing factor to the
side effects. To address this, we introduce the utility-preserving attack as a
viable mitigation strategy to maintain the attack stealthiness, while ensuring
decent attack performance. Our findings underscore the potential risks of
adopting text-to-image models in real-world scenarios, calling for future
research and safety measures in this space.


http://arxiv.org/abs/2310.16919
Wide Flat Minimum Watermarking for Robust Ownership Verification of GANs. (12%)
Jianwei Fei; Zhihua Xia; Benedetta Tondi; Mauro Barni
  We propose a novel multi-bit box-free watermarking method for the protection
of Intellectual Property Rights (IPR) of GANs with improved robustness against
white-box attacks like fine-tuning, pruning, quantization, and surrogate model
attacks. The watermark is embedded by adding an extra watermarking loss term
during GAN training, ensuring that the images generated by the GAN contain an
invisible watermark that can be retrieved by a pre-trained watermark decoder.
In order to improve the robustness against white-box model-level attacks, we
make sure that the model converges to a wide flat minimum of the watermarking
loss term, in such a way that any modification of the model parameters does not
erase the watermark. To do so, we add random noise vectors to the parameters of
the generator and require that the watermarking loss term is as invariant as
possible with respect to the presence of noise. This procedure forces the
generator to converge to a wide flat minimum of the watermarking loss. The
proposed method is architectureand dataset-agnostic, thus being applicable to
many different generation tasks and models, as well as to CNN-based image
processing architectures. We present the results of extensive experiments
showing that the presence of the watermark has a negligible impact on the
quality of the generated images, and proving the superior robustness of the
watermark against model modification and surrogate model attacks.


http://arxiv.org/abs/2310.16779
Multi-scale Diffusion Denoised Smoothing. (1%)
Jongheon Jeong; Jinwoo Shin
  Along with recent diffusion models, randomized smoothing has become one of a
few tangible approaches that offers adversarial robustness to models at scale,
e.g., those of large pre-trained models. Specifically, one can perform
randomized smoothing on any classifier via a simple "denoise-and-classify"
pipeline, so-called denoised smoothing, given that an accurate denoiser is
available - such as diffusion model. In this paper, we investigate the
trade-off between accuracy and certified robustness of denoised smoothing: for
example, we question on which representation of diffusion model would maximize
the certified robustness of denoised smoothing. We consider a new objective
that aims collective robustness of smoothed classifiers across multiple noise
levels at a shared diffusion model, which also suggests a new way to compensate
the cost of accuracy in randomized smoothing for its certified robustness. This
objective motivates us to fine-tune diffusion model (a) to perform consistent
denoising whenever the original image is recoverable, but (b) to generate
rather diverse outputs otherwise. Our experiments show that this fine-tuning
scheme of diffusion models combined with the multi-scale smoothing enables a
strong certified robustness possible at highest noise level while maintaining
the accuracy closer to non-smoothed classifiers.


http://arxiv.org/abs/2310.16838
SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation. (1%)
Qianxu Wang; Haotong Zhang; Congyue Deng; Yang You; Hao Dong; Yixin Zhu; Leonidas Guibas
  Humans excel at transferring manipulation skills across diverse object
shapes, poses, and appearances due to their understanding of semantic
correspondences between different instances. To endow robots with a similar
high-level understanding, we develop a Distilled Feature Field (DFF) for 3D
scenes, leveraging large 2D vision models to distill semantic features from
multiview images. While current research demonstrates advanced performance in
reconstructing DFFs from dense views, the development of learning a DFF from
sparse views is relatively nascent, despite its prevalence in numerous
manipulation tasks with fixed cameras. In this work, we introduce SparseDFF, a
novel method for acquiring view-consistent 3D DFFs from sparse RGBD
observations, enabling one-shot learning of dexterous manipulations that are
transferable to novel scenes. Specifically, we map the image features to the 3D
point cloud, allowing for propagation across the 3D space to establish a dense
feature field. At the core of SparseDFF is a lightweight feature refinement
network, optimized with a contrastive loss between pairwise views after
back-projecting the image features onto the 3D point cloud. Additionally, we
implement a point-pruning mechanism to augment feature continuity within each
local neighborhood. By establishing coherent feature fields on both source and
target scenes, we devise an energy function that facilitates the minimization
of feature discrepancies w.r.t. the end-effector parameters between the
demonstration and the target manipulation. We evaluate our approach using a
dexterous hand, mastering real-world manipulations on both rigid and deformable
objects, and showcase robust generalization in the face of object and
scene-context variations.


http://arxiv.org/abs/2311.12857
Adversarial sample generation and training using geometric masks for accurate and resilient license plate character recognition. (99%)
Bishal Shrestha; Griwan Khakurel; Kritika Simkhada; Badri Adhikari
  Reading dirty license plates accurately in moving vehicles is challenging for
automatic license plate recognition systems. Moreover, license plates are often
intentionally tampered with a malicious intent to avoid police apprehension.
Usually, such groups and individuals know how to fool the existing recognition
systems by making minor unnoticeable plate changes. Designing and developing
deep learning methods resilient to such real-world 'attack' practices remains
an active research problem. As a solution, this work develops a resilient
method to recognize license plate characters. Extracting 1057 character images
from 160 Nepalese vehicles, as the first step, we trained several standard deep
convolutional neural networks to obtain 99.5% character classification
accuracy. On adversarial images generated to simulate malicious tampering,
however, our model's accuracy dropped to 25%. Next, we enriched our dataset by
generating and adding geometrically masked images, retrained our models, and
investigated the models' predictions. The proposed approach of training with
generated adversarial images helped our adversarial attack-aware license plate
character recognition (AA-LPCR) model achieves an accuracy of 99.7%. This
near-perfect accuracy demonstrates that the proposed idea of random geometric
masking is highly effective for improving the accuracy of license plate
recognition models. Furthermore, by performing interpretability studies to
understand why our models work, we identify and highlight attack-prone regions
in the input character images. In sum, although Nepal's embossed license plate
detection systems are vulnerable to malicious attacks, our findings suggest
that these systems can be upgraded to close to 100% resilience.


http://arxiv.org/abs/2311.12858
RAEDiff: Denoising Diffusion Probabilistic Models Based Reversible Adversarial Examples Self-Generation and Self-Recovery. (92%)
Fan Xing; Xiaoyi Zhou; Xuefeng Fan; Zhuo Tian; Yan Zhao
  Collected and annotated datasets, which are obtained through extensive
efforts, are effective for training Deep Neural Network (DNN) models. However,
these datasets are susceptible to be misused by unauthorized users, resulting
in infringement of Intellectual Property (IP) rights owned by the dataset
creators. Reversible Adversarial Exsamples (RAE) can help to solve the issues
of IP protection for datasets. RAEs are adversarial perturbed images that can
be restored to the original. As a cutting-edge approach, RAE scheme can serve
the purposes of preventing unauthorized users from engaging in malicious model
training, as well as ensuring the legitimate usage of authorized users.
Nevertheless, in the existing work, RAEs still rely on the embedded auxiliary
information for restoration, which may compromise their adversarial abilities.
In this paper, a novel self-generation and self-recovery method, named as
RAEDiff, is introduced for generating RAEs based on a Denoising Diffusion
Probabilistic Models (DDPM). It diffuses datasets into a Biased Gaussian
Distribution (BGD) and utilizes the prior knowledge of the DDPM for generating
and recovering RAEs. The experimental results demonstrate that RAEDiff
effectively self-generates adversarial perturbations for DNN models, including
Artificial Intelligence Generated Content (AIGC) models, while also exhibiting
significant self-recovery capabilities.


http://arxiv.org/abs/2310.16335
Defense Against Model Extraction Attacks on Recommender Systems. (92%)
Sixiao Zhang; Hongzhi Yin; Hongxu Chen; Cheng Long
  The robustness of recommender systems has become a prominent topic within the
research community. Numerous adversarial attacks have been proposed, but most
of them rely on extensive prior knowledge, such as all the white-box attacks or
most of the black-box attacks which assume that certain external knowledge is
available. Among these attacks, the model extraction attack stands out as a
promising and practical method, involving training a surrogate model by
repeatedly querying the target model. However, there is a significant gap in
the existing literature when it comes to defending against model extraction
attacks on recommender systems. In this paper, we introduce Gradient-based
Ranking Optimization (GRO), which is the first defense strategy designed to
counter such attacks. We formalize the defense as an optimization problem,
aiming to minimize the loss of the protected target model while maximizing the
loss of the attacker's surrogate model. Since top-k ranking lists are
non-differentiable, we transform them into swap matrices which are instead
differentiable. These swap matrices serve as input to a student model that
emulates the surrogate model's behavior. By back-propagating the loss of the
student model, we obtain gradients for the swap matrices. These gradients are
used to compute a swap loss, which maximizes the loss of the student model. We
conducted experiments on three benchmark datasets to evaluate the performance
of GRO, and the results demonstrate its superior effectiveness in defending
against model extraction attacks.


http://arxiv.org/abs/2310.16061
Segue: Side-information Guided Generative Unlearnable Examples for Facial Privacy Protection in Real World. (89%)
Zhiling Zhang; Jie Zhang; Kui Zhang; Wenbo Zhou; Weiming Zhang; Nenghai Yu
  The widespread use of face recognition technology has given rise to privacy
concerns, as many individuals are worried about the collection and utilization
of their facial data. To address these concerns, researchers are actively
exploring the concept of ``unlearnable examples", by adding imperceptible
perturbation to data in the model training stage, which aims to prevent the
model from learning discriminate features of the target face. However, current
methods are inefficient and cannot guarantee transferability and robustness at
the same time, causing impracticality in the real world. To remedy it, we
propose a novel method called Segue: Side-information guided generative
unlearnable examples. Specifically, we leverage a once-trained multiple-used
model to generate the desired perturbation rather than the time-consuming
gradient-based method. To improve transferability, we introduce side
information such as true labels and pseudo labels, which are inherently
consistent across different scenarios. For robustness enhancement, a distortion
layer is integrated into the training pipeline. Extensive experiments
demonstrate that the proposed Segue is much faster than previous methods
(1000$\times$) and achieves transferable effectiveness across different
datasets and model architectures. Furthermore, it can resist JPEG compression,
adversarial training, and some standard data augmentations.


http://arxiv.org/abs/2310.16221
Hierarchical Randomized Smoothing. (75%)
Yan Scholten; Jan Schuchardt; Aleksandar Bojchevski; Stephan Günnemann
  Real-world data is complex and often consists of objects that can be
decomposed into multiple entities (e.g. images into pixels, graphs into
interconnected nodes). Randomized smoothing is a powerful framework for making
models provably robust against small changes to their inputs - by guaranteeing
robustness of the majority vote when randomly adding noise before
classification. Yet, certifying robustness on such complex data via randomized
smoothing is challenging when adversaries do not arbitrarily perturb entire
objects (e.g. images) but only a subset of their entities (e.g. pixels). As a
solution, we introduce hierarchical randomized smoothing: We partially smooth
objects by adding random noise only on a randomly selected subset of their
entities. By adding noise in a more targeted manner than existing methods we
obtain stronger robustness guarantees while maintaining high accuracy. We
initialize hierarchical smoothing using different noising distributions,
yielding novel robustness certificates for discrete and continuous domains. We
experimentally demonstrate the importance of hierarchical smoothing in image
and node classification, where it yields superior robustness-accuracy
trade-offs. Overall, hierarchical smoothing is an important contribution
towards models that are both - certifiably robust to perturbations and
accurate.


http://arxiv.org/abs/2310.15656
Momentum Gradient-based Untargeted Attack on Hypergraph Neural Networks. (73%)
Yang Chen; Stjepan Picek; Zhonglin Ye; Zhaoyang Wang; Haixing Zhao
  Hypergraph Neural Networks (HGNNs) have been successfully applied in various
hypergraph-related tasks due to their excellent higher-order representation
capabilities. Recent works have shown that deep learning models are vulnerable
to adversarial attacks. Most studies on graph adversarial attacks have focused
on Graph Neural Networks (GNNs), and the study of adversarial attacks on HGNNs
remains largely unexplored. In this paper, we try to reduce this gap. We design
a new HGNNs attack model for the untargeted attack, namely MGHGA, which focuses
on modifying node features. We consider the process of HGNNs training and use a
surrogate model to implement the attack before hypergraph modeling.
Specifically, MGHGA consists of two parts: feature selection and feature
modification. We use a momentum gradient mechanism to choose the attack node
features in the feature selection module. In the feature modification module,
we use two feature generation approaches (direct modification and sign
gradient) to enable MGHGA to be employed on discrete and continuous datasets.
We conduct extensive experiments on five benchmark datasets to validate the
attack performance of MGHGA in the node and the visual object classification
tasks. The results show that MGHGA improves performance by an average of 2%
compared to the than the baselines.


http://arxiv.org/abs/2310.16332
Corrupting Neuron Explanations of Deep Visual Features. (41%)
Divyansh Srivastava; Tuomas Oikarinen; Tsui-Wei Weng
  The inability of DNNs to explain their black-box behavior has led to a recent
surge of explainability methods. However, there are growing concerns that these
explainability methods are not robust and trustworthy. In this work, we perform
the first robustness analysis of Neuron Explanation Methods under a unified
pipeline and show that these explanations can be significantly corrupted by
random noises and well-designed perturbations added to their probing data. We
find that even adding small random noise with a standard deviation of 0.02 can
already change the assigned concepts of up to 28% neurons in the deeper layers.
Furthermore, we devise a novel corruption algorithm and show that our algorithm
can manipulate the explanation of more than 80% neurons by poisoning less than
10% of probing data. This raises the concern of trusting Neuron Explanation
Methods in real-life safety and fairness critical applications.


http://arxiv.org/abs/2310.18360
Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers. (10%)
Mosh Levy; Shauli Ravfogel; Yoav Goldberg
  Recent applications of LLMs in Machine Reading Comprehension (MRC) systems
have shown impressive results, but the use of shortcuts, mechanisms triggered
by features spuriously correlated to the true label, has emerged as a potential
threat to their reliability. We analyze the problem from two angles: LLMs as
editors, guided to edit text to mislead LLMs; and LLMs as readers, who answer
questions based on the edited text. We introduce a framework that guides an
editor to add potential shortcuts-triggers to samples. Using GPT4 as the
editor, we find it can successfully edit trigger shortcut in samples that fool
LLMs. Analysing LLMs as readers, we observe that even capable LLMs can be
deceived using shortcut knowledge. Strikingly, we discover that GPT4 can be
deceived by its own edits (15% drop in F1). Our findings highlight inherent
vulnerabilities of LLMs to shortcut manipulations. We publish ShortcutQA, a
curated dataset generated by our framework for future research.


http://arxiv.org/abs/2310.15654
A Survey on Detection of LLMs-Generated Content. (1%)
Xianjun Yang; Liangming Pan; Xuandong Zhao; Haifeng Chen; Linda Petzold; William Yang Wang; Wei Cheng
  The burgeoning capabilities of advanced large language models (LLMs) such as
ChatGPT have led to an increase in synthetic content generation with
implications across a variety of sectors, including media, cybersecurity,
public discourse, and education. As such, the ability to detect LLMs-generated
content has become of paramount importance. We aim to provide a detailed
overview of existing detection strategies and benchmarks, scrutinizing their
differences and identifying key challenges and prospects in the field,
advocating for more adaptable and robust models to enhance detection accuracy.
We also posit the necessity for a multi-faceted approach to defend against
various attacks to counter the rapidly advancing capabilities of LLMs. To the
best of our knowledge, this work is the first comprehensive survey on the
detection in the era of LLMs. We hope it will provide a broad understanding of
the current landscape of LLMs-generated content detection, offering a guiding
reference for researchers and practitioners striving to uphold the integrity of
digital information in an era increasingly dominated by synthetic content. The
relevant papers are summarized and will be consistently updated at
https://github.com/Xianjun-Yang/Awesome_papers_on_LLMs_detection.git.


http://arxiv.org/abs/2310.15991
White-box Compiler Fuzzing Empowered by Large Language Models. (1%)
Chenyuan Yang; Yinlin Deng; Runyu Lu; Jiayi Yao; Jiawei Liu; Reyhaneh Jabbarvand; Lingming Zhang
  Compiler correctness is crucial, as miscompilation falsifying the program
behaviors can lead to serious consequences. In the literature, fuzzing has been
extensively studied to uncover compiler defects. However, compiler fuzzing
remains challenging: Existing arts focus on black- and grey-box fuzzing, which
generates tests without sufficient understanding of internal compiler
behaviors. As such, they often fail to construct programs to exercise
conditions of intricate optimizations. Meanwhile, traditional white-box
techniques are computationally inapplicable to the giant codebase of compilers.
Recent advances demonstrate that Large Language Models (LLMs) excel in code
generation/understanding tasks and have achieved state-of-the-art performance
in black-box fuzzing. Nonetheless, prompting LLMs with compiler source-code
information remains a missing piece of research in compiler testing.
  To this end, we propose WhiteFox, the first white-box compiler fuzzer using
LLMs with source-code information to test compiler optimization. WhiteFox
adopts a dual-model framework: (i) an analysis LLM examines the low-level
optimization source code and produces requirements on the high-level test
programs that can trigger the optimization; (ii) a generation LLM produces test
programs based on the summarized requirements. Additionally,
optimization-triggering tests are used as feedback to further enhance the test
generation on the fly. Our evaluation on four popular compilers shows that
WhiteFox can generate high-quality tests to exercise deep optimizations
requiring intricate conditions, practicing up to 80 more optimizations than
state-of-the-art fuzzers. To date, WhiteFox has found in total 96 bugs, with 80
confirmed as previously unknown and 51 already fixed. Beyond compiler testing,
WhiteFox can also be adapted for white-box fuzzing of other complex, real-world
software systems in general.


http://arxiv.org/abs/2310.16263
Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation. (1%)
Jiexin Wang; Liuwen Cao; Xitong Luo; Zhiping Zhou; Jiayuan Xie; Adam Jatowt; Yi Cai
  Large language models (LLMs) have brought significant advancements to code
generation, benefiting both novice and experienced developers. However, their
training using unsanitized data from open-source repositories, like GitHub,
introduces the risk of inadvertently propagating security vulnerabilities. To
effectively mitigate this concern, this paper presents a comprehensive study
focused on evaluating and enhancing code LLMs from a software security
perspective. We introduce SecuCoGen\footnote{SecuCoGen has been uploaded as
supplemental material and will be made publicly available after publication.},
a meticulously curated dataset targeting 21 critical vulnerability types.
SecuCoGen comprises 180 samples and serves as the foundation for conducting
experiments on three crucial code-related tasks: code generation, code repair
and vulnerability classification, with a strong emphasis on security. Our
experimental results reveal that existing models often overlook security
concerns during code generation, leading to the generation of vulnerable code.
To address this, we propose effective approaches to mitigate the security
vulnerabilities and enhance the overall robustness of code generated by LLMs.
Moreover, our study identifies weaknesses in existing models' ability to repair
vulnerable code, even when provided with vulnerability information.
Additionally, certain vulnerability types pose challenges for the models,
hindering their performance in vulnerability classification. Based on these
findings, we believe our study will have a positive impact on the software
engineering community, inspiring the development of improved methods for
training and utilizing LLMs, thereby leading to safer and more trustworthy
model deployment.


http://arxiv.org/abs/2310.14637
Semantic-Aware Adversarial Training for Reliable Deep Hashing Retrieval. (99%)
Xu Yuan; Zheng Zhang; Xunguang Wang; Lin Wu
  Deep hashing has been intensively studied and successfully applied in
large-scale image retrieval systems due to its efficiency and effectiveness.
Recent studies have recognized that the existence of adversarial examples poses
a security threat to deep hashing models, that is, adversarial vulnerability.
Notably, it is challenging to efficiently distill reliable semantic
representatives for deep hashing to guide adversarial learning, and thereby it
hinders the enhancement of adversarial robustness of deep hashing-based
retrieval models. Moreover, current researches on adversarial training for deep
hashing are hard to be formalized into a unified minimax structure. In this
paper, we explore Semantic-Aware Adversarial Training (SAAT) for improving the
adversarial robustness of deep hashing models. Specifically, we conceive a
discriminative mainstay features learning (DMFL) scheme to construct semantic
representatives for guiding adversarial learning in deep hashing. Particularly,
our DMFL with the strict theoretical guarantee is adaptively optimized in a
discriminative learning manner, where both discriminative and semantic
properties are jointly considered. Moreover, adversarial examples are
fabricated by maximizing the Hamming distance between the hash codes of
adversarial samples and mainstay features, the efficacy of which is validated
in the adversarial attack trials. Further, we, for the first time, formulate
the formalized adversarial training of deep hashing into a unified minimax
optimization under the guidance of the generated mainstay codes. Extensive
experiments on benchmark datasets show superb attack performance against the
state-of-the-art algorithms, meanwhile, the proposed adversarial training can
effectively eliminate adversarial perturbations for trustworthy deep
hashing-based retrieval. Our code is available at
https://github.com/xandery-geek/SAAT.


http://arxiv.org/abs/2310.14561
F$^2$AT: Feature-Focusing Adversarial Training via Disentanglement of Natural and Perturbed Patterns. (99%)
Yaguan Qian; Chenyu Zhao; Zhaoquan Gu; Bin Wang; Shouling Ji; Wei Wang; Boyang Zhou; Pan Zhou
  Deep neural networks (DNNs) are vulnerable to adversarial examples crafted by
well-designed perturbations. This could lead to disastrous results on critical
applications such as self-driving cars, surveillance security, and medical
diagnosis. At present, adversarial training is one of the most effective
defenses against adversarial examples. However, traditional adversarial
training makes it difficult to achieve a good trade-off between clean accuracy
and robustness since spurious features are still learned by DNNs. The intrinsic
reason is that traditional adversarial training makes it difficult to fully
learn core features from adversarial examples when adversarial noise and clean
examples cannot be disentangled. In this paper, we disentangle the adversarial
examples into natural and perturbed patterns by bit-plane slicing. We assume
the higher bit-planes represent natural patterns and the lower bit-planes
represent perturbed patterns, respectively. We propose a Feature-Focusing
Adversarial Training (F$^2$AT), which differs from previous work in that it
enforces the model to focus on the core features from natural patterns and
reduce the impact of spurious features from perturbed patterns. The
experimental results demonstrated that F$^2$AT outperforms state-of-the-art
methods in clean accuracy and adversarial robustness.


http://arxiv.org/abs/2310.15444
Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks. (98%)
Xiaojun Jia; Jianshu Li; Jindong Gu; Yang Bai; Xiaochun Cao
  Adversarial training has shown promise in building robust models against
adversarial examples. A major drawback of adversarial training is the
computational overhead introduced by the generation of adversarial examples. To
overcome this limitation, adversarial training based on single-step attacks has
been explored. Previous work improves the single-step adversarial training from
different perspectives, e.g., sample initialization, loss regularization, and
training strategy. Almost all of them treat the underlying model as a black
box. In this work, we propose to exploit the interior building blocks of the
model to improve efficiency. Specifically, we propose to dynamically sample
lightweight subnetworks as a surrogate model during training. By doing this,
both the forward and backward passes can be accelerated for efficient
adversarial training. Besides, we provide theoretical analysis to show the
model robustness can be improved by the single-step adversarial training with
sampled subnetworks. Furthermore, we propose a novel sampling strategy where
the sampling varies from layer to layer and from iteration to iteration.
Compared with previous methods, our method not only reduces the training cost
but also achieves better model robustness. Evaluations on a series of popular
datasets demonstrate the effectiveness of the proposed FB-Better. Our code has
been released at https://github.com/jiaxiaojunQAQ/FP-Better.


http://arxiv.org/abs/2310.15140
AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models. (92%)
Sicheng Zhu; Ruiyi Zhang; Bang An; Gang Wu; Joe Barrow; Zichao Wang; Furong Huang; Ani Nenkova; Tong Sun
  Safety alignment of Large Language Models (LLMs) can be compromised with
manual jailbreak attacks and (automatic) adversarial attacks. Recent work
suggests that patching LLMs against these attacks is possible: manual jailbreak
attacks are human-readable but often limited and public, making them easy to
block; adversarial attacks generate gibberish prompts that can be detected
using perplexity-based filters. In this paper, we show that these solutions may
be too optimistic. We propose an interpretable adversarial attack,
\texttt{AutoDAN}, that combines the strengths of both types of attacks. It
automatically generates attack prompts that bypass perplexity-based filters
while maintaining a high attack success rate like manual jailbreak attacks.
These prompts are interpretable and diverse, exhibiting strategies commonly
used in manual jailbreak attacks, and transfer better than their non-readable
counterparts when using limited training data or a single proxy model. We also
customize \texttt{AutoDAN}'s objective to leak system prompts, another
jailbreak application not addressed in the adversarial attack literature. %,
demonstrating the versatility of the approach. We can also customize the
objective of \texttt{AutoDAN} to leak system prompts, beyond the ability to
elicit harmful content from the model, demonstrating the versatility of the
approach. Our work provides a new way to red-team LLMs and to understand the
mechanism of jailbreak attacks.


http://arxiv.org/abs/2310.15085
On the Detection of Image-Scaling Attacks in Machine Learning. (15%)
Erwin Quiring; Andreas Müller; Konrad Rieck
  Image scaling is an integral part of machine learning and computer vision
systems. Unfortunately, this preprocessing step is vulnerable to so-called
image-scaling attacks where an attacker makes unnoticeable changes to an image
so that it becomes a new image after scaling. This opens up new ways for
attackers to control the prediction or to improve poisoning and backdoor
attacks. While effective techniques exist to prevent scaling attacks, their
detection has not been rigorously studied yet. Consequently, it is currently
not possible to reliably spot these attacks in practice.
  This paper presents the first in-depth systematization and analysis of
detection methods for image-scaling attacks. We identify two general detection
paradigms and derive novel methods from them that are simple in design yet
significantly outperform previous work. We demonstrate the efficacy of these
methods in a comprehensive evaluation with all major learning platforms and
scaling algorithms. First, we show that image-scaling attacks modifying the
entire scaled image can be reliably detected even under an adaptive adversary.
Second, we find that our methods provide strong detection performance even if
only minor parts of the image are manipulated. As a result, we can introduce a
novel protection layer against image-scaling attacks.


http://arxiv.org/abs/2310.15171
RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions. (1%)
Lingdong Kong; Shaoyuan Xie; Hanjiang Hu; Lai Xing Ng; Benoit R. Cottereau; Wei Tsang Ooi
  Depth estimation from monocular images is pivotal for real-world visual
perception systems. While current learning-based depth estimation models train
and test on meticulously curated data, they often overlook out-of-distribution
(OoD) situations. Yet, in practical settings -- especially safety-critical ones
like autonomous driving -- common corruptions can arise. Addressing this
oversight, we introduce a comprehensive robustness test suite, RoboDepth,
encompassing 18 corruptions spanning three categories: i) weather and lighting
conditions; ii) sensor failures and movement; and iii) data processing
anomalies. We subsequently benchmark 42 depth estimation models across indoor
and outdoor scenes to assess their resilience to these corruptions. Our
findings underscore that, in the absence of a dedicated robustness evaluation
framework, many leading depth estimation models may be susceptible to typical
corruptions. We delve into design considerations for crafting more robust depth
estimation models, touching upon pre-training, augmentation, modality, model
capacity, and learning paradigms. We anticipate our benchmark will establish a
foundational platform for advancing robust OoD depth estimation.


http://arxiv.org/abs/2310.14270
Diffusion-Based Adversarial Purification for Speaker Verification. (99%)
Yibo Bai; Xiao-Lei Zhang
  Recently, automatic speaker verification (ASV) based on deep learning is
easily contaminated by adversarial attacks, which is a new type of attack that
injects imperceptible perturbations to audio signals so as to make ASV produce
wrong decisions. This poses a significant threat to the security and
reliability of ASV systems. To address this issue, we propose a Diffusion-Based
Adversarial Purification (DAP) method that enhances the robustness of ASV
systems against such adversarial attacks. Our method leverages a conditional
denoising diffusion probabilistic model to effectively purify the adversarial
examples and mitigate the impact of perturbations. DAP first introduces
controlled noise into adversarial examples, and then performs a reverse
denoising process to reconstruct clean audio. Experimental results demonstrate
the efficacy of the proposed DAP in enhancing the security of ASV and meanwhile
minimizing the distortion of the purified audio signals.


http://arxiv.org/abs/2310.14265
CT-GAT: Cross-Task Generative Adversarial Attack based on Transferability. (99%)
Minxuan Lv; Chengwei Dai; Kun Li; Wei Zhou; Songlin Hu
  Neural network models are vulnerable to adversarial examples, and adversarial
transferability further increases the risk of adversarial attacks. Current
methods based on transferability often rely on substitute models, which can be
impractical and costly in real-world scenarios due to the unavailability of
training data and the victim model's structural details. In this paper, we
propose a novel approach that directly constructs adversarial examples by
extracting transferable features across various tasks. Our key insight is that
adversarial transferability can extend across different tasks. Specifically, we
train a sequence-to-sequence generative model named CT-GAT using adversarial
sample data collected from multiple tasks to acquire universal adversarial
features and generate adversarial examples for different tasks. We conduct
experiments on ten distinct datasets, and the results demonstrate that our
method achieves superior attack performance with small cost.


http://arxiv.org/abs/2311.16118
Imperceptible CMOS camera dazzle for adversarial attacks on deep neural networks. (92%)
Zvi Stein; Adrian Stern
  Despite the outstanding performance of deep neural networks, they are
vulnerable to adversarial attacks. While there are many invisible attacks in
the digital domain, most physical world adversarial attacks are visible. Here
we present an invisible optical adversarial attack that uses a light source to
dazzle a CMOS camera with a rolling shutter. We present the photopic conditions
required to keep the attacking light source completely invisible while
sufficiently jamming the captured image so that a deep neural network applied
to it is deceived.


http://arxiv.org/abs/2310.14504
ADoPT: LiDAR Spoofing Attack Detection Based on Point-Level Temporal Consistency. (26%)
Minkyoung Cho; Yulong Cao; Zixiang Zhou; Z. Morley Mao
  Deep neural networks (DNNs) are increasingly integrated into LiDAR (Light
Detection and Ranging)-based perception systems for autonomous vehicles (AVs),
requiring robust performance under adversarial conditions. We aim to address
the challenge of LiDAR spoofing attacks, where attackers inject fake objects
into LiDAR data and fool AVs to misinterpret their environment and make
erroneous decisions. However, current defense algorithms predominantly depend
on perception outputs (i.e., bounding boxes) thus face limitations in detecting
attackers given the bounding boxes are generated by imperfect perception models
processing limited points, acquired based on the ego vehicle's viewpoint. To
overcome these limitations, we propose a novel framework, named ADoPT (Anomaly
Detection based on Point-level Temporal consistency), which quantitatively
measures temporal consistency across consecutive frames and identifies abnormal
objects based on the coherency of point clusters. In our evaluation using the
nuScenes dataset, our algorithm effectively counters various LiDAR spoofing
attacks, achieving a low (< 10%) false positive ratio (FPR) and high (> 85%)
true positive ratio (TPR), outperforming existing state-of-the-art defense
methods, CARLO and 3D-TC2. Furthermore, our evaluation demonstrates the
promising potential for accurate attack detection across various road
environments.


http://arxiv.org/abs/2310.14480
Attention-Enhancing Backdoor Attacks Against BERT-based Models. (13%)
Weimin Lyu; Songzhu Zheng; Lu Pang; Haibin Ling; Chao Chen
  Recent studies have revealed that \textit{Backdoor Attacks} can threaten the
safety of natural language processing (NLP) models. Investigating the
strategies of backdoor attacks will help to understand the model's
vulnerability. Most existing textual backdoor attacks focus on generating
stealthy triggers or modifying model weights. In this paper, we directly target
the interior structure of neural networks and the backdoor mechanism. We
propose a novel Trojan Attention Loss (TAL), which enhances the Trojan behavior
by directly manipulating the attention patterns. Our loss can be applied to
different attacking methods to boost their attack efficacy in terms of attack
successful rates and poisoning rates. It applies to not only traditional
dirty-label attacks, but also the more challenging clean-label attacks. We
validate our method on different backbone models (BERT, RoBERTa, and
DistilBERT) and various tasks (Sentiment Analysis, Toxic Detection, and Topic
Classification).


http://arxiv.org/abs/2310.14369
MoPe: Model Perturbation-based Privacy Attacks on Language Models. (9%)
Marvin Li; Jason Wang; Jeffrey Wang; Seth Neel
  Recent work has shown that Large Language Models (LLMs) can unintentionally
leak sensitive information present in their training data. In this paper, we
present Model Perturbations (MoPe), a new method to identify with high
confidence if a given text is in the training data of a pre-trained language
model, given white-box access to the models parameters. MoPe adds noise to the
model in parameter space and measures the drop in log-likelihood at a given
point $x$, a statistic we show approximates the trace of the Hessian matrix
with respect to model parameters. Across language models ranging from $70$M to
$12$B parameters, we show that MoPe is more effective than existing loss-based
attacks and recently proposed perturbation-based methods. We also examine the
role of training point order and model size in attack success, and empirically
demonstrate that MoPe accurately approximate the trace of the Hessian in
practice. Our results show that the loss of a point alone is insufficient to
determine extractability -- there are training points we can recover using our
method that have average loss. This casts some doubt on prior works that use
the loss of a point as evidence of memorization or unlearning.


http://arxiv.org/abs/2310.13950
Adversarial Image Generation by Spatial Transformation in Perceptual Colorspaces. (99%)
Ayberk Aydin; Alptekin Temizel
  Deep neural networks are known to be vulnerable to adversarial perturbations.
The amount of these perturbations are generally quantified using $L_p$ metrics,
such as $L_0$, $L_2$ and $L_\infty$. However, even when the measured
perturbations are small, they tend to be noticeable by human observers since
$L_p$ distance metrics are not representative of human perception. On the other
hand, humans are less sensitive to changes in colorspace. In addition, pixel
shifts in a constrained neighborhood are hard to notice. Motivated by these
observations, we propose a method that creates adversarial examples by applying
spatial transformations, which creates adversarial examples by changing the
pixel locations independently to chrominance channels of perceptual colorspaces
such as $YC_{b}C_{r}$ and $CIELAB$, instead of making an additive perturbation
or manipulating pixel values directly. In a targeted white-box attack setting,
the proposed method is able to obtain competitive fooling rates with very high
confidence. The experimental evaluations show that the proposed method has
favorable results in terms of approximate perceptual distance between benign
and adversarially generated images. The source code is publicly available at
https://github.com/ayberkydn/stadv-torch


http://arxiv.org/abs/2310.14045
Training Image Derivatives: Increased Accuracy and Universal Robustness. (5%)
Vsevolod I. Avrutskiy
  Derivative training is a well-known method to improve the accuracy of neural
networks. In the forward pass, not only the output values are computed, but
also their derivatives, and their deviations from the target derivatives are
included in the cost function, which is minimized with respect to the weights
by a gradient-based algorithm. So far, this method has been implemented for
relatively low-dimensional tasks. In this study, we apply the approach to the
problem of image analysis. We consider the task of reconstructing the vertices
of a cube based on its image. By training the derivatives with respect to the 6
degrees of freedom of the cube, we obtain 25 times more accurate results for
noiseless inputs. The derivatives also provide important insights into the
robustness problem, which is currently understood in terms of two types of
network vulnerabilities. The first type is small perturbations that
dramatically change the output, and the second type is substantial image
changes that the network erroneously ignores. They are currently considered as
conflicting goals, since conventional training methods produce a trade-off. The
first type can be analyzed via the gradient of the network, but the second type
requires human evaluation of the inputs, which is an oracle substitute. For the
task at hand, the nearest neighbor oracle can be defined, and the knowledge of
derivatives allows it to be expanded into Taylor series. This allows to perform
the first-order robustness analysis that unifies both types of vulnerabilities,
and to implement robust training that eliminates any trade-offs, so that
accuracy and robustness are limited only by network capacity.


http://arxiv.org/abs/2310.13321
Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting. (99%)
Zecheng Tang; Kaifeng Qi; Juntao Li; Min Zhang
  Recent studies have revealed that grammatical error correction methods in the
sequence-to-sequence paradigm are vulnerable to adversarial attack, and simply
utilizing adversarial examples in the pre-training or post-training process can
significantly enhance the robustness of GEC models to certain types of attack
without suffering too much performance loss on clean data. In this paper, we
further conduct a thorough robustness evaluation of cutting-edge GEC methods
for four different types of adversarial attacks and propose a simple yet very
effective Cycle Self-Augmenting (CSA) method accordingly. By leveraging the
augmenting data from the GEC models themselves in the post-training process and
introducing regularization data for cycle training, our proposed method can
effectively improve the model robustness of well-trained GEC models with only a
few more training epochs as an extra cost. More concretely, further training on
the regularization data can prevent the GEC models from over-fitting on
easy-to-learn samples and thus can improve the generalization capability and
robustness towards unseen data (adversarial noise/samples). Meanwhile, the
self-augmented data can provide more high-quality pseudo pairs to improve model
performance on the original testing data. Experiments on four benchmark
datasets and seven strong models indicate that our proposed training method can
significantly enhance the robustness of four types of attacks without using
purposely built adversarial examples in training. Evaluation results on clean
data further confirm that our proposed CSA method significantly improves the
performance of four baselines and yields nearly comparable results with other
state-of-the-art models. Our code is available at
https://github.com/ZetangForward/CSA-GEC.


http://arxiv.org/abs/2310.13345
An LLM can Fool Itself: A Prompt-Based Adversarial Attack. (99%)
Xilie Xu; Keyi Kong; Ning Liu; Lizhen Cui; Di Wang; Jingfeng Zhang; Mohan Kankanhalli
  The wide-ranging applications of large language models (LLMs), especially in
safety-critical domains, necessitate the proper evaluation of the LLM's
adversarial robustness. This paper proposes an efficient tool to audit the
LLM's adversarial robustness via a prompt-based adversarial attack
(PromptAttack). PromptAttack converts adversarial textual attacks into an
attack prompt that can cause the victim LLM to output the adversarial sample to
fool itself. The attack prompt is composed of three important components: (1)
original input (OI) including the original sample and its ground-truth label,
(2) attack objective (AO) illustrating a task description of generating a new
sample that can fool itself without changing the semantic meaning, and (3)
attack guidance (AG) containing the perturbation instructions to guide the LLM
on how to complete the task by perturbing the original sample at character,
word, and sentence levels, respectively. Besides, we use a fidelity filter to
ensure that PromptAttack maintains the original semantic meanings of the
adversarial examples. Further, we enhance the attack power of PromptAttack by
ensembling adversarial examples at different perturbation levels. Comprehensive
empirical results using Llama2 and GPT-3.5 validate that PromptAttack
consistently yields a much higher attack success rate compared to AdvGLUE and
AdvGLUE++. Interesting findings include that a simple emoji can easily mislead
GPT-3.5 to make wrong predictions.


http://arxiv.org/abs/2310.13828
Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models. (56%)
Shawn Shan; Wenxin Ding; Josephine Passananti; Haitao Zheng; Ben Y. Zhao
  Data poisoning attacks manipulate training data to introduce unexpected
behaviors into machine learning models at training time. For text-to-image
generative models with massive training datasets, current understanding of
poisoning attacks suggests that a successful attack would require injecting
millions of poison samples into their training pipeline. In this paper, we show
that poisoning attacks can be successful on generative models. We observe that
training data per concept can be quite limited in these models, making them
vulnerable to prompt-specific poisoning attacks, which target a model's ability
to respond to individual prompts.
  We introduce Nightshade, an optimized prompt-specific poisoning attack where
poison samples look visually identical to benign images with matching text
prompts. Nightshade poison samples are also optimized for potency and can
corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade
poison effects "bleed through" to related concepts, and multiple attacks can
composed together in a single prompt. Surprisingly, we show that a moderate
number of Nightshade attacks can destabilize general features in a
text-to-image generative model, effectively disabling its ability to generate
meaningful images. Finally, we propose the use of Nightshade` and similar tools
as a last defense for content creators against web scrapers that ignore
opt-out/do-not-crawl directives, and discuss possible implications for model
trainers and content creators.


http://arxiv.org/abs/2310.13893
The Hidden Adversarial Vulnerabilities of Medical Federated Learning. (45%)
Erfan Darzi; Florian Dubost; Nanna. M. Sijtsema; Ooijen P. M. A van
  In this paper, we delve into the susceptibility of federated medical image
analysis systems to adversarial attacks. Our analysis uncovers a novel
exploitation avenue: using gradient information from prior global model
updates, adversaries can enhance the efficiency and transferability of their
attacks. Specifically, we demonstrate that single-step attacks (e.g. FGSM),
when aptly initialized, can outperform the efficiency of their iterative
counterparts but with reduced computational demand. Our findings underscore the
need to revisit our understanding of AI security in federated healthcare
settings.


http://arxiv.org/abs/2310.13822
Adversarial Attacks on Fairness of Graph Neural Networks. (26%)
Binchi Zhang; Yushun Dong; Chen Chen; Yada Zhu; Minnan Luo; Jundong Li
  Fairness-aware graph neural networks (GNNs) have gained a surge of attention
as they can reduce the bias of predictions on any demographic group (e.g.,
female) in graph-based applications. Although these methods greatly improve the
algorithmic fairness of GNNs, the fairness can be easily corrupted by carefully
designed adversarial attacks. In this paper, we investigate the problem of
adversarial attacks on fairness of GNNs and propose G-FairAttack, a general
framework for attacking various types of fairness-aware GNNs in terms of
fairness with an unnoticeable effect on prediction utility. In addition, we
propose a fast computation technique to reduce the time complexity of
G-FairAttack. The experimental study demonstrates that G-FairAttack
successfully corrupts the fairness of different types of GNNs while keeping the
attack unnoticeable. Our study on fairness attacks sheds light on potential
vulnerabilities in fairness-aware GNNs and guides further research on the
robustness of GNNs in terms of fairness. The open-source code is available at
https://github.com/zhangbinchi/G-FairAttack.


http://arxiv.org/abs/2310.13424
FLTracer: Accurate Poisoning Attack Provenance in Federated Learning. (26%)
Xinyu Zhang; Qingyu Liu; Zhongjie Ba; Yuan Hong; Tianhang Zheng; Feng Lin; Li Lu; Kui Ren
  Federated Learning (FL) is a promising distributed learning approach that
enables multiple clients to collaboratively train a shared global model.
However, recent studies show that FL is vulnerable to various poisoning
attacks, which can degrade the performance of global models or introduce
backdoors into them. In this paper, we first conduct a comprehensive study on
prior FL attacks and detection methods. The results show that all existing
detection methods are only effective against limited and specific attacks. Most
detection methods suffer from high false positives, which lead to significant
performance degradation, especially in not independent and identically
distributed (non-IID) settings. To address these issues, we propose FLTracer,
the first FL attack provenance framework to accurately detect various attacks
and trace the attack time, objective, type, and poisoned location of updates.
Different from existing methodologies that rely solely on cross-client anomaly
detection, we propose a Kalman filter-based cross-round detection to identify
adversaries by seeking the behavior changes before and after the attack. Thus,
this makes it resilient to data heterogeneity and is effective even in non-IID
settings. To further improve the accuracy of our detection method, we employ
four novel features and capture their anomalies with the joint decisions.
Extensive evaluations show that FLTracer achieves an average true positive rate
of over $96.88\%$ at an average false positive rate of less than $2.67\%$,
significantly outperforming SOTA detection methods. \footnote{Code is available
at \url{https://github.com/Eyr3/FLTracer}.}


http://arxiv.org/abs/2311.03369
Can We Trust the Similarity Measurement in Federated Learning? (15%)
Zhilin Wang; Qin Hu; Xukai Zou
  Is it secure to measure the reliability of local models by similarity in
federated learning (FL)? This paper delves into an unexplored security threat
concerning applying similarity metrics, such as the L_2 norm, Euclidean
distance, and cosine similarity, in protecting FL. We first uncover the
deficiencies of similarity metrics that high-dimensional local models,
including benign and poisoned models, may be evaluated to have the same
similarity while being significantly different in the parameter values. We then
leverage this finding to devise a novel untargeted model poisoning attack,
Faker, which launches the attack by simultaneously maximizing the evaluated
similarity of the poisoned local model and the difference in the parameter
values. Experimental results based on seven datasets and eight defenses show
that Faker outperforms the state-of-the-art benchmark attacks by 1.1-9.0X in
reducing accuracy and 1.2-8.0X in saving time cost, which even holds for the
case of a single malicious client with limited knowledge about the FL system.
Moreover, Faker can degrade the performance of the global model by attacking
only once. We also preliminarily explore extending Faker to other attacks, such
as backdoor attacks and Sybil attacks. Lastly, we provide a model evaluation
strategy, called the similarity of partial parameters (SPP), to defend against
Faker. Given that numerous mechanisms in FL utilize similarity metrics to
assess local models, this work suggests that we should be vigilant regarding
the potential risks of using these metrics.


http://arxiv.org/abs/2310.13782
Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images. (4%)
Logan Frank; Jim Davis
  Knowledge distillation (KD) has been a popular and effective method for model
compression. One important assumption of KD is that the original training
dataset is always available. However, this is not always the case due to
privacy concerns and more. In recent years, "data-free" KD has emerged as a
growing research topic which focuses on the scenario of performing KD when no
data is provided. Many methods rely on a generator network to synthesize
examples for distillation (which can be difficult to train) and can frequently
produce images that are visually similar to the original dataset, which raises
questions surrounding whether privacy is completely preserved. In this work, we
propose a new approach to data-free KD that utilizes unnatural OpenGL images,
combined with large amounts of data augmentation and adversarial attacks, to
train a student network. We demonstrate that our approach achieves
state-of-the-art results for a variety of datasets/networks and is more stable
than existing generator-based data-free KD methods. Source code will be
available in the future.


http://arxiv.org/abs/2310.13894
VOICE-ZEUS: Impersonating Zoom's E2EE-Protected Static Media and Textual Communications via Simple Voice Manipulations. (4%)
Mashari Alatawi; Nitesh Saxena
  The authentication ceremony plays a crucial role in verifying the identities
of users before exchanging messages in end-to-end encryption (E2EE)
applications, thus preventing impersonation and man-in-the-middle (MitM)
attacks. Once authenticated, the subsequent communications in E2EE apps benefit
from the protection provided by the authentication ceremony. However, the
current implementation of the authentication ceremony in the Zoom application
introduces a potential vulnerability that can make it highly susceptible to
impersonation attacks. The existence of this vulnerability may undermine the
integrity of E2EE, posing a potential security risk when E2EE becomes a
mandatory feature in the Zoom application. In this paper, we examine and
evaluate this vulnerability in two attack scenarios, one where the attacker is
a malicious participant and another where the attacker is a malicious Zoom
server with control over Zoom's server infrastructure and cloud providers. Our
study aims to comprehensively examine the Zoom authentication ceremony, with a
specific focus on the potential for impersonation attacks in static media and
textual communications. We simulate a new session injection attack on Zoom E2EE
meetings to evaluate the system's susceptibility to simple voice manipulations.
Our simulation experiments show that Zoom's authentication ceremony is
vulnerable to a simple voice manipulation, called a VOICE-ZEUS attack, by
malicious participants and the malicious Zoom server. In this VOICE-ZEUS
attack, an attacker creates a fingerprint in a victim's voice by reordering
previously recorded digits spoken by the victim. We show how an attacker can
record and reorder snippets of digits to generate a new security code that
compromises a future Zoom meeting. We conclude that stronger security measures
are necessary during the group authentication ceremony in Zoom to prevent
impersonation attacks.


http://arxiv.org/abs/2310.12708
Generating Robust Adversarial Examples against Online Social Networks (OSNs). (98%)
Jun Liu; Jiantao Zhou; Haiwei Wu; Weiwei Sun; Jinyu Tian
  Online Social Networks (OSNs) have blossomed into prevailing transmission
channels for images in the modern era. Adversarial examples (AEs) deliberately
designed to mislead deep neural networks (DNNs) are found to be fragile against
the inevitable lossy operations conducted by OSNs. As a result, the AEs would
lose their attack capabilities after being transmitted over OSNs. In this work,
we aim to design a new framework for generating robust AEs that can survive the
OSN transmission; namely, the AEs before and after the OSN transmission both
possess strong attack capabilities. To this end, we first propose a
differentiable network termed SImulated OSN (SIO) to simulate the various
operations conducted by an OSN. Specifically, the SIO network consists of two
modules: 1) a differentiable JPEG layer for approximating the ubiquitous JPEG
compression and 2) an encoder-decoder subnetwork for mimicking the remaining
operations. Based upon the SIO network, we then formulate an optimization
framework to generate robust AEs by enforcing model outputs with and without
passing through the SIO to be both misled. Extensive experiments conducted over
Facebook, WeChat and QQ demonstrate that our attack methods produce more robust
AEs than existing approaches, especially under small distortion constraints;
the performance gain in terms of Attack Success Rate (ASR) could be more than
60%. Furthermore, we build a public dataset containing more than 10,000 pairs
of AEs processed by Facebook, WeChat or QQ, facilitating future research in the
robust AEs generation. The dataset and code are available at
https://github.com/csjunjun/RobustOSNAttack.git.


http://arxiv.org/abs/2310.12707
Recoverable Privacy-Preserving Image Classification through Noise-like Adversarial Examples. (98%)
Jun Liu; Jiantao Zhou; Jinyu Tian; Weiwei Sun
  With the increasing prevalence of cloud computing platforms, ensuring data
privacy during the cloud-based image related services such as classification
has become crucial. In this study, we propose a novel privacypreserving image
classification scheme that enables the direct application of classifiers
trained in the plaintext domain to classify encrypted images, without the need
of retraining a dedicated classifier. Moreover, encrypted images can be
decrypted back into their original form with high fidelity (recoverable) using
a secret key. Specifically, our proposed scheme involves utilizing a feature
extractor and an encoder to mask the plaintext image through a newly designed
Noise-like Adversarial Example (NAE). Such an NAE not only introduces a
noise-like visual appearance to the encrypted image but also compels the target
classifier to predict the ciphertext as the same label as the original
plaintext image. At the decoding phase, we adopt a Symmetric Residual Learning
(SRL) framework for restoring the plaintext image with minimal degradation.
Extensive experiments demonstrate that 1) the classification accuracy of the
classifier trained in the plaintext domain remains the same in both the
ciphertext and plaintext domains; 2) the encrypted images can be recovered into
their original form with an average PSNR of up to 51+ dB for the SVHN dataset
and 48+ dB for the VGGFace2 dataset; 3) our system exhibits satisfactory
generalization capability on the encryption, decryption and classification
tasks across datasets that are different from the training one; and 4) a
high-level of security is achieved against three potential threat models. The
code is available at https://github.com/csjunjun/RIC.git.


http://arxiv.org/abs/2310.12713
Learn from the Past: A Proxy based Adversarial Defense Framework to Boost Robustness. (98%)
Yaohua Liu; Jiaxin Gao; Zhu Liu; Xianghao Jiao; Xin Fan; Risheng Liu
  In light of the vulnerability of deep learning models to adversarial samples
and the ensuing security issues, a range of methods, including Adversarial
Training (AT) as a prominent representative, aimed at enhancing model
robustness against various adversarial attacks, have seen rapid development.
However, existing methods essentially assist the current state of target model
to defend against parameter-oriented adversarial attacks with explicit or
implicit computation burdens, which also suffers from unstable convergence
behavior due to inconsistency of optimization trajectories. Diverging from
previous work, this paper reconsiders the update rule of target model and
corresponding deficiency to defend based on its current state. By introducing
the historical state of the target model as a proxy, which is endowed with much
prior information for defense, we formulate a two-stage update rule, resulting
in a general adversarial defense framework, which we refer to as `LAST' ({\bf
L}earn from the P{\bf ast}). Besides, we devise a Self Distillation (SD) based
defense objective to constrain the update process of the proxy model without
the introduction of larger teacher models. Experimentally, we demonstrate
consistent and significant performance enhancements by refining a series of
single-step and multi-step AT methods (e.g., up to $\bf 9.2\%$ and $\bf 20.5\%$
improvement of Robust Accuracy (RA) on CIFAR10 and CIFAR100 datasets,
respectively) across various datasets, backbones and attack modalities, and
validate its ability to enhance training stability and ameliorate catastrophic
overfitting issues meanwhile.


http://arxiv.org/abs/2310.12793
OODRobustBench: benchmarking and analyzing adversarial robustness under distribution shift. (97%)
Lin Li; Yifei Wang; Chawin Sitawarin; Michael Spratling
  Existing works have made great progress in improving adversarial robustness,
but typically test their method only on data from the same distribution as the
training data, i.e. in-distribution (ID) testing. As a result, it is unclear
how such robustness generalizes under input distribution shifts, i.e.
out-of-distribution (OOD) testing. This is a concerning omission as such
distribution shifts are unavoidable when methods are deployed in the wild. To
address this issue we propose a benchmark named OODRobustBench to
comprehensively assess OOD adversarial robustness using 23 dataset-wise shifts
(i.e. naturalistic shifts in input distribution) and 6 threat-wise shifts
(i.e., unforeseen adversarial threat models). OODRobustBench is used to assess
706 robust models using 60.7K adversarial evaluations. This large-scale
analysis shows that: 1) adversarial robustness suffers from a severe OOD
generalization issue; 2) ID robustness correlates strongly with OOD robustness,
in a positive linear way, under many distribution shifts. The latter enables
the prediction of OOD robustness from ID robustness. Based on this, we are able
to predict the upper limit of OOD robustness for existing robust training
schemes. The results suggest that achieving OOD robustness requires designing
novel methods beyond the conventional ones. Last, we discover that extra data,
data augmentation, advanced model architectures and particular regularization
approaches can improve OOD robustness. Noticeably, the discovered training
schemes, compared to the baseline, exhibit dramatically higher robustness under
threat shift while keeping high ID robustness, demonstrating new promising
solutions for robustness against both multi-attack and unforeseen attacks.


http://arxiv.org/abs/2310.12516
Automatic Hallucination Assessment for Aligned Large Language Models via Transferable Adversarial Attacks. (97%)
Xiaodong Yu; Hao Cheng; Xiaodong Liu; Dan Roth; Jianfeng Gao
  Although remarkable progress has been achieved in preventing large language
model (LLM) hallucinations using instruction tuning and retrieval augmentation,
it remains challenging to measure the reliability of LLMs using human-crafted
evaluation data which is not available for many tasks and domains and could
suffer from data leakage. Inspired by adversarial machine learning, this paper
aims to develop a method of automatically generating evaluation data by
appropriately modifying existing data on which LLMs behave faithfully.
Specifically, this paper presents AutoDebug, an LLM-based framework to use
prompting chaining to generate transferable adversarial attacks in the form of
question-answering examples. We seek to understand the extent to which these
examples trigger the hallucination behaviors of LLMs.
  We implement AutoDebug using ChatGPT and evaluate the resulting two variants
of a popular open-domain question-answering dataset, Natural Questions (NQ), on
a collection of open-source and proprietary LLMs under various prompting
settings. Our generated evaluation data is human-readable and, as we show,
humans can answer these modified questions well. Nevertheless, we observe
pronounced accuracy drops across multiple LLMs including GPT-4. Our
experimental results show that LLMs are likely to hallucinate in two categories
of question-answering scenarios where (1) there are conflicts between knowledge
given in the prompt and their parametric knowledge, or (2) the knowledge
expressed in the prompt is complex. Finally, we find that the adversarial
examples generated by our method are transferable across all considered LLMs.
The examples generated by a small model can be used to debug a much larger
model, making our approach cost-effective.


http://arxiv.org/abs/2310.13076
PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defenses. (97%)
Chong Xiang; Tong Wu; Sihui Dai; Jonathan Petit; Suman Jana; Prateek Mittal
  State-of-the-art defenses against adversarial patch attacks can now achieve
strong certifiable robustness with a marginal drop in model utility. However,
this impressive performance typically comes at the cost of 10-100x more
inference-time computation compared to undefended models -- the research
community has witnessed an intense three-way trade-off between certifiable
robustness, model utility, and computation efficiency. In this paper, we
propose a defense framework named PatchCURE to approach this trade-off problem.
PatchCURE provides sufficient "knobs" for tuning defense performance and allows
us to build a family of defenses: the most robust PatchCURE instance can match
the performance of any existing state-of-the-art defense (without efficiency
considerations); the most efficient PatchCURE instance has similar inference
efficiency as undefended models. Notably, PatchCURE achieves state-of-the-art
robustness and utility performance across all different efficiency levels,
e.g., 16-23% absolute clean accuracy and certified robust accuracy advantages
over prior defenses when requiring computation efficiency to be close to
undefended models. The family of PatchCURE defenses enables us to flexibly
choose appropriate defenses to satisfy given computation and/or utility
constraints in practice.


http://arxiv.org/abs/2310.12815
Prompt Injection Attacks and Defenses in LLM-Integrated Applications. (47%)
Yupei Liu; Yuqi Jia; Runpeng Geng; Jinyuan Jia; Neil Zhenqiang Gong
  Large Language Models (LLMs) are increasingly deployed as the backend for a
variety of real-world applications called LLM-Integrated Applications. Multiple
recent works showed that LLM-Integrated Applications are vulnerable to prompt
injection attacks, in which an attacker injects malicious instruction/data into
the input of those applications such that they produce results as the attacker
desires. However, existing works are limited to case studies. As a result, the
literature lacks a systematic understanding of prompt injection attacks and
their defenses. We aim to bridge the gap in this work. In particular, we
propose a general framework to formalize prompt injection attacks. Existing
attacks, which are discussed in research papers and blog posts, are special
cases in our framework. Our framework enables us to design a new attack by
combining existing attacks. Moreover, we also propose a framework to
systematize defenses against prompt injection attacks. Using our frameworks, we
conduct a systematic evaluation on prompt injection attacks and their defenses
with 10 LLMs and 7 tasks. We hope our frameworks can inspire future research in
this field. Our code is available at
https://github.com/liu00222/Open-Prompt-Injection.


http://arxiv.org/abs/2310.12505
Attack Prompt Generation for Red Teaming and Defending Large Language Models. (15%)
Boyi Deng; Wenjie Wang; Fuli Feng; Yang Deng; Qifan Wang; Xiangnan He
  Large language models (LLMs) are susceptible to red teaming attacks, which
can induce LLMs to generate harmful content. Previous research constructs
attack prompts via manual or automatic methods, which have their own
limitations on construction cost and quality. To address these issues, we
propose an integrated approach that combines manual and automatic methods to
economically generate high-quality attack prompts. Specifically, considering
the impressive capabilities of newly emerged LLMs, we propose an attack
framework to instruct LLMs to mimic human-generated prompts through in-context
learning. Furthermore, we propose a defense framework that fine-tunes victim
LLMs through iterative interactions with the attack framework to enhance their
safety against red teaming attacks. Extensive experiments on different LLMs
validate the effectiveness of our proposed attack and defense frameworks.
Additionally, we release a series of attack prompts datasets named SAP with
varying sizes, facilitating the safety evaluation and enhancement of more LLMs.
Our code and dataset is available on https://github.com/Aatrox103/SAP .


http://arxiv.org/abs/2310.12665
SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models. (5%)
Boyang Zhang; Zheng Li; Ziqing Yang; Xinlei He; Michael Backes; Mario Fritz; Yang Zhang
  While advanced machine learning (ML) models are deployed in numerous
real-world applications, previous works demonstrate these models have security
and privacy vulnerabilities. Various empirical research has been done in this
field. However, most of the experiments are performed on target ML models
trained by the security researchers themselves. Due to the high computational
resource requirement for training advanced models with complex architectures,
researchers generally choose to train a few target models using relatively
simple architectures on typical experiment datasets. We argue that to
understand ML models' vulnerabilities comprehensively, experiments should be
performed on a large set of models trained with various purposes (not just the
purpose of evaluating ML attacks and defenses). To this end, we propose using
publicly available models with weights from the Internet (public models) for
evaluating attacks and defenses on ML models. We establish a database, namely
SecurityNet, containing 910 annotated image classification models. We then
analyze the effectiveness of several representative attacks/defenses, including
model stealing attacks, membership inference attacks, and backdoor detection on
these public models. Our evaluation empirically shows the performance of these
attacks/defenses can vary significantly on public models compared to
self-trained models. We share SecurityNet with the research community. and
advocate researchers to perform experiments on public models to better
demonstrate their proposed methods' effectiveness in the future.


http://arxiv.org/abs/2310.13061
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets. (1%)
Darshil Doshi; Aritra Das; Tianyu He; Andrey Gromov
  Robust generalization is a major challenge in deep learning, particularly
when the number of trainable parameters is very large. In general, it is very
difficult to know if the network has memorized a particular set of examples or
understood the underlying rule (or both). Motivated by this challenge, we study
an interpretable model where generalizing representations are understood
analytically, and are easily distinguishable from the memorizing ones. Namely,
we consider two-layer neural networks trained on modular arithmetic tasks where
($\xi \cdot 100\%$) of labels are corrupted (\emph{i.e.} some results of the
modular operations in the training set are incorrect). We show that (i) it is
possible for the network to memorize the corrupted labels \emph{and} achieve
$100\%$ generalization at the same time; (ii) the memorizing neurons can be
identified and pruned, lowering the accuracy on corrupted data and improving
the accuracy on uncorrupted data; (iii) regularization methods such as weight
decay, dropout and BatchNorm force the network to ignore the corrupted data
during optimization, and achieve $100\%$ accuracy on the uncorrupted dataset;
and (iv) the effect of these regularization methods is (``mechanistically'')
interpretable: weight decay and dropout force all the neurons to learn
generalizing representations, while BatchNorm de-amplifies the output of
memorizing neurons and amplifies the output of the generalizing ones. Finally,
we show that in the presence of regularization, the training dynamics involves
two consecutive stages: first, the network undergoes the \emph{grokking}
dynamics reaching high train \emph{and} test accuracy; second, it unlearns the
memorizing representations, where train accuracy suddenly jumps from $100\%$ to
$100 (1-\xi)\%$.


http://arxiv.org/abs/2310.13252
Detecting Shared Data Manipulation in Distributed Optimization Algorithms. (1%)
Mohannad Alkhraijah; Rachel Harris; Samuel Litchfield; David Huggins; Daniel K. Molzahn
  This paper investigates the vulnerability of the Alternating Direction Method
of Multipliers (ADMM) algorithm to shared data manipulation, with a focus on
solving optimal power flow (OPF) problems. Deliberate data manipulation may
cause the ADMM algorithm to converge to suboptimal solutions. We derive two
sufficient conditions for detecting data manipulation based on the theoretical
convergence trajectory of the ADMM algorithm. We evaluate the detection
conditions' performance on three data manipulation strategies we previously
proposed: simple, feedback, and bilevel optimization attacks. We then extend
these three data manipulation strategies to avoid detection by considering both
the detection conditions and a neural network (NN) detection model in the
attacks. We also propose an adversarial NN training framework to detect shared
data manipulation. We illustrate the performance of our data manipulation
strategy and detection framework on OPF problems. The results show that the
proposed detection conditions successfully detect most of the data manipulation
attacks. However, a bilevel optimization attack strategy that incorporates the
detection methods may avoid being detected. Countering this, our proposed
adversarial training framework detects all the instances of the bilevel
optimization attack.


http://arxiv.org/abs/2310.13191
Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models. (1%)
Jianwei Li; Qi Lei; Wei Cheng; Dongkuan Xu
  The pruning objective has recently extended beyond accuracy and sparsity to
robustness in language models. Despite this, existing methods struggle to
enhance robustness against adversarial attacks when continually increasing
model sparsity and require a retraining process. As humans step into the era of
large language models, these issues become increasingly prominent. This paper
proposes that the robustness of language models is proportional to the extent
of pre-trained knowledge they encompass. Accordingly, we introduce a
post-training pruning strategy designed to faithfully replicate the embedding
space and feature space of dense language models, aiming to conserve more
pre-trained knowledge during the pruning process. In this setup, each layer's
reconstruction error not only originates from itself but also includes
cumulative error from preceding layers, followed by an adaptive rectification.
Compared to other state-of-art baselines, our approach demonstrates a superior
balance between accuracy, sparsity, robustness, and pruning cost with BERT on
datasets SST2, IMDB, and AGNews, marking a significant stride towards robust
pruning in language models.


http://arxiv.org/abs/2310.12017
Exploring Decision-based Black-box Attacks on Face Forgery Detection. (99%)
Zhaoyu Chen; Bo Li; Kaixun Jiang; Shuang Wu; Shouhong Ding; Wenqiang Zhang
  Face forgery generation technologies generate vivid faces, which have raised
public concerns about security and privacy. Many intelligent systems, such as
electronic payment and identity verification, rely on face forgery detection.
Although face forgery detection has successfully distinguished fake faces,
recent studies have demonstrated that face forgery detectors are very
vulnerable to adversarial examples. Meanwhile, existing attacks rely on network
architectures or training datasets instead of the predicted labels, which leads
to a gap in attacking deployed applications. To narrow this gap, we first
explore the decision-based attacks on face forgery detection. However, applying
existing decision-based attacks directly suffers from perturbation
initialization failure and low image quality. First, we propose cross-task
perturbation to handle initialization failures by utilizing the high
correlation of face features on different tasks. Then, inspired by using
frequency cues by face forgery detection, we propose the frequency
decision-based attack. We add perturbations in the frequency domain and then
constrain the visual quality in the spatial domain. Finally, extensive
experiments demonstrate that our method achieves state-of-the-art attack
performance on FaceForensics++, CelebDF, and industrial APIs, with high query
efficiency and guaranteed image quality. Further, the fake faces by our method
can pass face forgery detection and face recognition, which exposes the
security problems of face forgery detectors.


http://arxiv.org/abs/2310.12431
Segment Anything Meets Universal Adversarial Perturbation. (99%)
Dongshen Han; Sheng Zheng; Chaoning Zhang
  As Segment Anything Model (SAM) becomes a popular foundation model in
computer vision, its adversarial robustness has become a concern that cannot be
ignored. This works investigates whether it is possible to attack SAM with
image-agnostic Universal Adversarial Perturbation (UAP). In other words, we
seek a single perturbation that can fool the SAM to predict invalid masks for
most (if not all) images. We demonstrate convetional image-centric attack
framework is effective for image-independent attacks but fails for universal
adversarial attack. To this end, we propose a novel perturbation-centric
framework that results in a UAP generation method based on self-supervised
contrastive learning (CL), where the UAP is set to the anchor sample and the
positive sample is augmented from the UAP. The representations of negative
samples are obtained from the image encoder in advance and saved in a memory
bank. The effectiveness of our proposed CL-based UAP generation method is
validated by both quantitative and qualitative results. On top of the ablation
study to understand various components in our proposed method, we shed light on
the roles of positive and negative samples in making the generated UAP
effective for attacking SAM.


http://arxiv.org/abs/2310.11890
IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks. (99%)
Yue Cao; Tianlin Li; Xiaofeng Cao; Ivor Tsang; Yang Liu; Qing Guo
  We introduce a novel approach to counter adversarial attacks, namely, image
resampling. Image resampling transforms a discrete image into a new one,
simulating the process of scene recapturing or rerendering as specified by a
geometrical transformation. The underlying rationale behind our idea is that
image resampling can alleviate the influence of adversarial perturbations while
preserving essential semantic information, thereby conferring an inherent
advantage in defending against adversarial attacks. To validate this concept,
we present a comprehensive study on leveraging image resampling to defend
against adversarial attacks. We have developed basic resampling methods that
employ interpolation strategies and coordinate shifting magnitudes. Our
analysis reveals that these basic methods can partially mitigate adversarial
attacks. However, they come with apparent limitations: the accuracy of clean
images noticeably decreases, while the improvement in accuracy on adversarial
examples is not substantial. We propose implicit representation-driven image
resampling (IRAD) to overcome these limitations. First, we construct an
implicit continuous representation that enables us to represent any input image
within a continuous coordinate space. Second, we introduce SampleNet, which
automatically generates pixel-wise shifts for resampling in response to
different inputs. Furthermore, we can extend our approach to the
state-of-the-art diffusion-based method, accelerating it with fewer time steps
while preserving its defense capability. Extensive experiments demonstrate that
our method significantly enhances the adversarial robustness of diverse deep
models against various attacks while maintaining high accuracy on clean images.


http://arxiv.org/abs/2310.11850
Revisiting Transferable Adversarial Image Examples: Attack Categorization, Evaluation Guidelines, and New Insights. (99%)
Zhengyu Zhao; Hanwei Zhang; Renjue Li; Ronan Sicre; Laurent Amsaleg; Michael Backes; Qi Li; Chao Shen
  Transferable adversarial examples raise critical security concerns in
real-world, black-box attack scenarios. However, in this work, we identify two
main problems in common evaluation practices: (1) For attack transferability,
lack of systematic, one-to-one attack comparison and fair hyperparameter
settings. (2) For attack stealthiness, simply no comparisons. To address these
problems, we establish new evaluation guidelines by (1) proposing a novel
attack categorization strategy and conducting systematic and fair
intra-category analyses on transferability, and (2) considering diverse
imperceptibility metrics and finer-grained stealthiness characteristics from
the perspective of attack traceback. To this end, we provide the first
large-scale evaluation of transferable adversarial examples on ImageNet,
involving 23 representative attacks against 9 representative defenses. Our
evaluation leads to a number of new insights, including consensus-challenging
ones: (1) Under a fair attack hyperparameter setting, one early attack method,
DI, actually outperforms all the follow-up methods. (2) A state-of-the-art
defense, DiffPure, actually gives a false sense of (white-box) security since
it is indeed largely bypassed by our (black-box) transferable attacks. (3) Even
when all attacks are bounded by the same $L_p$ norm, they lead to dramatically
different stealthiness performance, which negatively correlates with their
transferability performance. Overall, our work demonstrates that existing
problematic evaluations have indeed caused misleading conclusions and missing
points, and as a result, hindered the assessment of the actual progress in this
field.


http://arxiv.org/abs/2310.13019
Tailoring Adversarial Attacks on Deep Neural Networks for Targeted Class Manipulation Using DeepFool Algorithm. (99%)
S. M. Fazle Rabby Labib; Joyanta Jyoti Mondal; Meem Arafat Manab
  Deep neural networks (DNNs) have significantly advanced various domains, but
their vulnerability to adversarial attacks poses serious concerns.
Understanding these vulnerabilities and developing effective defense mechanisms
is crucial. DeepFool, an algorithm proposed by Moosavi-Dezfooli et al. (2016),
finds minimal perturbations to misclassify input images. However, DeepFool
lacks a targeted approach, making it less effective in specific attack
scenarios. Also, in previous related works, researchers primarily focus on
success, not considering how much an image is getting distorted; the integrity
of the image quality, and the confidence level to misclassifying. So, in this
paper, we propose Enhanced Targeted DeepFool, an augmented version of DeepFool
that allows targeting specific classes for misclassification and also introduce
a minimum confidence score requirement hyperparameter to enhance flexibility.
Our experiments demonstrate the effectiveness and efficiency of the proposed
method across different deep neural network architectures while preserving
image integrity as much and perturbation rate as less as possible. By using our
approach, the behavior of models can be manipulated arbitrarily using the
perturbed images, as we can specify both the target class and the associated
confidence score, unlike other DeepFool-derivative works, such as Targeted
DeepFool by Gajjar et al. (2022). Results show that one of the deep
convolutional neural network architectures, AlexNet, and one of the
state-of-the-art model Vision Transformer exhibit high robustness to getting
fooled. This approach can have larger implication, as our tuning of confidence
level can expose the robustness of image recognition models. Our code will be
made public upon acceptance of the paper.


http://arxiv.org/abs/2310.11901
Malicious Agent Detection for Robust Multi-Agent Collaborative Perception. (87%)
Yangheng Zhao; Zhen Xiang; Sheng Yin; Xianghe Pang; Siheng Chen; Yanfeng Wang
  Recently, multi-agent collaborative (MAC) perception has been proposed and
outperformed the traditional single-agent perception in many applications, such
as autonomous driving. However, MAC perception is more vulnerable to
adversarial attacks than single-agent perception due to the information
exchange. The attacker can easily degrade the performance of a victim agent by
sending harmful information from a malicious agent nearby. In this paper, we
extend adversarial attacks to an important perception task -- MAC object
detection, where generic defenses such as adversarial training are no longer
effective against these attacks. More importantly, we propose Malicious Agent
Detection (MADE), a reactive defense specific to MAC perception that can be
deployed by each agent to accurately detect and then remove any potential
malicious agent in its local collaboration network. In particular, MADE
inspects each agent in the network independently using a semi-supervised
anomaly detector based on a double-hypothesis test with the Benjamini-Hochberg
procedure to control the false positive rate of the inference. For the two
hypothesis tests, we propose a match loss statistic and a collaborative
reconstruction loss statistic, respectively, both based on the consistency
between the agent to be inspected and the ego agent where our detector is
deployed. We conduct comprehensive evaluations on a benchmark 3D dataset
V2X-sim and a real-road dataset DAIR-V2X and show that with the protection of
MADE, the drops in the average precision compared with the best-case "oracle"
defender against our attack are merely 1.28% and 0.34%, respectively, much
lower than 8.92% and 10.00% for adversarial training, respectively.


http://arxiv.org/abs/2310.12063
Black-Box Training Data Identification in GANs via Detector Networks. (82%)
Lukman Olagoke; Salil Vadhan; Seth Neel
  Since their inception Generative Adversarial Networks (GANs) have been
popular generative models across images, audio, video, and tabular data. In
this paper we study whether given access to a trained GAN, as well as fresh
samples from the underlying distribution, if it is possible for an attacker to
efficiently identify if a given point is a member of the GAN's training data.
This is of interest for both reasons related to copyright, where a user may
want to determine if their copyrighted data has been used to train a GAN, and
in the study of data privacy, where the ability to detect training set
membership is known as a membership inference attack. Unlike the majority of
prior work this paper investigates the privacy implications of using GANs in
black-box settings, where the attack only has access to samples from the
generator, rather than access to the discriminator as well. We introduce a
suite of membership inference attacks against GANs in the black-box setting and
evaluate our attacks on image GANs trained on the CIFAR10 dataset and tabular
GANs trained on genomic data. Our most successful attack, called The Detector,
involve training a second network to score samples based on their likelihood of
being generated by the GAN, as opposed to a fresh sample from the distribution.
We prove under a simple model of the generator that the detector is an
approximately optimal membership inference attack. Across a wide range of
tabular and image datasets, attacks, and GAN architectures, we find that
adversaries can orchestrate non-trivial privacy attacks when provided with
access to samples from the generator. At the same time, the attack success
achievable against GANs still appears to be lower compared to other generative
and discriminative models; this leaves the intriguing open question of whether
GANs are in fact more private, or if it is a matter of developing stronger
attacks.


http://arxiv.org/abs/2310.11789
Adversarial Training for Physics-Informed Neural Networks. (81%)
Yao Li; Shengzhu Shi; Zhichang Guo; Boying Wu
  Physics-informed neural networks have shown great promise in solving partial
differential equations. However, due to insufficient robustness, vanilla PINNs
often face challenges when solving complex PDEs, especially those involving
multi-scale behaviors or solutions with sharp or oscillatory characteristics.
To address these issues, based on the projected gradient descent adversarial
attack, we proposed an adversarial training strategy for PINNs termed by
AT-PINNs. AT-PINNs enhance the robustness of PINNs by fine-tuning the model
with adversarial samples, which can accurately identify model failure locations
and drive the model to focus on those regions during training. AT-PINNs can
also perform inference with temporal causality by selecting the initial
collocation points around temporal initial values. We implement AT-PINNs to the
elliptic equation with multi-scale coefficients, Poisson equation with
multi-peak solutions, Burgers equation with sharp solutions and the Allen-Cahn
equation. The results demonstrate that AT-PINNs can effectively locate and
reduce failure regions. Moreover, AT-PINNs are suitable for solving complex
PDEs, since locating failure regions through adversarial attacks is independent
of the size of failure regions or the complexity of the distribution.


http://arxiv.org/abs/2310.12243
REVAMP: Automated Simulations of Adversarial Attacks on Arbitrary Objects in Realistic Scenes. (80%)
Matthew Hull; Zijie J. Wang; Duen Horng Chau
  Deep Learning models, such as those used in an autonomous vehicle are
vulnerable to adversarial attacks where an attacker could place an adversarial
object in the environment, leading to mis-classification. Generating these
adversarial objects in the digital space has been extensively studied, however
successfully transferring these attacks from the digital realm to the physical
realm has proven challenging when controlling for real-world environmental
factors. In response to these limitations, we introduce REVAMP, an easy-to-use
Python library that is the first-of-its-kind tool for creating attack scenarios
with arbitrary objects and simulating realistic environmental factors,
lighting, reflection, and refraction. REVAMP enables researchers and
practitioners to swiftly explore various scenarios within the digital realm by
offering a wide range of configurable options for designing experiments and
using differentiable rendering to reproduce physically plausible adversarial
objects. We will demonstrate and invite the audience to try REVAMP to produce
an adversarial texture on a chosen object while having control over various
scene parameters. The audience will choose a scene, an object to attack, the
desired attack class, and the number of camera positions to use. Then, in real
time, we show how this altered texture causes the chosen object to be
mis-classified, showcasing the potential of REVAMP in real-world scenarios.
REVAMP is open-source and available at https://github.com/poloclub/revamp.


http://arxiv.org/abs/2310.11970
Quantifying Privacy Risks of Prompts in Visual Prompt Learning. (76%)
Yixin Wu; Rui Wen; Michael Backes; Pascal Berrang; Mathias Humbert; Yun Shen; Yang Zhang
  Large-scale pre-trained models are increasingly adapted to downstream tasks
through a new paradigm called prompt learning. In contrast to fine-tuning,
prompt learning does not update the pre-trained model's parameters. Instead, it
only learns an input perturbation, namely prompt, to be added to the downstream
task data for predictions. Given the fast development of prompt learning, a
well-generalized prompt inevitably becomes a valuable asset as significant
effort and proprietary data are used to create it. This naturally raises the
question of whether a prompt may leak the proprietary information of its
training data. In this paper, we perform the first comprehensive privacy
assessment of prompts learned by visual prompt learning through the lens of
property inference and membership inference attacks. Our empirical evaluation
shows that the prompts are vulnerable to both attacks. We also demonstrate that
the adversary can mount a successful property inference attack with limited
cost. Moreover, we show that membership inference attacks against prompts can
be successful with relaxed adversarial assumptions. We further make some
initial investigations on the defenses and observe that our method can mitigate
the membership inference attacks with a decent utility-defense trade-off but
fails to defend against property inference attacks. We hope our results can
shed light on the privacy risks of the popular prompt learning paradigm. To
facilitate the research in this direction, we will share our code and models
with the community.


http://arxiv.org/abs/2310.11868
To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now. (47%)
Yimeng Zhang; Jinghan Jia; Xin Chen; Aochuan Chen; Yihua Zhang; Jiancheng Liu; Ke Ding; Sijia Liu
  The recent advances in diffusion models (DMs) have revolutionized the
generation of complex and diverse images. However, these models also introduce
potential safety hazards, such as the production of harmful content and
infringement of data copyrights. Although there have been efforts to create
safety-driven unlearning methods to counteract these challenges, doubts remain
about their capabilities. To bridge this uncertainty, we propose an evaluation
framework built upon adversarial attacks (also referred to as adversarial
prompts), in order to discern the trustworthiness of these safety-driven
unlearned DMs. Specifically, our research explores the (worst-case) robustness
of unlearned DMs in eradicating unwanted concepts, styles, and objects,
assessed by the generation of adversarial prompts. We develop a novel
adversarial learning approach called UnlearnDiff that leverages the inherent
classification capabilities of DMs to streamline the generation of adversarial
prompts, making it as simple for DMs as it is for image classification attacks.
This technique streamlines the creation of adversarial prompts, making the
process as intuitive for generative modeling as it is for image classification
assaults. Through comprehensive benchmarking, we assess the unlearning
robustness of five prevalent unlearned DMs across multiple tasks. Our results
underscore the effectiveness and efficiency of UnlearnDiff when compared to
state-of-the-art adversarial prompting methods. Codes are available at
https://github.com/OPTML-Group/Diffusion-MU-Attack. WARNING: This paper
contains model outputs that may be offensive in nature.


http://arxiv.org/abs/2310.12432
CAT: Closed-loop Adversarial Training for Safe End-to-End Driving. (2%)
Linrui Zhang; Zhenghao Peng; Quanyi Li; Bolei Zhou
  Driving safety is a top priority for autonomous vehicles. Orthogonal to prior
work handling accident-prone traffic events by algorithm designs at the policy
level, we investigate a Closed-loop Adversarial Training (CAT) framework for
safe end-to-end driving in this paper through the lens of environment
augmentation. CAT aims to continuously improve the safety of driving agents by
training the agent on safety-critical scenarios that are dynamically generated
over time. A novel resampling technique is developed to turn log-replay
real-world driving scenarios into safety-critical ones via probabilistic
factorization, where the adversarial traffic generation is modeled as the
multiplication of standard motion prediction sub-problems. Consequently, CAT
can launch more efficient physical attacks compared to existing safety-critical
scenario generation methods and yields a significantly less computational cost
in the iterative learning pipeline. We incorporate CAT into the MetaDrive
simulator and validate our approach on hundreds of driving scenarios imported
from real-world driving datasets. Experimental results demonstrate that CAT can
effectively generate adversarial scenarios countering the agent being trained.
After training, the agent can achieve superior driving safety in both
log-replay and safety-critical traffic scenarios on the held-out test set. Code
and data are available at https://metadriverse.github.io/cat.


http://arxiv.org/abs/2310.12214
PrivInfer: Privacy-Preserving Inference for Black-box Large Language Model. (1%)
Meng Tong; Kejiang Chen; Yuang Qi; Jie Zhang; Weiming Zhang; Nenghai Yu
  Large language models (LLMs), such as ChatGPT, have simplified text
generation tasks, yet their inherent privacy risks are increasingly garnering
attention. Existing solutions for privacy-preserving inference face significant
challenges in practical deployment and implementation. In this paper, we
propose PrivInfer, the first practical framework for privacy-preserving
inference. It comprises two modules specifically designed for black-box LLMs in
text generation. The perturbation module, employing differential privacy,
generates perturbed prompts, thus enabling privacy-preserving inference with
black-box LLMs. The restoration module extracts coherent and meaningful
responses from obtained perturbed results, thus ensuring the accomplishment of
the text generation tasks. Additionally, to enhance privacy and utility
further, we develop RANTEXT, a novel differential privacy mechanism integrated
into the perturbation module of PrivInfer. This mechanism is specifically
tailored for LLMs and utilizes random adjacency in text perturbations.
Experimental results indicate that PrivInfer is comparable to GPT-4 in text
generation quality, and RANTEXT outperforms the current leading scheme in
privacy protection, even under its adaptive attack, our proposed GPT inference
attack.


http://arxiv.org/abs/2310.11597
The Efficacy of Transformer-based Adversarial Attacks in Security Domains. (99%)
Kunyang Li; Kyle Domico; Jean-Charles Noirot Ferrand; Patrick McDaniel
  Today, the security of many domains rely on the use of Machine Learning to
detect threats, identify vulnerabilities, and safeguard systems from attacks.
Recently, transformer architectures have improved the state-of-the-art
performance on a wide range of tasks such as malware detection and network
intrusion detection. But, before abandoning current approaches to transformers,
it is crucial to understand their properties and implications on cybersecurity
applications. In this paper, we evaluate the robustness of transformers to
adversarial samples for system defenders (i.e., resiliency to adversarial
perturbations generated on different types of architectures) and their
adversarial strength for system attackers (i.e., transferability of adversarial
samples generated by transformers to other target models). To that effect, we
first fine-tune a set of pre-trained transformer, Convolutional Neural Network
(CNN), and hybrid (an ensemble of transformer and CNN) models to solve
different downstream image-based tasks. Then, we use an attack algorithm to
craft 19,367 adversarial examples on each model for each task. The
transferability of these adversarial examples is measured by evaluating each
set on other models to determine which models offer more adversarial strength,
and consequently, more robustness against these attacks. We find that the
adversarial examples crafted on transformers offer the highest transferability
rate (i.e., 25.7% higher than the average) onto other models. Similarly,
adversarial examples crafted on other models have the lowest rate of
transferability (i.e., 56.7% lower than the average) onto transformers. Our
work emphasizes the importance of studying transformer architectures for
attacking and defending models in security domains, and suggests using them as
the primary architecture in transfer attack settings.


http://arxiv.org/abs/2310.11594
Adversarial Robustness Unhardening via Backdoor Attacks in Federated Learning. (93%)
Taejin Kim; Jiarui Li; Shubhranshu Singh; Nikhil Madaan; Carlee Joe-Wong
  In today's data-driven landscape, the delicate equilibrium between
safeguarding user privacy and unleashing data potential stands as a paramount
concern. Federated learning, which enables collaborative model training without
necessitating data sharing, has emerged as a privacy-centric solution. This
decentralized approach brings forth security challenges, notably poisoning and
backdoor attacks where malicious entities inject corrupted data. Our research,
initially spurred by test-time evasion attacks, investigates the intersection
of adversarial training and backdoor attacks within federated learning,
introducing Adversarial Robustness Unhardening (ARU). ARU is employed by a
subset of adversaries to intentionally undermine model robustness during
decentralized training, rendering models susceptible to a broader range of
evasion attacks. We present extensive empirical experiments evaluating ARU's
impact on adversarial training and existing robust aggregation defenses against
poisoning and backdoor attacks. Our findings inform strategies for enhancing
ARU to counter current defensive measures and highlight the limitations of
existing defenses, offering insights into bolstering defenses against ARU.


http://arxiv.org/abs/2310.11595
WaveAttack: Asymmetric Frequency Obfuscation-based Backdoor Attacks Against Deep Neural Networks. (15%)
Jun Xia; Zhihao Yue; Yingbo Zhou; Zhiwei Ling; Xian Wei; Mingsong Chen
  Due to the popularity of Artificial Intelligence (AI) technology, numerous
backdoor attacks are designed by adversaries to mislead deep neural network
predictions by manipulating training samples and training processes. Although
backdoor attacks are effective in various real scenarios, they still suffer
from the problems of both low fidelity of poisoned samples and non-negligible
transfer in latent space, which make them easily detectable by existing
backdoor detection algorithms. To overcome the weakness, this paper proposes a
novel frequency-based backdoor attack method named WaveAttack, which obtains
image high-frequency features through Discrete Wavelet Transform (DWT) to
generate backdoor triggers. Furthermore, we introduce an asymmetric frequency
obfuscation method, which can add an adaptive residual in the training and
inference stage to improve the impact of triggers and further enhance the
effectiveness of WaveAttack. Comprehensive experimental results show that
WaveAttack not only achieves higher stealthiness and effectiveness, but also
outperforms state-of-the-art (SOTA) backdoor attack methods in the fidelity of
images by up to 28.27\% improvement in PSNR, 1.61\% improvement in SSIM, and
70.59\% reduction in IS.


http://arxiv.org/abs/2310.11105
Generalizability of CNN Architectures for Face Morph Presentation Attack. (1%)
Sherko R. HmaSalah; Aras Asaad
  Automatic border control systems are wide spread in modern airports
worldwide. Morphing attacks on face biometrics is a serious threat that
undermines the security and reliability of face recognition systems deployed in
airports and border controls. Therefore, developing a robust Machine Learning
(ML) system is necessary to prevent criminals crossing borders with fake
identifications especially since it has been shown that security officers
cannot detect morphs better than machines. In this study, we investigate the
generalization power of Convolutional Neural Network (CNN) architectures
against morphing attacks. The investigation utilizes 5 distinct CNNs namely
ShuffleNet, DenseNet201, VGG16, EffecientNet-B0 and InceptionResNet-v2. Each
CNN architecture represents a well-known family of CNN models in terms of
number of parameters, architectural design and performance across various
computer vision applications. To ensure robust evaluation, we employ 4
different datasets (Utrecht, London, Defacto and KurdFace) that contain a
diverse range of digital face images which cover variations in ethnicity,
gender, age, lighting condition and camera setting. One of the fundamental
concepts of ML system design is the ability to generalize effectively to
previously unseen data, hence not only we evaluate the performance of CNN
models within individual datasets but also explore their performance across
combined datasets and investigating each dataset in testing phase only.
Experimental results on more than 8 thousand images (genuine and morph) from
the 4 datasets show that InceptionResNet-v2 generalizes better to unseen data
and outperforms the other 4 CNN models.


http://arxiv.org/abs/2310.10844
Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks. (98%)
Erfan Shayegani; Md Abdullah Al Mamun; Yu Fu; Pedram Zaree; Yue Dong; Nael Abu-Ghazaleh
  Large Language Models (LLMs) are swiftly advancing in architecture and
capability, and as they integrate more deeply into complex systems, the urgency
to scrutinize their security properties grows. This paper surveys research in
the emerging interdisciplinary field of adversarial attacks on LLMs, a subfield
of trustworthy ML, combining the perspectives of Natural Language Processing
and Security. Prior work has shown that even safety-aligned LLMs (via
instruction tuning and reinforcement learning through human feedback) can be
susceptible to adversarial attacks, which exploit weaknesses and mislead AI
systems, as evidenced by the prevalence of `jailbreak' attacks on models like
ChatGPT and Bard. In this survey, we first provide an overview of large
language models, describe their safety alignment, and categorize existing
research based on various learning structures: textual-only attacks,
multi-modal attacks, and additional attack methods specifically targeting
complex systems, such as federated learning or multi-agent systems. We also
offer comprehensive remarks on works that focus on the fundamental sources of
vulnerabilities and potential defenses. To make this field more accessible to
newcomers, we present a systematic review of existing works, a structured
typology of adversarial attack concepts, and additional resources, including
slides for presentations on related topics at the 62nd Annual Meeting of the
Association for Computational Linguistics (ACL'24).


http://arxiv.org/abs/2310.10807
Regularization properties of adversarially-trained linear regression. (92%)
Antônio H. Ribeiro; Dave Zachariah; Francis Bach; Thomas B. Schön
  State-of-the-art machine learning models can be vulnerable to very small
input perturbations that are adversarially constructed. Adversarial training is
an effective approach to defend against it. Formulated as a min-max problem, it
searches for the best solution when the training data were corrupted by the
worst-case attacks. Linear models are among the simple models where
vulnerabilities can be observed and are the focus of our study. In this case,
adversarial training leads to a convex optimization problem which can be
formulated as the minimization of a finite sum. We provide a comparative
analysis between the solution of adversarial training in linear regression and
other regularization methods. Our main findings are that: (A) Adversarial
training yields the minimum-norm interpolating solution in the
overparameterized regime (more parameters than data), as long as the maximum
disturbance radius is smaller than a threshold. And, conversely, the
minimum-norm interpolator is the solution to adversarial training with a given
radius. (B) Adversarial training can be equivalent to parameter shrinking
methods (ridge regression and Lasso). This happens in the underparametrized
region, for an appropriate choice of adversarial radius and zero-mean
symmetrically distributed covariates. (C) For $\ell_\infty$-adversarial
training -- as in square-root Lasso -- the choice of adversarial radius for
optimal bounds does not depend on the additive noise variance. We confirm our
theoretical findings with numerical examples.


http://arxiv.org/abs/2310.10744
Fast Adversarial Label-Flipping Attack on Tabular Data. (84%)
Xinglong Chang; Gillian Dobbie; Jörg Wicker
  Machine learning models are increasingly used in fields that require high
reliability such as cybersecurity. However, these models remain vulnerable to
various attacks, among which the adversarial label-flipping attack poses
significant threats. In label-flipping attacks, the adversary maliciously flips
a portion of training labels to compromise the machine learning model. This
paper raises significant concerns as these attacks can camouflage a highly
skewed dataset as an easily solvable classification problem, often misleading
machine learning practitioners into lower defenses and miscalculations of
potential risks. This concern amplifies in tabular data settings, where
identifying true labels requires expertise, allowing malicious label-flipping
attacks to easily slip under the radar. To demonstrate this risk is inherited
in the adversary's objective, we propose FALFA (Fast Adversarial Label-Flipping
Attack), a novel efficient attack for crafting adversarial labels. FALFA is
based on transforming the adversary's objective and employs linear programming
to reduce computational complexity. Using ten real-world tabular datasets, we
demonstrate FALFA's superior attack potential, highlighting the need for robust
defenses against such threats.


http://arxiv.org/abs/2310.10126
A Non-monotonic Smooth Activation Function. (83%)
Koushik Biswas; Meghana Karri; Ulaş Bağcı
  Activation functions are crucial in deep learning models since they introduce
non-linearity into the networks, allowing them to learn from errors and make
adjustments, which is essential for learning complex patterns. The essential
purpose of activation functions is to transform unprocessed input signals into
significant output activations, promoting information transmission throughout
the neural network. In this study, we propose a new activation function called
Sqish, which is a non-monotonic and smooth function and an alternative to
existing ones. We showed its superiority in classification, object detection,
segmentation tasks, and adversarial robustness experiments. We got an 8.21%
improvement over ReLU on the CIFAR100 dataset with the ShuffleNet V2 model in
the FGSM adversarial attack. We also got a 5.87% improvement over ReLU on image
classification on the CIFAR100 dataset with the ShuffleNet V2 model.


http://arxiv.org/abs/2310.10610
Quantifying Assistive Robustness Via the Natural-Adversarial Frontier. (68%)
Jerry Zhi-Yang He; Zackory Erickson; Daniel S. Brown; Anca D. Dragan
  Our ultimate goal is to build robust policies for robots that assist people.
What makes this hard is that people can behave unexpectedly at test time,
potentially interacting with the robot outside its training distribution and
leading to failures. Even just measuring robustness is a challenge. Adversarial
perturbations are the default, but they can paint the wrong picture: they can
correspond to human motions that are unlikely to occur during natural
interactions with people. A robot policy might fail under small adversarial
perturbations but work under large natural perturbations. We propose that
capturing robustness in these interactive settings requires constructing and
analyzing the entire natural-adversarial frontier: the Pareto-frontier of human
policies that are the best trade-offs between naturalness and low robot
performance. We introduce RIGID, a method for constructing this frontier by
training adversarial human policies that trade off between minimizing robot
reward and acting human-like (as measured by a discriminator). On an Assistive
Gym task, we use RIGID to analyze the performance of standard collaborative
Reinforcement Learning, as well as the performance of existing methods meant to
increase robustness. We also compare the frontier RIGID identifies with the
failures identified in expert adversarial interaction, and with
naturally-occurring failures during user interaction. Overall, we find evidence
that RIGID can provide a meaningful measure of robustness predictive of
deployment performance, and uncover failure cases in human-robot interaction
that are difficult to find manually. https://ood-human.github.io.


http://arxiv.org/abs/2310.10124
A Comprehensive Study of Privacy Risks in Curriculum Learning. (67%)
Joann Qiongna Chen; Xinlei He; Zheng Li; Yang Zhang; Zhou Li
  Training a machine learning model with data following a meaningful order,
i.e., from easy to hard, has been proven to be effective in accelerating the
training process and achieving better model performance. The key enabling
technique is curriculum learning (CL), which has seen great success and has
been deployed in areas like image and text classification. Yet, how CL affects
the privacy of machine learning is unclear. Given that CL changes the way a
model memorizes the training data, its influence on data privacy needs to be
thoroughly evaluated. To fill this knowledge gap, we perform the first study
and leverage membership inference attack (MIA) and attribute inference attack
(AIA) as two vectors to quantify the privacy leakage caused by CL.
  Our evaluation of nine real-world datasets with attack methods (NN-based,
metric-based, label-only MIA, and NN-based AIA) revealed new insights about CL.
First, MIA becomes slightly more effective when CL is applied, but the impact
is much more prominent to a subset of training samples ranked as difficult.
Second, a model trained under CL is less vulnerable under AIA, compared to MIA.
Third, the existing defense techniques like DP-SGD, MemGuard, and MixupMMD are
still effective under CL, though DP-SGD has a significant impact on target
model accuracy. Finally, based on our insights into CL, we propose a new MIA,
termed Diff-Cali, which exploits the difficulty scores for result calibration
and is demonstrated to be effective against all CL methods and the normal
training method. With this study, we hope to draw the community's attention to
the unintended privacy risks of emerging machine-learning techniques and
develop new attack benchmarks and defense solutions.


http://arxiv.org/abs/2310.10427
DANAA: Towards transferable attacks with double adversarial neuron attribution. (26%)
Zhibo Jin; Zhiyu Zhu; Xinyi Wang; Jiayu Zhang; Jun Shen; Huaming Chen
  While deep neural networks have excellent results in many fields, they are
susceptible to interference from attacking samples resulting in erroneous
judgments. Feature-level attacks are one of the effective attack types, which
targets the learnt features in the hidden layers to improve its transferability
across different models. Yet it is observed that the transferability has been
largely impacted by the neuron importance estimation results. In this paper, a
double adversarial neuron attribution attack method, termed `DANAA', is
proposed to obtain more accurate feature importance estimation. In our method,
the model outputs are attributed to the middle layer based on an adversarial
non-linear path. The goal is to measure the weight of individual neurons and
retain the features that are more important towards transferability. We have
conducted extensive experiments on the benchmark datasets to demonstrate the
state-of-the-art performance of our method. Our code is available at:
https://github.com/Davidjinzb/DANAA


http://arxiv.org/abs/2310.10780
Demystifying Poisoning Backdoor Attacks from a Statistical Perspective. (9%)
Ganghua Wang; Xun Xian; Jayanth Srinivasa; Ashish Kundu; Xuan Bi; Mingyi Hong; Jie Ding
  The growing dependence on machine learning in real-world applications
emphasizes the importance of understanding and ensuring its safety. Backdoor
attacks pose a significant security risk due to their stealthy nature and
potentially serious consequences. Such attacks involve embedding triggers
within a learning model with the intention of causing malicious behavior when
an active trigger is present while maintaining regular functionality without
it. This paper evaluates the effectiveness of any backdoor attack incorporating
a constant trigger, by establishing tight lower and upper boundaries for the
performance of the compromised model on both clean and backdoor test data. The
developed theory answers a series of fundamental but previously underexplored
problems, including (1) what are the determining factors for a backdoor
attack's success, (2) what is the direction of the most effective backdoor
attack, and (3) when will a human-imperceptible trigger succeed. Our derived
understanding applies to both discriminative and generative models. We also
demonstrate the theory by conducting experiments using benchmark datasets and
state-of-the-art backdoor attack scenarios.


http://arxiv.org/abs/2310.10077
Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks. (4%)
Shuyu Jiang; Xingshu Chen; Rui Tang
  Recently, Large language models (LLMs) with powerful general capabilities
have been increasingly integrated into various Web applications, while
undergoing alignment training to ensure that the generated content aligns with
user intent and ethics. Unfortunately, they remain the risk of generating
harmful content like hate speech and criminal activities in practical
applications. Current approaches primarily rely on detecting, collecting, and
training against harmful prompts to prevent such risks. However, they typically
focused on the "superficial" harmful prompts with a solitary intent, ignoring
composite attack instructions with multiple intentions that can easily elicit
harmful content in real-world scenarios. In this paper, we introduce an
innovative technique for obfuscating harmful instructions: Compositional
Instruction Attacks (CIA), which refers to attacking by combination and
encapsulation of multiple instructions. CIA hides harmful prompts within
instructions of harmless intentions, making it impossible for the model to
identify underlying malicious intentions. Furthermore, we implement two
transformation methods, known as T-CIA and W-CIA, to automatically disguise
harmful instructions as talking or writing tasks, making them appear harmless
to LLMs. We evaluated CIA on GPT-4, ChatGPT, and ChatGLM2 with two safety
assessment datasets and two harmful prompt datasets. It achieves an attack
success rate of 95%+ on safety assessment datasets, and 83%+ for GPT-4, 91%+
for ChatGPT (gpt-3.5-turbo backed) and ChatGLM2-6B on harmful prompt datasets.
Our approach reveals the vulnerability of LLMs to such compositional
instruction attacks that harbor underlying harmful intentions, contributing
significantly to LLM security development. Warning: this paper may contain
offensive or upsetting content!


http://arxiv.org/abs/2310.10483
Passive Inference Attacks on Split Learning via Adversarial Regularization. (3%)
Xiaochen Zhu; Xinjian Luo; Yuncheng Wu; Yangfan Jiang; Xiaokui Xiao; Beng Chin Ooi
  Split Learning (SL) has emerged as a practical and efficient alternative to
traditional federated learning. While previous attempts to attack SL have often
relied on overly strong assumptions or targeted easily exploitable models, we
seek to develop more practical attacks. We introduce SDAR, a novel attack
framework against SL with an honest-but-curious server. SDAR leverages
auxiliary data and adversarial regularization to learn a decodable simulator of
the client's private model, which can effectively infer the client's private
features under the vanilla SL, and both features and labels under the U-shaped
SL. We perform extensive experiments in both configurations to validate the
effectiveness of our proposed attacks. Notably, in challenging but practical
scenarios where existing passive attacks struggle to reconstruct the client's
private data effectively, SDAR consistently achieves attack performance
comparable to active attacks. On CIFAR-10, at the deep split level of 7, SDAR
achieves private feature reconstruction with less than 0.025 mean squared error
in both the vanilla and the U-shaped SL, and attains a label inference accuracy
of over 98% in the U-shaped setting, while existing attacks fail to produce
non-trivial results.


http://arxiv.org/abs/2310.10810
Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms. (3%)
Alexander Bukharin; Yan Li; Yue Yu; Qingru Zhang; Zhehui Chen; Simiao Zuo; Chao Zhang; Songan Zhang; Tuo Zhao
  Multi-Agent Reinforcement Learning (MARL) has shown promising results across
several domains. Despite this promise, MARL policies often lack robustness and
are therefore sensitive to small changes in their environment. This presents a
serious concern for the real world deployment of MARL algorithms, where the
testing environment may slightly differ from the training environment. In this
work we show that we can gain robustness by controlling a policy's Lipschitz
constant, and under mild conditions, establish the existence of a Lipschitz and
close-to-optimal policy. Based on these insights, we propose a new robust MARL
framework, ERNIE, that promotes the Lipschitz continuity of the policies with
respect to the state observations and actions by adversarial regularization.
The ERNIE framework provides robustness against noisy observations, changing
transition dynamics, and malicious actions of agents. However, ERNIE's
adversarial regularization may introduce some training instability. To reduce
this instability, we reformulate adversarial regularization as a Stackelberg
game. We demonstrate the effectiveness of the proposed framework with extensive
experiments in traffic light control and particle environments. In addition, we
extend ERNIE to mean-field MARL with a formulation based on distributionally
robust optimization that outperforms its non-robust counterpart and is of
independent interest. Our code is available at
https://github.com/abukharin3/ERNIE.


http://arxiv.org/abs/2310.10490
On the Transferability of Learning Models for Semantic Segmentation for Remote Sensing Data. (2%)
Rongjun Qin; Guixiang Zhang; Yang Tang
  Recent deep learning-based methods outperform traditional learning methods on
remote sensing (RS) semantic segmentation/classification tasks. However, they
require large training datasets and are generally known for lack of
transferability due to the highly disparate RS image content across different
geographical regions. Yet, there is no comprehensive analysis of their
transferability, i.e., to which extent a model trained on a source domain can
be readily applicable to a target domain. Therefore, in this paper, we aim to
investigate the raw transferability of traditional and deep learning (DL)
models, as well as the effectiveness of domain adaptation (DA) approaches in
enhancing the transferability of the DL models (adapted transferability). By
utilizing four highly diverse RS datasets, we train six models with and without
three DA approaches to analyze their transferability between these datasets
quantitatively. Furthermore, we developed a straightforward method to quantify
the transferability of a model using the spectral indices as a medium and have
demonstrated its effectiveness in evaluating the model transferability at the
target domain when the labels are unavailable. Our experiments yield several
generally important yet not well-reported observations regarding the raw and
adapted transferability. Moreover, our proposed label-free transferability
assessment method is validated to be better than posterior model confidence.
The findings can guide the future development of generalized RS learning
models. The trained models are released under this link:
https://github.com/GDAOSU/Transferability-Remote-Sensing


http://arxiv.org/abs/2310.10090
Orthogonal Uncertainty Representation of Data Manifold for Robust Long-Tailed Learning. (1%)
Yanbiao Ma; Licheng Jiao; Fang Liu; Shuyuan Yang; Xu Liu; Lingling Li
  In scenarios with long-tailed distributions, the model's ability to identify
tail classes is limited due to the under-representation of tail samples. Class
rebalancing, information augmentation, and other techniques have been proposed
to facilitate models to learn the potential distribution of tail classes. The
disadvantage is that these methods generally pursue models with balanced class
accuracy on the data manifold, while ignoring the ability of the model to
resist interference. By constructing noisy data manifold, we found that the
robustness of models trained on unbalanced data has a long-tail phenomenon.
That is, even if the class accuracy is balanced on the data domain, it still
has bias on the noisy data manifold. However, existing methods cannot
effectively mitigate the above phenomenon, which makes the model vulnerable in
long-tailed scenarios. In this work, we propose an Orthogonal Uncertainty
Representation (OUR) of feature embedding and an end-to-end training strategy
to improve the long-tail phenomenon of model robustness. As a general
enhancement tool, OUR has excellent compatibility with other methods and does
not require additional data generation, ensuring fast and efficient training.
Comprehensive evaluations on long-tailed datasets show that our method
significantly improves the long-tail phenomenon of robustness, bringing
consistent performance gains to other long-tailed learning methods.


http://arxiv.org/abs/2310.10865
Will the Prince Get True Love's Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts. (1%)
Christina Chance; Da Yin; Dakuo Wang; Kai-Wei Chang
  Recent studies show that traditional fairytales are rife with harmful gender
biases. To help mitigate these gender biases in fairytales, this work aims to
assess learned biases of language models by evaluating their robustness against
gender perturbations. Specifically, we focus on Question Answering (QA) tasks
in fairytales. Using counterfactual data augmentation to the FairytaleQA
dataset, we evaluate model robustness against swapped gender character
information, and then mitigate learned biases by introducing counterfactual
gender stereotypes during training time. We additionally introduce a novel
approach that utilizes the massive vocabulary of language models to support
text genres beyond fairytales. Our experimental results suggest that models are
sensitive to gender perturbations, with significant performance drops compared
to the original testing set. However, when first fine-tuned on a counterfactual
training dataset, models are less sensitive to the later introduced anti-gender
stereotyped text.


http://arxiv.org/abs/2310.09891
Towards Deep Learning Models Resistant to Transfer-based Adversarial Attacks via Data-centric Robust Learning. (99%)
Yulong Yang; Chenhao Lin; Xiang Ji; Qiwei Tian; Qian Li; Hongshan Yang; Zhibo Wang; Chao Shen
  Transfer-based adversarial attacks raise a severe threat to real-world deep
learning systems since they do not require access to target models. Adversarial
training (AT), which is recognized as the strongest defense against white-box
attacks, has also guaranteed high robustness to (black-box) transfer-based
attacks. However, AT suffers from heavy computational overhead since it
optimizes the adversarial examples during the whole training process. In this
paper, we demonstrate that such heavy optimization is not necessary for AT
against transfer-based attacks. Instead, a one-shot adversarial augmentation
prior to training is sufficient, and we name this new defense paradigm
Data-centric Robust Learning (DRL). Our experimental results show that DRL
outperforms widely-used AT techniques (e.g., PGD-AT, TRADES, EAT, and FAT) in
terms of black-box robustness and even surpasses the top-1 defense on
RobustBench when combined with diverse data augmentations and loss
regularizations. We also identify other benefits of DRL, for instance, the
model generalization capability and robust fairness.


http://arxiv.org/abs/2310.09792
SCME: A Self-Contrastive Method for Data-free and Query-Limited Model Extraction Attack. (99%)
Renyang Liu; Jinhong Zhang; Kwok-Yan Lam; Jun Zhao; Wei Zhou
  Previous studies have revealed that artificial intelligence (AI) systems are
vulnerable to adversarial attacks. Among them, model extraction attacks fool
the target model by generating adversarial examples on a substitute model. The
core of such an attack is training a substitute model as similar to the target
model as possible, where the simulation process can be categorized in a
data-dependent and data-free manner. Compared with the data-dependent method,
the data-free one has been proven to be more practical in the real world since
it trains the substitute model with synthesized data. However, the distribution
of these fake data lacks diversity and cannot detect the decision boundary of
the target model well, resulting in the dissatisfactory simulation effect.
Besides, these data-free techniques need a vast number of queries to train the
substitute model, increasing the time and computing consumption and the risk of
exposure. To solve the aforementioned problems, in this paper, we propose a
novel data-free model extraction method named SCME (Self-Contrastive Model
Extraction), which considers both the inter- and intra-class diversity in
synthesizing fake data. In addition, SCME introduces the Mixup operation to
augment the fake data, which can explore the target model's decision boundary
effectively and improve the simulating capacity. Extensive experiments show
that the proposed method can yield diversified fake data. Moreover, our method
has shown superiority in many different attack settings under the query-limited
scenario, especially for untargeted attacks, the SCME outperforms SOTA methods
by 11.43\% on average for five baseline datasets.


http://arxiv.org/abs/2310.09795
AFLOW: Developing Adversarial Examples under Extremely Noise-limited Settings. (99%)
Renyang Liu; Jinhong Zhang; Haoran Li; Jin Zhang; Yuanyu Wang; Wei Zhou
  Extensive studies have demonstrated that deep neural networks (DNNs) are
vulnerable to adversarial attacks. Despite the significant progress in the
attack success rate that has been made recently, the adversarial noise
generated by most of the existing attack methods is still too conspicuous to
the human eyes and proved to be easily detected by defense mechanisms.
Resulting that these malicious examples cannot contribute to exploring the
vulnerabilities of existing DNNs sufficiently. Thus, to better reveal the
defects of DNNs and further help enhance their robustness under noise-limited
situations, a new inconspicuous adversarial examples generation method is
exactly needed to be proposed. To bridge this gap, we propose a novel Normalize
Flow-based end-to-end attack framework, called AFLOW, to synthesize
imperceptible adversarial examples under strict constraints. Specifically,
rather than the noise-adding manner, AFLOW directly perturbs the hidden
representation of the corresponding image to craft the desired adversarial
examples. Compared with existing methods, extensive experiments on three
benchmark datasets show that the adversarial examples built by AFLOW exhibit
superiority in imperceptibility, image quality and attack capability. Even on
robust models, AFLOW can still achieve higher attack results than previous
methods.


http://arxiv.org/abs/2310.10010
Black-box Targeted Adversarial Attack on Segment Anything (SAM). (99%)
Sheng Zheng; Chaoning Zhang
  Deep recognition models are widely vulnerable to adversarial examples, which
change the model output by adding quasi-imperceptible perturbation to the image
input. Recently, Segment Anything Model (SAM) has emerged to become a popular
foundation model in computer vision due to its impressive generalization to
unseen data and tasks. Realizing flexible attacks on SAM is beneficial for
understanding the robustness of SAM in the adversarial context. To this end,
this work aims to achieve a targeted adversarial attack (TAA) on SAM.
Specifically, under a certain prompt, the goal is to make the predicted mask of
an adversarial example resemble that of a given target image. The task of TAA
on SAM has been realized in a recent arXiv work in the white-box setup by
assuming access to prompt and model, which is thus less practical. To address
the issue of prompt dependence, we propose a simple yet effective approach by
only attacking the image encoder. Moreover, we propose a novel regularization
loss to enhance the cross-model transferability by increasing the feature
dominance of adversarial images over random natural images. Extensive
experiments verify the effectiveness of our proposed simple techniques to
conduct a successful black-box TAA on SAM.


http://arxiv.org/abs/2310.10036
Evading Detection Actively: Toward Anti-Forensics against Forgery Localization. (97%)
Long Zhuo; Shenghai Luo; Shunquan Tan; Han Chen; Bin Li; Jiwu Huang
  Anti-forensics seeks to eliminate or conceal traces of tampering artifacts.
Typically, anti-forensic methods are designed to deceive binary detectors and
persuade them to misjudge the authenticity of an image. However, to the best of
our knowledge, no attempts have been made to deceive forgery detectors at the
pixel level and mis-locate forged regions. Traditional adversarial attack
methods cannot be directly used against forgery localization due to the
following defects: 1) they tend to just naively induce the target forensic
models to flip their pixel-level pristine or forged decisions; 2) their
anti-forensics performance tends to be severely degraded when faced with the
unseen forensic models; 3) they lose validity once the target forensic models
are retrained with the anti-forensics images generated by them. To tackle the
three defects, we propose SEAR (Self-supErvised Anti-foRensics), a novel
self-supervised and adversarial training algorithm that effectively trains
deep-learning anti-forensic models against forgery localization. SEAR sets a
pretext task to reconstruct perturbation for self-supervised learning. In
adversarial training, SEAR employs a forgery localization model as a supervisor
to explore tampering features and constructs a deep-learning concealer to erase
corresponding traces. We have conducted largescale experiments across diverse
datasets. The experimental results demonstrate that, through the combination of
self-supervised learning and adversarial learning, SEAR successfully deceives
the state-of-the-art forgery localization methods, as well as tackle the three
defects regarding traditional adversarial attack methods mentioned above.


http://arxiv.org/abs/2310.09744
Explore the Effect of Data Selection on Poison Efficiency in Backdoor Attacks. (61%)
Ziqiang Li; Pengfei Xia; Hong Sun; Yueqi Zeng; Wei Zhang; Bin Li
  As the number of parameters in Deep Neural Networks (DNNs) scales, the thirst
for training data also increases. To save costs, it has become common for users
and enterprises to delegate time-consuming data collection to third parties.
Unfortunately, recent research has shown that this practice raises the risk of
DNNs being exposed to backdoor attacks. Specifically, an attacker can
maliciously control the behavior of a trained model by poisoning a small
portion of the training data. In this study, we focus on improving the
poisoning efficiency of backdoor attacks from the sample selection perspective.
The existing attack methods construct such poisoned samples by randomly
selecting some clean data from the benign set and then embedding a trigger into
them. However, this random selection strategy ignores that each sample may
contribute differently to the backdoor injection, thereby reducing the
poisoning efficiency. To address the above problem, a new selection strategy
named Improved Filtering and Updating Strategy (FUS++) is proposed.
Specifically, we adopt the forgetting events of the samples to indicate the
contribution of different poisoned samples and use the curvature of the loss
surface to analyses the effectiveness of this phenomenon. Accordingly, we
combine forgetting events and curvature of different samples to conduct a
simple yet efficient sample selection strategy. The experimental results on
image classification (CIFAR-10, CIFAR-100, ImageNet-10), text classification
(AG News), audio classification (ESC-50), and age regression (Facial Age)
consistently demonstrate the effectiveness of the proposed strategy: the attack
performance using FUS++ is significantly higher than that using random
selection for the same poisoning ratio.


http://arxiv.org/abs/2310.10012
Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? (5%)
Yu-Lin Tsai; Chia-Yi Hsu; Chulin Xie; Chih-Hsun Lin; Jia-You Chen; Bo Li; Pin-Yu Chen; Chia-Mu Yu; Chun-Ying Huang
  Diffusion models for text-to-image (T2I) synthesis, such as Stable Diffusion
(SD), have recently demonstrated exceptional capabilities for generating
high-quality content. However, this progress has raised several concerns of
potential misuse, particularly in creating copyrighted, prohibited, and
restricted content, or NSFW (not safe for work) images. While efforts have been
made to mitigate such problems, either by implementing a safety filter at the
evaluation stage or by fine-tuning models to eliminate undesirable concepts or
styles, the effectiveness of these safety measures in dealing with a wide range
of prompts remains largely unexplored. In this work, we aim to investigate
these safety mechanisms by proposing one novel concept retrieval algorithm for
evaluation. We introduce Ring-A-Bell, a model-agnostic red-teaming tool for T2I
diffusion models, where the whole evaluation can be prepared in advance without
prior knowledge of the target model. Specifically, Ring-A-Bell first performs
concept extraction to obtain holistic representations for sensitive and
inappropriate concepts. Subsequently, by leveraging the extracted concept,
Ring-A-Bell automatically identifies problematic prompts for diffusion models
with the corresponding generation of inappropriate content, allowing the user
to assess the reliability of deployed safety mechanisms. Finally, we
empirically validate our method by testing online services such as Midjourney
and various methods of concept removal. Our results show that Ring-A-Bell, by
manipulating safe prompting benchmarks, can transform prompts that were
originally regarded as safe to evade existing safety mechanisms, thus revealing
the defects of the so-called safety mechanisms which could practically lead to
the generation of harmful contents.


http://arxiv.org/abs/2310.09827
VFLAIR: A Research Library and Benchmark for Vertical Federated Learning. (3%)
Tianyuan Zou; Zixuan Gu; Yu He; Hideaki Takahashi; Yang Liu; Guangnan Ye; Ya-Qin Zhang
  Vertical Federated Learning (VFL) has emerged as a collaborative training
paradigm that allows participants with different features of the same group of
users to accomplish cooperative training without exposing their raw data or
model parameters. VFL has gained significant attention for its research
potential and real-world applications in recent years, but still faces
substantial challenges, such as in defending various kinds of data inference
and backdoor attacks. Moreover, most of existing VFL projects are
industry-facing and not easily used for keeping track of the current research
progress. To address this need, we present an extensible and lightweight VFL
framework VFLAIR (available at https://github.com/FLAIR-THU/VFLAIR), which
supports VFL training with a variety of models, datasets and protocols, along
with standardized modules for comprehensive evaluations of attacks and defense
strategies. We also benchmark 11 attacks and 8 defenses performance under
different communication and model partition settings and draw concrete insights
and recommendations on the choice of defense strategies for different practical
VFL deployment scenario.


http://arxiv.org/abs/2310.09652
BufferSearch: Generating Black-Box Adversarial Texts With Lower Queries. (98%)
Wenjie Lv; Zhen Wang; Yitao Zheng; Zhehua Zhong; Qi Xuan; Tianyi Chen
  Machine learning security has recently become a prominent topic in the
natural language processing (NLP) area. The existing black-box adversarial
attack suffers prohibitively from the high model querying complexity, resulting
in easily being captured by anti-attack monitors. Meanwhile, how to eliminate
redundant model queries is rarely explored. In this paper, we propose a
query-efficient approach BufferSearch to effectively attack general intelligent
NLP systems with the minimal number of querying requests. In general,
BufferSearch makes use of historical information and conducts statistical test
to avoid incurring model queries frequently. Numerically, we demonstrate the
effectiveness of BufferSearch on various benchmark text-classification
experiments by achieving the competitive attacking performance but with a
significant reduction of query quantity. Furthermore, BufferSearch performs
multiple times better than competitors within restricted query budget. Our work
establishes a strong benchmark for the future study of query-efficiency in NLP
adversarial attacks.


http://arxiv.org/abs/2310.09361
Is Certifying $\ell_p$ Robustness Still Worthwhile? (99%)
Ravi Mangal; Klas Leino; Zifan Wang; Kai Hu; Weicheng Yu; Corina Pasareanu; Anupam Datta; Matt Fredrikson
  Over the years, researchers have developed myriad attacks that exploit the
ubiquity of adversarial examples, as well as defenses that aim to guard against
the security vulnerabilities posed by such attacks. Of particular interest to
this paper are defenses that provide provable guarantees against the class of
$\ell_p$-bounded attacks. Certified defenses have made significant progress,
taking robustness certification from toy models and datasets to large-scale
problems like ImageNet classification. While this is undoubtedly an interesting
academic problem, as the field has matured, its impact in practice remains
unclear, thus we find it useful to revisit the motivation for continuing this
line of research. There are three layers to this inquiry, which we address in
this paper: (1) why do we care about robustness research? (2) why do we care
about the $\ell_p$-bounded threat model? And (3) why do we care about
certification as opposed to empirical defenses? In brief, we take the position
that local robustness certification indeed confers practical value to the field
of machine learning. We focus especially on the latter two questions from
above. With respect to the first of the two, we argue that the $\ell_p$-bounded
threat model acts as a minimal requirement for safe application of models in
security-critical domains, while at the same time, evidence has mounted
suggesting that local robustness may lead to downstream external benefits not
immediately related to robustness. As for the second, we argue that (i)
certification provides a resolution to the cat-and-mouse game of adversarial
attacks; and furthermore, that (ii) perhaps contrary to popular belief, there
may not exist a fundamental trade-off between accuracy, robustness, and
certifiability, while moreover, certified training techniques constitute a
particularly promising way for learning robust models.


http://arxiv.org/abs/2310.09266
User Inference Attacks on Large Language Models. (16%)
Nikhil Kandpal; Krishna Pillutla; Alina Oprea; Peter Kairouz; Christopher A. Choquette-Choo; Zheng Xu
  Fine-tuning is a common and effective method for tailoring large language
models (LLMs) to specialized tasks and applications. In this paper, we study
the privacy implications of fine-tuning LLMs on user data. To this end, we
define a realistic threat model, called user inference, wherein an attacker
infers whether or not a user's data was used for fine-tuning. We implement
attacks for this threat model that require only a small set of samples from a
user (possibly different from the samples used for training) and black-box
access to the fine-tuned LLM. We find that LLMs are susceptible to user
inference attacks across a variety of fine-tuning datasets, at times with near
perfect attack success rates. Further, we investigate which properties make
users vulnerable to user inference, finding that outlier users (i.e. those with
data distributions sufficiently different from other users) and users who
contribute large quantities of data are most susceptible to attack. Finally, we
explore several heuristics for mitigating privacy attacks. We find that
interventions in the training algorithm, such as batch or per-example gradient
clipping and early stopping fail to prevent user inference. However, limiting
the number of fine-tuning samples from a single user can reduce attack
effectiveness, albeit at the cost of reducing the total amount of fine-tuning
data.


http://arxiv.org/abs/2310.08847
On the Over-Memorization During Natural, Robust and Catastrophic Overfitting. (1%)
Runqi Lin; Chaojian Yu; Bo Han; Tongliang Liu
  Overfitting negatively impacts the generalization ability of deep neural
networks (DNNs) in both natural and adversarial training. Existing methods
struggle to consistently address different types of overfitting, typically
designing strategies that focus separately on either natural or adversarial
patterns. In this work, we adopt a unified perspective by solely focusing on
natural patterns to explore different types of overfitting. Specifically, we
examine the memorization effect in DNNs and reveal a shared behaviour termed
over-memorization, which impairs their generalization capacity. This behaviour
manifests as DNNs suddenly becoming high-confidence in predicting certain
training patterns and retaining a persistent memory for them. Furthermore, when
DNNs over-memorize an adversarial pattern, they tend to simultaneously exhibit
high-confidence prediction for the corresponding natural pattern. These
findings motivate us to holistically mitigate different types of overfitting by
hindering the DNNs from over-memorization natural patterns. To this end, we
propose a general framework, Distraction Over-Memorization (DOM), which
explicitly prevents over-memorization by either removing or augmenting the
high-confidence natural patterns. Extensive experiments demonstrate the
effectiveness of our proposed method in mitigating overfitting across various
training paradigms.


http://arxiv.org/abs/2310.08073
Samples on Thin Ice: Re-Evaluating Adversarial Pruning of Neural Networks. (99%)
Giorgio Piras; Maura Pintor; Ambra Demontis; Battista Biggio
  Neural network pruning has shown to be an effective technique for reducing
the network size, trading desirable properties like generalization and
robustness to adversarial attacks for higher sparsity. Recent work has claimed
that adversarial pruning methods can produce sparse networks while also
preserving robustness to adversarial examples. In this work, we first
re-evaluate three state-of-the-art adversarial pruning methods, showing that
their robustness was indeed overestimated. We then compare pruned and dense
versions of the same models, discovering that samples on thin ice, i.e., closer
to the unpruned model's decision boundary, are typically misclassified after
pruning. We conclude by discussing how this intuition may lead to designing
more effective adversarial pruning methods in future work.


http://arxiv.org/abs/2310.08292
Concealed Electronic Countermeasures of Radar Signal with Adversarial Examples. (93%)
Ruinan Ma; Canjie Zhu; Mingfeng Lu; Yunjie Li; Yu-an Tan; Ruibin Zhang; Ran Tao
  Electronic countermeasures involving radar signals are an important aspect of
modern warfare. Traditional electronic countermeasures techniques typically add
large-scale interference signals to ensure interference effects, which can lead
to attacks being too obvious. In recent years, AI-based attack methods have
emerged that can effectively solve this problem, but the attack scenarios are
currently limited to time domain radar signal classification. In this paper, we
focus on the time-frequency images classification scenario of radar signals. We
first propose an attack pipeline under the time-frequency images scenario and
DITIMI-FGSM attack algorithm with high transferability. Then, we propose
STFT-based time domain signal attack(STDS) algorithm to solve the problem of
non-invertibility in time-frequency analysis, thus obtaining the time-domain
representation of the interference signal. A large number of experiments show
that our attack pipeline is feasible and the proposed attack method has a high
success rate.


http://arxiv.org/abs/2310.08808
Attacks Meet Interpretability (AmI) Evaluation and Findings. (92%)
Qian Ma; Ziping Ye; Shagufta Mehnaz
  To investigate the effectiveness of the model explanation in detecting
adversarial examples, we reproduce the results of two papers, Attacks Meet
Interpretability: Attribute-steered Detection of Adversarial Samples and Is AmI
(Attacks Meet Interpretability) Robust to Adversarial Examples. And then
conduct experiments and case studies to identify the limitations of both works.
We find that Attacks Meet Interpretability(AmI) is highly dependent on the
selection of hyperparameters. Therefore, with a different hyperparameter
choice, AmI is still able to detect Nicholas Carlini's attack. Finally, we
propose recommendations for future work on the evaluation of defense techniques
such as AmI.


http://arxiv.org/abs/2310.08177
Improving Fast Minimum-Norm Attacks with Hyperparameter Optimization. (68%)
Giuseppe Floris; Raffaele Mura; Luca Scionis; Giorgio Piras; Maura Pintor; Ambra Demontis; Battista Biggio
  Evaluating the adversarial robustness of machine learning models using
gradient-based attacks is challenging. In this work, we show that
hyperparameter optimization can improve fast minimum-norm attacks by automating
the selection of the loss function, the optimizer and the step-size scheduler,
along with the corresponding hyperparameters. Our extensive evaluation
involving several robust models demonstrates the improved efficacy of fast
minimum-norm attacks when hyper-up with hyperparameter optimization. We release
our open-source code at https://github.com/pralab/HO-FMN.


http://arxiv.org/abs/2310.08681
Fed-Safe: Securing Federated Learning in Healthcare Against Adversarial Attacks. (64%)
Erfan Darzi; Nanna M. Sijtsema; Ooijen P. M. A van
  This paper explores the security aspects of federated learning applications
in medical image analysis. Current robustness-oriented methods like adversarial
training, secure aggregation, and homomorphic encryption often risk privacy
compromises. The central aim is to defend the network against potential privacy
breaches while maintaining model robustness against adversarial manipulations.
We show that incorporating distributed noise, grounded in the privacy
guarantees in federated settings, enables the development of a adversarially
robust model that also meets federated privacy standards. We conducted
comprehensive evaluations across diverse attack scenarios, parameters, and use
cases in cancer imaging, concentrating on pathology, meningioma, and glioma.
The results reveal that the incorporation of distributed noise allows for the
attainment of security levels comparable to those of conventional adversarial
training while requiring fewer retraining samples to establish a robust model.


http://arxiv.org/abs/2310.08732
Provably Robust Cost-Sensitive Learning via Randomized Smoothing. (45%)
Yuan Xin; Michael Backes; Xiao Zhang
  We focus on learning adversarially robust classifiers under a cost-sensitive
scenario, where the potential harm of different classwise adversarial
transformations is encoded in a binary cost matrix. Existing methods are either
empirical that cannot certify robustness or suffer from inherent scalability
issues. In this work, we study whether randomized smoothing, a more scalable
robustness certification framework, can be leveraged to certify cost-sensitive
robustness. Built upon a notion of cost-sensitive certified radius, we show how
to adapt the standard randomized smoothing certification pipeline to produce
tight robustness guarantees for any cost matrix. In addition, with fine-grained
certified radius optimization schemes specifically designed for different data
subgroups, we propose an algorithm to train smoothed classifiers that are
optimized for cost-sensitive robustness. Extensive experiments on image
benchmarks and a real-world medical dataset demonstrate the superiority of our
method in achieving significantly improved performance of certified
cost-sensitive robustness while having a negligible impact on overall accuracy.


http://arxiv.org/abs/2310.08571
Bucks for Buckets (B4B): Active Defenses Against Stealing Encoders. (31%)
Jan Dubiński; Stanisław Pawlak; Franziska Boenisch; Tomasz Trzciński; Adam Dziedzic
  Machine Learning as a Service (MLaaS) APIs provide ready-to-use and
high-utility encoders that generate vector representations for given inputs.
Since these encoders are very costly to train, they become lucrative targets
for model stealing attacks during which an adversary leverages query access to
the API to replicate the encoder locally at a fraction of the original training
costs. We propose Bucks for Buckets (B4B), the first active defense that
prevents stealing while the attack is happening without degrading
representation quality for legitimate API users. Our defense relies on the
observation that the representations returned to adversaries who try to steal
the encoder's functionality cover a significantly larger fraction of the
embedding space than representations of legitimate users who utilize the
encoder to solve a particular downstream task.vB4B leverages this to adaptively
adjust the utility of the returned representations according to a user's
coverage of the embedding space. To prevent adaptive adversaries from eluding
our defense by simply creating multiple user accounts (sybils), B4B also
individually transforms each user's representations. This prevents the
adversary from directly aggregating representations over multiple accounts to
create their stolen encoder copy. Our active defense opens a new path towards
securely sharing and democratizing encoders over public APIs.


http://arxiv.org/abs/2310.08097
Sentinel: An Aggregation Function to Secure Decentralized Federated Learning. (11%)
Chao Feng; Alberto Huertas Celdran; Janosch Baltensperger; Enrique Tomas Matınez Bertran; Gerome Bovet; Burkhard Stiller
  The rapid integration of Federated Learning (FL) into networking encompasses
various aspects such as network management, quality of service, and
cybersecurity while preserving data privacy. In this context, Decentralized
Federated Learning (DFL) emerges as an innovative paradigm to train
collaborative models, addressing the single point of failure limitation.
However, the security and trustworthiness of FL and DFL are compromised by
poisoning attacks, negatively impacting its performance. Existing defense
mechanisms have been designed for centralized FL and they do not adequately
exploit the particularities of DFL. Thus, this work introduces Sentinel, a
defense strategy to counteract poisoning attacks in DFL. Sentinel leverages the
accessibility of local data and defines a three-step aggregation protocol
consisting of similarity filtering, bootstrap validation, and normalization to
safeguard against malicious model updates. Sentinel has been evaluated with
diverse datasets and various poisoning attack types and threat levels,
improving the state-of-the-art performance against both untargeted and targeted
poisoning attacks.


http://arxiv.org/abs/2310.08772
Investigating the Robustness and Properties of Detection Transformers (DETR) Toward Difficult Images. (9%)
Zhao Ning Zou; Yuhang Zhang; Robert Wijaya
  Transformer-based object detectors (DETR) have shown significant performance
across machine vision tasks, ultimately in object detection. This detector is
based on a self-attention mechanism along with the transformer encoder-decoder
architecture to capture the global context in the image. The critical issue to
be addressed is how this model architecture can handle different image
nuisances, such as occlusion and adversarial perturbations. We studied this
issue by measuring the performance of DETR with different experiments and
benchmarking the network with convolutional neural network (CNN) based
detectors like YOLO and Faster-RCNN. We found that DETR performs well when it
comes to resistance to interference from information loss in occlusion images.
Despite that, we found that the adversarial stickers put on the image require
the network to produce a new unnecessary set of keys, queries, and values,
which in most cases, results in a misdirection of the network. DETR also
performed poorer than YOLOv5 in the image corruption benchmark. Furthermore, we
found that DETR depends heavily on the main query when making a prediction,
which leads to imbalanced contributions between queries since the main query
receives most of the gradient flow.


http://arxiv.org/abs/2310.08708
Polynomial Time Cryptanalytic Extraction of Neural Network Models. (3%)
Adi Shamir; Isaac Canales-Martinez; Anna Hambitzer; Jorge Chavez-Saab; Francisco Rodrigez-Henriquez; Nitin Satpute
  Billions of dollars and countless GPU hours are currently spent on training
Deep Neural Networks (DNNs) for a variety of tasks. Thus, it is essential to
determine the difficulty of extracting all the parameters of such neural
networks when given access to their black-box implementations. Many versions of
this problem have been studied over the last 30 years, and the best current
attack on ReLU-based deep neural networks was presented at Crypto 2020 by
Carlini, Jagielski, and Mironov. It resembles a differential chosen plaintext
attack on a cryptosystem, which has a secret key embedded in its black-box
implementation and requires a polynomial number of queries but an exponential
amount of time (as a function of the number of neurons). In this paper, we
improve this attack by developing several new techniques that enable us to
extract with arbitrarily high precision all the real-valued parameters of a
ReLU-based DNN using a polynomial number of queries and a polynomial amount of
time. We demonstrate its practical efficiency by applying it to a full-sized
neural network for classifying the CIFAR10 dataset, which has 3072 inputs, 8
hidden layers with 256 neurons each, and over million neuronal parameters. An
attack following the approach by Carlini et al. requires an exhaustive search
over 2 to the power 256 possibilities. Our attack replaces this with our new
techniques, which require only 30 minutes on a 256-core computer.


http://arxiv.org/abs/2310.08320
Defending Our Privacy With Backdoors. (3%)
Dominik Hintersdorf; Lukas Struppek; Daniel Neider; Kristian Kersting
  The proliferation of large AI models trained on uncurated, often sensitive
web-scraped data has raised significant privacy concerns. One of the concerns
is that adversaries can extract information about the training data using
privacy attacks. Unfortunately, the task of removing specific information from
the models without sacrificing performance is not straightforward and has
proven to be challenging. We propose a rather easy yet effective defense based
on backdoor attacks to remove private information such as names of individuals
from models, and focus in this work on text encoders. Specifically, through
strategic insertion of backdoors, we align the embeddings of sensitive phrases
with those of neutral terms-"a person" instead of the person's name. Our
empirical results demonstrate the effectiveness of our backdoor-based defense
on CLIP by assessing its performance using a specialized privacy attack for
zero-shot classifiers. Our approach provides not only a new "dual-use"
perspective on backdoor attacks, but also presents a promising avenue to
enhance the privacy of individuals within models trained on uncurated
web-scraped data.


http://arxiv.org/abs/2310.08040
SEE-OoD: Supervised Exploration For Enhanced Out-of-Distribution Detection. (1%)
Xiaoyang Song; Wenbo Sun; Maher Nouiehed; Raed Al Kontar; Judy Jin
  Current techniques for Out-of-Distribution (OoD) detection predominantly rely
on quantifying predictive uncertainty and incorporating model regularization
during the training phase, using either real or synthetic OoD samples. However,
methods that utilize real OoD samples lack exploration and are prone to overfit
the OoD samples at hand. Whereas synthetic samples are often generated based on
features extracted from training data, rendering them less effective when the
training and OoD data are highly overlapped in the feature space. In this work,
we propose a Wasserstein-score-based generative adversarial training scheme to
enhance OoD detection accuracy, which, for the first time, performs data
augmentation and exploration simultaneously under the supervision of limited
OoD samples. Specifically, the generator explores OoD spaces and generates
synthetic OoD samples using feedback from the discriminator, while the
discriminator exploits both the observed and synthesized samples for OoD
detection using a predefined Wasserstein score. We provide theoretical
guarantees that the optimal solutions of our generative scheme are
statistically achievable through adversarial training in empirical settings. We
then demonstrate that the proposed method outperforms state-of-the-art
techniques on various computer vision datasets and exhibits superior
generalizability to unseen OoD data.


http://arxiv.org/abs/2310.08537
XAI Benchmark for Visual Explanation. (1%)
Yifei Zhang; Siyi Gu; James Song; Bo Pan; Liang Zhao
  The rise of deep learning algorithms has led to significant advancements in
computer vision tasks, but their "black box" nature has raised concerns
regarding interpretability. Explainable AI (XAI) has emerged as a critical area
of research aiming to open this "black box", and shed light on the
decision-making process of AI models. Visual explanations, as a subset of
Explainable Artificial Intelligence (XAI), provide intuitive insights into the
decision-making processes of AI models handling visual data by highlighting
influential areas in an input image. Despite extensive research conducted on
visual explanations, most evaluations are model-centered since the availability
of corresponding real-world datasets with ground truth explanations is scarce
in the context of image data. To bridge this gap, we introduce an XAI Benchmark
comprising a dataset collection from diverse topics that provide both class
labels and corresponding explanation annotations for images. We have processed
data from diverse domains to align with our unified visual explanation
framework. We introduce a comprehensive Visual Explanation pipeline, which
integrates data loading, preprocessing, experimental setup, and model
evaluation processes. This structure enables researchers to conduct fair
comparisons of various visual explanation techniques. In addition, we provide a
comprehensive review of over 10 evaluation methods for visual explanation to
assist researchers in effectively utilizing our dataset collection. To further
assess the performance of existing visual explanation methods, we conduct
experiments on selected datasets using various model-centered and ground
truth-centered evaluation metrics. We envision this benchmark could facilitate
the advancement of visual explanation models. The XAI dataset collection and
easy-to-use code for evaluation are publicly accessible at
https://xaidataset.github.io.


http://arxiv.org/abs/2310.08419
Jailbreaking Black Box Large Language Models in Twenty Queries. (1%)
Patrick Chao; Alexander Robey; Edgar Dobriban; Hamed Hassani; George J. Pappas; Eric Wong
  There is growing interest in ensuring that large language models (LLMs) align
with human values. However, the alignment of such models is vulnerable to
adversarial jailbreaks, which coax LLMs into overriding their safety
guardrails. The identification of these vulnerabilities is therefore
instrumental in understanding inherent weaknesses and preventing future misuse.
To this end, we propose Prompt Automatic Iterative Refinement (PAIR), an
algorithm that generates semantic jailbreaks with only black-box access to an
LLM. PAIR -- which is inspired by social engineering attacks -- uses an
attacker LLM to automatically generate jailbreaks for a separate targeted LLM
without human intervention. In this way, the attacker LLM iteratively queries
the target LLM to update and refine a candidate jailbreak. Empirically, PAIR
often requires fewer than twenty queries to produce a jailbreak, which is
orders of magnitude more efficient than existing algorithms. PAIR also achieves
competitive jailbreaking success rates and transferability on open and
closed-source LLMs, including GPT-3.5/4, Vicuna, and PaLM-2.


http://arxiv.org/abs/2310.07492
Boosting Black-box Attack to Deep Neural Networks with Conditional Diffusion Models. (99%)
Renyang Liu; Wei Zhou; Tianwei Zhang; Kangjie Chen; Jun Zhao; Kwok-Yan Lam
  Existing black-box attacks have demonstrated promising potential in creating
adversarial examples (AE) to deceive deep learning models. Most of these
attacks need to handle a vast optimization space and require a large number of
queries, hence exhibiting limited practical impacts in real-world scenarios. In
this paper, we propose a novel black-box attack strategy, Conditional Diffusion
Model Attack (CDMA), to improve the query efficiency of generating AEs under
query-limited situations. The key insight of CDMA is to formulate the task of
AE synthesis as a distribution transformation problem, i.e., benign examples
and their corresponding AEs can be regarded as coming from two distinctive
distributions and can transform from each other with a particular converter.
Unlike the conventional \textit{query-and-optimization} approach, we generate
eligible AEs with direct conditional transform using the aforementioned data
converter, which can significantly reduce the number of queries needed. CDMA
adopts the conditional Denoising Diffusion Probabilistic Model as the
converter, which can learn the transformation from clean samples to AEs, and
ensure the smooth development of perturbed noise resistant to various defense
strategies. We demonstrate the effectiveness and efficiency of CDMA by
comparing it with nine state-of-the-art black-box attacks across three
benchmark datasets. On average, CDMA can reduce the query count to a handful of
times; in most cases, the query count is only ONE. We also show that CDMA can
obtain $>99\%$ attack success rate for untarget attacks over all datasets and
targeted attack over CIFAR-10 with the noise budget of $\epsilon=16$.


http://arxiv.org/abs/2310.07780
Promoting Robustness of Randomized Smoothing: Two Cost-Effective Approaches. (89%)
Linbo Liu; Trong Nghia Hoang; Lam M. Nguyen; Tsui-Wei Weng
  Randomized smoothing has recently attracted attentions in the field of
adversarial robustness to provide provable robustness guarantees on smoothed
neural network classifiers. However, existing works show that vanilla
randomized smoothing usually does not provide good robustness performance and
often requires (re)training techniques on the base classifier in order to boost
the robustness of the resulting smoothed classifier. In this work, we propose
two cost-effective approaches to boost the robustness of randomized smoothing
while preserving its clean performance. The first approach introduces a new
robust training method AdvMacerwhich combines adversarial training and
robustness certification maximization for randomized smoothing. We show that
AdvMacer can improve the robustness performance of randomized smoothing
classifiers compared to SOTA baselines, while being 3x faster to train than
MACER baseline. The second approach introduces a post-processing method EsbRS
which greatly improves the robustness certificate based on building model
ensembles. We explore different aspects of model ensembles that has not been
studied by prior works and propose a novel design methodology to further
improve robustness of the ensemble based on our theoretical analysis.


http://arxiv.org/abs/2310.07325
An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l. (13%)
James Dao; Yeu-Tong Lao; Can Rager; Jett Janiak
  We provide concrete evidence for memory management in a 4-layer transformer.
Specifically, we identify clean-up behavior, in which model components
consistently remove the output of preceeding components during a forward pass.
Our findings suggest that the interpretability technique Direct Logit
Attribution provides misleading results. We show explicit examples where this
technique is inaccurate, as it does not account for clean-up behavior.


http://arxiv.org/abs/2310.07632
Prompt Backdoors in Visual Prompt Learning. (11%)
Hai Huang; Zhengyu Zhao; Michael Backes; Yun Shen; Yang Zhang
  Fine-tuning large pre-trained computer vision models is infeasible for
resource-limited users. Visual prompt learning (VPL) has thus emerged to
provide an efficient and flexible alternative to model fine-tuning through
Visual Prompt as a Service (VPPTaaS). Specifically, the VPPTaaS provider
optimizes a visual prompt given downstream data, and downstream users can use
this prompt together with the large pre-trained model for prediction. However,
this new learning paradigm may also pose security risks when the VPPTaaS
provider instead provides a malicious visual prompt. In this paper, we take the
first step to explore such risks through the lens of backdoor attacks.
Specifically, we propose BadVisualPrompt, a simple yet effective backdoor
attack against VPL. For example, poisoning $5\%$ CIFAR10 training data leads to
above $99\%$ attack success rates with only negligible model accuracy drop by
$1.5\%$. In particular, we identify and then address a new technical challenge
related to interactions between the backdoor trigger and visual prompt, which
does not exist in conventional, model-level backdoors. Moreover, we provide
in-depth analyses of seven backdoor defenses from model, prompt, and input
levels. Overall, all these defenses are either ineffective or impractical to
mitigate our BadVisualPrompt, implying the critical vulnerability of VPL.


http://arxiv.org/abs/2310.08015
Why Train More? Effective and Efficient Membership Inference via Memorization. (10%)
Jihye Choi; Shruti Tople; Varun Chandrasekaran; Somesh Jha
  Membership Inference Attacks (MIAs) aim to identify specific data samples
within the private training dataset of machine learning models, leading to
serious privacy violations and other sophisticated threats. Many practical
black-box MIAs require query access to the data distribution (the same
distribution where the private data is drawn) to train shadow models. By doing
so, the adversary obtains models trained "with" or "without" samples drawn from
the distribution, and analyzes the characteristics of the samples under
consideration. The adversary is often required to train more than hundreds of
shadow models to extract the signals needed for MIAs; this becomes the
computational overhead of MIAs. In this paper, we propose that by strategically
choosing the samples, MI adversaries can maximize their attack success while
minimizing the number of shadow models. First, our motivational experiments
suggest memorization as the key property explaining disparate sample
vulnerability to MIAs. We formalize this through a theoretical bound that
connects MI advantage with memorization. Second, we show sample complexity
bounds that connect the number of shadow models needed for MIAs with
memorization. Lastly, we confirm our theoretical arguments with comprehensive
experiments; by utilizing samples with high memorization scores, the adversary
can (a) significantly improve its efficacy regardless of the MIA used, and (b)
reduce the number of shadow models by nearly two orders of magnitude compared
to state-of-the-art approaches.


http://arxiv.org/abs/2310.07958
Towards Causal Deep Learning for Vulnerability Detection. (4%)
Md Mahbubur Rahman; Ira Ceka; Chengzhi Mao; Saikat Chakraborty; Baishakhi Ray; Wei Le
  Deep learning vulnerability detection has shown promising results in recent
years. However, an important challenge that still blocks it from being very
useful in practice is that the model is not robust under perturbation and it
cannot generalize well over the out-of-distribution (OOD) data, e.g., applying
a trained model to unseen projects in real world. We hypothesize that this is
because the model learned non-robust features, e.g., variable names, that have
spurious correlations with labels. When the perturbed and OOD datasets no
longer have the same spurious features, the model prediction fails. To address
the challenge, in this paper, we introduced causality into deep learning
vulnerability detection. Our approach CausalVul consists of two phases. First,
we designed novel perturbations to discover spurious features that the model
may use to make predictions. Second, we applied the causal learning algorithms,
specifically, do-calculus, on top of existing deep learning models to
systematically remove the use of spurious features and thus promote causal
based prediction. Our results show that CausalVul consistently improved the
model accuracy, robustness and OOD performance for all the state-of-the-art
models and datasets we experimented. To the best of our knowledge, this is the
first work that introduces do calculus based causal learning to software
engineering models and shows it's indeed useful for improving the model
accuracy, robustness and generalization. Our replication package is located at
https://figshare.com/s/0ffda320dcb96c249ef2.


http://arxiv.org/abs/2310.07745
Deep Reinforcement Learning for Autonomous Cyber Operations: A Survey. (3%)
Gregory Palmer; Chris Parry; Daniel J. B. Harrold; Chris Willis
  The rapid increase in the number of cyber-attacks in recent years raises the
need for principled methods for defending networks against malicious actors.
Deep reinforcement learning (DRL) has emerged as a promising approach for
mitigating these attacks. However, while DRL has shown much potential for
cyber-defence, numerous challenges must be overcome before DRL can be applied
to autonomous cyber-operations (ACO) at scale. Principled methods are required
for environments that confront learners with very high-dimensional state
spaces, large multi-discrete action spaces, and adversarial learning. Recent
works have reported success in solving these problems individually. There have
also been impressive engineering efforts towards solving all three for
real-time strategy games. However, applying DRL to the full ACO problem remains
an open challenge. Here, we survey the relevant DRL literature and
conceptualize an idealised ACO-DRL agent. We provide: i.) A summary of the
domain properties that define the ACO problem; ii.) A comprehensive evaluation
of the extent to which domains used for benchmarking DRL approaches are
comparable to ACO; iii.) An overview of state-of-the-art approaches for scaling
DRL to domains that confront learners with the curse of dimensionality, and;
iv.) A survey and critique of current methods for limiting the exploitability
of agents within adversarial settings from the perspective of ACO. We conclude
with open research questions that we hope will motivate future directions for
researchers and practitioners working on ACO.


http://arxiv.org/abs/2310.06468
A Geometrical Approach to Evaluate the Adversarial Robustness of Deep Neural Networks. (99%)
Yang Wang; Bo Dong; Ke Xu; Haiyin Piao; Yufei Ding; Baocai Yin; Xin Yang
  Deep Neural Networks (DNNs) are widely used for computer vision tasks.
However, it has been shown that deep models are vulnerable to adversarial
attacks, i.e., their performances drop when imperceptible perturbations are
made to the original inputs, which may further degrade the following visual
tasks or introduce new problems such as data and privacy security. Hence,
metrics for evaluating the robustness of deep models against adversarial
attacks are desired. However, previous metrics are mainly proposed for
evaluating the adversarial robustness of shallow networks on the small-scale
datasets. Although the Cross Lipschitz Extreme Value for nEtwork Robustness
(CLEVER) metric has been proposed for large-scale datasets (e.g., the ImageNet
dataset), it is computationally expensive and its performance relies on a
tractable number of samples. In this paper, we propose the Adversarial
Converging Time Score (ACTS), an attack-dependent metric that quantifies the
adversarial robustness of a DNN on a specific input. Our key observation is
that local neighborhoods on a DNN's output surface would have different shapes
given different inputs. Hence, given different inputs, it requires different
time for converging to an adversarial sample. Based on this geometry meaning,
ACTS measures the converging time as an adversarial robustness metric. We
validate the effectiveness and generalization of the proposed ACTS metric
against different adversarial attacks on the large-scale ImageNet dataset using
state-of-the-art deep networks. Extensive experiments show that our ACTS metric
is an efficient and effective adversarial metric over the previous CLEVER
metric.


http://arxiv.org/abs/2310.07159
My Brother Helps Me: Node Injection Based Adversarial Attack on Social Bot Detection. (98%)
Lanjun Wang; Xinran Qiao; Yanwei Xie; Weizhi Nie; Yongdong Zhang; Anan Liu
  Social platforms such as Twitter are under siege from a multitude of
fraudulent users. In response, social bot detection tasks have been developed
to identify such fake users. Due to the structure of social networks, the
majority of methods are based on the graph neural network(GNN), which is
susceptible to attacks. In this study, we propose a node injection-based
adversarial attack method designed to deceive bot detection models. Notably,
neither the target bot nor the newly injected bot can be detected when a new
bot is added around the target bot. This attack operates in a black-box
fashion, implying that any information related to the victim model remains
unknown. To our knowledge, this is the first study exploring the resilience of
bot detection through graph node injection. Furthermore, we develop an
attribute recovery module to revert the injected node embedding from the graph
embedding space back to the original feature space, enabling the adversary to
manipulate node perturbation effectively. We conduct adversarial attacks on
four commonly used GNN structures for bot detection on two widely used
datasets: Cresci-2015 and TwiBot-22. The attack success rate is over 73\% and
the rate of newly injected nodes being detected as bots is below 13\% on these
two datasets.


http://arxiv.org/abs/2310.06396
Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach. (83%)
Kai Zhao; Qiyu Kang; Yang Song; Rui She; Sijie Wang; Wee Peng Tay
  Graph neural networks (GNNs) are vulnerable to adversarial perturbations,
including those that affect both node features and graph topology. This paper
investigates GNNs derived from diverse neural flows, concentrating on their
connection to various stability notions such as BIBO stability, Lyapunov
stability, structural stability, and conservative stability. We argue that
Lyapunov stability, despite its common use, does not necessarily ensure
adversarial robustness. Inspired by physics principles, we advocate for the use
of conservative Hamiltonian neural flows to construct GNNs that are robust to
adversarial attacks. The adversarial robustness of different neural flow GNNs
is empirically compared on several benchmark datasets under a variety of
adversarial attacks. Extensive numerical experiments demonstrate that GNNs
leveraging conservative Hamiltonian flows with Lyapunov stability substantially
improve robustness against adversarial perturbations. The implementation code
of experiments is available at
https://github.com/zknus/NeurIPS-2023-HANG-Robustness.


http://arxiv.org/abs/2310.06956
Adversarial optimization leads to over-optimistic security-constrained dispatch, but sampling can help. (76%)
Charles Dawson; Chuchu Fan
  To ensure safe, reliable operation of the electrical grid, we must be able to
predict and mitigate likely failures. This need motivates the classic
security-constrained AC optimal power flow (SCOPF) problem. SCOPF is commonly
solved using adversarial optimization, where the dispatcher and an adversary
take turns optimizing a robust dispatch and adversarial attack, respectively.
We show that adversarial optimization is liable to severely overestimate the
robustness of the optimized dispatch (when the adversary encounters a local
minimum), leading the operator to falsely believe that their dispatch is
secure.
  To prevent this overconfidence, we develop a novel adversarial sampling
approach that prioritizes diversity in the predicted attacks. We find that our
method not only substantially improves the robustness of the optimized dispatch
but also avoids overconfidence, accurately characterizing the likelihood of
voltage collapse under a given threat model. We demonstrate a proof-of-concept
on small-scale transmission systems with 14 and 57 nodes.


http://arxiv.org/abs/2310.07152
No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML. (62%)
Ziqi Zhang; Chen Gong; Yifeng Cai; Yuanyuan Yuan; Bingyan Liu; Ding Li; Yao Guo; Xiangqun Chen
  On-device ML introduces new security challenges: DNN models become white-box
accessible to device users. Based on white-box information, adversaries can
conduct effective model stealing (MS) and membership inference attack (MIA).
Using Trusted Execution Environments (TEEs) to shield on-device DNN models aims
to downgrade (easy) white-box attacks to (harder) black-box attacks. However,
one major shortcoming is the sharply increased latency (up to 50X). To
accelerate TEE-shield DNN computation with GPUs, researchers proposed several
model partition techniques. These solutions, referred to as TEE-Shielded DNN
Partition (TSDP), partition a DNN model into two parts, offloading the
privacy-insensitive part to the GPU while shielding the privacy-sensitive part
within the TEE. This paper benchmarks existing TSDP solutions using both MS and
MIA across a variety of DNN models, datasets, and metrics. We show important
findings that existing TSDP solutions are vulnerable to privacy-stealing
attacks and are not as safe as commonly believed. We also unveil the inherent
difficulty in deciding optimal DNN partition configurations (i.e., the highest
security with minimal utility cost) for present TSDP solutions. The experiments
show that such ``sweet spot'' configurations vary across datasets and models.
Based on lessons harvested from the experiments, we present TEESlice, a novel
TSDP method that defends against MS and MIA during DNN inference. TEESlice
follows a partition-before-training strategy, which allows for accurate
separation between privacy-related weights from public weights. TEESlice
delivers the same security protection as shielding the entire DNN model inside
TEE (the ``upper-bound'' security guarantees) with over 10X less overhead (in
both experimental and real-world environments) than prior TSDP solutions and no
accuracy loss.


http://arxiv.org/abs/2310.06958
Comparing the robustness of modern no-reference image- and video-quality metrics to adversarial attacks. (45%)
Anastasia Antsiferova; Khaled Abud; Aleksandr Gushchin; Sergey Lavrushkin; Ekaterina Shumitskaya; Maksim Velikanov; Dmitriy Vatolin
  Nowadays neural-network-based image- and video-quality metrics show better
performance compared to traditional methods. However, they also became more
vulnerable to adversarial attacks that increase metrics' scores without
improving visual quality. The existing benchmarks of quality metrics compare
their performance in terms of correlation with subjective quality and
calculation time. However, the adversarial robustness of image-quality metrics
is also an area worth researching. In this paper, we analyse modern metrics'
robustness to different adversarial attacks. We adopted adversarial attacks
from computer vision tasks and compared attacks' efficiency against 15
no-reference image/video-quality metrics. Some metrics showed high resistance
to adversarial attacks which makes their usage in benchmarks safer than
vulnerable metrics. The benchmark accepts new metrics submissions for
researchers who want to make their metrics more robust to attacks or to find
such metrics for their needs. Try our benchmark using pip install
robustness-benchmark.


http://arxiv.org/abs/2310.07100
GraphCloak: Safeguarding Task-specific Knowledge within Graph-structured Data from Unauthorized Exploitation. (22%)
Yixin Liu; Chenrui Fan; Xun Chen; Pan Zhou; Lichao Sun
  As Graph Neural Networks (GNNs) become increasingly prevalent in a variety of
fields, from social network analysis to protein-protein interaction studies,
growing concerns have emerged regarding the unauthorized utilization of
personal data. Recent studies have shown that imperceptible poisoning attacks
are an effective method of protecting image data from such misuse. However, the
efficacy of this approach in the graph domain remains unexplored. To bridge
this gap, this paper introduces GraphCloak to safeguard against the
unauthorized usage of graph data. Compared with prior work, GraphCloak offers
unique significant innovations: (1) graph-oriented, the perturbations are
applied to both topological structures and descriptive features of the graph;
(2) effective and stealthy, our cloaking method can bypass various inspections
while causing a significant performance drop in GNNs trained on the cloaked
graphs; and (3) stable across settings, our methods consistently perform
effectively under a range of practical settings with limited knowledge. To
address the intractable bi-level optimization problem, we propose two
error-minimizing-based poisoning methods that target perturbations on the
structural and feature space, along with a subgraph injection poisoning method.
Our comprehensive evaluation of these methods underscores their effectiveness,
stealthiness, and stability. We also delve into potential countermeasures and
provide analytical justification for their effectiveness, paving the way for
intriguing future research.


http://arxiv.org/abs/2310.06668
Latent Diffusion Counterfactual Explanations. (5%)
Karim Farid; Simon Schrodi; Max Argus; Thomas Brox
  Counterfactual explanations have emerged as a promising method for
elucidating the behavior of opaque black-box models. Recently, several works
leveraged pixel-space diffusion models for counterfactual generation. To handle
noisy, adversarial gradients during counterfactual generation -- causing
unrealistic artifacts or mere adversarial perturbations -- they required either
auxiliary adversarially robust models or computationally intensive guidance
schemes. However, such requirements limit their applicability, e.g., in
scenarios with restricted access to the model's training data. To address these
limitations, we introduce Latent Diffusion Counterfactual Explanations (LDCE).
LDCE harnesses the capabilities of recent class- or text-conditional foundation
latent diffusion models to expedite counterfactual generation and focus on the
important, semantic parts of the data. Furthermore, we propose a novel
consensus guidance mechanism to filter out noisy, adversarial gradients that
are misaligned with the diffusion model's implicit classifier. We demonstrate
the versatility of LDCE across a wide spectrum of models trained on diverse
datasets with different learning paradigms. Finally, we showcase how LDCE can
provide insights into model errors, enhancing our understanding of black-box
model behavior.


http://arxiv.org/abs/2310.06588
FTFT: efficient and robust Fine-Tuning by transFerring Training dynamics. (2%)
Yupei Du; Albert Gatt; Dong Nguyen
  Despite the massive success of fine-tuning large Pre-trained Language Models
(PLMs) on a wide range of Natural Language Processing (NLP) tasks, they remain
susceptible to out-of-distribution (OOD) and adversarial inputs. Data map (DM)
is a simple yet effective dual-model approach that enhances the robustness of
fine-tuned PLMs, which involves fine-tuning a model on the original training
set (i.e. reference model), selecting a specified fraction of important
training examples according to the training dynamics of the reference model,
and fine-tuning the same model on these selected examples (i.e. main model).
However, it suffers from the drawback of requiring fine-tuning the same model
twice, which is computationally expensive for large models. In this paper, we
first show that 1) training dynamics are highly transferable across different
model sizes and different pre-training methods, and that 2) main models
fine-tuned using DM learn faster than when using conventional Empirical Risk
Minimization (ERM). Building on these observations, we propose a novel
fine-tuning approach based on the DM method: Fine-Tuning by transFerring
Training dynamics (FTFT). Compared with DM, FTFT uses more efficient reference
models and then fine-tunes more capable main models for fewer steps. Our
experiments show that FTFT achieves better generalization robustness than ERM
while spending less than half of the training cost.


http://arxiv.org/abs/2310.07084
Investigating the Adversarial Robustness of Density Estimation Using the Probability Flow ODE. (2%)
Marius Arvinte; Cory Cornelius; Jason Martin; Nageen Himayat
  Beyond their impressive sampling capabilities, score-based diffusion models
offer a powerful analysis tool in the form of unbiased density estimation of a
query sample under the training data distribution. In this work, we investigate
the robustness of density estimation using the probability flow (PF) neural
ordinary differential equation (ODE) model against gradient-based likelihood
maximization attacks and the relation to sample complexity, where the
compressed size of a sample is used as a measure of its complexity. We
introduce and evaluate six gradient-based log-likelihood maximization attacks,
including a novel reverse integration attack. Our experimental evaluations on
CIFAR-10 show that density estimation using the PF ODE is robust against
high-complexity, high-likelihood attacks, and that in some cases adversarial
samples are semantically meaningful, as expected from a robust estimator.


http://arxiv.org/abs/2310.06387
Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. (1%)
Zeming Wei; Yifei Wang; Yisen Wang
  Large Language Models (LLMs) have shown remarkable success in various tasks,
but concerns about their safety and the potential for generating malicious
content have emerged. In this paper, we explore the power of In-Context
Learning (ICL) in manipulating the alignment ability of LLMs. We find that by
providing just few in-context demonstrations without fine-tuning, LLMs can be
manipulated to increase or decrease the probability of jailbreaking, i.e.
answering malicious prompts. Based on these observations, we propose In-Context
Attack (ICA) and In-Context Defense (ICD) methods for jailbreaking and guarding
aligned language model purposes. ICA crafts malicious contexts to guide models
in generating harmful outputs, while ICD enhances model robustness by
demonstrations of rejecting to answer harmful prompts. Our experiments show the
effectiveness of ICA and ICD in increasing or reducing the success rate of
adversarial jailbreaking attacks. Overall, we shed light on the potential of
ICL to influence LLM behavior and provide a new perspective for enhancing the
safety and alignment of LLMs.


http://arxiv.org/abs/2310.06182
PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization. (92%)
Jiancong Xiao; Ruoyu Sun; Zhi- Quan Luo
  Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is
found empirically that adversarially robust generalization is crucial in
establishing defense algorithms against adversarial attacks. Therefore, it is
interesting to study the theoretical guarantee of robust generalization. This
paper focuses on norm-based complexity, based on a PAC-Bayes approach
(Neyshabur et al., 2017). The main challenge lies in extending the key
ingredient, which is a weight perturbation bound in standard settings, to the
robust settings. Existing attempts heavily rely on additional strong
assumptions, leading to loose bounds. In this paper, we address this issue and
provide a spectrally-normalized robust generalization bound for DNNs. Compared
to existing bounds, our bound offers two significant advantages: Firstly, it
does not depend on additional assumptions. Secondly, it is considerably
tighter, aligning with the bounds of standard generalization. Therefore, our
result provides a different perspective on understanding robust generalization:
The mismatch terms between standard and robust generalization bounds shown in
previous studies do not contribute to the poor robust generalization. Instead,
these disparities solely due to mathematical issues. Finally, we extend the
main result to adversarial robustness against general non-$\ell_p$ attacks and
other neural network architectures.


http://arxiv.org/abs/2310.14942
Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Hand. (22%)
Junfeng Guo; Yiming Li; Lixu Wang; Shu-Tao Xia; Heng Huang; Cong Liu; Bo Li
  The prosperity of deep neural networks (DNNs) is largely benefited from
open-source datasets, based on which users can evaluate and improve their
methods. In this paper, we revisit backdoor-based dataset ownership
verification (DOV), which is currently the only feasible approach to protect
the copyright of open-source datasets. We reveal that these methods are
fundamentally harmful given that they could introduce malicious
misclassification behaviors to watermarked DNNs by the adversaries. In this
paper, we design DOV from another perspective by making watermarked models
(trained on the protected dataset) correctly classify some `hard' samples that
will be misclassified by the benign model. Our method is inspired by the
generalization property of DNNs, where we find a \emph{hardly-generalized
domain} for the original dataset (as its \emph{domain watermark}). It can be
easily learned with the protected dataset containing modified samples.
Specifically, we formulate the domain generation as a bi-level optimization and
propose to optimize a set of visually-indistinguishable clean-label modified
data with similar effects to domain-watermarked samples from the
hardly-generalized domain to ensure watermark stealthiness. We also design a
hypothesis-test-guided ownership verification via our domain watermark and
provide the theoretical analyses of our method. Extensive experiments on three
benchmark datasets are conducted, which verify the effectiveness of our method
and its resistance to potential adaptive methods. The code for reproducing main
experiments is available at
\url{https://github.com/JunfengGo/Domain-Watermark}.


http://arxiv.org/abs/2310.06112
Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach. (5%)
Shaopeng Fu; Di Wang
  Adversarial training (AT) is a canonical method for enhancing the robustness
of deep neural networks (DNNs). However, recent studies empirically
demonstrated that it suffers from robust overfitting, i.e., a long time AT can
be detrimental to the robustness of DNNs. This paper presents a theoretical
explanation of robust overfitting for DNNs. Specifically, we non-trivially
extend the neural tangent kernel (NTK) theory to AT and prove that an
adversarially trained wide DNN can be well approximated by a linearized DNN.
Moreover, for squared loss, closed-form AT dynamics for the linearized DNN can
be derived, which reveals a new AT degeneration phenomenon: a long-term AT will
result in a wide DNN degenerates to that obtained without AT and thus cause
robust overfitting. Based on our theoretical results, we further design a
method namely Adv-NTK, the first AT algorithm for infinite-width DNNs.
Experiments on real-world datasets show that Adv-NTK can help infinite-width
DNNs enhance comparable robustness to that of their finite-width counterparts,
which in turn justifies our theoretical findings. The code is available at
https://github.com/fshp971/adv-ntk.


http://arxiv.org/abs/2310.06227
Exploring adversarial attacks in federated learning for medical imaging. (2%)
Erfan Darzi; Florian Dubost; N. M. Sijtsema; Ooijen P. M. A van
  Federated learning offers a privacy-preserving framework for medical image
analysis but exposes the system to adversarial attacks. This paper aims to
evaluate the vulnerabilities of federated learning networks in medical image
analysis against such attacks. Employing domain-specific MRI tumor and
pathology imaging datasets, we assess the effectiveness of known threat
scenarios in a federated learning environment. Our tests reveal that
domain-specific configurations can increase the attacker's success rate
significantly. The findings emphasize the urgent need for effective defense
mechanisms and suggest a critical re-evaluation of current security protocols
in federated medical image analysis systems.


http://arxiv.org/abs/2310.05354
An Initial Investigation of Neural Replay Simulator for Over-the-Air Adversarial Perturbations to Automatic Speaker Verification. (99%)
Jiaqi Li; Li Wang; Liumeng Xue; Lei Wang; Zhizheng Wu
  Deep Learning has advanced Automatic Speaker Verification (ASV) in the past
few years. Although it is known that deep learning-based ASV systems are
vulnerable to adversarial examples in digital access, there are few studies on
adversarial attacks in the context of physical access, where a replay process
(i.e., over the air) is involved. An over-the-air attack involves a
loudspeaker, a microphone, and a replaying environment that impacts the
movement of the sound wave. Our initial experiment confirms that the replay
process impacts the effectiveness of the over-the-air attack performance. This
study performs an initial investigation towards utilizing a neural replay
simulator to improve over-the-air adversarial attack robustness. This is
achieved by using a neural waveform synthesizer to simulate the replay process
when estimating the adversarial perturbations. Experiments conducted on the
ASVspoof2019 dataset confirm that the neural replay simulator can considerably
increase the success rates of over-the-air adversarial attacks. This raises the
concern for adversarial attacks on speaker verification in physical access
applications.


http://arxiv.org/abs/2310.05057
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models. (26%)
Yifan Jiang; Filip Ilievski; Kaixin Ma; Zhivar Sourati
  The success of language models has inspired the NLP community to attend to
tasks that require implicit and complex reasoning, relying on human-like
commonsense mechanisms. While such vertical thinking tasks have been relatively
popular, lateral thinking puzzles have received little attention. To bridge
this gap, we devise BRAINTEASER: a multiple-choice Question Answering task
designed to test the model's ability to exhibit lateral thinking and defy
default commonsense associations. We design a three-step procedure for creating
the first lateral thinking benchmark, consisting of data collection, distractor
generation, and generation of adversarial examples, leading to 1,100 puzzles
with high-quality annotations. To assess the consistency of lateral reasoning
by models, we enrich BRAINTEASER based on a semantic and contextual
reconstruction of its questions. Our experiments with state-of-the-art
instruction- and commonsense language models reveal a significant gap between
human and model performance, which is further widened when consistency across
adversarial formats is considered. We make all of our code and data available
to stimulate work on developing and evaluating lateral thinking models.


http://arxiv.org/abs/2310.04780
IPMix: Label-Preserving Data Augmentation Method for Training Robust Classifiers. (76%)
Zhenglin Huang; Xianan Bao; Na Zhang; Qingqi Zhang; Xiaomei Tu; Biao Wu; Xi Yang
  Data augmentation has been proven effective for training high-accuracy
convolutional neural network classifiers by preventing overfitting. However,
building deep neural networks in real-world scenarios requires not only high
accuracy on clean data but also robustness when data distributions shift. While
prior methods have proposed that there is a trade-off between accuracy and
robustness, we propose IPMix, a simple data augmentation approach to improve
robustness without hurting clean accuracy. IPMix integrates three levels of
data augmentation (image-level, patch-level, and pixel-level) into a coherent
and label-preserving technique to increase the diversity of training data with
limited computational overhead. To further improve the robustness, IPMix
introduces structural complexity at different levels to generate more diverse
images and adopts the random mixing method for multi-scale information fusion.
Experiments demonstrate that IPMix outperforms state-of-the-art corruption
robustness on CIFAR-C and ImageNet-C. In addition, we show that IPMix also
significantly improves the other safety measures, including robustness to
adversarial perturbations, calibration, prediction consistency, and anomaly
detection, achieving state-of-the-art or comparable results on several
benchmarks, including ImageNet-R, ImageNet-A, and ImageNet-O.


http://arxiv.org/abs/2310.04655
VLAttack: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models. (98%)
Ziyi Yin; Muchao Ye; Tianrong Zhang; Tianyu Du; Jinguo Zhu; Han Liu; Jinghui Chen; Ting Wang; Fenglong Ma
  Vision-Language (VL) pre-trained models have shown their superiority on many
multimodal tasks. However, the adversarial robustness of such models has not
been fully explored. Existing approaches mainly focus on exploring the
adversarial robustness under the white-box setting, which is unrealistic. In
this paper, we aim to investigate a new yet practical task to craft image and
text perturbations using pre-trained VL models to attack black-box fine-tuned
models on different downstream tasks. Towards this end, we propose VLAttack to
generate adversarial samples by fusing perturbations of images and texts from
both single-modal and multimodal levels. At the single-modal level, we propose
a new block-wise similarity attack (BSA) strategy to learn image perturbations
for disrupting universal representations. Besides, we adopt an existing text
attack strategy to generate text perturbations independent of the image-modal
attack. At the multimodal level, we design a novel iterative cross-search
attack (ICSA) method to update adversarial image-text pairs periodically,
starting with the outputs from the single-modal level. We conduct extensive
experiments to attack three widely-used VL pretrained models for six tasks on
eight datasets. Experimental results show that the proposed VLAttack framework
achieves the highest attack success rates on all tasks compared with
state-of-the-art baselines, which reveals a significant blind spot in the
deployment of pre-trained VL models. Codes will be released soon.


http://arxiv.org/abs/2310.03707
OMG-ATTACK: Self-Supervised On-Manifold Generation of Transferable Evasion Attacks. (99%)
Ofir Bar Tal; Adi Haviv; Amit H. Bermano
  Evasion Attacks (EA) are used to test the robustness of trained neural
networks by distorting input data to misguide the model into incorrect
classifications. Creating these attacks is a challenging task, especially with
the ever-increasing complexity of models and datasets. In this work, we
introduce a self-supervised, computationally economical method for generating
adversarial examples, designed for the unseen black-box setting. Adapting
techniques from representation learning, our method generates on-manifold EAs
that are encouraged to resemble the data distribution. These attacks are
comparable in effectiveness compared to the state-of-the-art when attacking the
model trained on, but are significantly more effective when attacking unseen
models, as the attacks are more related to the data rather than the model
itself. Our experiments consistently demonstrate the method is effective across
various models, unseen data categories, and even defended models, suggesting a
significant role for on-manifold EAs when targeting unseen models.


http://arxiv.org/abs/2310.03334
Untargeted White-box Adversarial Attack with Heuristic Defence Methods in Real-time Deep Learning based Network Intrusion Detection System. (99%)
Khushnaseeb Roshan; Aasim Zafar; Sheikh Burhan Ul Haque
  Network Intrusion Detection System (NIDS) is a key component in securing the
computer network from various cyber security threats and network attacks.
However, consider an unfortunate situation where the NIDS is itself attacked
and vulnerable more specifically, we can say, How to defend the defender?. In
Adversarial Machine Learning (AML), the malicious actors aim to fool the
Machine Learning (ML) and Deep Learning (DL) models to produce incorrect
predictions with intentionally crafted adversarial examples. These adversarial
perturbed examples have become the biggest vulnerability of ML and DL based
systems and are major obstacles to their adoption in real-time and
mission-critical applications such as NIDS. AML is an emerging research domain,
and it has become a necessity for the in-depth study of adversarial attacks and
their defence strategies to safeguard the computer network from various cyber
security threads. In this research work, we aim to cover important aspects
related to NIDS, adversarial attacks and its defence mechanism to increase the
robustness of the ML and DL based NIDS. We implemented four powerful
adversarial attack techniques, namely, Fast Gradient Sign Method (FGSM),
Jacobian Saliency Map Attack (JSMA), Projected Gradient Descent (PGD) and
Carlini & Wagner (C&W) in NIDS. We analyzed its performance in terms of various
performance metrics in detail. Furthermore, the three heuristics defence
strategies, i.e., Adversarial Training (AT), Gaussian Data Augmentation (GDA)
and High Confidence (HC), are implemented to improve the NIDS robustness under
adversarial attack situations. The complete workflow is demonstrated in
real-time network with data packet flow. This research work provides the
overall background for the researchers interested in AML and its implementation
from a computer network security point of view.


http://arxiv.org/abs/2310.03358
Enhancing Robust Representation in Adversarial Training: Alignment and Exclusion Criteria. (99%)
Nuoyan Zhou; Nannan Wang; Decheng Liu; Dawei Zhou; Xinbo Gao
  Deep neural networks are vulnerable to adversarial noise. Adversarial
Training (AT) has been demonstrated to be the most effective defense strategy
to protect neural networks from being fooled. However, we find AT omits to
learning robust features, resulting in poor performance of adversarial
robustness. To address this issue, we highlight two criteria of robust
representation: (1) Exclusion: \emph{the feature of examples keeps away from
that of other classes}; (2) Alignment: \emph{the feature of natural and
corresponding adversarial examples is close to each other}. These motivate us
to propose a generic framework of AT to gain robust representation, by the
asymmetric negative contrast and reverse attention. Specifically, we design an
asymmetric negative contrast based on predicted probabilities, to push away
examples of different classes in the feature space. Moreover, we propose to
weight feature by parameters of the linear classifier as the reverse attention,
to obtain class-aware feature and pull close the feature of the same class.
Empirical evaluations on three benchmark datasets show our methods greatly
advance the robustness of AT and achieve state-of-the-art performance.


http://arxiv.org/abs/2310.03349
An Integrated Algorithm for Robust and Imperceptible Audio Adversarial Examples. (98%)
Armin Ettenhofer; Jan-Philipp Schulze; Karla Pizzi
  Audio adversarial examples are audio files that have been manipulated to fool
an automatic speech recognition (ASR) system, while still sounding benign to a
human listener. Most methods to generate such samples are based on a two-step
algorithm: first, a viable adversarial audio file is produced, then, this is
fine-tuned with respect to perceptibility and robustness. In this work, we
present an integrated algorithm that uses psychoacoustic models and room
impulse responses (RIR) in the generation step. The RIRs are dynamically
created by a neural network during the generation process to simulate a
physical environment to harden our examples against transformations experienced
in over-the-air attacks. We compare the different approaches in three
experiments: in a simulated environment and in a realistic over-the-air
scenario to evaluate the robustness, and in a human study to evaluate the
perceptibility. Our algorithms considering psychoacoustics only or in addition
to the robustness show an improvement in the signal-to-noise ratio (SNR) as
well as in the human perception study, at the cost of an increased word error
rate (WER).


http://arxiv.org/abs/2310.03614
Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally. (98%)
Shawqi Al-Maliki; Adnan Qayyum; Hassan Ali; Mohamed Abdallah; Junaid Qadir; Dinh Thai Hoang; Dusit Niyato; Ala Al-Fuqaha
  Deep Neural Networks (DNNs) have been the driving force behind many of the
recent advances in machine learning. However, research has shown that DNNs are
vulnerable to adversarial examples -- input samples that have been perturbed to
force DNN-based models to make errors. As a result, Adversarial Machine
Learning (AdvML) has gained a lot of attention, and researchers have
investigated these vulnerabilities in various settings and modalities. In
addition, DNNs have also been found to incorporate embedded bias and often
produce unexplainable predictions, which can result in anti-social AI
applications. The emergence of new AI technologies that leverage Large Language
Models (LLMs), such as ChatGPT and GPT-4, increases the risk of producing
anti-social applications at scale. AdvML for Social Good (AdvML4G) is an
emerging field that repurposes the AdvML bug to invent pro-social applications.
Regulators, practitioners, and researchers should collaborate to encourage the
development of pro-social applications and hinder the development of
anti-social ones. In this work, we provide the first comprehensive review of
the emerging field of AdvML4G. This paper encompasses a taxonomy that
highlights the emergence of AdvML4G, a discussion of the differences and
similarities between AdvML4G and AdvML, a taxonomy covering social good-related
concepts and aspects, an exploration of the motivations behind the emergence of
AdvML4G at the intersection of ML4G and AdvML, and an extensive summary of the
works that utilize AdvML4G as an auxiliary tool for innovating pro-social
applications. Finally, we elaborate upon various challenges and open research
issues that require significant attention from the research community.


http://arxiv.org/abs/2310.03684
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks. (87%)
Alexander Robey; Eric Wong; Hamed Hassani; George J. Pappas
  Despite efforts to align large language models (LLMs) with human values,
widely-used LLMs such as GPT, Llama, Claude, and PaLM are susceptible to
jailbreaking attacks, wherein an adversary fools a targeted LLM into generating
objectionable content. To address this vulnerability, we propose SmoothLLM, the
first algorithm designed to mitigate jailbreaking attacks on LLMs. Based on our
finding that adversarially-generated prompts are brittle to character-level
changes, our defense first randomly perturbs multiple copies of a given input
prompt, and then aggregates the corresponding predictions to detect adversarial
inputs. SmoothLLM reduces the attack success rate on numerous popular LLMs to
below one percentage point, avoids unnecessary conservatism, and admits
provable guarantees on attack mitigation. Moreover, our defense uses
exponentially fewer queries than existing attacks and is compatible with any
LLM.


http://arxiv.org/abs/2310.03578
Targeted Adversarial Attacks on Generalizable Neural Radiance Fields. (56%)
Andras Horvath; Csaba M. Jozsa
  Neural Radiance Fields (NeRFs) have recently emerged as a powerful tool for
3D scene representation and rendering. These data-driven models can learn to
synthesize high-quality images from sparse 2D observations, enabling realistic
and interactive scene reconstructions. However, the growing usage of NeRFs in
critical applications such as augmented reality, robotics, and virtual
environments could be threatened by adversarial attacks.
  In this paper we present how generalizable NeRFs can be attacked by both
low-intensity adversarial attacks and adversarial patches, where the later
could be robust enough to be used in real world applications. We also
demonstrate targeted attacks, where a specific, predefined output scene is
generated by these attack with success.


http://arxiv.org/abs/2310.03664
Certification of Deep Learning Models for Medical Image Segmentation. (15%)
Othmane Laousy; Alexandre Araujo; Guillaume Chassagnon; Nikos Paragios; Marie-Pierre Revel; Maria Vakalopoulou
  In medical imaging, segmentation models have known a significant improvement
in the past decade and are now used daily in clinical practice. However,
similar to classification models, segmentation models are affected by
adversarial attacks. In a safety-critical field like healthcare, certifying
model predictions is of the utmost importance. Randomized smoothing has been
introduced lately and provides a framework to certify models and obtain
theoretical guarantees. In this paper, we present for the first time a
certified segmentation baseline for medical imaging based on randomized
smoothing and diffusion models. Our results show that leveraging the power of
denoising diffusion probabilistic models helps us overcome the limits of
randomized smoothing. We conduct extensive experiments on five public datasets
of chest X-rays, skin lesions, and colonoscopies, and empirically show that we
are able to maintain high certified Dice scores even for highly perturbed
images. Our work represents the first attempt to certify medical image
segmentation models, and we aspire for it to set a foundation for future
benchmarks in this crucial and largely uncharted area.


http://arxiv.org/abs/2310.03312
Certifiably Robust Graph Contrastive Learning. (5%)
Minhua Lin; Teng Xiao; Enyan Dai; Xiang Zhang; Suhang Wang
  Graph Contrastive Learning (GCL) has emerged as a popular unsupervised graph
representation learning method. However, it has been shown that GCL is
vulnerable to adversarial attacks on both the graph structure and node
attributes. Although empirical approaches have been proposed to enhance the
robustness of GCL, the certifiable robustness of GCL is still remain
unexplored. In this paper, we develop the first certifiably robust framework in
GCL. Specifically, we first propose a unified criteria to evaluate and certify
the robustness of GCL. We then introduce a novel technique, RES (Randomized
Edgedrop Smoothing), to ensure certifiable robustness for any GCL model, and
this certified robustness can be provably preserved in downstream tasks.
Furthermore, an effective training method is proposed for robust GCL. Extensive
experiments on real-world datasets demonstrate the effectiveness of our
proposed method in providing effective certifiable robustness and enhancing the
robustness of any GCL model. The source code of RES is available at
https://github.com/ventr1c/RES-GCL.


http://arxiv.org/abs/2310.03518
Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations. (2%)
Jiachi Liu; Liwen Wang; Guanting Dong; Xiaoshuai Song; Zechen Wang; Zhengyang Wang; Shanglin Lei; Jinzheng Zhao; Keqing He; Bo Xiao; Weiran Xu
  In real dialogue scenarios, as there are unknown input noises in the
utterances, existing supervised slot filling models often perform poorly in
practical applications. Even though there are some studies on noise-robust
models, these works are only evaluated on rule-based synthetic datasets, which
is limiting, making it difficult to promote the research of noise-robust
methods. In this paper, we introduce a noise robustness evaluation dataset
named Noise-SF for slot filling task. The proposed dataset contains five types
of human-annotated noise, and all those noises are exactly existed in real
extensive robust-training methods of slot filling into the proposed framework.
By conducting exhaustive empirical evaluation experiments on Noise-SF, we find
that baseline models have poor performance in robustness evaluation, and the
proposed framework can effectively improve the robustness of models. Based on
the empirical experimental results, we make some forward-looking suggestions to
fuel the research in this direction. Our dataset Noise-SF will be released at
https://github.com/dongguanting/Noise-SF.


http://arxiv.org/abs/2310.02997
Optimizing Key-Selection for Face-based One-Time Biometrics via Morphing. (98%)
Daile Osorio-Roig; Mahdi Ghafourian; Christian Rathgeb; Ruben Vera-Rodriguez; Christoph Busch; Julian Fierrez
  Nowadays, facial recognition systems are still vulnerable to adversarial
attacks. These attacks vary from simple perturbations of the input image to
modifying the parameters of the recognition model to impersonate an authorised
subject. So-called privacy-enhancing facial recognition systems have been
mostly developed to provide protection of stored biometric reference data, i.e.
templates. In the literature, privacy-enhancing facial recognition approaches
have focused solely on conventional security threats at the template level,
ignoring the growing concern related to adversarial attacks. Up to now, few
works have provided mechanisms to protect face recognition against adversarial
attacks while maintaining high security at the template level. In this paper,
we propose different key selection strategies to improve the security of a
competitive cancelable scheme operating at the signal level. Experimental
results show that certain strategies based on signal-level key selection can
lead to complete blocking of the adversarial attack based on an iterative
optimization for the most secure threshold, while for the most practical
threshold, the attack success chance can be decreased to approximately 5.0%.


http://arxiv.org/abs/2310.03185
Misusing Tools in Large Language Models With Visual Adversarial Examples. (97%)
Xiaohan Fu; Zihan Wang; Shuheng Li; Rajesh K. Gupta; Niloofar Mireshghallah; Taylor Berg-Kirkpatrick; Earlence Fernandes
  Large Language Models (LLMs) are being enhanced with the ability to use tools
and to process multiple modalities. These new capabilities bring new benefits
and also new security risks. In this work, we show that an attacker can use
visual adversarial examples to cause attacker-desired tool usage. For example,
the attacker could cause a victim LLM to delete calendar events, leak private
conversations and book hotels. Different from prior work, our attacks can
affect the confidentiality and integrity of user resources connected to the LLM
while being stealthy and generalizable to multiple input prompts. We construct
these attacks using gradient-based adversarial training and characterize
performance along multiple dimensions. We find that our adversarial images can
manipulate the LLM to invoke tools following real-world syntax almost always
(~98%) while maintaining high similarity to clean images (~0.9 SSIM).
Furthermore, using human scoring and automated metrics, we find that the
attacks do not noticeably affect the conversation (and its semantics) between
the user and the LLM.


http://arxiv.org/abs/2310.03285
Burning the Adversarial Bridges: Robust Windows Malware Detection Against Binary-level Mutations. (82%)
Ahmed Abusnaina; Yizhen Wang; Sunpreet Arora; Ke Wang; Mihai Christodorescu; David Mohaisen
  Toward robust malware detection, we explore the attack surface of existing
malware detection systems. We conduct root-cause analyses of the practical
binary-level black-box adversarial malware examples. Additionally, we uncover
the sensitivity of volatile features within the detection engines and exhibit
their exploitability. Highlighting volatile information channels within the
software, we introduce three software pre-processing steps to eliminate the
attack surface, namely, padding removal, software stripping, and inter-section
information resetting. Further, to counter the emerging section injection
attacks, we propose a graph-based section-dependent information extraction
scheme for software representation. The proposed scheme leverages aggregated
information within various sections in the software to enable robust malware
detection and mitigate adversarial settings. Our experimental results show that
traditional malware detection models are ineffective against adversarial
threats. However, the attack surface can be largely reduced by eliminating the
volatile information. Therefore, we propose simple-yet-effective methods to
mitigate the impacts of binary manipulation attacks. Overall, our graph-based
malware detection scheme can accurately detect malware with an area under the
curve score of 88.32\% and a score of 88.19% under a combination of binary
manipulation attacks, exhibiting the efficiency of our proposed scheme.


http://arxiv.org/abs/2310.03166
Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors. (81%)
Biagio Montaruli; Luca Demetrio; Maura Pintor; Luca Compagna; Davide Balzarotti; Battista Biggio
  Machine-learning phishing webpage detectors (ML-PWD) have been shown to
suffer from adversarial manipulations of the HTML code of the input webpage.
Nevertheless, the attacks recently proposed have demonstrated limited
effectiveness due to their lack of optimizing the usage of the adopted
manipulations, and they focus solely on specific elements of the HTML code. In
this work, we overcome these limitations by first designing a novel set of
fine-grained manipulations which allow to modify the HTML code of the input
phishing webpage without compromising its maliciousness and visual appearance,
i.e., the manipulations are functionality- and rendering-preserving by design.
We then select which manipulations should be applied to bypass the target
detector by a query-efficient black-box optimization algorithm. Our experiments
show that our attacks are able to raze to the ground the performance of current
state-of-the-art ML-PWD using just 30 queries, thus overcoming the weaker
attacks developed in previous work, and enabling a much fairer robustness
evaluation of ML-PWD.


http://arxiv.org/abs/2310.03125
Shielding the Unseen: Privacy Protection through Poisoning NeRF with Spatial Deformation. (10%)
Yihan Wu; Brandon Y. Feng; Heng Huang
  In this paper, we introduce an innovative method of safeguarding user privacy
against the generative capabilities of Neural Radiance Fields (NeRF) models.
Our novel poisoning attack method induces changes to observed views that are
imperceptible to the human eye, yet potent enough to disrupt NeRF's ability to
accurately reconstruct a 3D scene. To achieve this, we devise a bi-level
optimization algorithm incorporating a Projected Gradient Descent (PGD)-based
spatial deformation. We extensively test our approach on two common NeRF
benchmark datasets consisting of 29 real-world scenes with high-quality images.
Our results compellingly demonstrate that our privacy-preserving method
significantly impairs NeRF's performance across these benchmark datasets.
Additionally, we show that our method is adaptable and versatile, functioning
across various perturbation strengths and NeRF architectures. This work offers
valuable insights into NeRF's vulnerabilities and emphasizes the need to
account for such potential privacy risks when developing robust 3D scene
reconstruction algorithms. Our study contributes to the larger conversation
surrounding responsible AI and generative machine learning, aiming to protect
user privacy and respect creative ownership in the digital age.


http://arxiv.org/abs/2310.02480
Splitting the Difference on Adversarial Training. (99%)
Matan Levi; Aryeh Kontorovich
  The existence of adversarial examples points to a basic weakness of deep
neural networks. One of the most effective defenses against such examples,
adversarial training, entails training models with some degree of robustness,
usually at the expense of a degraded natural accuracy. Most adversarial
training methods aim to learn a model that finds, for each class, a common
decision boundary encompassing both the clean and perturbed examples. In this
work, we take a fundamentally different approach by treating the perturbed
examples of each class as a separate class to be learned, effectively splitting
each class into two classes: "clean" and "adversarial." This split doubles the
number of classes to be learned, but at the same time considerably simplifies
the decision boundaries. We provide a theoretical plausibility argument that
sheds some light on the conditions under which our approach can be expected to
be beneficial. Likewise, we empirically demonstrate that our method learns
robust models while attaining optimal or near-optimal natural accuracy, e.g.,
on CIFAR-10 we obtain near-optimal natural accuracy of $95.01\%$ alongside
significant robustness across multiple tasks. The ability to achieve such
near-optimal natural accuracy, while maintaining a significant level of
robustness, makes our method applicable to real-world applications where
natural accuracy is at a premium. As a whole, our main contribution is a
general method that confers a significant level of robustness upon classifiers
with only minor or negligible degradation of their natural accuracy.


http://arxiv.org/abs/2310.02025
DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training. (96%)
Aochuan Chen; Yimeng Zhang; Jinghan Jia; James Diffenderfer; Jiancheng Liu; Konstantinos Parasyris; Yihua Zhang; Zheng Zhang; Bhavya Kailkhura; Sijia Liu
  Zeroth-order (ZO) optimization has become a popular technique for solving
machine learning (ML) problems when first-order (FO) information is difficult
or impossible to obtain. However, the scalability of ZO optimization remains an
open problem: Its use has primarily been limited to relatively small-scale ML
problems, such as sample-wise adversarial attack generation. To our best
knowledge, no prior work has demonstrated the effectiveness of ZO optimization
in training deep neural networks (DNNs) without a significant decrease in
performance. To overcome this roadblock, we develop DeepZero, a principled ZO
deep learning (DL) framework that can scale ZO optimization to DNN training
from scratch through three primary innovations. First, we demonstrate the
advantages of coordinate-wise gradient estimation (CGE) over randomized
vector-wise gradient estimation in training accuracy and computational
efficiency. Second, we propose a sparsity-induced ZO training protocol that
extends the model pruning methodology using only finite differences to explore
and exploit the sparse DL prior in CGE. Third, we develop the methods of
feature reuse and forward parallelization to advance the practical
implementations of ZO training. Our extensive experiments show that DeepZero
achieves state-of-the-art (SOTA) accuracy on ResNet-20 trained on CIFAR-10,
approaching FO training performance for the first time. Furthermore, we show
the practical utility of DeepZero in applications of certified adversarial
defense and DL-based partial differential equation error correction, achieving
10-20% improvement over SOTA. We believe our results will inspire future
research on scalable ZO optimization and contribute to advancing DL with black
box.


http://arxiv.org/abs/2310.02544
SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformers. (86%)
KL Navaneet; Soroush Abbasi Koohpayegani; Essam Sleiman; Hamed Pirsiavash
  Recently, there has been a lot of progress in reducing the computation of
deep models at inference time. These methods can reduce both the computational
needs and power usage of deep models. Some of these approaches adaptively scale
the compute based on the input instance. We show that such models can be
vulnerable to a universal adversarial patch attack, where the attacker
optimizes for a patch that when pasted on any image, can increase the compute
and power consumption of the model. We run experiments with three different
efficient vision transformer methods showing that in some cases, the attacker
can increase the computation to the maximum possible level by simply pasting a
patch that occupies only 8\% of the image area. We also show that a standard
adversarial training defense method can reduce some of the attack's success. We
believe adaptive efficient methods will be necessary for the future to lower
the power usage of deep models, so we hope our paper encourages the community
to study the robustness of these methods and develop better defense methods for
the proposed attack.


http://arxiv.org/abs/2310.01875
Towards Stable Backdoor Purification through Feature Shift Tuning. (83%)
Rui Min; Zeyu Qin; Li Shen; Minhao Cheng
  It has been widely observed that deep neural networks (DNN) are vulnerable to
backdoor attacks where attackers could manipulate the model behavior
maliciously by tampering with a small set of training samples. Although a line
of defense methods is proposed to mitigate this threat, they either require
complicated modifications to the training process or heavily rely on the
specific model architecture, which makes them hard to deploy into real-world
applications. Therefore, in this paper, we instead start with fine-tuning, one
of the most common and easy-to-deploy backdoor defenses, through comprehensive
evaluations against diverse attack scenarios. Observations made through initial
experiments show that in contrast to the promising defensive results on high
poisoning rates, vanilla tuning methods completely fail at low poisoning rate
scenarios. Our analysis shows that with the low poisoning rate, the
entanglement between backdoor and clean features undermines the effect of
tuning-based defenses. Therefore, it is necessary to disentangle the backdoor
and clean features in order to improve backdoor purification. To address this,
we introduce Feature Shift Tuning (FST), a method for tuning-based backdoor
purification. Specifically, FST encourages feature shifts by actively deviating
the classifier weights from the originally compromised weights. Extensive
experiments demonstrate that our FST provides consistently stable performance
under different attack settings. Additionally, it is also convenient to deploy
in real-world scenarios with significantly reduced computation costs. Our codes
are available at
\url{https://github.com/AISafety-HKUST/stable_backdoor_purification}.


http://arxiv.org/abs/2310.02417
Jailbreaker in Jail: Moving Target Defense for Large Language Models. (73%)
Bocheng Chen; Advait Paliwal; Qiben Yan
  Large language models (LLMs), known for their capability in understanding and
following instructions, are vulnerable to adversarial attacks. Researchers have
found that current commercial LLMs either fail to be "harmless" by presenting
unethical answers, or fail to be "helpful" by refusing to offer meaningful
answers when faced with adversarial queries. To strike a balance between being
helpful and harmless, we design a moving target defense (MTD) enhanced LLM
system. The system aims to deliver non-toxic answers that align with outputs
from multiple model candidates, making them more robust against adversarial
attacks. We design a query and output analysis model to filter out unsafe or
non-responsive answers. %to achieve the two objectives of randomly selecting
outputs from different LLMs. We evaluate over 8 most recent chatbot models with
state-of-the-art adversarial queries. Our MTD-enhanced LLM system reduces the
attack success rate from 37.5\% to 0\%. Meanwhile, it decreases the response
refusal rate from 50\% to 0\%.


http://arxiv.org/abs/2310.01959
Beyond Labeling Oracles: What does it mean to steal ML models? (47%)
Avital Shafran; Ilia Shumailov; Murat A. Erdogdu; Nicolas Papernot
  Model extraction attacks are designed to steal trained models with only query
access, as is often provided through APIs that ML-as-a-Service providers offer.
ML models are expensive to train, in part because data is hard to obtain, and a
primary incentive for model extraction is to acquire a model while incurring
less cost than training from scratch. Literature on model extraction commonly
claims or presumes that the attacker is able to save on both data acquisition
and labeling costs. We show that the attacker often does not. This is because
current attacks implicitly rely on the adversary being able to sample from the
victim model's data distribution. We thoroughly evaluate factors influencing
the success of model extraction. We discover that prior knowledge of the
attacker, i.e. access to in-distribution data, dominates other factors like the
attack policy the adversary follows to choose which queries to make to the
victim model API. Thus, an adversary looking to develop an equally capable
model with a fixed budget has little practical incentive to perform model
extraction, since for the attack to work they need to collect in-distribution
data, saving only on the cost of labeling. With low labeling costs in the
current market, the usefulness of such attacks is questionable. Ultimately, we
demonstrate that the effect of prior knowledge needs to be explicitly decoupled
from the attack policy. To this end, we propose a benchmark to evaluate attack
policy directly.


http://arxiv.org/abs/2310.02237
Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness. (13%)
Yanzhao Wu; Ka-Ho Chow; Wenqi Wei; Ling Liu
  Deep neural network ensembles hold the potential of improving generalization
performance for complex learning tasks. This paper presents formal analysis and
empirical evaluation to show that heterogeneous deep ensembles with high
ensemble diversity can effectively leverage model learning heterogeneity to
boost ensemble robustness. We first show that heterogeneous DNN models trained
for solving the same learning problem, e.g., object detection, can
significantly strengthen the mean average precision (mAP) through our weighted
bounding box ensemble consensus method. Second, we further compose ensembles of
heterogeneous models for solving different learning problems, e.g., object
detection and semantic segmentation, by introducing the connected component
labeling (CCL) based alignment. We show that this two-tier heterogeneity driven
ensemble construction method can compose an ensemble team that promotes high
ensemble diversity and low negative correlation among member models of the
ensemble, strengthening ensemble robustness against both negative examples and
adversarial attacks. Third, we provide a formal analysis of the ensemble
robustness in terms of negative correlation. Extensive experiments validate the
enhanced robustness of heterogeneous ensembles in both benign and adversarial
settings. The source codes are available on GitHub at
https://github.com/git-disl/HeteRobust.


http://arxiv.org/abs/2310.02113
FLEDGE: Ledger-based Federated Learning Resilient to Inference and Backdoor Attacks. (11%)
Jorge Castillo; Phillip Rieger; Hossein Fereidooni; Qian Chen; Ahmad Sadeghi
  Federated learning (FL) is a distributed learning process that uses a trusted
aggregation server to allow multiple parties (or clients) to collaboratively
train a machine learning model without having them share their private data.
Recent research, however, has demonstrated the effectiveness of inference and
poisoning attacks on FL. Mitigating both attacks simultaneously is very
challenging. State-of-the-art solutions have proposed the use of poisoning
defenses with Secure Multi-Party Computation (SMPC) and/or Differential Privacy
(DP). However, these techniques are not efficient and fail to address the
malicious intent behind the attacks, i.e., adversaries (curious servers and/or
compromised clients) seek to exploit a system for monetization purposes. To
overcome these limitations, we present a ledger-based FL framework known as
FLEDGE that allows making parties accountable for their behavior and achieve
reasonable efficiency for mitigating inference and poisoning attacks. Our
solution leverages crypto-currency to increase party accountability by
penalizing malicious behavior and rewarding benign conduct. We conduct an
extensive evaluation on four public datasets: Reddit, MNIST, Fashion-MNIST, and
CIFAR-10. Our experimental results demonstrate that (1) FLEDGE provides strong
privacy guarantees for model updates without sacrificing model utility; (2)
FLEDGE can successfully mitigate different poisoning attacks without degrading
the performance of the global model; and (3) FLEDGE offers unique reward
mechanisms to promote benign behavior during model training and/or model
aggregation.


http://arxiv.org/abs/2310.01818
AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Framework. (3%)
Xilie Xu; Jingfeng Zhang; Mohan Kankanhalli
  Robust Fine-Tuning (RFT) is a low-cost strategy to obtain adversarial
robustness in downstream applications, without requiring a lot of computational
resources and collecting significant amounts of data. This paper uncovers an
issue with the existing RFT, where optimizing both adversarial and natural
objectives through the feature extractor (FE) yields significantly divergent
gradient directions. This divergence introduces instability in the optimization
process, thereby hindering the attainment of adversarial robustness and
rendering RFT highly sensitive to hyperparameters. To mitigate this issue, we
propose a low-rank (LoRa) branch that disentangles RFT into two distinct
components: optimizing natural objectives via the LoRa branch and adversarial
objectives via the FE. Besides, we introduce heuristic strategies for
automating the scheduling of the learning rate and the scalars of loss terms.
Extensive empirical evaluations demonstrate that our proposed automated RFT
disentangled via the LoRa branch (AutoLoRa) achieves new state-of-the-art
results across a range of downstream tasks. AutoLoRa holds significant
practical utility, as it automatically converts a pre-trained FE into an
adversarially robust model for downstream tasks without the need for searching
hyperparameters.


http://arxiv.org/abs/2310.01452
Fooling the Textual Fooler via Randomizing Latent Representations. (99%)
Duy C. Hoang; Quang H. Nguyen; Saurav Manchanda; MinLong Peng; Kok-Seng Wong; Khoa D. Doan
  Despite outstanding performance in a variety of NLP tasks, recent studies
have revealed that NLP models are vulnerable to adversarial attacks that
slightly perturb the input to cause the models to misbehave. Among these
attacks, adversarial word-level perturbations are well-studied and effective
attack strategies. Since these attacks work in black-box settings, they do not
require access to the model architecture or model parameters and thus can be
detrimental to existing NLP applications. To perform an attack, the adversary
queries the victim model many times to determine the most important words in an
input text and to replace these words with their corresponding synonyms. In
this work, we propose a lightweight and attack-agnostic defense whose main goal
is to perplex the process of generating an adversarial example in these
query-based black-box attacks; that is to fool the textual fooler. This
defense, named AdvFooler, works by randomizing the latent representation of the
input at inference time. Different from existing defenses, AdvFooler does not
necessitate additional computational overhead during training nor relies on
assumptions about the potential adversarial perturbation set while having a
negligible impact on the model's accuracy. Our theoretical and empirical
analyses highlight the significance of robustness resulting from confusing the
adversary via randomizing the latent space, as well as the impact of
randomization on clean accuracy. Finally, we empirically demonstrate near
state-of-the-art robustness of AdvFooler against representative adversarial
word-level attacks on two benchmark datasets.


http://arxiv.org/abs/2310.01537
Adversarial Client Detection via Non-parametric Subspace Monitoring in the Internet of Federated Things. (92%)
Xianjian Xie; Xiaochen Xian; Dan Li; Andi Wang
  The Internet of Federated Things (IoFT) represents a network of
interconnected systems with federated learning as the backbone, facilitating
collaborative knowledge acquisition while ensuring data privacy for individual
systems. The wide adoption of IoFT, however, is hindered by security concerns,
particularly the susceptibility of federated learning networks to adversarial
attacks. In this paper, we propose an effective non-parametric approach FedRR,
which leverages the low-rank features of the transmitted parameter updates
generated by federated learning to address the adversarial attack problem.
Besides, our proposed method is capable of accurately detecting adversarial
clients and controlling the false alarm rate under the scenario with no attack
occurring. Experiments based on digit recognition using the MNIST datasets
validated the advantages of our approach.


http://arxiv.org/abs/2310.04445
LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model. (87%)
Muhammad Ahmed Shah; Roshan Sharma; Hira Dhamyal; Raphael Olivier; Ankit Shah; Joseph Konan; Dareen Alharthi; Hazim T Bukhari; Massa Baali; Soham Deshmukh; Michael Kuhlmann; Bhiksha Raj; Rita Singh
  It has been shown that Large Language Model (LLM) alignments can be
circumvented by appending specially crafted attack suffixes with harmful
queries to elicit harmful responses. To conduct attacks against private target
models whose characterization is unknown, public models can be used as proxies
to fashion the attack, with successful attacks being transferred from public
proxies to private target models. The success rate of attack depends on how
closely the proxy model approximates the private model. We hypothesize that for
attacks to be transferrable, it is sufficient if the proxy can approximate the
target model in the neighborhood of the harmful query. Therefore, in this
paper, we propose \emph{Local Fine-Tuning (LoFT)}, \textit{i.e.}, fine-tuning
proxy models on similar queries that lie in the lexico-semantic neighborhood of
harmful queries to decrease the divergence between the proxy and target models.
First, we demonstrate three approaches to prompt private target models to
obtain similar queries given harmful queries. Next, we obtain data for local
fine-tuning by eliciting responses from target models for the generated similar
queries. Then, we optimize attack suffixes to generate attack prompts and
evaluate the impact of our local fine-tuning on the attack's success rate.
Experiments show that local fine-tuning of proxy models improves attack
transferability and increases attack success rate by $39\%$, $7\%$, and $0.5\%$
(absolute) on target models ChatGPT, GPT-4, and Claude respectively.


http://arxiv.org/abs/2310.01469
LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples. (87%)
Jia-Yu Yao; Kun-Peng Ning; Zhen-Hui Liu; Mu-Nan Ning; Li Yuan
  Large Language Models (LLMs), including GPT-3.5, LLaMA, and PaLM, seem to be
knowledgeable and able to adapt to many tasks. However, we still can not
completely trust their answer, since LLMs suffer from
hallucination--fabricating non-existent facts to cheat users without
perception. And the reasons for their existence and pervasiveness remain
unclear. In this paper, we demonstrate that non-sense prompts composed of
random tokens can also elicit the LLMs to respond with hallucinations. This
phenomenon forces us to revisit that hallucination may be another view of
adversarial examples, and it shares similar features with conventional
adversarial examples as the basic feature of LLMs. Therefore, we formalize an
automatic hallucination triggering method as the hallucination attack in an
adversarial way. Finally, we explore basic feature of attacked adversarial
prompts and propose a simple yet effective defense strategy. Our code is
released on GitHub.


http://arxiv.org/abs/2310.01166
Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models. (13%)
Zhou Yang; Zhipeng Zhao; Chenyu Wang; Jieke Shi; Dongsum Kim; Donggyun Han; David Lo
  Given large-scale source code datasets available in open-source projects and
advanced large language models, recent code models have been proposed to
address a series of critical software engineering tasks, such as program repair
and code completion. The training data of the code models come from various
sources, not only the publicly available source code, e.g., open-source
projects on GitHub but also the private data such as the confidential source
code from companies, which may contain sensitive information (for example, SSH
keys and personal information). As a result, the use of these code models may
raise new privacy concerns.
  In this paper, we focus on a critical yet not well-explored question on using
code models: what is the risk of membership information leakage in code models?
Membership information leakage refers to the risk that an attacker can infer
whether a given data point is included in (i.e., a member of) the training
data. To answer this question, we propose Gotcha, a novel membership inference
attack method specifically for code models. We investigate the membership
leakage risk of code models. Our results reveal a worrying fact that the risk
of membership leakage is high: although the previous attack methods are close
to random guessing, Gotcha can predict the data membership with a high true
positive rate of 0.95 and a low false positive rate of 0.10. We also show that
the attacker's knowledge of the victim model (e.g., the model architecture and
the pre-training data) impacts the success rate of attacks. Further analysis
demonstrates that changing the decoding strategy can mitigate the risk of
membership leakage. This study calls for more attention to understanding the
privacy of code models and developing more effective countermeasures against
such attacks.


http://arxiv.org/abs/2311.12832
Toward effective protection against diffusion based mimicry through score distillation. (3%)
Haotian Xue; Chumeng Liang; Xiaoyu Wu; Yongxin Chen
  While generative diffusion models excel in producing high-quality images,
they can also be misused to mimic authorized images, posing a significant
threat to AI systems. Efforts have been made to add calibrated perturbations to
protect images from diffusion-based mimicry pipelines. However, most of the
existing methods are too ineffective and even impractical to be used by
individual users due to their high computation and memory requirements. In this
work, we present novel findings on attacking latent diffusion models (LDM) and
propose new plug-and-play strategies for more effective protection. In
particular, we explore the bottleneck in attacking an LDM, discovering that the
encoder module rather than the denoiser module is the vulnerable point. Based
on this insight, we present our strategy using Score Distillation Sampling
(SDS) to double the speed of protection and reduce memory occupation by half
without compromising its strength. Additionally, we provide a robust protection
strategy by counterintuitively minimizing the semantic loss, which can assist
in generating more natural perturbations. Finally, we conduct extensive
experiments to substantiate our findings and comprehensively evaluate our newly
proposed strategies. We hope our insights and protective measures can
contribute to better defense against malicious diffusion-based mimicry,
advancing the development of secure AI systems. The code is available in
https://github.com/xavihart/Diff-Protect


http://arxiv.org/abs/2310.01651
Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations. (1%)
Yongshuo Zong; Tingyang Yu; Bingchen Zhao; Ruchika Chavhan; Timothy Hospedales
  Large language and vision-language models are rapidly being deployed in
practice thanks to their impressive capabilities in instruction following,
in-context learning, and so on. This raises an urgent need to carefully analyse
their robustness so that stakeholders can understand if and when such models
are trustworthy enough to be relied upon in any given application. In this
paper, we highlight a specific vulnerability in popular models, namely
permutation sensitivity in multiple-choice question answering (MCQA).
Specifically, we show empirically that popular models are vulnerable to
adversarial permutation in answer sets for multiple-choice prompting, which is
surprising as models should ideally be as invariant to prompt permutation as
humans are. These vulnerabilities persist across various model sizes, and exist
in very recent language and vision-language models. Code is available at
\url{https://github.com/ys-zong/FoolyourVLLMs}.


http://arxiv.org/abs/2310.00633
A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks. (99%)
Yanjie Li; Bin Xie; Songtao Guo; Yuanyuan Yang; Bin Xiao
  Benefiting from the rapid development of deep learning, 2D and 3D computer
vision applications are deployed in many safe-critical systems, such as
autopilot and identity authentication. However, deep learning models are not
trustworthy enough because of their limited robustness against adversarial
attacks. The physically realizable adversarial attacks further pose fatal
threats to the application and human safety. Lots of papers have emerged to
investigate the robustness and safety of deep learning models against
adversarial attacks. To lead to trustworthy AI, we first construct a general
threat model from different perspectives and then comprehensively review the
latest progress of both 2D and 3D adversarial attacks. We extend the concept of
adversarial examples beyond imperceptive perturbations and collate over 170
papers to give an overview of deep learning model robustness against various
adversarial attacks. To the best of our knowledge, we are the first to
systematically investigate adversarial attacks for 3D models, a flourishing
field applied to many real-world applications. In addition, we examine physical
adversarial attacks that lead to safety violations. Last but not least, we
summarize present popular topics, give insights on challenges, and shed light
on future research on trustworthy AI.


http://arxiv.org/abs/2310.00761
Counterfactual Image Generation for adversarially robust and interpretable Classifiers. (96%)
Rafael Bischof; Florian Scheidegger; Michael A. Kraus; A. Cristiano I. Malossi
  Neural Image Classifiers are effective but inherently hard to interpret and
susceptible to adversarial attacks. Solutions to both problems exist, among
others, in the form of counterfactual examples generation to enhance
explainability or adversarially augment training datasets for improved
robustness. However, existing methods exclusively address only one of the
issues. We propose a unified framework leveraging image-to-image translation
Generative Adversarial Networks (GANs) to produce counterfactual samples that
highlight salient regions for interpretability and act as adversarial samples
to augment the dataset for more robustness. This is achieved by combining the
classifier and discriminator into a single model that attributes real images to
their respective classes and flags generated images as "fake". We assess the
method's effectiveness by evaluating (i) the produced explainability masks on a
semantic segmentation task for concrete cracks and (ii) the model's resilience
against the Projected Gradient Descent (PGD) attack on a fruit defects
detection problem. Our produced saliency maps are highly descriptive, achieving
competitive IoU values compared to classical segmentation models despite being
trained exclusively on classification labels. Furthermore, the model exhibits
improved robustness to adversarial attacks, and we show how the discriminator's
"fakeness" value serves as an uncertainty measure of the predictions.


http://arxiv.org/abs/2310.00607
On the Onset of Robust Overfitting in Adversarial Training. (64%)
Chaojian Yu; Xiaolong Shi; Jun Yu; Bo Han; Tongliang Liu
  Adversarial Training (AT) is a widely-used algorithm for building robust
neural networks, but it suffers from the issue of robust overfitting, the
fundamental mechanism of which remains unclear. In this work, we consider
normal data and adversarial perturbation as separate factors, and identify that
the underlying causes of robust overfitting stem from the normal data through
factor ablation in AT. Furthermore, we explain the onset of robust overfitting
as a result of the model learning features that lack robust generalization,
which we refer to as non-effective features. Specifically, we provide a
detailed analysis of the generation of non-effective features and how they lead
to robust overfitting. Additionally, we explain various empirical behaviors
observed in robust overfitting and revisit different techniques to mitigate
robust overfitting from the perspective of non-effective features, providing a
comprehensive understanding of the robust overfitting phenomenon. This
understanding inspires us to propose two measures, attack strength and data
augmentation, to hinder the learning of non-effective features by the neural
network, thereby alleviating robust overfitting. Extensive experiments
conducted on benchmark datasets demonstrate the effectiveness of the proposed
methods in mitigating robust overfitting and enhancing adversarial robustness.


http://arxiv.org/abs/2310.00616
Understanding Adversarial Transferability in Federated Learning. (64%)
Yijiang Li; Ying Gao; Haohan Wang
  We investigate the robustness and security issues from a novel and practical
setting: a group of malicious clients has impacted the model during training by
disguising their identities and acting as benign clients, and only revealing
their adversary position after the training to conduct transferable adversarial
attacks with their data, which is usually a subset of the data that FL system
is trained with. Our aim is to offer a full understanding of the challenges the
FL system faces in this practical setting across a spectrum of configurations.
We notice that such an attack is possible, but the federated model is more
robust compared with its centralized counterpart when the accuracy on clean
images is comparable. Through our study, we hypothesized the robustness is from
two factors: the decentralized training on distributed data and the averaging
operation. We provide evidence from both the perspective of empirical
experiments and theoretical analysis. Our work has implications for
understanding the robustness of federated learning systems and poses a
practical question for federated learning applications.


http://arxiv.org/abs/2310.00626
GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learning. (61%)
Qiannan Wang; Changchun Yin; Zhe Liu; Liming Fang; Run Wang; Chenhao Lin
  Within the realm of computer vision, self-supervised learning (SSL) pertains
to training pre-trained image encoders utilizing a substantial quantity of
unlabeled images. Pre-trained image encoders can serve as feature extractors,
facilitating the construction of downstream classifiers for various tasks.
However, the use of SSL has led to an increase in security research related to
various backdoor attacks. Currently, the trigger patterns used in backdoor
attacks on SSL are mostly visible or static (sample-agnostic), making backdoors
less covert and significantly affecting the attack performance. In this work,
we propose GhostEncoder, the first dynamic invisible backdoor attack on SSL.
Unlike existing backdoor attacks on SSL, which use visible or static trigger
patterns, GhostEncoder utilizes image steganography techniques to encode hidden
information into benign images and generate backdoor samples. We then fine-tune
the pre-trained image encoder on a manipulation dataset to inject the backdoor,
enabling downstream classifiers built upon the backdoored encoder to inherit
the backdoor behavior for target downstream tasks. We evaluate GhostEncoder on
three downstream tasks and results demonstrate that GhostEncoder provides
practical stealthiness on images and deceives the victim model with a high
attack success rate without compromising its utility. Furthermore, GhostEncoder
withstands state-of-the-art defenses, including STRIP, STRIP-Cl, and
SSL-Cleanse.


http://arxiv.org/abs/2310.00648
Fewer is More: Trojan Attacks on Parameter-Efficient Fine-Tuning. (9%)
Lauren Hong; Ting Wang
  Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of
pre-trained language models (PLMs) to specific tasks. By tuning only a minimal
set of (extra) parameters, PEFT achieves performance comparable to full
fine-tuning. However, despite its prevalent use, the security implications of
PEFT remain largely unexplored. In this paper, we conduct a pilot study
revealing that PEFT exhibits unique vulnerability to trojan attacks.
Specifically, we present PETA, a novel attack that accounts for downstream
adaptation through bilevel optimization: the upper-level objective embeds the
backdoor into a PLM while the lower-level objective simulates PEFT to retain
the PLM's task-specific performance. With extensive evaluation across a variety
of downstream tasks and trigger designs, we demonstrate PETA's effectiveness in
terms of both attack success rate and unaffected clean accuracy, even after the
victim user performs PEFT over the backdoored PLM using untainted data.
Moreover, we empirically provide possible explanations for PETA's efficacy: the
bilevel optimization inherently 'orthogonalizes' the backdoor and PEFT modules,
thereby retaining the backdoor throughout PEFT. Based on this insight, we
explore a simple defense that omits PEFT in selected layers of the backdoored
PLM and unfreezes a subset of these layers' parameters, which is shown to
effectively neutralize PETA.


http://arxiv.org/abs/2310.00847
Can Pre-trained Networks Detect Familiar Out-of-Distribution Data? (1%)
Atsuyuki Miyai; Qing Yu; Go Irie; Kiyoharu Aizawa
  Out-of-distribution (OOD) detection is critical for safety-sensitive machine
learning applications and has been extensively studied, yielding a plethora of
methods developed in the literature. However, most studies for OOD detection
did not use pre-trained models and trained a backbone from scratch. In recent
years, transferring knowledge from large pre-trained models to downstream tasks
by lightweight tuning has become mainstream for training in-distribution (ID)
classifiers. To bridge the gap between the practice of OOD detection and
current classifiers, the unique and crucial problem is that the samples whose
information networks know often come as OOD input. We consider that such data
may significantly affect the performance of large pre-trained networks because
the discriminability of these OOD data depends on the pre-training algorithm.
Here, we define such OOD data as PT-OOD (Pre-Trained OOD) data. In this paper,
we aim to reveal the effect of PT-OOD on the OOD detection performance of
pre-trained networks from the perspective of pre-training algorithms. To
achieve this, we explore the PT-OOD detection performance of supervised and
self-supervised pre-training algorithms with linear-probing tuning, the most
common efficient tuning method. Through our experiments and analysis, we find
that the low linear separability of PT-OOD in the feature space heavily
degrades the PT-OOD detection performance, and self-supervised models are more
vulnerable to PT-OOD than supervised pre-trained models, even with
state-of-the-art detection methods. To solve this vulnerability, we further
propose a unique solution to large-scale pre-trained models: Leveraging
powerful instance-by-instance discriminative representations of pre-trained
models and detecting OOD in the feature space independent of the ID decision
boundaries. The code will be available via https://github.com/AtsuMiyai/PT-OOD.


http://arxiv.org/abs/2310.00710
How well does LLM generate security tests? (1%)
Ying Daphne Zhang; Wenjia Daphne Song; Zhengjie Daphne Ji; Daphne Danfeng; Yao; Na Meng
  Developers often build software on top of third-party libraries (Libs) to
improve programmer productivity and software quality. The libraries may contain
vulnerabilities exploitable by hackers to attack the applications (Apps) built
on top of them. People refer to such attacks as supply chain attacks, the
documented number of which has increased 742% in 2022. People created tools to
mitigate such attacks, by scanning the library dependencies of Apps,
identifying the usage of vulnerable library versions, and suggesting secure
alternatives to vulnerable dependencies. However, recent studies show that many
developers do not trust the reports by these tools; they ask for code or
evidence to demonstrate how library vulnerabilities lead to security exploits,
in order to assess vulnerability severity and modification necessity.
Unfortunately, manually crafting demos of application-specific attacks is
challenging and time-consuming, and there is insufficient tool support to
automate that procedure.
  In this study, we used ChatGPT-4.0 to generate security tests, and to
demonstrate how vulnerable library dependencies facilitate the supply chain
attacks to given Apps. We explored various prompt styles/templates, and found
that ChatGPT-4.0 generated tests for all 55 Apps, demonstrating 24 attacks
successfully. It outperformed two state-of-the-art security test generators --
TRANSFER and SIEGE -- by generating a lot more tests and achieving more
exploits. ChatGPT-4.0 worked better when prompts described more on the
vulnerabilities, possible exploits, and code context. Our research will shed
light on new research in security test generation. The generated tests will
help developers create secure by design and secure by default software.


http://arxiv.org/abs/2310.00567
Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks. (99%)
Quang H. Nguyen; Yingjie Lao; Tung Pham; Kok-Seng Wong; Khoa D. Doan
  Recent works have shown that deep neural networks are vulnerable to
adversarial examples that find samples close to the original image but can make
the model misclassify. Even with access only to the model's output, an attacker
can employ black-box attacks to generate such adversarial examples. In this
work, we propose a simple and lightweight defense against black-box attacks by
adding random noise to hidden features at intermediate layers of the model at
inference time. Our theoretical analysis confirms that this method effectively
enhances the model's resilience against both score-based and decision-based
black-box attacks. Importantly, our defense does not necessitate adversarial
training and has minimal impact on accuracy, rendering it applicable to any
pre-trained model. Our analysis also reveals the significance of selectively
adding noise to different parts of the model based on the gradient of the
adversarial objective function, which can be varied during the attack. We
demonstrate the robustness of our defense against multiple black-box attacks
through extensive empirical experiments involving diverse models with various
architectures.


http://arxiv.org/abs/2310.00438
Human-Producible Adversarial Examples. (98%)
David Khachaturov; Yue Gao; Ilia Shumailov; Robert Mullins; Ross Anderson; Kassem Fawaz
  Visual adversarial examples have so far been restricted to pixel-level image
manipulations in the digital world, or have required sophisticated equipment
such as 2D or 3D printers to be produced in the physical real world. We present
the first ever method of generating human-producible adversarial examples for
the real world that requires nothing more complicated than a marker pen. We
call them $\textbf{adversarial tags}$. First, building on top of differential
rendering, we demonstrate that it is possible to build potent adversarial
examples with just lines. We find that by drawing just $4$ lines we can disrupt
a YOLO-based model in $54.8\%$ of cases; increasing this to $9$ lines disrupts
$81.8\%$ of the cases tested. Next, we devise an improved method for line
placement to be invariant to human drawing error. We evaluate our system
thoroughly in both digital and analogue worlds and demonstrate that our tags
can be applied by untrained humans. We demonstrate the effectiveness of our
method for producing real-world adversarial examples by conducting a user study
where participants were asked to draw over printed images using digital
equivalents as guides. We further evaluate the effectiveness of both targeted
and untargeted attacks, and discuss various trade-offs and method limitations,
as well as the practical and ethical implications of our work. The source code
will be released publicly.


http://arxiv.org/abs/2310.00503
Black-box Attacks on Image Activity Prediction and its Natural Language Explanations. (98%)
Alina Elena Baia; Valentina Poggioni; Andrea Cavallaro
  Explainable AI (XAI) methods aim to describe the decision process of deep
neural networks. Early XAI methods produced visual explanations, whereas more
recent techniques generate multimodal explanations that include textual
information and visual representations. Visual XAI methods have been shown to
be vulnerable to white-box and gray-box adversarial attacks, with an attacker
having full or partial knowledge of and access to the target system. As the
vulnerabilities of multimodal XAI models have not been examined, in this paper
we assess for the first time the robustness to black-box attacks of the natural
language explanations generated by a self-rationalizing image-based activity
recognition model. We generate unrestricted, spatially variant perturbations
that disrupt the association between the predictions and the corresponding
explanations to mislead the model into generating unfaithful explanations. We
show that we can create adversarial images that manipulate the explanations of
an activity recognition model by having access only to its final output.


http://arxiv.org/abs/2310.00542
Horizontal Class Backdoor to Deep Learning. (56%)
Hua Ma; Shang Wang; Yansong Gao
  All existing backdoor attacks to deep learning (DL) models belong to the
vertical class backdoor (VCB). That is, any sample from a class will activate
the implanted backdoor in the presence of the secret trigger, regardless of
source-class-agnostic or source-class-specific backdoor. Current trends of
existing defenses are overwhelmingly devised for VCB attacks especially the
source-class-agnostic backdoor, which essentially neglects other potential
simple but general backdoor types, thus giving false security implications. It
is thus urgent to discover unknown backdoor types.
  This work reveals a new, simple, and general horizontal class backdoor (HCB)
attack. We show that the backdoor can be naturally bounded with innocuous
natural features that are common and pervasive in the real world. Note that an
innocuous feature (e.g., expression) is irrelevant to the main task of the
model (e.g., recognizing a person from one to another). The innocuous feature
spans across classes horizontally but is exhibited by partial samples per class
-- satisfying the horizontal class (HC) property. Only when the trigger is
concurrently presented with the HC innocuous feature, can the backdoor be
effectively activated. Extensive experiments on attacking performance in terms
of high attack success rates with tasks of 1) MNIST, 2) facial recognition, 3)
traffic sign recognition, and 4) object detection demonstrate that the HCB is
highly efficient and effective. We extensively evaluate the HCB evasiveness
against a (chronologically) series of 9 influential countermeasures of
Fine-Pruning (RAID 18'), STRIP (ACSAC 19'), Neural Cleanse (Oakland 19'), ABS
(CCS 19'), Februus (ACSAC 20'), MNTD (Oakland 21'), SCAn (USENIX SEC 21'), MOTH
(Oakland 22'), and Beatrix (NDSS 23'), where none of them can succeed even when
a simplest trigger is used.


http://arxiv.org/abs/2310.00416
Refutation of Shapley Values for XAI -- Additional Evidence. (8%)
Xuanxiang Huang; Joao Marques-Silva
  Recent work demonstrated the inadequacy of Shapley values for explainable
artificial intelligence (XAI). Although to disprove a theory a single
counterexample suffices, a possible criticism of earlier work is that the focus
was solely on Boolean classifiers. To address such possible criticism, this
paper demonstrates the inadequacy of Shapley values for families of classifiers
where features are not boolean, but also for families of classifiers for which
multiple classes can be picked. Furthermore, the paper shows that the features
changed in any minimal $l_0$ distance adversarial examples do not include
irrelevant features, thus offering further arguments regarding the inadequacy
of Shapley values for XAI.


http://arxiv.org/abs/2310.00076
Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks. (99%)
Mehrdad Saberi; Vinu Sankar Sadasivan; Keivan Rezaei; Aounon Kumar; Atoosa Chegini; Wenxiao Wang; Soheil Feizi
  In light of recent advancements in generative AI models, it has become
essential to distinguish genuine content from AI-generated one to prevent the
malicious usage of fake materials as authentic ones and vice versa. Various
techniques have been introduced for identifying AI-generated images, with
watermarking emerging as a promising approach. In this paper, we analyze the
robustness of various AI-image detectors including watermarking and
classifier-based deepfake detectors. For watermarking methods that introduce
subtle image perturbations (i.e., low perturbation budget methods), we reveal a
fundamental trade-off between the evasion error rate (i.e., the fraction of
watermarked images detected as non-watermarked ones) and the spoofing error
rate (i.e., the fraction of non-watermarked images detected as watermarked
ones) upon an application of a diffusion purification attack. In this regime,
we also empirically show that diffusion purification effectively removes
watermarks with minimal changes to images. For high perturbation watermarking
methods where notable changes are applied to images, the diffusion purification
attack is not effective. In this case, we develop a model substitution
adversarial attack that can successfully remove watermarks. Moreover, we show
that watermarking methods are vulnerable to spoofing attacks where the attacker
aims to have real images (potentially obscene) identified as watermarked ones,
damaging the reputation of the developers. In particular, by just having
black-box access to the watermarking method, we show that one can generate a
watermarked noise image which can be added to the real images to have them
falsely flagged as watermarked ones. Finally, we extend our theory to
characterize a fundamental trade-off between the robustness and reliability of
classifier-based deep fake detectors and demonstrate it through experiments.


http://arxiv.org/abs/2309.17348
Efficient Biologically Plausible Adversarial Training. (98%)
Matilde Tristany Farinha; Thomas Ortner; Giorgia Dellaferrera; Benjamin Grewe; Angeliki Pantazi
  Artificial Neural Networks (ANNs) trained with Backpropagation (BP) show
astounding performance and are increasingly often used in performing our daily
life tasks. However, ANNs are highly vulnerable to adversarial attacks, which
alter inputs with small targeted perturbations that drastically disrupt the
models' performance. The most effective method to make ANNs robust against
these attacks is adversarial training, in which the training dataset is
augmented with exemplary adversarial samples. Unfortunately, this approach has
the drawback of increased training complexity since generating adversarial
samples is very computationally demanding. In contrast to ANNs, humans are not
susceptible to adversarial attacks. Therefore, in this work, we investigate
whether biologically-plausible learning algorithms are more robust against
adversarial attacks than BP. In particular, we present an extensive comparative
analysis of the adversarial robustness of BP and Present the Error to Perturb
the Input To modulate Activity (PEPITA), a recently proposed
biologically-plausible learning algorithm, on various computer vision tasks. We
observe that PEPITA has higher intrinsic adversarial robustness and, with
adversarial training, has a more favourable natural-vs-adversarial performance
trade-off as, for the same natural accuracies, PEPITA's adversarial accuracies
decrease in average by 0.26% and BP's by 8.05%.


http://arxiv.org/abs/2309.17410
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. (96%)
Vaidehi Patil; Peter Hase; Mohit Bansal
  Pretrained language models sometimes possess knowledge that we do not wish
them to, including memorized personal information and knowledge that could be
used to harm people. They can also output toxic or harmful text. To mitigate
these safety and informational issues, we propose an attack-and-defense
framework for studying the task of deleting sensitive information directly from
model weights. We study direct edits to model weights because (1) this approach
should guarantee that particular deleted information is never extracted by
future prompt attacks, and (2) it should protect against whitebox attacks,
which is necessary for making claims about safety/privacy in a setting where
publicly available model weights could be used to elicit sensitive information.
Our threat model assumes that an attack succeeds if the answer to a sensitive
question is located among a set of B generated candidates, based on scenarios
where the information would be insecure if the answer is among B candidates.
Experimentally, we show that even state-of-the-art model editing methods such
as ROME struggle to truly delete factual information from models like GPT-J, as
our whitebox and blackbox attacks can recover "deleted" information from an
edited model 38% of the time. These attacks leverage two key observations: (1)
that traces of deleted information can be found in intermediate model hidden
states, and (2) that applying an editing method for one question may not delete
information across rephrased versions of the question. Finally, we provide new
defense methods that protect against some extraction attacks, but we do not
find a single universally effective defense method. Our results suggest that
truly deleting sensitive information is a tractable but difficult problem,
since even relatively low attack success rates have potentially severe societal
implications for real-world deployment of language models.


http://arxiv.org/abs/2309.17048
On Continuity of Robust and Accurate Classifiers. (93%)
Ramin Barati; Reza Safabakhsh; Mohammad Rahmati
  The reliability of a learning model is key to the successful deployment of
machine learning in various applications. Creating a robust model, particularly
one unaffected by adversarial attacks, requires a comprehensive understanding
of the adversarial examples phenomenon. However, it is difficult to describe
the phenomenon due to the complicated nature of the problems in machine
learning. It has been shown that adversarial training can improve the
robustness of the hypothesis. However, this improvement comes at the cost of
decreased performance on natural samples. Hence, it has been suggested that
robustness and accuracy of a hypothesis are at odds with each other. In this
paper, we put forth the alternative proposal that it is the continuity of a
hypothesis that is incompatible with its robustness and accuracy. In other
words, a continuous function cannot effectively learn the optimal robust
hypothesis. To this end, we will introduce a framework for a rigorous study of
harmonic and holomorphic hypothesis in learning theory terms and provide
empirical evidence that continuous hypotheses does not perform as well as
discontinuous hypotheses in some common machine learning tasks. From a
practical point of view, our results suggests that a robust and accurate
learning rule would train different continuous hypotheses for different regions
of the domain. From a theoretical perspective, our analysis explains the
adversarial examples phenomenon as a conflict between the continuity of a
sequence of functions and its uniform convergence to a discontinuous function.


http://arxiv.org/abs/2309.17401
Adversarial Machine Learning in Latent Representations of Neural Networks. (93%)
Milin Zhang; Mohammad Abdi; Francesco Restuccia
  Distributed deep neural networks (DNNs) have been shown to reduce the
computational burden of mobile devices and decrease the end-to-end inference
latency in edge computing scenarios. While distributed DNNs have been studied,
to the best of our knowledge the resilience of distributed DNNs to adversarial
action still remains an open problem. In this paper, we fill the existing
research gap by rigorously analyzing the robustness of distributed DNNs against
adversarial action. We cast this problem in the context of information theory
and introduce two new measurements for distortion and robustness. Our
theoretical findings indicate that (i) assuming the same level of information
distortion, latent features are always more robust than input representations;
(ii) the adversarial robustness is jointly determined by the feature dimension
and the generalization capability of the DNN. To test our theoretical findings,
we perform extensive experimental analysis by considering 6 different DNN
architectures, 6 different approaches for distributed DNN and 10 different
adversarial attacks to the ImageNet-1K dataset. Our experimental results
support our theoretical findings by showing that the compressed latent
representations can reduce the success rate of adversarial attacks by 88% in
the best case and by 57% on the average compared to attacks to the input space.


http://arxiv.org/abs/2310.00116
Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization. (92%)
Mahyar Fazlyab; Taha Entesari; Aniket Roy; Rama Chellappa
  To improve the robustness of deep classifiers against adversarial
perturbations, many approaches have been proposed, such as designing new
architectures with better robustness properties (e.g., Lipschitz-capped
networks), or modifying the training process itself (e.g., min-max
optimization, constrained learning, or regularization). These approaches,
however, might not be effective at increasing the margin in the input (feature)
space. As a result, there has been an increasing interest in developing
training procedures that can directly manipulate the decision boundary in the
input space. In this paper, we build upon recent developments in this category
by developing a robust training algorithm whose objective is to increase the
margin in the output (logit) space while regularizing the Lipschitz constant of
the model along vulnerable directions. We show that these two objectives can
directly promote larger margins in the input space. To this end, we develop a
scalable method for calculating guaranteed differentiable upper bounds on the
Lipschitz constant of neural networks accurately and efficiently. The relative
accuracy of the bounds prevents excessive regularization and allows for more
direct manipulation of the decision boundary. Furthermore, our Lipschitz
bounding algorithm exploits the monotonicity and Lipschitz continuity of the
activation layers, and the resulting bounds can be used to design new layers
with controllable bounds on their Lipschitz constant. Experiments on the MNIST,
CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm
obtains competitively improved results compared to the state-of-the-art.


http://arxiv.org/abs/2309.17278
Toward Robust Recommendation via Real-time Vicinal Defense. (82%)
Yichang Xu; Chenwang Wu; Defu Lian
  Recommender systems have been shown to be vulnerable to poisoning attacks,
where malicious data is injected into the dataset to cause the recommender
system to provide biased recommendations. To defend against such attacks,
various robust learning methods have been proposed. However, most methods are
model-specific or attack-specific, making them lack generality, while other
methods, such as adversarial training, are oriented towards evasion attacks and
thus have a weak defense strength in poisoning attacks.
  In this paper, we propose a general method, Real-time Vicinal Defense (RVD),
which leverages neighboring training data to fine-tune the model before making
a recommendation for each user. RVD works in the inference phase to ensure the
robustness of the specific sample in real-time, so there is no need to change
the model structure and training process, making it more practical. Extensive
experimental results demonstrate that RVD effectively mitigates targeted
poisoning attacks across various models without sacrificing accuracy. Moreover,
the defensive effect can be further amplified when our method is combined with
other strategies.


http://arxiv.org/abs/2310.00070
Adversarial Explainability: Utilizing Explainable Machine Learning in Bypassing IoT Botnet Detection Systems. (31%)
Mohammed M. Alani; Atefeh Mashatan; Ali Miri
  Botnet detection based on machine learning have witnessed significant leaps
in recent years, with the availability of large and reliable datasets that are
extracted from real-life scenarios. Consequently, adversarial attacks on
machine learning-based cybersecurity systems are posing a significant threat to
the practicality of these solutions. In this paper, we introduce a novel attack
that utilizes machine learning model's explainability in evading detection by
botnet detection systems. The proposed attack utilizes information obtained
from model's explainability to build adversarial samples that can evade
detection in a blackbox setting. The proposed attack was tested on a trained
IoT botnet detection systems and was capable of bypassing the botnet detection
with 0% detection by altering one feature only to generate the adversarial
samples.


http://arxiv.org/abs/2310.00108
Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study. (13%)
Myeongseob Ko; Ming Jin; Chenguang Wang; Ruoxi Jia
  Membership inference attacks (MIAs) aim to infer whether a data point has
been used to train a machine learning model. These attacks can be employed to
identify potential privacy vulnerabilities and detect unauthorized use of
personal data. While MIAs have been traditionally studied for simple
classification models, recent advancements in multi-modal pre-training, such as
CLIP, have demonstrated remarkable zero-shot performance across a range of
computer vision tasks. However, the sheer scale of data and models presents
significant computational challenges for performing the attacks.
  This paper takes a first step towards developing practical MIAs against
large-scale multi-modal models. We introduce a simple baseline strategy by
thresholding the cosine similarity between text and image features of a target
point and propose further enhancing the baseline by aggregating cosine
similarity across transformations of the target. We also present a new weakly
supervised attack method that leverages ground-truth non-members (e.g.,
obtained by using the publication date of a target model and the timestamps of
the open data) to further enhance the attack. Our evaluation shows that CLIP
models are susceptible to our attack strategies, with our simple baseline
achieving over $75\%$ membership identification accuracy. Furthermore, our
enhanced attacks outperform the baseline across multiple models and datasets,
with the weakly supervised attack demonstrating an average-case performance
improvement of $17\%$ and being at least $7$X more effective at low
false-positive rates. These findings highlight the importance of protecting the
privacy of multi-modal foundational models, which were previously assumed to be
less susceptible to MIAs due to less overfitting. Our code is available at
https://github.com/ruoxi-jia-group/CLIP-MIA.


http://arxiv.org/abs/2309.17301
Distributed Resilient Control of DC Microgrids Under Generally Unbounded FDI Attacks. (1%)
Yichao Wang; Mohamadamin Rajabinezhad; Omar A. Beg; Shan Zuo
  Due to the nature of distributed secondary control paradigm, DC microgrids
are prone to malicious cyber-physical attacks, which could be unbounded to
maximize their damage. Existing resilient secondary control methods addressing
unbounded attacks require that the first time derivatives of cyber-physical
attack signals be bounded. The secondary defense strategy presented in this
letter relax such a strict constraint by addressing more generally unbounded
attack signals and hence, enhance the resilience of DC microgrids in
adversarial environments. Rigorous proofs, based on Lyapunov techniques, show
that the proposed method guarantees the uniformly ultimately bounded
convergence for both voltage regulation and proportional load sharing under
generally unbounded attacks. Comparative case studies further validate the
enhanced resilience of the proposed attack-resilient control strategy.


http://arxiv.org/abs/2310.00222
Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learning. (1%)
Hongsheng Hu; Xuyun Zhang; Zoran Salcic; Lichao Sun; Kim-Kwang Raymond Choo; Gillian Dobbie
  Federated learning (FL) is a popular approach to facilitate privacy-aware
machine learning since it allows multiple clients to collaboratively train a
global model without granting others access to their private data. It is,
however, known that FL can be vulnerable to membership inference attacks
(MIAs), where the training records of the global model can be distinguished
from the testing records. Surprisingly, research focusing on the investigation
of the source inference problem appears to be lacking. We also observe that
identifying a training record's source client can result in privacy breaches
extending beyond MIAs. For example, consider an FL application where multiple
hospitals jointly train a COVID-19 diagnosis model, membership inference
attackers can identify the medical records that have been used for training,
and any additional identification of the source hospital can result the patient
from the particular hospital more prone to discrimination. Seeking to
contribute to the literature gap, we take the first step to investigate source
privacy in FL. Specifically, we propose a new inference attack (hereafter
referred to as source inference attack -- SIA), designed to facilitate an
honest-but-curious server to identify the training record's source client. The
proposed SIAs leverage the Bayesian theorem to allow the server to implement
the attack in a non-intrusive manner without deviating from the defined FL
protocol. We then evaluate SIAs in three different FL frameworks to show that
in existing FL frameworks, the clients sharing gradients, model parameters, or
predictions on a public dataset will leak such source information to the
server. We also conduct extensive experiments on various datasets to
investigate the key factors in an SIA. The experimental results validate the
efficacy of the proposed SIAs.


http://arxiv.org/abs/2309.16878
Investigating Human-Identifiable Features Hidden in Adversarial Perturbations. (98%)
Dennis Y. Menn; Tzu-hsun Feng; Sriram Vishwanath; Hung-yi Lee
  Neural networks perform exceedingly well across various machine learning
tasks but are not immune to adversarial perturbations. This vulnerability has
implications for real-world applications. While much research has been
conducted, the underlying reasons why neural networks fall prey to adversarial
attacks are not yet fully understood. Central to our study, which explores up
to five attack algorithms across three datasets, is the identification of
human-identifiable features in adversarial perturbations. Additionally, we
uncover two distinct effects manifesting within human-identifiable features.
Specifically, the masking effect is prominent in untargeted attacks, while the
generation effect is more common in targeted attacks. Using pixel-level
annotations, we extract such features and demonstrate their ability to
compromise target models. In addition, our findings indicate a notable extent
of similarity in perturbations across different attack algorithms when averaged
over multiple models. This work also provides insights into phenomena
associated with adversarial perturbations, such as transferability and model
interpretability. Our study contributes to a deeper understanding of the
underlying mechanisms behind adversarial attacks and offers insights for the
development of more resilient defense strategies for neural networks.


http://arxiv.org/abs/2309.16207
Parameter-Saving Adversarial Training: Reinforcing Multi-Perturbation Robustness via Hypernetworks. (98%)
Huihui Gong; Minjing Dong; Siqi Ma; Seyit Camtepe; Surya Nepal; Chang Xu
  Adversarial training serves as one of the most popular and effective methods
to defend against adversarial perturbations. However, most defense mechanisms
only consider a single type of perturbation while various attack methods might
be adopted to perform stronger adversarial attacks against the deployed model
in real-world scenarios, e.g., $\ell_2$ or $\ell_\infty$. Defending against
various attacks can be a challenging problem since multi-perturbation
adversarial training and its variants only achieve suboptimal robustness
trade-offs, due to the theoretical limit to multi-perturbation robustness for a
single model. Besides, it is impractical to deploy large models in some
storage-efficient scenarios. To settle down these drawbacks, in this paper we
propose a novel multi-perturbation adversarial training framework,
parameter-saving adversarial training (PSAT), to reinforce multi-perturbation
robustness with an advantageous side effect of saving parameters, which
leverages hypernetworks to train specialized models against a single
perturbation and aggregate these specialized models to defend against multiple
perturbations. Eventually, we extensively evaluate and compare our proposed
method with state-of-the-art single/multi-perturbation robust methods against
various latest attack methods on different datasets, showing the robustness
superiority and parameter efficiency of our proposed method, e.g., for the
CIFAR-10 dataset with ResNet-50 as the backbone, PSAT saves approximately 80\%
of parameters with achieving the state-of-the-art robustness trade-off
accuracy.


http://arxiv.org/abs/2309.16487
Towards Poisoning Fair Representations. (70%)
Tianci Liu; Haoyu Wang; Feijie Wu; Hengtong Zhang; Pan Li; Lu Su; Jing Gao
  Fair machine learning seeks to mitigate model prediction bias against certain
demographic subgroups such as elder and female. Recently, fair representation
learning (FRL) trained by deep neural networks has demonstrated superior
performance, whereby representations containing no demographic information are
inferred from the data and then used as the input to classification or other
downstream tasks. Despite the development of FRL methods, their vulnerability
under data poisoning attack, a popular protocol to benchmark model robustness
under adversarial scenarios, is under-explored. Data poisoning attacks have
been developed for classical fair machine learning methods which incorporate
fairness constraints into shallow-model classifiers. Nonetheless, these attacks
fall short in FRL due to notably different fairness goals and model
architectures. This work proposes the first data poisoning framework attacking
FRL. We induce the model to output unfair representations that contain as much
demographic information as possible by injecting carefully crafted poisoning
samples into the training data. This attack entails a prohibitive bilevel
optimization, wherefore an effective approximated solution is proposed. A
theoretical analysis on the needed number of poisoning samples is derived and
sheds light on defending against the attack. Experiments on benchmark fairness
datasets and state-of-the-art fair representation learning models demonstrate
the superiority of our attack.


http://arxiv.org/abs/2309.16452
On the Trade-offs between Adversarial Robustness and Actionable Explanations. (68%)
Satyapriya Krishna; Chirag Agarwal; Himabindu Lakkaraju
  As machine learning models are increasingly being employed in various
high-stakes settings, it becomes important to ensure that predictions of these
models are not only adversarially robust, but also readily explainable to
relevant stakeholders. However, it is unclear if these two notions can be
simultaneously achieved or if there exist trade-offs between them. In this
work, we make one of the first attempts at studying the impact of adversarially
robust models on actionable explanations which provide end users with a means
for recourse. We theoretically and empirically analyze the cost (ease of
implementation) and validity (probability of obtaining a positive model
prediction) of recourses output by state-of-the-art algorithms when the
underlying models are adversarially robust vs. non-robust. More specifically,
we derive theoretical bounds on the differences between the cost and the
validity of the recourses generated by state-of-the-art algorithms for
adversarially robust vs. non-robust linear and non-linear models. Our empirical
results with multiple real-world datasets validate our theoretical results and
show the impact of varying degrees of model robustness on the cost and validity
of the resulting recourses. Our analyses demonstrate that adversarially robust
models significantly increase the cost and reduce the validity of the resulting
recourses, thus shedding light on the inherent trade-offs between adversarial
robustness and actionable explanations


http://arxiv.org/abs/2309.16827
Post-Training Overfitting Mitigation in DNN Classifiers. (41%)
Hang Wang; David J. Miller; George Kesidis
  Well-known (non-malicious) sources of overfitting in deep neural net (DNN)
classifiers include: i) large class imbalances; ii) insufficient training-set
diversity; and iii) over-training. In recent work, it was shown that backdoor
data-poisoning also induces overfitting, with unusually large classification
margins to the attacker's target class, mediated particularly by (unbounded)
ReLU activations that allow large signals to propagate in the DNN. Thus, an
effective post-training (with no knowledge of the training set or training
process) mitigation approach against backdoors was proposed, leveraging a small
clean dataset, based on bounding neural activations. Improving upon that work,
we threshold activations specifically to limit maximum margins (MMs), which
yields performance gains in backdoor mitigation. We also provide some
analytical support for this mitigation approach. Most importantly, we show that
post-training MM-based regularization substantially mitigates non-malicious
overfitting due to class imbalances and overtraining. Thus, unlike adversarial
training, which provides some resilience against attacks but which harms clean
(attack-free) generalization, we demonstrate an approach originating from
adversarial learning that helps clean generalization accuracy. Experiments on
CIFAR-10 and CIFAR-100, in comparison with peer methods, demonstrate strong
performance of our methods.


http://arxiv.org/abs/2309.16883
The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing. (26%)
Blaise Delattre; Alexandre Araujo; Quentin Barthélemy; Alexandre Allauzen
  Real-life applications of deep neural networks are hindered by their unsteady
predictions when faced with noisy inputs and adversarial attacks. The certified
radius is in this context a crucial indicator of the robustness of models.
However how to design an efficient classifier with a sufficient certified
radius? Randomized smoothing provides a promising framework by relying on noise
injection in inputs to obtain a smoothed and more robust classifier. In this
paper, we first show that the variance introduced by randomized smoothing
closely interacts with two other important properties of the classifier,
\textit{i.e.} its Lipschitz constant and margin. More precisely, our work
emphasizes the dual impact of the Lipschitz constant of the base classifier, on
both the smoothed classifier and the empirical variance. Moreover, to increase
the certified robust radius, we introduce a different simplex projection
technique for the base classifier to leverage the variance-margin trade-off
thanks to Bernstein's concentration inequality, along with an enhanced
Lipschitz bound. Experimental results show a significant improvement in
certified accuracy compared to current state-of-the-art methods. Our novel
certification procedure allows us to use pre-trained models that are used with
randomized smoothing, effectively improving the current certification radius in
a zero-shot manner.


http://arxiv.org/abs/2309.16952
Leveraging Optimization for Adaptive Attacks on Image Watermarks. (10%)
Nils Lukas; Abdulrahman Diaa; Lucas Fenaux; Florian Kerschbaum
  Untrustworthy users can misuse image generators to synthesize high-quality
deepfakes and engage in online spam or disinformation campaigns. Watermarking
deters misuse by marking generated content with a hidden message, enabling its
detection using a secret watermarking key. A core security property of
watermarking is robustness, which states that an attacker can only evade
detection by substantially degrading image quality. Assessing robustness
requires designing an adaptive attack for the specific watermarking algorithm.
A challenge when evaluating watermarking algorithms and their (adaptive)
attacks is to determine whether an adaptive attack is optimal, i.e., it is the
best possible attack. We solve this problem by defining an objective function
and then approach adaptive attacks as an optimization problem. The core idea of
our adaptive attacks is to replicate secret watermarking keys locally by
creating surrogate keys that are differentiable and can be used to optimize the
attack's parameters. We demonstrate for Stable Diffusion models that such an
attacker can break all five surveyed watermarking methods at negligible
degradation in image quality. These findings emphasize the need for more
rigorous robustness testing against adaptive, learnable attackers.


http://arxiv.org/abs/2309.16172
Random and Safe Cache Architecture to Defeat Cache Timing Attacks. (9%)
Guangyuan Hu; Ruby B. Lee
  Caches have been exploited to leak secret information due to the different
times they take to handle memory accesses. Cache timing attacks include
non-speculative cache side and covert channel attacks and cache-based
speculative execution attacks. We first present a systematic view of the attack
and defense space and show that no existing defense has addressed both
speculative and non-speculative cache timing attack families, which we do in
this paper. We propose Random and Safe (RaS) cache architectures to decorrelate
the cache state changes from memory requests. RaS fills the cache with ``safe''
cache lines that are likely to be used in the future, rather than with
demand-fetched, security-sensitive lines. RaS captures a group of safe
addresses during runtime and fetches addresses randomly displaced from these
addresses. Our proposed RaS architecture is flexible to allow
security-performance trade-offs. We show different designs of RaS architectures
that can defeat cache side-channel attacks and cache-based speculative
execution attacks. The RaS variant against cache-based speculative execution
attacks has 4.2% average performance overhead and other RaS variants against
both attack families have 7.9% to 45.2% average overhead. For some benchmarks,
RaS defenses improve the performance while providing security.


http://arxiv.org/abs/2309.16631
Robust Offline Reinforcement Learning -- Certify the Confidence Interval. (4%)
Jiarui Yao; Simon Shaolei Du
  Currently, reinforcement learning (RL), especially deep RL, has received more
and more attention in the research area. However, the security of RL has been
an obvious problem due to the attack manners becoming mature. In order to
defend against such adversarial attacks, several practical approaches are
developed, such as adversarial training, data filtering, etc. However, these
methods are mostly based on empirical algorithms and experiments, without
rigorous theoretical analysis of the robustness of the algorithms. In this
paper, we develop an algorithm to certify the robustness of a given policy
offline with random smoothing, which could be proven and conducted as
efficiently as ones without random smoothing. Experiments on different
environments confirm the correctness of our algorithm.


http://arxiv.org/abs/2309.16314
A Primer on Bayesian Neural Networks: Review and Debates. (2%)
Julyan Arbel; Konstantinos Pitas; Mariia Vladimirova; Vincent Fortuin
  Neural networks have achieved remarkable performance across various problem
domains, but their widespread applicability is hindered by inherent limitations
such as overconfidence in predictions, lack of interpretability, and
vulnerability to adversarial attacks. To address these challenges, Bayesian
neural networks (BNNs) have emerged as a compelling extension of conventional
neural networks, integrating uncertainty estimation into their predictive
capabilities.
  This comprehensive primer presents a systematic introduction to the
fundamental concepts of neural networks and Bayesian inference, elucidating
their synergistic integration for the development of BNNs. The target audience
comprises statisticians with a potential background in Bayesian methods but
lacking deep learning expertise, as well as machine learners proficient in deep
neural networks but with limited exposure to Bayesian statistics. We provide an
overview of commonly employed priors, examining their impact on model behavior
and performance. Additionally, we delve into the practical considerations
associated with training and inference in BNNs.
  Furthermore, we explore advanced topics within the realm of BNN research,
acknowledging the existence of ongoing debates and controversies. By offering
insights into cutting-edge developments, this primer not only equips
researchers and practitioners with a solid foundation in BNNs, but also
illuminates the potential applications of this dynamic field. As a valuable
resource, it fosters an understanding of BNNs and their promising prospects,
facilitating further advancements in the pursuit of knowledge and innovation.


http://arxiv.org/abs/2309.16096
Adversarial Examples Might be Avoidable: The Role of Data Concentration in Adversarial Robustness. (95%)
Ambar Pal; Jeremias Sulam; René Vidal
  The susceptibility of modern machine learning classifiers to adversarial
examples has motivated theoretical results suggesting that these might be
unavoidable. However, these results can be too general to be applicable to
natural data distributions. Indeed, humans are quite robust for tasks involving
vision. This apparent conflict motivates a deeper dive into the question: Are
adversarial examples truly unavoidable? In this work, we theoretically
demonstrate that a key property of the data distribution -- concentration on
small-volume subsets of the input space -- determines whether a robust
classifier exists. We further demonstrate that, for a data distribution
concentrated on a union of low-dimensional linear subspaces, exploiting data
structure naturally leads to classifiers that enjoy good robustness guarantees,
improving upon methods for provable certification in certain regimes.


http://arxiv.org/abs/2309.15519
Defending Against Physical Adversarial Patch Attacks on Infrared Human Detection. (92%)
Lukas Strack; Futa Waseda; Huy H. Nguyen; Yinqiang Zheng; Isao Echizen
  Infrared detection is an emerging technique for safety-critical tasks owing
to its remarkable anti-interference capability. However, recent studies have
revealed that it is vulnerable to physically-realizable adversarial patches,
posing risks in its real-world applications. To address this problem, we are
the first to investigate defense strategies against adversarial patch attacks
on infrared detection, especially human detection. We have devised a
straightforward defense strategy, patch-based occlusion-aware detection (POD),
which efficiently augments training samples with random patches and
subsequently detects them. POD not only robustly detects people but also
identifies adversarial patch locations. Surprisingly, while being extremely
computationally efficient, POD easily generalizes to state-of-the-art
adversarial patch attacks that are unseen during training. Furthermore, POD
improves detection precision even in a clean (i.e., no-patch) situation due to
the data augmentation effect. Evaluation demonstrated that POD is robust to
adversarial patches of various shapes and sizes. The effectiveness of our
baseline approach is shown to be a viable defense mechanism for real-world
infrared human detection systems, paving the way for exploring future research
directions.


http://arxiv.org/abs/2309.15669
On Computational Entanglement and Its Interpretation in Adversarial Machine Learning. (92%)
YenLung Lai; Xingbo Dong; Zhe Jin
  Adversarial examples in machine learning has emerged as a focal point of
research due to their remarkable ability to deceive models with seemingly
inconspicuous input perturbations, potentially resulting in severe
consequences. In this study, we embark on a comprehensive exploration of
adversarial machine learning models, shedding light on their intrinsic
complexity and interpretability. Our investigation reveals intriguing links
between machine learning model complexity and Einstein's theory of special
relativity, through the concept of entanglement. More specific, we define
entanglement computationally and demonstrate that distant feature samples can
exhibit strong correlations, akin to entanglement in quantum realm. This
revelation challenges conventional perspectives in describing the phenomenon of
adversarial transferability observed in contemporary machine learning models.
By drawing parallels with the relativistic effects of time dilation and length
contraction during computation, we gain deeper insights into adversarial
machine learning, paving the way for more robust and interpretable models in
this rapidly evolving field.


http://arxiv.org/abs/2309.15418
Automatic Feature Fairness in Recommendation via Adversaries. (33%)
Hengchang Hu; Yiming Cao; Zhankui He; Samson Tan; Min-Yen Kan
  Fairness is a widely discussed topic in recommender systems, but its
practical implementation faces challenges in defining sensitive features while
maintaining recommendation accuracy. We propose feature fairness as the
foundation to achieve equitable treatment across diverse groups defined by
various feature combinations. This improves overall accuracy through balanced
feature generalizability. We introduce unbiased feature learning through
adversarial training, using adversarial perturbation to enhance feature
representation. The adversaries improve model generalization for
under-represented features. We adapt adversaries automatically based on two
forms of feature biases: frequency and combination variety of feature values.
This allows us to dynamically adjust perturbation strengths and adversarial
training weights. Stronger perturbations are applied to feature values with
fewer combination varieties to improve generalization, while higher weights for
low-frequency features address training imbalances. We leverage the Adaptive
Adversarial perturbation based on the widely-applied Factorization Machine
(AAFM) as our backbone model. In experiments, AAFM surpasses strong baselines
in both fairness and accuracy measures. AAFM excels in providing item- and
user-fairness for single- and multi-feature tasks, showcasing their versatility
and scalability. To maintain good accuracy, we find that adversarial
perturbation must be well-managed: during training, perturbations should not
overly persist and their strengths should decay.


http://arxiv.org/abs/2309.15770
Generating Transferable Adversarial Simulation Scenarios for Self-Driving via Neural Rendering. (11%)
Yasasa Abeysirigoonawardena; Kevin Xie; Chuhan Chen; Salar Hosseini; Ruiting Chen; Ruiqi Wang; Florian Shkurti
  Self-driving software pipelines include components that are learned from a
significant number of training examples, yet it remains challenging to evaluate
the overall system's safety and generalization performance. Together with
scaling up the real-world deployment of autonomous vehicles, it is of critical
importance to automatically find simulation scenarios where the driving
policies will fail. We propose a method that efficiently generates adversarial
simulation scenarios for autonomous driving by solving an optimal control
problem that aims to maximally perturb the policy from its nominal trajectory.
  Given an image-based driving policy, we show that we can inject new objects
in a neural rendering representation of the deployment scene, and optimize
their texture in order to generate adversarial sensor inputs to the policy. We
demonstrate that adversarial scenarios discovered purely in the neural renderer
(surrogate scene) can often be successfully transferred to the deployment
scene, without further optimization. We demonstrate this transfer occurs both
in simulated and real environments, provided the learned surrogate scene is
sufficiently close to the deployment scene.


http://arxiv.org/abs/2309.15687
Breaking NoC Anonymity using Flow Correlation Attack. (2%)
Hansika Weerasena; Pan Zhixin; Khushboo Rani; Prabhat Mishra
  Network-on-Chip (NoC) is widely used as the internal communication fabric in
today's multicore System-on-Chip (SoC) designs. Security of the on-chip
communication is crucial because exploiting any vulnerability in shared NoC
would be a goldmine for an attacker. NoC security relies on effective
countermeasures against diverse attacks. We investigate the security strength
of existing anonymous routing protocols in NoC architectures. Specifically,
this paper makes two important contributions. We show that the existing
anonymous routing is vulnerable to machine learning (ML) based flow correlation
attacks on NoCs. We propose a lightweight anonymous routing that use traffic
obfuscation techniques which can defend against ML-based flow correlation
attacks. Experimental studies using both real and synthetic traffic reveal that
our proposed attack is successful against state-of-the-art anonymous routing in
NoC architectures with a high accuracy (up to 99%) for diverse traffic
patterns, while our lightweight countermeasure can defend against ML-based
attacks with minor hardware and performance overhead.


http://arxiv.org/abs/2310.06855
Genetic Algorithm-Based Dynamic Backdoor Attack on Federated Learning-Based Network Traffic Classification. (1%)
Mahmoud Nazzal; Nura Aljaafari; Ahmed Sawalmeh; Abdallah Khreishah; Muhammad Anan; Abdulelah Algosaibi; Mohammed Alnaeem; Adel Aldalbahi; Abdulaziz Alhumam; Conrado P. Vizcarra; Shadan Alhamed
  Federated learning enables multiple clients to collaboratively contribute to
the learning of a global model orchestrated by a central server. This learning
scheme promotes clients' data privacy and requires reduced communication
overheads. In an application like network traffic classification, this helps
hide the network vulnerabilities and weakness points. However, federated
learning is susceptible to backdoor attacks, in which adversaries inject
manipulated model updates into the global model. These updates inject a salient
functionality in the global model that can be launched with specific input
patterns. Nonetheless, the vulnerability of network traffic classification
models based on federated learning to these attacks remains unexplored. In this
paper, we propose GABAttack, a novel genetic algorithm-based backdoor attack
against federated learning for network traffic classification. GABAttack
utilizes a genetic algorithm to optimize the values and locations of backdoor
trigger patterns, ensuring a better fit with the input and the model. This
input-tailored dynamic attack is promising for improved attack evasiveness
while being effective. Extensive experiments conducted over real-world network
datasets validate the success of the proposed GABAttack in various situations
while maintaining almost invisible activity. This research serves as an
alarming call for network security experts and practitioners to develop robust
defense measures against such attacks.


http://arxiv.org/abs/2310.07726
Towards the Vulnerability of Watermarking Artificial Intelligence Generated Content. (1%)
Guanlin Li; Yifei Chen; Jie Zhang; Jiwei Li; Shangwei Guo; Tianwei Zhang
  Artificial Intelligence Generated Content (AIGC) is gaining great popularity
in social media, with many commercial services available. These services
leverage advanced generative models, such as latent diffusion models and large
language models, to generate creative content (e.g., realistic images, fluent
sentences) for users. The usage of such generated content needs to be highly
regulated, as the service providers need to ensure the users do not violate the
usage policies (e.g., abuse for commercialization, generating and distributing
unsafe content).
  Numerous watermarking approaches have been proposed recently. However, in
this paper, we show that an adversary can easily break these watermarking
mechanisms. Specifically, we consider two possible attacks. (1) Watermark
removal: the adversary can easily erase the embedded watermark from the
generated content and then use it freely without the regulation of the service
provider. (2) Watermark forge: the adversary can create illegal content with
forged watermarks from another user, causing the service provider to make wrong
attributions. We propose WMaGi, a unified framework to achieve both attacks in
a holistic way. The key idea is to leverage a pre-trained diffusion model for
content processing, and a generative adversarial network for watermark removing
or forging. We evaluate WMaGi on different datasets and embedding setups. The
results prove that it can achieve high success rates while maintaining the
quality of the generated content. Compared with existing diffusion model-based
attacks, WMaGi is 5,050$\sim$11,000$\times$ faster.


http://arxiv.org/abs/2309.14700
Structure Invariant Transformation for better Adversarial Transferability. (99%)
Xiaosen Wang; Zeliang Zhang; Jianping Zhang
  Given the severe vulnerability of Deep Neural Networks (DNNs) against
adversarial examples, there is an urgent need for an effective adversarial
attack to identify the deficiencies of DNNs in security-sensitive applications.
As one of the prevalent black-box adversarial attacks, the existing
transfer-based attacks still cannot achieve comparable performance with the
white-box attacks. Among these, input transformation based attacks have shown
remarkable effectiveness in boosting transferability. In this work, we find
that the existing input transformation based attacks transform the input image
globally, resulting in limited diversity of the transformed images. We
postulate that the more diverse transformed images result in better
transferability. Thus, we investigate how to locally apply various
transformations onto the input image to improve such diversity while preserving
the structure of image. To this end, we propose a novel input transformation
based attack, called Structure Invariant Attack (SIA), which applies a random
image transformation onto each image block to craft a set of diverse images for
gradient calculation. Extensive experiments on the standard ImageNet dataset
demonstrate that SIA exhibits much better transferability than the existing
SOTA input transformation based attacks on CNN-based and transformer-based
models, showing its generality and superiority in boosting transferability.
Code is available at https://github.com/xiaosen-wang/SIT.


http://arxiv.org/abs/2309.15087
Privacy-preserving and Privacy-attacking Approaches for Speech and Audio -- A Survey. (16%)
Yuchen Liu; Apu Kapadia; Donald Williamson
  In contemporary society, voice-controlled devices, such as smartphones and
home assistants, have become pervasive due to their advanced capabilities and
functionality. The always-on nature of their microphones offers users the
convenience of readily accessing these devices. However, recent research and
events have revealed that such voice-controlled devices are prone to various
forms of malicious attacks, hence making it a growing concern for both users
and researchers to safeguard against such attacks. Despite the numerous studies
that have investigated adversarial attacks and privacy preservation for images,
a conclusive study of this nature has not been conducted for the audio domain.
Therefore, this paper aims to examine existing approaches for
privacy-preserving and privacy-attacking strategies for audio and speech. To
achieve this goal, we classify the attack and defense scenarios into several
categories and provide detailed analysis of each approach. We also interpret
the dissimilarities between the various approaches, highlight their
contributions, and examine their limitations. Our investigation reveals that
voice-controlled devices based on neural networks are inherently susceptible to
specific types of attacks. Although it is possible to enhance the robustness of
such models to certain forms of attack, more sophisticated approaches are
required to comprehensively safeguard user privacy.


http://arxiv.org/abs/2309.15386
Neural Stochastic Differential Equations for Robust and Explainable Analysis of Electromagnetic Unintended Radiated Emissions. (2%)
Sumit Kumar Jha; Susmit Jha; Rickard Ewetz; Alvaro Velasquez
  We present a comprehensive evaluation of the robustness and explainability of
ResNet-like models in the context of Unintended Radiated Emission (URE)
classification and suggest a new approach leveraging Neural Stochastic
Differential Equations (SDEs) to address identified limitations. We provide an
empirical demonstration of the fragility of ResNet-like models to Gaussian
noise perturbations, where the model performance deteriorates sharply and its
F1-score drops to near insignificance at 0.008 with a Gaussian noise of only
0.5 standard deviation. We also highlight a concerning discrepancy where the
explanations provided by ResNet-like models do not reflect the inherent
periodicity in the input data, a crucial attribute in URE detection from stable
devices. In response to these findings, we propose a novel application of
Neural SDEs to build models for URE classification that are not only robust to
noise but also provide more meaningful and intuitive explanations. Neural SDE
models maintain a high F1-score of 0.93 even when exposed to Gaussian noise
with a standard deviation of 0.5, demonstrating superior resilience to ResNet
models. Neural SDE models successfully recover the time-invariant or periodic
horizontal bands from the input data, a feature that was conspicuously missing
in the explanations generated by ResNet-like models. This advancement presents
a small but significant step in the development of robust and interpretable
models for real-world URE applications where data is inherently noisy and
assurance arguments demand interpretable machine learning predictions.


http://arxiv.org/abs/2309.15224
Collaborative Watermarking for Adversarial Speech Synthesis. (1%)
Lauri Aalto University, Finland Juvela; Xin National Institute of Informatics, Japan Wang
  Advances in neural speech synthesis have brought us technology that is not
only close to human naturalness, but is also capable of instant voice cloning
with little data, and is highly accessible with pre-trained models available.
Naturally, the potential flood of generated content raises the need for
synthetic speech detection and watermarking. Recently, considerable research
effort in synthetic speech detection has been related to the Automatic Speaker
Verification and Spoofing Countermeasure Challenge (ASVspoof), which focuses on
passive countermeasures. This paper takes a complementary view to generated
speech detection: a synthesis system should make an active effort to watermark
the generated speech in a way that aids detection by another machine, but
remains transparent to a human listener. We propose a collaborative training
scheme for synthetic speech watermarking and show that a HiFi-GAN neural
vocoder collaborating with the ASVspoof 2021 baseline countermeasure models
consistently improves detection performance over conventional classifier
training. Furthermore, we demonstrate how collaborative training can be paired
with augmentation strategies for added robustness against noise and
time-stretching. Finally, listening tests demonstrate that collaborative
training has little adverse effect on perceptual quality of vocoded speech.


http://arxiv.org/abs/2309.14585
DifAttack: Query-Efficient Black-Box Attack via Disentangled Feature Space. (99%)
Liu Jun; Zhou Jiantao; Zeng Jiandian; Jinyu Tian
  This work investigates efficient score-based black-box adversarial attacks
with a high Attack Success Rate (ASR) and good generalizability. We design a
novel attack method based on a Disentangled Feature space, called DifAttack,
which differs significantly from the existing ones operating over the entire
feature space. Specifically, DifAttack firstly disentangles an image's latent
feature into an adversarial feature and a visual feature, where the former
dominates the adversarial capability of an image, while the latter largely
determines its visual appearance. We train an autoencoder for the
disentanglement by using pairs of clean images and their Adversarial Examples
(AEs) generated from available surrogate models via white-box attack methods.
Eventually, DifAttack iteratively optimizes the adversarial feature according
to the query feedback from the victim model until a successful AE is generated,
while keeping the visual feature unaltered. In addition, due to the avoidance
of using surrogate models' gradient information when optimizing AEs for
black-box models, our proposed DifAttack inherently possesses better attack
capability in the open-set scenario, where the training dataset of the victim
model is unknown. Extensive experimental results demonstrate that our method
achieves significant improvements in ASR and query efficiency simultaneously,
especially in the targeted attack and open-set scenarios. The code will be
available at https://github.com/csjunjun/DifAttack.git soon.


http://arxiv.org/abs/2309.14615
Gray-box Adversarial Attack of Deep Reinforcement Learning-based Trading Agents. (98%)
Foozhan Ataiefard; Hadi Hemmati
  In recent years, deep reinforcement learning (Deep RL) has been successfully
implemented as a smart agent in many systems such as complex games,
self-driving cars, and chat-bots. One of the interesting use cases of Deep RL
is its application as an automated stock trading agent. In general, any
automated trading agent is prone to manipulations by adversaries in the trading
environment. Thus studying their robustness is vital for their success in
practice. However, typical mechanism to study RL robustness, which is based on
white-box gradient-based adversarial sample generation techniques (like FGSM),
is obsolete for this use case, since the models are protected behind secure
international exchange APIs, such as NASDAQ. In this research, we demonstrate
that a "gray-box" approach for attacking a Deep RL-based trading agent is
possible by trading in the same stock market, with no extra access to the
trading agent. In our proposed approach, an adversary agent uses a hybrid Deep
Neural Network as its policy consisting of Convolutional layers and
fully-connected layers. On average, over three simulated trading market
configurations, the adversary policy proposed in this research is able to
reduce the reward values by 214.17%, which results in reducing the potential
profits of the baseline by 139.4%, ensemble method by 93.7%, and an automated
trading software developed by our industrial partner by 85.5%, while consuming
significantly less budget than the victims (427.77%, 187.16%, and 66.97%,
respectively).


http://arxiv.org/abs/2309.14122
SurrogatePrompt: Bypassing the Safety Filter of Text-To-Image Models via Substitution. (1%)
Zhongjie Ba; Jieming Zhong; Jiachen Lei; Peng Cheng; Qinglong Wang; Zhan Qin; Zhibo Wang; Kui Ren
  Advanced text-to-image models such as DALL-E 2 and Midjourney possess the
capacity to generate highly realistic images, raising significant concerns
regarding the potential proliferation of unsafe content. This includes adult,
violent, or deceptive imagery of political figures. Despite claims of rigorous
safety mechanisms implemented in these models to restrict the generation of
not-safe-for-work (NSFW) content, we successfully devise and exhibit the first
prompt attacks on Midjourney, resulting in the production of abundant
photorealistic NSFW images. We reveal the fundamental principles of such prompt
attacks and suggest strategically substituting high-risk sections within a
suspect prompt to evade closed-source safety measures. Our novel framework,
SurrogatePrompt, systematically generates attack prompts, utilizing large
language models, image-to-text, and image-to-image modules to automate attack
prompt creation at scale. Evaluation results disclose an 88% success rate in
bypassing Midjourney's proprietary safety filter with our attack prompts,
leading to the generation of counterfeit images depicting political figures in
violent scenarios. Both subjective and objective assessments validate that the
images generated from our attack prompts present considerable safety hazards.


http://arxiv.org/abs/2309.13857
Adversarial Attacks on Video Object Segmentation with Hard Region Discovery. (99%)
Ping Li; Yu Zhang; Li Yuan; Jian Zhao; Xianghua Xu; Xiaoqin Zhang
  Video object segmentation has been applied to various computer vision tasks,
such as video editing, autonomous driving, and human-robot interaction.
However, the methods based on deep neural networks are vulnerable to
adversarial examples, which are the inputs attacked by almost
human-imperceptible perturbations, and the adversary (i.e., attacker) will fool
the segmentation model to make incorrect pixel-level predictions. This will
rise the security issues in highly-demanding tasks because small perturbations
to the input video will result in potential attack risks. Though adversarial
examples have been extensively used for classification, it is rarely studied in
video object segmentation. Existing related methods in computer vision either
require prior knowledge of categories or cannot be directly applied due to the
special design for certain tasks, failing to consider the pixel-wise region
attack. Hence, this work develops an object-agnostic adversary that has
adversarial impacts on VOS by first-frame attacking via hard region discovery.
Particularly, the gradients from the segmentation model are exploited to
discover the easily confused region, in which it is difficult to identify the
pixel-wise objects from the background in a frame. This provides a hardness map
that helps to generate perturbations with a stronger adversarial power for
attacking the first frame. Empirical studies on three benchmarks indicate that
our attacker significantly degrades the performance of several state-of-the-art
video object segmentation models.


http://arxiv.org/abs/2309.13609
Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attacks. (98%)
Ao-Xiang Zhang; Yu Ran; Weixuan Tang; Yuan-Gen Wang
  No-Reference Video Quality Assessment (NR-VQA) plays an essential role in
improving the viewing experience of end-users. Driven by deep learning, recent
NR-VQA models based on Convolutional Neural Networks (CNNs) and Transformers
have achieved outstanding performance. To build a reliable and practical
assessment system, it is of great necessity to evaluate their robustness.
However, such issue has received little attention in the academic community. In
this paper, we make the first attempt to evaluate the robustness of NR-VQA
models against adversarial attacks, and propose a patch-based random search
method for black-box attack. Specifically, considering both the attack effect
on quality score and the visual quality of adversarial video, the attack
problem is formulated as misleading the estimated quality score under the
constraint of just-noticeable difference (JND). Built upon such formulation, a
novel loss function called Score-Reversed Boundary Loss is designed to push the
adversarial video's estimated quality score far away from its ground-truth
score towards a specific boundary, and the JND constraint is modeled as a
strict $L_2$ and $L_\infty$ norm restriction. By this means, both white-box and
black-box attacks can be launched in an effective and imperceptible manner. The
source code is available at https://github.com/GZHU-DVL/AttackVQA.


http://arxiv.org/abs/2309.13841
On the Effectiveness of Adversarial Samples against Ensemble Learning-based Windows PE Malware Detectors. (86%)
Trong-Nghia To; Danh Le Kim; Do Thi Thu Hien; Nghi Hoang Khoa; Hien Do Hoang; Phan The Duy; Van-Hau Pham
  Recently, there has been a growing focus and interest in applying machine
learning (ML) to the field of cybersecurity, particularly in malware detection
and prevention. Several research works on malware analysis have been proposed,
offering promising results for both academic and practical applications. In
these works, the use of Generative Adversarial Networks (GANs) or Reinforcement
Learning (RL) can aid malware creators in crafting metamorphic malware that
evades antivirus software. In this study, we propose a mutation system to
counteract ensemble learning-based detectors by combining GANs and an RL model,
overcoming the limitations of the MalGAN model. Our proposed FeaGAN model is
built based on MalGAN by incorporating an RL model called the Deep Q-network
anti-malware Engines Attacking Framework (DQEAF). The RL model addresses three
key challenges in performing adversarial attacks on Windows Portable Executable
malware, including format preservation, executability preservation, and
maliciousness preservation. In the FeaGAN model, ensemble learning is utilized
to enhance the malware detector's evasion ability, with the generated
adversarial patterns. The experimental results demonstrate that 100\% of the
selected mutant samples preserve the format of executable files, while certain
successes in both executability preservation and maliciousness preservation are
achieved, reaching a stable success rate.


http://arxiv.org/abs/2310.03033
Benchmarking Local Robustness of High-Accuracy Binary Neural Networks for Enhanced Traffic Sign Recognition. (80%)
Andreea Postovan; Mădălina Eraşcu
  Traffic signs play a critical role in road safety and traffic management for
autonomous driving systems. Accurate traffic sign classification is essential
but challenging due to real-world complexities like adversarial examples and
occlusions. To address these issues, binary neural networks offer promise in
constructing classifiers suitable for resource-constrained devices.
  In our previous work, we proposed high-accuracy BNN models for traffic sign
recognition, focusing on compact size for limited computation and energy
resources. To evaluate their local robustness, this paper introduces a set of
benchmark problems featuring layers that challenge state-of-the-art
verification tools. These layers include binarized convolutions, max pooling,
batch normalization, fully connected. The difficulty of the verification
problem is given by the high number of network parameters (905k - 1.7 M), of
the input dimension (2.7k-12k), and of the number of regions (43) as well by
the fact that the neural networks are not sparse.
  The proposed BNN models and local robustness properties can be checked at
https://github.com/ChristopherBrix/vnncomp2023_benchmarks/tree/main/benchmarks/traffic_signs_recognition.
  The results of the 4th International Verification of Neural Networks
Competition (VNN-COMP'23) revealed the fact that 4, out of 7, solvers can
handle many of our benchmarks randomly selected (minimum is 6, maximum is 36,
out of 45). Surprisingly, tools output also wrong results or missing
counterexample (ranging from 1 to 4). Currently, our focus lies in exploring
the possibility of achieving a greater count of solved instances by extending
the allotted time (previously set at 8 minutes). Furthermore, we are intrigued
by the reasons behind the erroneous outcomes provided by the tools for certain
benchmarks.


http://arxiv.org/abs/2309.13794
Projected Randomized Smoothing for Certified Adversarial Robustness. (76%)
Samuel Pfrommer; Brendon G. Anderson; Somayeh Sojoudi
  Randomized smoothing is the current state-of-the-art method for producing
provably robust classifiers. While randomized smoothing typically yields robust
$\ell_2$-ball certificates, recent research has generalized provable robustness
to different norm balls as well as anisotropic regions. This work considers a
classifier architecture that first projects onto a low-dimensional
approximation of the data manifold and then applies a standard classifier. By
performing randomized smoothing in the low-dimensional projected space, we
characterize the certified region of our smoothed composite classifier back in
the high-dimensional input space and prove a tractable lower bound on its
volume. We show experimentally on CIFAR-10 and SVHN that classifiers without
the initial projection are vulnerable to perturbations that are normal to the
data manifold and yet are captured by the certified regions of our method. We
compare the volume of our certified regions against various baselines and show
that our method improves on the state-of-the-art by many orders of magnitude.


http://arxiv.org/abs/2309.13763
Combining Two Adversarial Attacks Against Person Re-Identification Systems. (73%)
Eduardo de O. Andrade; Igor Garcia Ballhausen Sampaio; Joris Guérin; José Viterbo
  The field of Person Re-Identification (Re-ID) has received much attention
recently, driven by the progress of deep neural networks, especially for image
classification. The problem of Re-ID consists in identifying individuals
through images captured by surveillance cameras in different scenarios.
Governments and companies are investing a lot of time and money in Re-ID
systems for use in public safety and identifying missing persons. However,
several challenges remain for successfully implementing Re-ID, such as
occlusions and light reflections in people's images. In this work, we focus on
adversarial attacks on Re-ID systems, which can be a critical threat to the
performance of these systems. In particular, we explore the combination of
adversarial attacks against Re-ID models, trying to strengthen the decrease in
the classification results. We conduct our experiments on three datasets:
DukeMTMC-ReID, Market-1501, and CUHK03. We combine the use of two types of
adversarial attacks, P-FGSM and Deep Mis-Ranking, applied to two popular Re-ID
models: IDE (ResNet-50) and AlignedReID. The best result demonstrates a
decrease of 3.36% in the Rank-10 metric for AlignedReID applied to CUHK03. We
also try to use Dropout during the inference as a defense method.


http://arxiv.org/abs/2309.13579
Seeing Is Not Always Believing: Invisible Collision Attack and Defence on Pre-Trained Models. (2%)
Minghang Deng; Zhong Zhang; Junming Shao
  Large-scale pre-trained models (PTMs) such as BERT and GPT have achieved
great success in diverse fields. The typical paradigm is to pre-train a big
deep learning model on large-scale data sets, and then fine-tune the model on
small task-specific data sets for downstream tasks. Although PTMs have rapidly
progressed with wide real-world applications, they also pose significant risks
of potential attacks. Existing backdoor attacks or data poisoning methods often
build up the assumption that the attacker invades the computers of victims or
accesses the target data, which is challenging in real-world scenarios. In this
paper, we propose a novel framework for an invisible attack on PTMs with
enhanced MD5 collision. The key idea is to generate two equal-size models with
the same MD5 checksum by leveraging the MD5 chosen-prefix collision.
Afterwards, the two ``same" models will be deployed on public websites to
induce victims to download the poisoned model. Unlike conventional attacks on
deep learning models, this new attack is flexible, covert, and
model-independent. Additionally, we propose a simple defensive strategy for
recognizing the MD5 chosen-prefix collision and provide a theoretical
justification for its feasibility. We extensively validate the effectiveness
and stealthiness of our proposed attack and defensive method on different
models and data sets.


http://arxiv.org/abs/2309.13256
Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks. (61%)
Zhaohan Xi; Tianyu Du; Changjiang Li; Ren Pang; Shouling Ji; Jinghui Chen; Fenglong Ma; Ting Wang
  Pre-trained language models (PLMs) have demonstrated remarkable performance
as few-shot learners. However, their security risks under such settings are
largely unexplored. In this work, we conduct a pilot study showing that PLMs as
few-shot learners are highly vulnerable to backdoor attacks while existing
defenses are inadequate due to the unique challenges of few-shot scenarios. To
address such challenges, we advocate MDP, a novel lightweight, pluggable, and
effective defense for PLMs as few-shot learners. Specifically, MDP leverages
the gap between the masking-sensitivity of poisoned and clean samples: with
reference to the limited few-shot data as distributional anchors, it compares
the representations of given samples under varying masking and identifies
poisoned samples as ones with significant variations. We show analytically that
MDP creates an interesting dilemma for the attacker to choose between attack
effectiveness and detection evasiveness. The empirical evaluation using
benchmark datasets and representative attacks validates the efficacy of MDP.


http://arxiv.org/abs/2309.13475
Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers. (1%)
Aryaman Gupta; Kaustav Chakraborty; Somil Bansal
  Autonomous systems, such as self-driving cars and drones, have made
significant strides in recent years by leveraging visual inputs and machine
learning for decision-making and control. Despite their impressive performance,
these vision-based controllers can make erroneous predictions when faced with
novel or out-of-distribution inputs. Such errors can cascade to catastrophic
system failures and compromise system safety. In this work, we introduce a
run-time anomaly monitor to detect and mitigate such closed-loop, system-level
failures. Specifically, we leverage a reachability-based framework to
stress-test the vision-based controller offline and mine its system-level
failures. This data is then used to train a classifier that is leveraged online
to flag inputs that might cause system breakdowns. The anomaly detector
highlights issues that transcend individual modules and pertain to the safety
of the overall system. We also design a fallback controller that robustly
handles these detected anomalies to preserve system safety. We validate the
proposed approach on an autonomous aircraft taxiing system that uses a
vision-based controller for taxiing. Our results show the efficacy of the
proposed approach in identifying and handling system-level anomalies,
outperforming methods such as prediction error-based detection, and ensembling,
thereby enhancing the overall safety and robustness of autonomous systems.


http://arxiv.org/abs/2309.13444
Moving Target Defense based Secured Network Slicing System in the O-RAN Architecture. (1%)
Mojdeh Karbalaee Motalleb; Chafika Benzaïd; Tarik Taleb; Vahid Shah-Mansouri
  The open radio access network (O-RAN) architecture's native virtualization
and embedded intelligence facilitate RAN slicing and enable comprehensive
end-to-end services in post-5G networks. However, any vulnerabilities could
harm security. Therefore, artificial intelligence (AI) and machine learning
(ML) security threats can even threaten O-RAN benefits. This paper proposes a
novel approach to estimating the optimal number of predefined VNFs for each
slice while addressing secure AI/ML methods for dynamic service admission
control and power minimization in the O-RAN architecture. We solve this problem
on two-time scales using mathematical methods for determining the predefined
number of VNFs on a large time scale and the proximal policy optimization
(PPO), a Deep Reinforcement Learning algorithm, for solving dynamic service
admission control and power minimization for different slices on a small-time
scale. To secure the ML system for O-RAN, we implement a moving target defense
(MTD) strategy to prevent poisoning attacks by adding uncertainty to the
system. Our experimental results show that the proposed PPO-based service
admission control approach achieves an admission rate above 80\% and that the
MTD strategy effectively strengthens the robustness of the PPO method against
adversarial attacks.


http://arxiv.org/abs/2309.13245
RBFormer: Improve Adversarial Robustness of Transformer by Robust Bias. (99%)
Hao Cheng; Jinhao Duan; Hui Li; Lyutianyang Zhang; Jiahang Cao; Ping Wang; Jize Zhang; Kaidi Xu; Renjing Xu
  Recently, there has been a surge of interest and attention in
Transformer-based structures, such as Vision Transformer (ViT) and Vision
Multilayer Perceptron (VMLP). Compared with the previous convolution-based
structures, the Transformer-based structure under investigation showcases a
comparable or superior performance under its distinctive attention-based input
token mixer strategy. Introducing adversarial examples as a robustness
consideration has had a profound and detrimental impact on the performance of
well-established convolution-based structures. This inherent vulnerability to
adversarial attacks has also been demonstrated in Transformer-based structures.
In this paper, our emphasis lies on investigating the intrinsic robustness of
the structure rather than introducing novel defense measures against
adversarial attacks. To address the susceptibility to robustness issues, we
employ a rational structure design approach to mitigate such vulnerabilities.
Specifically, we enhance the adversarial robustness of the structure by
increasing the proportion of high-frequency structural robust biases. As a
result, we introduce a novel structure called Robust Bias Transformer-based
Structure (RBFormer) that shows robust superiority compared to several existing
baseline structures. Through a series of extensive experiments, RBFormer
outperforms the original structures by a significant margin, achieving an
impressive improvement of +16.12% and +5.04% across different evaluation
criteria on CIFAR-10 and ImageNet-1k, respectively.


http://arxiv.org/abs/2309.13190
Spatial-frequency channels, shape bias, and adversarial robustness. (69%)
Ajay Subramanian; Elena Sizikova; Najib J. Majaj; Denis G. Pelli
  What spatial frequency information do humans and neural networks use to
recognize objects? In neuroscience, critical band masking is an established
tool that can reveal the frequency-selective filters used for object
recognition. Critical band masking measures the sensitivity of recognition
performance to noise added at each spatial frequency. Existing critical band
masking studies show that humans recognize periodic patterns (gratings) and
letters by means of a spatial-frequency filter (or "channel'') that has a
frequency bandwidth of one octave (doubling of frequency). Here, we introduce
critical band masking as a task for network-human comparison and test 14 humans
and 76 neural networks on 16-way ImageNet categorization in the presence of
narrowband noise. We find that humans recognize objects in natural images using
the same one-octave-wide channel that they use for letters and gratings, making
it a canonical feature of human object recognition. On the other hand, the
neural network channel, across various architectures and training strategies,
is 2-4 times as wide as the human channel. In other words, networks are
vulnerable to high and low frequency noise that does not affect human
performance. Adversarial and augmented-image training are commonly used to
increase network robustness and shape bias. Does this training align network
and human object recognition channels? Three network channel properties
(bandwidth, center frequency, peak noise sensitivity) correlate strongly with
shape bias (53% variance explained) and with robustness of
adversarially-trained networks (74% variance explained). Adversarial training
increases robustness but expands the channel bandwidth even further away from
the human bandwidth. Thus, critical band masking reveals that the network
channel is more than twice as wide as the human channel, and that adversarial
training only increases this difference.


http://arxiv.org/abs/2309.12914
VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks. (69%)
Heitor R. Guimarães; Arthur Pimentel; Anderson Avila; Tiago H. Falk
  Keyword spotting (KWS) refers to the task of identifying a set of predefined
words in audio streams. With the advances seen recently with deep neural
networks, it has become a popular technology to activate and control small
devices, such as voice assistants. Relying on such models for edge devices,
however, can be challenging due to hardware constraints. Moreover, as
adversarial attacks have increased against voice-based technologies, developing
solutions robust to such attacks has become crucial. In this work, we propose
VIC-KD, a robust distillation recipe for model compression and adversarial
robustness. Using self-supervised speech representations, we show that imposing
geometric priors to the latent representations of both Teacher and Student
models leads to more robust target models. Experiments on the Google Speech
Commands datasets show that the proposed methodology improves upon current
state-of-the-art robust distillation methods, such as ARD and RSLAD, by 12% and
8% in robust accuracy, respectively.


http://arxiv.org/abs/2309.13016
Understanding Deep Gradient Leakage via Inversion Influence Functions. (11%)
Haobo Zhang; Junyuan Hong; Yuyang Deng; Mehrdad Mahdavi; Jiayu Zhou
  Deep Gradient Leakage (DGL) is a highly effective attack that recovers
private training images from gradient vectors. This attack casts significant
privacy challenges on distributed learning from clients with sensitive data,
where clients are required to share gradients. Defending against such attacks
requires but lacks an understanding of when and how privacy leakage happens,
mostly because of the black-box nature of deep networks. In this paper, we
propose a novel Inversion Influence Function (I$^2$F) that establishes a
closed-form connection between the recovered images and the private gradients
by implicitly solving the DGL problem. Compared to directly solving DGL, I$^2$F
is scalable for analyzing deep networks, requiring only oracle access to
gradients and Jacobian-vector products. We empirically demonstrate that I$^2$F
effectively approximated the DGL generally on different model architectures,
datasets, attack implementations, and noise-based defenses. With this novel
tool, we provide insights into effective gradient perturbation directions, the
unfairness of privacy protection, and privacy-preferred model initialization.
Our codes are provided in
https://github.com/illidanlab/inversion-influence-function.


http://arxiv.org/abs/2309.13150
Pixel-wise Smoothing for Certified Robustness against Camera Motion Perturbations. (10%)
Hanjiang Hu; Zuxin Liu; Linyi Li; Jiacheng Zhu; Ding Zhao
  In recent years, computer vision has made remarkable advancements in
autonomous driving and robotics. However, it has been observed that deep
learning-based visual perception models lack robustness when faced with camera
motion perturbations. The current certification process for assessing
robustness is costly and time-consuming due to the extensive number of image
projections required for Monte Carlo sampling in the 3D camera motion space. To
address these challenges, we present a novel, efficient, and practical
framework for certifying the robustness of 3D-2D projective transformations
against camera motion perturbations. Our approach leverages a smoothing
distribution over the 2D pixel space instead of in the 3D physical space,
eliminating the need for costly camera motion sampling and significantly
enhancing the efficiency of robustness certifications. With the pixel-wise
smoothed classifier, we are able to fully upper bound the projection errors
using a technique of uniform partitioning in camera motion space. Additionally,
we extend our certification framework to a more general scenario where only a
single-frame point cloud is required in the projection oracle. This is achieved
by deriving Lipschitz-based approximated partition intervals. Through extensive
experimentation, we validate the trade-off between effectiveness and efficiency
enabled by our proposed method. Remarkably, our approach achieves approximately
80% certified accuracy while utilizing only 30% of the projected image frames.


http://arxiv.org/abs/2309.13038
Privacy Assessment on Reconstructed Images: Are Existing Evaluation Metrics Faithful to Human Perception? (5%)
Xiaoxiao Sun; Nidham Gazagnadou; Vivek Sharma; Lingjuan Lyu; Hongdong Li; Liang Zheng
  Hand-crafted image quality metrics, such as PSNR and SSIM, are commonly used
to evaluate model privacy risk under reconstruction attacks. Under these
metrics, reconstructed images that are determined to resemble the original one
generally indicate more privacy leakage. Images determined as overall
dissimilar, on the other hand, indicate higher robustness against attack.
However, there is no guarantee that these metrics well reflect human opinions,
which, as a judgement for model privacy leakage, are more trustworthy. In this
paper, we comprehensively study the faithfulness of these hand-crafted metrics
to human perception of privacy information from the reconstructed images. On 5
datasets ranging from natural images, faces, to fine-grained classes, we use 4
existing attack methods to reconstruct images from many different
classification models and, for each reconstructed image, we ask multiple human
annotators to assess whether this image is recognizable. Our studies reveal
that the hand-crafted metrics only have a weak correlation with the human
evaluation of privacy leakage and that even these metrics themselves often
contradict each other. These observations suggest risks of current metrics in
the community. To address this potential risk, we propose a learning-based
measure called SemSim to evaluate the Semantic Similarity between the original
and reconstructed images. SemSim is trained with a standard triplet loss, using
an original image as an anchor, one of its recognizable reconstructed images as
a positive sample, and an unrecognizable one as a negative. By training on
human annotations, SemSim exhibits a greater reflection of privacy leakage on
the semantic level. We show that SemSim has a significantly higher correlation
with human judgment compared with existing metrics. Moreover, this strong
correlation generalizes to unseen datasets, models and attack methods.


http://arxiv.org/abs/2309.13002
Expressive variational quantum circuits provide inherent privacy in federated learning. (1%)
Niraj Kumar; Jamie Heredge; Changhao Li; Shaltiel Eloul; Shree Hari Sureshbabu; Marco Pistoia
  Federated learning has emerged as a viable distributed solution to train
machine learning models without the actual need to share data with the central
aggregator. However, standard neural network-based federated learning models
have been shown to be susceptible to data leakage from the gradients shared
with the server. In this work, we introduce federated learning with variational
quantum circuit model built using expressive encoding maps coupled with
overparameterized ans\"atze. We show that expressive maps lead to inherent
privacy against gradient inversion attacks, while overparameterization ensures
model trainability. Our privacy framework centers on the complexity of solving
the system of high-degree multivariate Chebyshev polynomials generated by the
gradients of quantum circuit. We present compelling arguments highlighting the
inherent difficulty in solving these equations, both in exact and approximate
scenarios. Additionally, we delve into machine learning-based attack strategies
and establish a direct connection between overparameterization in the original
federated learning model and underparameterization in the attack model.
Furthermore, we provide numerical scaling arguments showcasing that
underparameterization of the expressive map in the attack model leads to the
loss landscape being swamped with exponentially many spurious local minima
points, thus making it extremely hard to realize a successful attack. This
provides a strong claim, for the first time, that the nature of quantum machine
learning models inherently helps prevent data leakage in federated learning.


http://arxiv.org/abs/2309.12955
On Data Fabrication in Collaborative Vehicular Perception: Attacks and Countermeasures. (1%)
Qingzhao Zhang; Shuowei Jin; Ruiyang Zhu; Jiachen Sun; Xumiao Zhang; Qi Alfred Chen; Z. Morley Mao
  Collaborative perception, which greatly enhances the sensing capability of
connected and autonomous vehicles (CAVs) by incorporating data from external
resources, also brings forth potential security risks. CAVs' driving decisions
rely on remote untrusted data, making them susceptible to attacks carried out
by malicious participants in the collaborative perception system. However,
security analysis and countermeasures for such threats are absent. To
understand the impact of the vulnerability, we break the ground by proposing
various real-time data fabrication attacks in which the attacker delivers
crafted malicious data to victims in order to perturb their perception results,
leading to hard brakes or increased collision risks. Our attacks demonstrate a
high success rate of over 86% on high-fidelity simulated scenarios and are
realizable in real-world experiments. To mitigate the vulnerability, we present
a systematic anomaly detection approach that enables benign vehicles to jointly
reveal malicious fabrication. It detects 91.5% of attacks with a false positive
rate of 3% in simulated scenarios and significantly mitigates attack impacts in
real-world scenarios.


http://arxiv.org/abs/2309.12593
Improving Machine Learning Robustness via Adversarial Training. (99%)
Long Dang; Thushari Hapuarachchi; Kaiqi Xiong; Jing Lin
  As Machine Learning (ML) is increasingly used in solving various tasks in
real-world applications, it is crucial to ensure that ML algorithms are robust
to any potential worst-case noises, adversarial attacks, and highly unusual
situations when they are designed. Studying ML robustness will significantly
help in the design of ML algorithms. In this paper, we investigate ML
robustness using adversarial training in centralized and decentralized
environments, where ML training and testing are conducted in one or multiple
computers. In the centralized environment, we achieve a test accuracy of 65.41%
and 83.0% when classifying adversarial examples generated by Fast Gradient Sign
Method and DeepFool, respectively. Comparing to existing studies, these results
demonstrate an improvement of 18.41% for FGSM and 47% for DeepFool. In the
decentralized environment, we study Federated learning (FL) robustness by using
adversarial training with independent and identically distributed (IID) and
non-IID data, respectively, where CIFAR-10 is used in this research. In the IID
data case, our experimental results demonstrate that we can achieve such a
robust accuracy that it is comparable to the one obtained in the centralized
environment. Moreover, in the non-IID data case, the natural accuracy drops
from 66.23% to 57.82%, and the robust accuracy decreases by 25% and 23.4% in
C&W and Projected Gradient Descent (PGD) attacks, compared to the IID data
case, respectively. We further propose an IID data-sharing approach, which
allows for increasing the natural accuracy to 85.04% and the robust accuracy
from 57% to 72% in C&W attacks and from 59% to 67% in PGD attacks.


http://arxiv.org/abs/2309.11830
A Chinese Prompt Attack Dataset for LLMs with Evil Content. (62%)
Chengyuan Liu; Fubang Zhao; Lizhi Qing; Yangyang Kang; Changlong Sun; Kun Kuang; Fei Wu
  Large Language Models (LLMs) present significant priority in text
understanding and generation. However, LLMs suffer from the risk of generating
harmful contents especially while being employed to applications. There are
several black-box attack methods, such as Prompt Attack, which can change the
behaviour of LLMs and induce LLMs to generate unexpected answers with harmful
contents. Researchers are interested in Prompt Attack and Defense with LLMs,
while there is no publicly available dataset to evaluate the abilities of
defending prompt attack. In this paper, we introduce a Chinese Prompt Attack
Dataset for LLMs, called CPAD. Our prompts aim to induce LLMs to generate
unexpected outputs with several carefully designed prompt attack approaches and
widely concerned attacking contents. Different from previous datasets involving
safety estimation, We construct the prompts considering three dimensions:
contents, attacking methods and goals, thus the responses can be easily
evaluated and analysed. We run several well-known Chinese LLMs on our dataset,
and the results show that our prompts are significantly harmful to LLMs, with
around 70% attack success rate. We will release CPAD to encourage further
studies on prompt attack and defense.


http://arxiv.org/abs/2309.12481
HANS, are you clever? Clever Hans Effect Analysis of Neural Systems. (45%)
Leonardo Ranaldi; Fabio Massimo Zanzotto
  Instruction-tuned Large Language Models (It-LLMs) have been exhibiting
outstanding abilities to reason around cognitive states, intentions, and
reactions of all people involved, letting humans guide and comprehend
day-to-day social interactions effectively. In fact, several multiple-choice
questions (MCQ) benchmarks have been proposed to construct solid assessments of
the models' abilities. However, earlier works are demonstrating the presence of
inherent "order bias" in It-LLMs, posing challenges to the appropriate
evaluation. In this paper, we investigate It-LLMs' resilience abilities towards
a series of probing tests using four MCQ benchmarks. Introducing adversarial
examples, we show a significant performance gap, mainly when varying the order
of the choices, which reveals a selection bias and brings into discussion
reasoning abilities. Following a correlation between first positions and model
choices due to positional bias, we hypothesized the presence of structural
heuristics in the decision-making process of the It-LLMs, strengthened by
including significant examples in few-shot scenarios. Finally, by using the
Chain-of-Thought (CoT) technique, we elicit the model to reason and mitigate
the bias by obtaining more robust models.


http://arxiv.org/abs/2309.12263
On the Relationship between Skill Neurons and Robustness in Prompt Tuning. (10%)
Leon Ackermann; Xenia Ohmer
  Prompt Tuning is a popular parameter-efficient finetuning method for
pre-trained large language models (PLMs). Recently, based on experiments with
RoBERTa, it has been suggested that Prompt Tuning activates specific neurons in
the transformer's feed-forward networks, that are highly predictive and
selective for the given task. In this paper, we study the robustness of Prompt
Tuning in relation to these "skill neurons", using RoBERTa and T5. We show that
prompts tuned for a specific task are transferable to tasks of the same type
but are not very robust to adversarial data, with higher robustness for T5 than
RoBERTa. At the same time, we replicate the existence of skill neurons in
RoBERTa and further show that skill neurons also seem to exist in T5.
Interestingly, the skill neurons of T5 determined on non-adversarial data are
also among the most predictive neurons on the adversarial data, which is not
the case for RoBERTa. We conclude that higher adversarial robustness may be
related to a model's ability to activate the relevant skill neurons on
adversarial data.


http://arxiv.org/abs/2309.11894
DeepTheft: Stealing DNN Model Architectures through Power Side Channel. (1%)
Yansong Gao; Huming Qiu; Zhi Zhang; Binghui Wang; Hua Ma; Alsharif Abuadbba; Minhui Xue; Anmin Fu; Surya Nepal
  Deep Neural Network (DNN) models are often deployed in resource-sharing
clouds as Machine Learning as a Service (MLaaS) to provide inference
services.To steal model architectures that are of valuable intellectual
properties, a class of attacks has been proposed via different side-channel
leakage, posing a serious security challenge to MLaaS.
  Also targeting MLaaS, we propose a new end-to-end attack, DeepTheft, to
accurately recover complex DNN model architectures on general processors via
the RAPL-based power side channel. However, an attacker can acquire only a low
sampling rate (1 KHz) of the time-series energy traces from the RAPL interface,
rendering existing techniques ineffective in stealing large and deep DNN
models. To this end, we design a novel and generic learning-based framework
consisting of a set of meta-models, based on which DeepTheft is demonstrated to
have high accuracy in recovering a large number (thousands) of models
architectures from different model families including the deepest ResNet152.
Particularly, DeepTheft has achieved a Levenshtein Distance Accuracy of 99.75%
in recovering network structures, and a weighted average F1 score of 99.60% in
recovering diverse layer-wise hyperparameters. Besides, our proposed learning
framework is general to other time-series side-channel signals. To validate its
generalization, another existing side channel is exploited, i.e., CPU
frequency. Different from RAPL, CPU frequency is accessible to unprivileged
users in bare-metal OSes. By using our generic learning framework trained
against CPU frequency traces, DeepTheft has shown similarly high attack
performance in stealing model architectures.


http://arxiv.org/abs/2309.11751
How Robust is Google's Bard to Adversarial Image Attacks? (99%)
Yinpeng Dong; Huanran Chen; Jiawei Chen; Zhengwei Fang; Xiao Yang; Yichi Zhang; Yu Tian; Hang Su; Jun Zhu
  Multimodal Large Language Models (MLLMs) that integrate text and other
modalities (especially vision) have achieved unprecedented performance in
various multimodal tasks. However, due to the unsolved adversarial robustness
problem of vision models, MLLMs can have more severe safety and security risks
by introducing the vision inputs. In this work, we study the adversarial
robustness of Google's Bard, a competitive chatbot to ChatGPT that released its
multimodal capability recently, to better understand the vulnerabilities of
commercial MLLMs. By attacking white-box surrogate vision encoders or MLLMs,
the generated adversarial examples can mislead Bard to output wrong image
descriptions with a 22% success rate based solely on the transferability. We
show that the adversarial examples can also attack other MLLMs, e.g., a 26%
attack success rate against Bing Chat and a 86% attack success rate against
ERNIE bot. Moreover, we identify two defense mechanisms of Bard, including face
detection and toxicity detection of images. We design corresponding attacks to
evade these defenses, demonstrating that the current defenses of Bard are also
vulnerable. We hope this work can deepen our understanding on the robustness of
MLLMs and facilitate future research on defenses. Our code is available at
https://github.com/thu-ml/Attack-Bard.


http://arxiv.org/abs/2309.11111
PRAT: PRofiling Adversarial aTtacks. (99%)
Rahul Ambati; Naveed Akhtar; Ajmal Mian; Yogesh Singh Rawat
  Intrinsic susceptibility of deep learning to adversarial examples has led to
a plethora of attack techniques with a broad common objective of fooling deep
models. However, we find slight compositional differences between the
algorithms achieving this objective. These differences leave traces that
provide important clues for attacker profiling in real-life scenarios. Inspired
by this, we introduce a novel problem of PRofiling Adversarial aTtacks (PRAT).
Given an adversarial example, the objective of PRAT is to identify the attack
used to generate it. Under this perspective, we can systematically group
existing attacks into different families, leading to the sub-problem of attack
family identification, which we also study. To enable PRAT analysis, we
introduce a large Adversarial Identification Dataset (AID), comprising over
180k adversarial samples generated with 13 popular attacks for image
specific/agnostic white/black box setups. We use AID to devise a novel
framework for the PRAT objective. Our framework utilizes a Transformer based
Global-LOcal Feature (GLOF) module to extract an approximate signature of the
adversarial attack, which in turn is used for the identification of the attack.
Using AID and our framework, we provide multiple interesting benchmark results
for the PRAT problem.


http://arxiv.org/abs/2309.11196
When to Trust AI: Advances and Challenges for Certification of Neural Networks. (64%)
Marta Kwiatkowska; Xiyue Zhang
  Artificial intelligence (AI) has been advancing at a fast pace and it is now
poised for deployment in a wide range of applications, such as autonomous
systems, medical diagnosis and natural language processing. Early adoption of
AI technology for real-world applications has not been without problems,
particularly for neural networks, which may be unstable and susceptible to
adversarial examples. In the longer term, appropriate safety assurance
techniques need to be developed to reduce potential harm due to avoidable
system failures and ensure trustworthiness. Focusing on certification and
explainability, this paper provides an overview of techniques that have been
developed to ensure safety of AI decisions and discusses future challenges.


http://arxiv.org/abs/2309.11462
AudioFool: Fast, Universal and synchronization-free Cross-Domain Attack on Speech Recognition. (54%)
Mohamad Fakih; Rouwaida Kanj; Fadi Kurdahi; Mohammed E. Fouda
  Automatic Speech Recognition systems have been shown to be vulnerable to
adversarial attacks that manipulate the command executed on the device. Recent
research has focused on exploring methods to create such attacks, however, some
issues relating to Over-The-Air (OTA) attacks have not been properly addressed.
In our work, we examine the needed properties of robust attacks compatible with
the OTA model, and we design a method of generating attacks with arbitrary such
desired properties, namely the invariance to synchronization, and the
robustness to filtering: this allows a Denial-of-Service (DoS) attack against
ASR systems. We achieve these characteristics by constructing attacks in a
modified frequency domain through an inverse Fourier transform. We evaluate our
method on standard keyword classification tasks and analyze it in OTA, and we
analyze the properties of the cross-domain attacks to explain the efficiency of
the approach.


http://arxiv.org/abs/2309.11667
Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation. (54%)
Krishna Kanth Nakka; Mathieu Salzmann
  As 3D human pose estimation can now be achieved with very high accuracy in
the supervised learning scenario, tackling the case where 3D pose annotations
are not available has received increasing attention. In particular, several
methods have proposed to learn image representations in a self-supervised
fashion so as to disentangle the appearance information from the pose one. The
methods then only need a small amount of supervised data to train a pose
regressor using the pose-related latent vector as input, as it should be free
of appearance information. In this paper, we carry out in-depth analysis to
understand to what degree the state-of-the-art disentangled representation
learning methods truly separate the appearance information from the pose one.
First, we study disentanglement from the perspective of the self-supervised
network, via diverse image synthesis experiments. Second, we investigate
disentanglement with respect to the 3D pose regressor following an adversarial
attack perspective. Specifically, we design an adversarial strategy focusing on
generating natural appearance changes of the subject, and against which we
could expect a disentangled network to be robust. Altogether, our analyses show
that disentanglement in the three state-of-the-art disentangled representation
learning frameworks if far from complete, and that their pose codes contain
significant appearance information. We believe that our approach provides a
valuable testbed to evaluate the degree of disentanglement of pose from
appearance in self-supervised 3D human pose estimation.


http://arxiv.org/abs/2309.11053
Fed-LSAE: Thwarting Poisoning Attacks against Federated Cyber Threat Detection System via Autoencoder-based Latent Space Inspection. (5%)
Tran Duc Luong; Vuong Minh Tien; Nguyen Huu Quyen; Do Thi Thu Hien; Phan The Duy; Van-Hau Pham
  The significant rise of security concerns in conventional centralized
learning has promoted federated learning (FL) adoption in building intelligent
applications without privacy breaches. In cybersecurity, the sensitive data
along with the contextual information and high-quality labeling in each
enterprise organization play an essential role in constructing high-performance
machine learning (ML) models for detecting cyber threats. Nonetheless, the
risks coming from poisoning internal adversaries against FL systems have raised
discussions about designing robust anti-poisoning frameworks. Whereas defensive
mechanisms in the past were based on outlier detection, recent approaches tend
to be more concerned with latent space representation. In this paper, we
investigate a novel robust aggregation method for FL, namely Fed-LSAE, which
takes advantage of latent space representation via the penultimate layer and
Autoencoder to exclude malicious clients from the training process. The
experimental results on the CIC-ToN-IoT and N-BaIoT datasets confirm the
feasibility of our defensive mechanism against cutting-edge poisoning attacks
for developing a robust FL-based threat detector in the context of IoT. More
specifically, the FL evaluation witnesses an upward trend of approximately 98%
across all metrics when integrating with our Fed-LSAE defense.


http://arxiv.org/abs/2309.16577
Compilation as a Defense: Enhancing DL Model Attack Robustness via Tensor Optimization. (2%)
Stefan Trawicki; William Hackett; Lewis Birch; Neeraj Suri; Peter Garraghan
  Adversarial Machine Learning (AML) is a rapidly growing field of security
research, with an often overlooked area being model attacks through
side-channels. Previous works show such attacks to be serious threats, though
little progress has been made on efficient remediation strategies that avoid
costly model re-engineering. This work demonstrates a new defense against AML
side-channel attacks using model compilation techniques, namely tensor
optimization. We show relative model attack effectiveness decreases of up to
43% using tensor optimization, discuss the implications, and direction of
future work.


http://arxiv.org/abs/2309.10348
Language Guided Adversarial Purification. (99%)
Himanshu Singh; A V Subramanyam
  Adversarial purification using generative models demonstrates strong
adversarial defense performance. These methods are classifier and
attack-agnostic, making them versatile but often computationally intensive.
Recent strides in diffusion and score networks have improved image generation
and, by extension, adversarial purification. Another highly efficient class of
adversarial defense methods known as adversarial training requires specific
knowledge of attack vectors, forcing them to be trained extensively on
adversarial examples. To overcome these limitations, we introduce a new
framework, namely Language Guided Adversarial Purification (LGAP), utilizing
pre-trained diffusion models and caption generators to defend against
adversarial attacks. Given an input image, our method first generates a
caption, which is then used to guide the adversarial purification process
through a diffusion network. Our approach has been evaluated against strong
adversarial attacks, proving its effectiveness in enhancing adversarial
robustness. Our results indicate that LGAP outperforms most existing
adversarial defense techniques without requiring specialized network training.
This underscores the generalizability of models trained on large datasets,
highlighting a promising direction for further research.


http://arxiv.org/abs/2309.10916
What Learned Representations and Influence Functions Can Tell Us About Adversarial Examples. (99%)
Shakila Mahjabin Tonni; Mark Dras
  Adversarial examples, deliberately crafted using small perturbations to fool
deep neural networks, were first studied in image processing and more recently
in NLP. While approaches to detecting adversarial examples in NLP have largely
relied on search over input perturbations, image processing has seen a range of
techniques that aim to characterise adversarial subspaces over the learned
representations.
  In this paper, we adapt two such approaches to NLP, one based on nearest
neighbors and influence functions and one on Mahalanobis distances. The former
in particular produces a state-of-the-art detector when compared against
several strong baselines; moreover, the novel use of influence functions
provides insight into how the nature of adversarial example subspaces in NLP
relate to those in image processing, and also how they differ depending on the
kind of NLP task.


http://arxiv.org/abs/2309.10586
Adversarial Attacks Against Uncertainty Quantification. (99%)
Emanuele Ledda; Daniele Angioni; Giorgio Piras; Giorgio Fumera; Battista Biggio; Fabio Roli
  Machine-learning models can be fooled by adversarial examples, i.e.,
carefully-crafted input perturbations that force models to output wrong
predictions. While uncertainty quantification has been recently proposed to
detect adversarial inputs, under the assumption that such attacks exhibit a
higher prediction uncertainty than pristine data, it has been shown that
adaptive attacks specifically aimed at reducing also the uncertainty estimate
can easily bypass this defense mechanism. In this work, we focus on a different
adversarial scenario in which the attacker is still interested in manipulating
the uncertainty estimate, but regardless of the correctness of the prediction;
in particular, the goal is to undermine the use of machine-learning models when
their outputs are consumed by a downstream module or by a human operator.
Following such direction, we: \textit{(i)} design a threat model for attacks
targeting uncertainty quantification; \textit{(ii)} devise different attack
strategies on conceptually different UQ techniques spanning for both
classification and semantic segmentation problems; \textit{(iii)} conduct a
first complete and extensive analysis to compare the differences between some
of the most employed UQ approaches under attack. Our extensive experimental
analysis shows that our attacks are more effective in manipulating uncertainty
quantification measures than attacks aimed to also induce misclassifications.


http://arxiv.org/abs/2309.10544
Model Leeching: An Extraction Attack Targeting LLMs. (76%)
Lewis Birch; William Hackett; Stefan Trawicki; Neeraj Suri; Peter Garraghan
  Model Leeching is a novel extraction attack targeting Large Language Models
(LLMs), capable of distilling task-specific knowledge from a target LLM into a
reduced parameter model. We demonstrate the effectiveness of our attack by
extracting task capability from ChatGPT-3.5-Turbo, achieving 73% Exact Match
(EM) similarity, and SQuAD EM and F1 accuracy scores of 75% and 87%,
respectively for only $50 in API cost. We further demonstrate the feasibility
of adversarial attack transferability from an extracted model extracted via
Model Leeching to perform ML attack staging against a target LLM, resulting in
an 11% increase to attack success rate when applied to ChatGPT-3.5-Turbo.


http://arxiv.org/abs/2309.11022
Information Leakage from Data Updates in Machine Learning Models. (16%)
Tian Hui; Farhad Farokhi; Olga Ohrimenko
  In this paper we consider the setting where machine learning models are
retrained on updated datasets in order to incorporate the most up-to-date
information or reflect distribution shifts. We investigate whether one can
infer information about these updates in the training data (e.g., changes to
attribute values of records). Here, the adversary has access to snapshots of
the machine learning model before and after the change in the dataset occurs.
Contrary to the existing literature, we assume that an attribute of a single or
multiple training data points are changed rather than entire data records are
removed or added. We propose attacks based on the difference in the prediction
confidence of the original model and the updated model. We evaluate our attack
methods on two public datasets along with multi-layer perceptron and logistic
regression models. We validate that two snapshots of the model can result in
higher information leakage in comparison to having access to only the updated
model. Moreover, we observe that data records with rare values are more
vulnerable to attacks, which points to the disparate vulnerability of privacy
attacks in the update setting. When multiple records with the same original
attribute value are updated to the same new value (i.e., repeated changes), the
attacker is more likely to correctly guess the updated values since repeated
changes leave a larger footprint on the trained model. These observations point
to vulnerability of machine learning models to attribute inference attacks in
the update setting.


http://arxiv.org/abs/2309.10644
Robin: A Novel Method to Produce Robust Interpreters for Deep Learning-Based Code Classifiers. (16%)
Zhen Li; Ruqian Zhang; Deqing Zou; Ning Wang; Yating Li; Shouhuai Xu; Chen Chen; Hai Jin
  Deep learning has been widely used in source code classification tasks, such
as code classification according to their functionalities, code authorship
attribution, and vulnerability detection. Unfortunately, the black-box nature
of deep learning makes it hard to interpret and understand why a classifier
(i.e., classification model) makes a particular prediction on a given example.
This lack of interpretability (or explainability) might have hindered their
adoption by practitioners because it is not clear when they should or should
not trust a classifier's prediction. The lack of interpretability has motivated
a number of studies in recent years. However, existing methods are neither
robust nor able to cope with out-of-distribution examples. In this paper, we
propose a novel method to produce \underline{Rob}ust \underline{in}terpreters
for a given deep learning-based code classifier; the method is dubbed Robin.
The key idea behind Robin is a novel hybrid structure combining an interpreter
and two approximators, while leveraging the ideas of adversarial training and
data augmentation. Experimental results show that on average the interpreter
produced by Robin achieves a 6.11\% higher fidelity (evaluated on the
classifier), 67.22\% higher fidelity (evaluated on the approximator), and
15.87x higher robustness than that of the three existing interpreters we
evaluated. Moreover, the interpreter is 47.31\% less affected by
out-of-distribution examples than that of LEMNA.


http://arxiv.org/abs/2309.10607
SPFL: A Self-purified Federated Learning Method Against Poisoning Attacks. (12%)
Zizhen Liu; Weiyang He; Chip-Hong Chang; Jing Ye; Huawei Li; Xiaowei Li
  While Federated learning (FL) is attractive for pulling privacy-preserving
distributed training data, the credibility of participating clients and
non-inspectable data pose new security threats, of which poisoning attacks are
particularly rampant and hard to defend without compromising privacy,
performance or other desirable properties of FL. To tackle this problem, we
propose a self-purified FL (SPFL) method that enables benign clients to exploit
trusted historical features of locally purified model to supervise the training
of aggregated model in each iteration. The purification is performed by an
attention-guided self-knowledge distillation where the teacher and student
models are optimized locally for task loss, distillation loss and
attention-based loss simultaneously. SPFL imposes no restriction on the
communication protocol and aggregator at the server. It can work in tandem with
any existing secure aggregation algorithms and protocols for augmented security
and privacy guarantee. We experimentally demonstrate that SPFL outperforms
state-of-the-art FL defenses against various poisoning attacks. The attack
success rate of SPFL trained model is at most 3$\%$ above that of a clean
model, even if the poisoning attack is launched in every iteration with all but
one malicious clients in the system. Meantime, it improves the model quality on
normal inputs compared to FedAvg, either under attack or in the absence of an
attack.


http://arxiv.org/abs/2309.11005
It's Simplex! Disaggregating Measures to Improve Certified Robustness. (11%)
Andrew C. Cullen; Paul Montague; Shijie Liu; Sarah M. Erfani; Benjamin I. P. Rubinstein
  Certified robustness circumvents the fragility of defences against
adversarial attacks, by endowing model predictions with guarantees of class
invariance for attacks up to a calculated size. While there is value in these
certifications, the techniques through which we assess their performance do not
present a proper accounting of their strengths and weaknesses, as their
analysis has eschewed consideration of performance over individual samples in
favour of aggregated measures. By considering the potential output space of
certified models, this work presents two distinct approaches to improve the
analysis of certification mechanisms, that allow for both dataset-independent
and dataset-dependent measures of certification performance. Embracing such a
perspective uncovers new certification approaches, which have the potential to
more than double the achievable radius of certification, relative to current
state-of-the-art. Empirical evaluation verifies that our new approach can
certify $9\%$ more samples at noise scale $\sigma = 1$, with greater relative
improvements observed as the difficulty of the predictive task increases.


http://arxiv.org/abs/2310.10664
Nebula: Self-Attention for Dynamic Malware Analysis. (5%)
Dmitrijs Trizna; Luca Demetrio; Battista Biggio; Fabio Roli
  Dynamic analysis enables detecting Windows malware by executing programs in a
controlled environment, and storing their actions in log reports. Previous work
has started training machine learning models on such reports to perform either
malware detection or malware classification. However, most of the approaches
(i) have only considered convolutional and long-short term memory networks,
(ii) they have been built focusing only on APIs called at runtime, without
considering other relevant though heterogeneous sources of information like
network and file operations, and (iii) the code and pretrained models are
hardly available, hindering reproducibility of results in this research area.
In this work, we overcome these limitations by presenting Nebula, a versatile,
self-attention transformer-based neural architecture that can generalize across
different behavior representations and formats, combining heterogeneous
information from dynamic log reports. We show the efficacy of Nebula on three
distinct data collections from different dynamic analysis platforms, comparing
its performance with previous state-of-the-art models developed for malware
detection and classification tasks. We produce an extensive ablation study that
showcases how the components of Nebula influence its predictive performance,
while enabling it to outperform some competing approaches at very low false
positive rates. We conclude our work by inspecting the behavior of Nebula
through the application of explainability methods, which highlight that Nebula
correctly focuses more on portions of reports that contain malicious
activities. We release our code and models at github.com/dtrizna/nebula.


http://arxiv.org/abs/2310.07725
Extreme Image Transformations Facilitate Robust Latent Object Representations. (1%)
Girik Malik; Dakarai Crowder; Ennio Mingolla
  Adversarial attacks can affect the object recognition capabilities of
machines in wild. These can often result from spurious correlations between
input and class labels, and are prone to memorization in large networks. While
networks are expected to do automated feature selection, it is not effective at
the scale of the object. Humans, however, are able to select the minimum set of
features required to form a robust representation of an object. In this work,
we show that finetuning any pretrained off-the-shelf network with Extreme Image
Transformations (EIT) not only helps in learning a robust latent
representation, it also improves the performance of these networks against
common adversarial attacks of various intensities. Our EIT trained networks
show strong activations in the object regions even when tested with more
intense noise, showing promising generalizations across different kinds of
adversarial attacks.


http://arxiv.org/abs/2309.09480
Stealthy Physical Masked Face Recognition Attack via Adversarial Style Optimization. (99%)
Huihui Gong; Minjing Dong; Siqi Ma; Seyit Camtepe; Surya Nepal; Chang Xu
  Deep neural networks (DNNs) have achieved state-of-the-art performance on
face recognition (FR) tasks in the last decade. In real scenarios, the
deployment of DNNs requires taking various face accessories into consideration,
like glasses, hats, and masks. In the COVID-19 pandemic era, wearing face masks
is one of the most effective ways to defend against the novel coronavirus.
However, DNNs are known to be vulnerable to adversarial examples with a small
but elaborated perturbation. Thus, a facial mask with adversarial perturbations
may pose a great threat to the widely used deep learning-based FR models. In
this paper, we consider a challenging adversarial setting: targeted attack
against FR models. We propose a new stealthy physical masked FR attack via
adversarial style optimization. Specifically, we train an adversarial style
mask generator that hides adversarial perturbations inside style masks.
Moreover, to ameliorate the phenomenon of sub-optimization with one fixed
style, we propose to discover the optimal style given a target through style
optimization in a continuous relaxation manner. We simultaneously optimize the
generator and the style selection for generating strong and stealthy
adversarial style masks. We evaluated the effectiveness and transferability of
our proposed method via extensive white-box and black-box digital experiments.
Furthermore, we also conducted physical attack experiments against local FR
models and online platforms.


http://arxiv.org/abs/2309.10243
Transferable Adversarial Attack on Image Tampering Localization. (99%)
Yuqi Wang; Gang Cao; Zijie Lou; Haochen Zhu
  It is significant to evaluate the security of existing digital image
tampering localization algorithms in real-world applications. In this paper, we
propose an adversarial attack scheme to reveal the reliability of such
tampering localizers, which would be fooled and fail to predict altered regions
correctly. Specifically, the adversarial examples based on optimization and
gradient are implemented for white/black-box attacks. Correspondingly, the
adversarial example is optimized via reverse gradient propagation, and the
perturbation is added adaptively in the direction of gradient rising. The
black-box attack is achieved by relying on the transferability of such
adversarial examples to different localizers. Extensive evaluations verify that
the proposed attack sharply reduces the localization accuracy while preserving
high visual quality of the attacked images.


http://arxiv.org/abs/2309.10136
Efficient Low-Rank GNN Defense Against Structural Attacks. (96%)
Abdullah Alchihabi; Qing En; Yuhong Guo
  Graph Neural Networks (GNNs) have been shown to possess strong representation
abilities over graph data. However, GNNs are vulnerable to adversarial attacks,
and even minor perturbations to the graph structure can significantly degrade
their performance. Existing methods either are ineffective against
sophisticated attacks or require the optimization of dense adjacency matrices,
which is time-consuming and prone to local minima. To remedy this problem, we
propose an Efficient Low-Rank Graph Neural Network (ELR-GNN) defense method,
which aims to learn low-rank and sparse graph structures for defending against
adversarial attacks, ensuring effective defense with greater efficiency.
Specifically, ELR-GNN consists of two modules: a Coarse Low-Rank Estimation
Module and a Fine-Grained Estimation Module. The first module adopts the
truncated Singular Value Decomposition (SVD) to initialize the low-rank
adjacency matrix estimation, which serves as a starting point for optimizing
the low-rank matrix. In the second module, the initial estimate is refined by
jointly learning a low-rank sparse graph structure with the GNN model. Sparsity
is incorporated into the learned low-rank adjacency matrix by pruning weak
connections, which can reduce redundant data while maintaining valuable
information. As a result, instead of using the dense adjacency matrix directly,
ELR-GNN can learn a low-rank and sparse estimate of it in a simple, efficient
and easy to optimize manner. The experimental results demonstrate that ELR-GNN
outperforms the state-of-the-art GNN defense methods in the literature, in
addition to being very efficient and easy to train.


http://arxiv.org/abs/2309.09928
Evaluating Adversarial Robustness with Expected Viable Performance. (45%)
Ryan McCoppin; Colin Dawson; Sean M. Kennedy; Leslie M. Blaha
  We introduce a metric for evaluating the robustness of a classifier, with
particular attention to adversarial perturbations, in terms of expected
functionality with respect to possible adversarial perturbations. A classifier
is assumed to be non-functional (that is, has a functionality of zero) with
respect to a perturbation bound if a conventional measure of performance, such
as classification accuracy, is less than a minimally viable threshold when the
classifier is tested on examples from that perturbation bound. Defining
robustness in terms of an expected value is motivated by a domain general
approach to robustness quantification.


http://arxiv.org/abs/2309.10058
Dual Student Networks for Data-Free Model Stealing. (26%)
James Beetham; Navid Kardan; Ajmal Mian; Mubarak Shah
  Existing data-free model stealing methods use a generator to produce samples
in order to train a student model to match the target model outputs. To this
end, the two main challenges are estimating gradients of the target model
without access to its parameters, and generating a diverse set of training
samples that thoroughly explores the input space. We propose a Dual Student
method where two students are symmetrically trained in order to provide the
generator a criterion to generate samples that the two students disagree on. On
one hand, disagreement on a sample implies at least one student has classified
the sample incorrectly when compared to the target model. This incentive
towards disagreement implicitly encourages the generator to explore more
diverse regions of the input space. On the other hand, our method utilizes
gradients of student models to indirectly estimate gradients of the target
model. We show that this novel training objective for the generator network is
equivalent to optimizing a lower bound on the generator's loss if we had access
to the target model gradients. We show that our new optimization framework
provides more accurate gradient estimation of the target model and better
accuracies on benchmark classification datasets. Additionally, our approach
balances improved query efficiency with training computation cost. Finally, we
demonstrate that our method serves as a better proxy model for transfer-based
adversarial attacks than existing data-free model stealing methods.


http://arxiv.org/abs/2309.09700
Securing Fixed Neural Network Steganography. (5%)
Zicong Luo; Sheng Li; Guobiao Li; Zhenxing Qian; Xinpeng Zhang
  Image steganography is the art of concealing secret information in images in
a way that is imperceptible to unauthorized parties. Recent advances show that
is possible to use a fixed neural network (FNN) for secret embedding and
extraction. Such fixed neural network steganography (FNNS) achieves high
steganographic performance without training the networks, which could be more
useful in real-world applications. However, the existing FNNS schemes are
vulnerable in the sense that anyone can extract the secret from the
stego-image. To deal with this issue, we propose a key-based FNNS scheme to
improve the security of the FNNS, where we generate key-controlled
perturbations from the FNN for data embedding. As such, only the receiver who
possesses the key is able to correctly extract the secret from the stego-image
using the FNN. In order to improve the visual quality and undetectability of
the stego-image, we further propose an adaptive perturbation optimization
strategy by taking the perturbation cost into account. Experimental results
show that our proposed scheme is capable of preventing unauthorized secret
extraction from the stego-images. Furthermore, our scheme is able to generate
stego-images with higher visual quality than the state-of-the-art FNNS scheme,
especially when the FNN is a neural network for ordinary learning tasks.


http://arxiv.org/abs/2309.10253
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. (4%)
Jiahao Yu; Xingwei Lin; Zheng Yu; Xinyu Xing
  Large language models (LLMs) have recently experienced tremendous popularity
and are widely used from casual conversations to AI-driven programming.
However, despite their considerable success, LLMs are not entirely reliable and
can give detailed guidance on how to conduct harmful or illegal activities.
While safety measures can reduce the risk of such outputs, adversarial
jailbreak attacks can still exploit LLMs to produce harmful content. These
jailbreak templates are typically manually crafted, making large-scale testing
challenging.
  In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing
framework inspired by the AFL fuzzing framework. Instead of manual engineering,
GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs.
At its core, GPTFuzz starts with human-written templates as initial seeds, then
mutates them to produce new templates. We detail three key components of
GPTFuzz: a seed selection strategy for balancing efficiency and variability,
mutate operators for creating semantically equivalent or similar sentences, and
a judgment model to assess the success of a jailbreak attack.
  We evaluate GPTFuzz against various commercial and open-source LLMs,
including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our
results indicate that GPTFuzz consistently produces jailbreak templates with a
high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz
achieves over 90% attack success rates against ChatGPT and Llama-2 models, even
with suboptimal initial seed templates. We anticipate that GPTFuzz will be
instrumental for researchers and practitioners in examining LLM robustness and
will encourage further exploration into enhancing LLM safety.


http://arxiv.org/abs/2309.09586
Spoofing attack augmentation: can differently-trained attack models improve generalisation? (3%)
Wanying Ge; Xin Wang; Junichi Yamagishi; Massimiliano Todisco; Nicholas Evans
  A reliable deepfake detector or spoofing countermeasure (CM) should be robust
in the face of unpredictable spoofing attacks. To encourage the learning of
more generaliseable artefacts, rather than those specific only to known
attacks, CMs are usually exposed to a broad variety of different attacks during
training. Even so, the performance of deep-learning-based CM solutions are
known to vary, sometimes substantially, when they are retrained with different
initialisations, hyper-parameters or training data partitions. We show in this
paper that the potency of spoofing attacks, also deep-learning-based, can
similarly vary according to training conditions, sometimes resulting in
substantial degradations to detection performance. Nevertheless, while a
RawNet2 CM model is vulnerable when only modest adjustments are made to the
attack algorithm, those based upon graph attention networks and self-supervised
learning are reassuringly robust. The focus upon training data generated with
different attack algorithms might not be sufficient on its own to ensure
generaliability; some form of spoofing attack augmentation at the algorithm
level can be complementary.


http://arxiv.org/abs/2309.09837
Frame-to-Utterance Convergence: A Spectra-Temporal Approach for Unified Spoofing Detection. (1%)
Awais Khan; Khalid Mahmood Malik; Shah Nawaz
  Voice spoofing attacks pose a significant threat to automated speaker
verification systems. Existing anti-spoofing methods often simulate specific
attack types, such as synthetic or replay attacks. However, in real-world
scenarios, the countermeasures are unaware of the generation schema of the
attack, necessitating a unified solution. Current unified solutions struggle to
detect spoofing artifacts, especially with recent spoofing mechanisms. For
instance, the spoofing algorithms inject spectral or temporal anomalies, which
are challenging to identify. To this end, we present a spectra-temporal fusion
leveraging frame-level and utterance-level coefficients. We introduce a novel
local spectral deviation coefficient (SDC) for frame-level inconsistencies and
employ a bi-LSTM-based network for sequential temporal coefficients (STC),
which capture utterance-level artifacts. Our spectra-temporal fusion strategy
combines these coefficients, and an auto-encoder generates spectra-temporal
deviated coefficients (STDC) to enhance robustness. Our proposed approach
addresses multiple spoofing categories, including synthetic, replay, and
partial deepfake attacks. Extensive evaluation on diverse datasets
(ASVspoof2019, ASVspoof2021, VSDC, partial spoofs, and in-the-wild deepfakes)
demonstrated its robustness for a wide range of voice applications.


http://arxiv.org/abs/2309.09464
Reducing Adversarial Training Cost with Gradient Approximation. (99%)
Huihui Gong; Shuo Yang; Siqi Ma; Seyit Camtepe; Surya Nepal; Chang Xu
  Deep learning models have achieved state-of-the-art performances in various
domains, while they are vulnerable to the inputs with well-crafted but small
perturbations, which are named after adversarial examples (AEs). Among many
strategies to improve the model robustness against AEs, Projected Gradient
Descent (PGD) based adversarial training is one of the most effective methods.
Unfortunately, the prohibitive computational overhead of generating strong
enough AEs, due to the maximization of the loss function, sometimes makes the
regular PGD adversarial training impractical when using larger and more
complicated models. In this paper, we propose that the adversarial loss can be
approximated by the partial sum of Taylor series. Furthermore, we approximate
the gradient of adversarial loss and propose a new and efficient adversarial
training method, adversarial training with gradient approximation (GAAT), to
reduce the cost of building up robust models. Additionally, extensive
experiments demonstrate that this efficiency improvement can be achieved
without any or with very little loss in accuracy on natural and adversarial
examples, which show that our proposed method saves up to 60\% of the training
time with comparable model test accuracy on MNIST, CIFAR-10 and CIFAR-100
datasets.


http://arxiv.org/abs/2309.14348
Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM. (61%)
Bochuan Cao; Yuanpu Cao; Lu Lin; Jinghui Chen
  Recently, Large Language Models (LLMs) have made significant advancements and
are now widely used across various domains. Unfortunately, there has been a
rising concern that LLMs can be misused to generate harmful or malicious
content. Though a line of research has focused on aligning LLMs with human
values and preventing them from producing inappropriate content, such
alignments are usually vulnerable and can be bypassed by alignment-breaking
attacks via adversarially optimized or handcrafted jailbreaking prompts. In
this work, we introduce a Robustly Aligned LLM (RA-LLM) to defend against
potential alignment-breaking attacks. RA-LLM can be directly constructed upon
an existing aligned LLM with a robust alignment checking function, without
requiring any expensive retraining or fine-tuning process of the original LLM.
Furthermore, we also provide a theoretical analysis for RA-LLM to verify its
effectiveness in defending against alignment-breaking attacks. Through
real-world experiments on open-source large language models, we demonstrate
that RA-LLM can successfully defend against both state-of-the-art adversarial
prompts and popular handcrafted jailbreaking prompts by reducing their attack
success rates from nearly 100\% to around 10\% or less.


http://arxiv.org/abs/2309.08999
Context-aware Adversarial Attack on Named Entity Recognition. (99%)
Shuguang Chen; Leonardo Neves; Thamar Solorio
  In recent years, large pre-trained language models (PLMs) have achieved
remarkable performance on many natural language processing benchmarks. Despite
their success, prior studies have shown that PLMs are vulnerable to attacks
from adversarial examples. In this work, we focus on the named entity
recognition task and study context-aware adversarial attack methods to examine
the model's robustness. Specifically, we propose perturbing the most
informative words for recognizing entities to create adversarial examples and
investigate different candidate replacement methods to generate natural and
plausible adversarial examples. Experiments and analyses show that our methods
are more effective in deceiving the model into making wrong predictions than
strong baselines.


http://arxiv.org/abs/2309.08945
Inverse classification with logistic and softmax classifiers: efficient optimization. (56%)
Miguel Á. Carreira-Perpiñán; Suryabhan Singh Hada
  In recent years, a certain type of problems have become of interest where one
wants to query a trained classifier. Specifically, one wants to find the
closest instance to a given input instance such that the classifier's predicted
label is changed in a desired way. Examples of these ``inverse classification''
problems are counterfactual explanations, adversarial examples and model
inversion. All of them are fundamentally optimization problems over the input
instance vector involving a fixed classifier, and it is of interest to achieve
a fast solution for interactive or real-time applications. We focus on solving
this problem efficiently for two of the most widely used classifiers: logistic
regression and softmax classifiers. Owing to special properties of these
models, we show that the optimization can be solved in closed form for logistic
regression, and iteratively but extremely fast for the softmax classifier. This
allows us to solve either case exactly (to nearly machine precision) in a
runtime of milliseconds to around a second even for very high-dimensional
instances and many classes.


http://arxiv.org/abs/2309.08953
Robust Backdoor Attacks on Object Detection in Real World. (11%)
Yaguan Qian; Boyuan Ji; Shuke He; Shenhui Huang; Xiang Ling; Bin Wang; Wei Wang
  Deep learning models are widely deployed in many applications, such as object
detection in various security fields. However, these models are vulnerable to
backdoor attacks. Most backdoor attacks were intensively studied on classified
models, but little on object detection. Previous works mainly focused on the
backdoor attack in the digital world, but neglect the real world. Especially,
the backdoor attack's effect in the real world will be easily influenced by
physical factors like distance and illumination. In this paper, we proposed a
variable-size backdoor trigger to adapt to the different sizes of attacked
objects, overcoming the disturbance caused by the distance between the viewing
point and attacked object. In addition, we proposed a backdoor training named
malicious adversarial training, enabling the backdoor object detector to learn
the feature of the trigger with physical noise. The experiment results show
this robust backdoor attack (RBA) could enhance the attack success rate in the
real world.


http://arxiv.org/abs/2309.09123
Conditional Mutual Information Constrained Deep Learning for Classification. (5%)
En-Hui Yang; Shayan Mohajer Hamidi; Linfeng Ye; Renhao Tan; Beverly Yang
  The concepts of conditional mutual information (CMI) and normalized
conditional mutual information (NCMI) are introduced to measure the
concentration and separation performance of a classification deep neural
network (DNN) in the output probability distribution space of the DNN, where
CMI and the ratio between CMI and NCMI represent the intra-class concentration
and inter-class separation of the DNN, respectively. By using NCMI to evaluate
popular DNNs pretrained over ImageNet in the literature, it is shown that their
validation accuracies over ImageNet validation data set are more or less
inversely proportional to their NCMI values. Based on this observation, the
standard deep learning (DL) framework is further modified to minimize the
standard cross entropy function subject to an NCMI constraint, yielding CMI
constrained deep learning (CMIC-DL). A novel alternating learning algorithm is
proposed to solve such a constrained optimization problem. Extensive experiment
results show that DNNs trained within CMIC-DL outperform the state-of-the-art
models trained within the standard DL and other loss functions in the
literature in terms of both accuracy and robustness against adversarial
attacks. In addition, visualizing the evolution of learning process through the
lens of CMI and NCMI is also advocated.


http://arxiv.org/abs/2309.08650
Adversarial Attacks on Tables with Entity Swap. (92%)
Aneta Koleva; Martin Ringsquandl; Volker Tresp
  The capabilities of large language models (LLMs) have been successfully
applied in the context of table representation learning. The recently proposed
tabular language models have reported state-of-the-art results across various
tasks for table interpretation. However, a closer look into the datasets
commonly used for evaluation reveals an entity leakage from the train set into
the test set. Motivated by this observation, we explore adversarial attacks
that represent a more realistic inference setup. Adversarial attacks on text
have been shown to greatly affect the performance of LLMs, but currently, there
are no attacks targeting tabular language models. In this paper, we propose an
evasive entity-swap attack for the column type annotation (CTA) task. Our CTA
attack is the first black-box attack on tables, where we employ a
similarity-based sampling strategy to generate adversarial examples. The
experimental results show that the proposed attack generates up to a 70% drop
in performance.


http://arxiv.org/abs/2309.08549
HINT: Healthy Influential-Noise based Training to Defend against Data Poisoning Attacks. (87%)
Minh-Hao Van; Alycia N. Carey; Xintao Wu
  While numerous defense methods have been proposed to prohibit potential
poisoning attacks from untrusted data sources, most research works only defend
against specific attacks, which leaves many avenues for an adversary to
exploit. In this work, we propose an efficient and robust training approach to
defend against data poisoning attacks based on influence functions, named
Healthy Influential-Noise based Training. Using influence functions, we craft
healthy noise that helps to harden the classification model against poisoning
attacks without significantly affecting the generalization ability on test
data. In addition, our method can perform effectively when only a subset of the
training data is modified, instead of the current method of adding noise to all
examples that has been used in several previous works. We conduct comprehensive
evaluations over two image datasets with state-of-the-art poisoning attacks
under different realistic attack scenarios. Our empirical results show that
HINT can efficiently protect deep learning models against the effect of both
untargeted and targeted poisoning attacks.


http://arxiv.org/abs/2309.08230
A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services. (1%)
Hongsheng Hu; Shuo Wang; Jiamin Chang; Haonan Zhong; Ruoxi Sun; Shuang Hao; Haojin Zhu; Minhui Xue
  The right to be forgotten requires the removal or "unlearning" of a user's
data from machine learning models. However, in the context of Machine Learning
as a Service (MLaaS), retraining a model from scratch to fulfill the unlearning
request is impractical due to the lack of training data on the service
provider's side (the server). Furthermore, approximate unlearning further
embraces a complex trade-off between utility (model performance) and privacy
(unlearning performance). In this paper, we try to explore the potential
threats posed by unlearning services in MLaaS, specifically over-unlearning,
where more information is unlearned than expected. We propose two strategies
that leverage over-unlearning to measure the impact on the trade-off balancing,
under black-box access settings, in which the existing machine unlearning
attacks are not applicable. The effectiveness of these strategies is evaluated
through extensive experiments on benchmark datasets, across various model
architectures and representative unlearning approaches. Results indicate
significant potential for both strategies to undermine model efficacy in
unlearning scenarios. This study uncovers an underexplored gap between
unlearning and contemporary MLaaS, highlighting the need for careful
considerations in balancing data unlearning, model utility, and security.


http://arxiv.org/abs/2309.08825
Distributionally Robust Post-hoc Classifiers under Prior Shifts. (1%)
Jiaheng Wei; Harikrishna Narasimhan; Ehsan Amid; Wen-Sheng Chu; Yang Liu; Abhishek Kumar
  The generalization ability of machine learning models degrades significantly
when the test distribution shifts away from the training distribution. We
investigate the problem of training models that are robust to shifts caused by
changes in the distribution of class-priors or group-priors. The presence of
skewed training priors can often lead to the models overfitting to spurious
features. Unlike existing methods, which optimize for either the worst or the
average performance over classes or groups, our work is motivated by the need
for finer control over the robustness properties of the model. We present an
extremely lightweight post-hoc approach that performs scaling adjustments to
predictions from a pre-trained model, with the goal of minimizing a
distributionally robust loss around a chosen target distribution. These
adjustments are computed by solving a constrained optimization problem on a
validation set and applied to the model during test time. Our constrained
optimization objective is inspired by a natural notion of robustness to
controlled distribution shifts. Our method comes with provable guarantees and
empirically makes a strong case for distributional robust post-hoc classifiers.
An empirical implementation is available at
https://github.com/weijiaheng/Drops.


http://arxiv.org/abs/2309.08058
Unleashing the Adversarial Facet of Software Debloating. (98%)
Do-Men Su; Mohannad Alhanahnah
  Software debloating techniques are applied to craft a specialized version of
the program based on the user's requirements and remove irrelevant code
accordingly. The debloated programs presumably maintain better performance and
reduce the attack surface in contrast to the original programs. This work
unleashes the effectiveness of applying software debloating techniques on the
robustness of machine learning systems in the malware classification domain. We
empirically study how an adversarial can leverage software debloating
techniques to mislead machine learning malware classification models. We apply
software debloating techniques to generate adversarial examples and demonstrate
these adversarial examples can reduce the detection rate of VirusTotal. Our
study opens new directions for research into adversarial machine learning not
only in malware detection/classification but also in other software domains.


http://arxiv.org/abs/2309.07983
SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systems. (76%)
Guangke Chen; Yedi Zhang; Fu Song
  Membership inference attacks allow adversaries to determine whether a
particular example was contained in the model's training dataset. While
previous works have confirmed the feasibility of such attacks in various
applications, none has focused on speaker recognition (SR), a promising
voice-based biometric recognition technique. In this work, we propose SLMIA-SR,
the first membership inference attack tailored to SR. In contrast to
conventional example-level attack, our attack features speaker-level membership
inference, i.e., determining if any voices of a given speaker, either the same
as or different from the given inference voices, have been involved in the
training of a model. It is particularly useful and practical since the training
and inference voices are usually distinct, and it is also meaningful
considering the open-set nature of SR, namely, the recognition speakers were
often not present in the training data. We utilize intra-similarity and
inter-dissimilarity, two training objectives of SR, to characterize the
differences between training and non-training speakers and quantify them with
two groups of features driven by carefully-established feature engineering to
mount the attack. To improve the generalizability of our attack, we propose a
novel mixing ratio training strategy to train attack models. To enhance the
attack performance, we introduce voice chunk splitting to cope with the limited
number of inference voices and propose to train attack models dependent on the
number of inference voices. Our attack is versatile and can work in both
white-box and black-box scenarios. Additionally, we propose two novel
techniques to reduce the number of black-box queries while maintaining the
attack performance. Extensive experiments demonstrate the effectiveness of
SLMIA-SR.


http://arxiv.org/abs/2311.16113
BAGEL: Backdoor Attacks against Federated Contrastive Learning. (16%)
Yao Huang; Kongyang Chen; Jiannong Cao; Jiaxing Shen; Shaowei Wang; Yun Peng; Weilong Peng; Kechao Cai
  Federated Contrastive Learning (FCL) is an emerging privacy-preserving
paradigm in distributed learning for unlabeled data. In FCL, distributed
parties collaboratively learn a global encoder with unlabeled data, and the
global encoder could be widely used as a feature extractor to build models for
many downstream tasks. However, FCL is also vulnerable to many security threats
(e.g., backdoor attacks) due to its distributed nature, which are seldom
investigated in existing solutions. In this paper, we study the backdoor attack
against FCL as a pioneer research, to illustrate how backdoor attacks on
distributed local clients act on downstream tasks. Specifically, in our system,
malicious clients can successfully inject a backdoor into the global encoder by
uploading poisoned local updates, thus downstream models built with this global
encoder will also inherit the backdoor. We also investigate how to inject
backdoors into multiple downstream models, in terms of two different backdoor
attacks, namely the \textit{centralized attack} and the \textit{decentralized
attack}. Experiment results show that both the centralized and the
decentralized attacks can inject backdoors into downstream models effectively
with high attack success rates. Finally, we evaluate two defense methods
against our proposed backdoor attacks in FCL, which indicates that the
decentralized backdoor attack is more stealthy and harder to defend.


http://arxiv.org/abs/2309.07808
What Matters to Enhance Traffic Rule Compliance of Imitation Learning for Automated Driving. (13%)
Hongkuan Zhou; Aifen Sui; Wei Cao; Letian Shi
  More research attention has recently been given to end-to-end autonomous
driving technologies where the entire driving pipeline is replaced with a
single neural network because of its simpler structure and faster inference
time. Despite this appealing approach largely reducing the components in
driving pipeline, its simplicity also leads to interpretability problems and
safety issues arXiv:2003.06404. The trained policy is not always compliant with
the traffic rules and it is also hard to discover the reason for the
misbehavior because of the lack of intermediate outputs. Meanwhile, Sensors are
also critical to autonomous driving's security and feasibility to perceive the
surrounding environment under complex driving scenarios. In this paper, we
proposed P-CSG, a novel penalty-based imitation learning approach with cross
semantics generation sensor fusion technologies to increase the overall
performance of End-to-End Autonomous Driving. We conducted an assessment of our
model's performance using the Town 05 Long benchmark, achieving an impressive
driving score improvement of over 15%. Furthermore, we conducted robustness
evaluations against adversarial attacks like FGSM and Dot attacks, revealing a
substantial increase in robustness compared to baseline models.More detailed
information, such as code-based resources, ablation studies and videos can be
found at https://hk-zh.github.io/p-csg-plus.


http://arxiv.org/abs/2309.07428
Physical Invisible Backdoor Based on Camera Imaging. (2%)
Yusheng Guo; Nan Zhong; Zhenxing Qian; Xinpeng Zhang
  Backdoor attack aims to compromise a model, which returns an adversary-wanted
output when a specific trigger pattern appears yet behaves normally for clean
inputs. Current backdoor attacks require changing pixels of clean images, which
results in poor stealthiness of attacks and increases the difficulty of the
physical implementation. This paper proposes a novel physical invisible
backdoor based on camera imaging without changing nature image pixels.
Specifically, a compromised model returns a target label for images taken by a
particular camera, while it returns correct results for other images. To
implement and evaluate the proposed backdoor, we take shots of different
objects from multi-angles using multiple smartphones to build a new dataset of
21,500 images. Conventional backdoor attacks work ineffectively with some
classical models, such as ResNet18, over the above-mentioned dataset.
Therefore, we propose a three-step training strategy to mount the backdoor
attack. First, we design and train a camera identification model with the phone
IDs to extract the camera fingerprint feature. Subsequently, we elaborate a
special network architecture, which is easily compromised by our backdoor
attack, by leveraging the attributes of the CFA interpolation algorithm and
combining it with the feature extraction block in the camera identification
model. Finally, we transfer the backdoor from the elaborated special network
architecture to the classical architecture model via teacher-student
distillation learning. Since the trigger of our method is related to the
specific phone, our attack works effectively in the physical world. Experiment
results demonstrate the feasibility of our proposed approach and robustness
against various backdoor defenses.


http://arxiv.org/abs/2309.07973
M3Dsynth: A dataset of medical 3D images with AI-generated local manipulations. (1%)
Giada Zingarini; Davide Cozzolino; Riccardo Corvi; Giovanni Poggi; Luisa Verdoliva
  The ability to detect manipulated visual content is becoming increasingly
important in many application fields, given the rapid advances in image
synthesis methods. Of particular concern is the possibility of modifying the
content of medical images, altering the resulting diagnoses. Despite its
relevance, this issue has received limited attention from the research
community. One reason is the lack of large and curated datasets to use for
development and benchmarking purposes. Here, we investigate this issue and
propose M3Dsynth, a large dataset of manipulated Computed Tomography (CT) lung
images. We create manipulated images by injecting or removing lung cancer
nodules in real CT scans, using three different methods based on Generative
Adversarial Networks (GAN) or Diffusion Models (DM), for a total of 8,577
manipulated samples. Experiments show that these images easily fool automated
diagnostic tools. We also tested several state-of-the-art forensic detectors
and demonstrated that, once trained on the proposed dataset, they are able to
accurately detect and localize manipulated synthetic content, including when
training and test sets are not aligned, showing good generalization ability.
Dataset and code will be publicly available at
https://grip-unina.github.io/M3Dsynth/.


http://arxiv.org/abs/2309.07398
Semantic Adversarial Attacks via Diffusion Models. (99%)
Chenan Wang; Jinhao Duan; Chaowei Xiao; Edward Kim; Matthew Stamm; Kaidi Xu
  Traditional adversarial attacks concentrate on manipulating clean examples in
the pixel space by adding adversarial perturbations. By contrast, semantic
adversarial attacks focus on changing semantic attributes of clean examples,
such as color, context, and features, which are more feasible in the real
world. In this paper, we propose a framework to quickly generate a semantic
adversarial attack by leveraging recent diffusion models since semantic
information is included in the latent space of well-trained diffusion models.
Then there are two variants of this framework: 1) the Semantic Transformation
(ST) approach fine-tunes the latent space of the generated image and/or the
diffusion model itself; 2) the Latent Masking (LM) approach masks the latent
space with another target image and local backpropagation-based interpretation
methods. Additionally, the ST approach can be applied in either white-box or
black-box settings. Extensive experiments are conducted on CelebA-HQ and AFHQ
datasets, and our framework demonstrates great fidelity, generalizability, and
transferability compared to other baselines. Our approaches achieve
approximately 100% attack success rate in multiple settings with the best FID
as 36.61. Code is available at
https://github.com/steven202/semantic_adv_via_dm.


http://arxiv.org/abs/2309.07106
Hardening RGB-D Object Recognition Systems against Adversarial Patch Attacks. (99%)
Yang Zheng; Luca Demetrio; Antonio Emanuele Cinà; Xiaoyi Feng; Zhaoqiang Xia; Xiaoyue Jiang; Ambra Demontis; Battista Biggio; Fabio Roli
  RGB-D object recognition systems improve their predictive performances by
fusing color and depth information, outperforming neural network architectures
that rely solely on colors. While RGB-D systems are expected to be more robust
to adversarial examples than RGB-only systems, they have also been proven to be
highly vulnerable. Their robustness is similar even when the adversarial
examples are generated by altering only the original images' colors. Different
works highlighted the vulnerability of RGB-D systems; however, there is a
lacking of technical explanations for this weakness. Hence, in our work, we
bridge this gap by investigating the learned deep representation of RGB-D
systems, discovering that color features make the function learned by the
network more complex and, thus, more sensitive to small perturbations. To
mitigate this problem, we propose a defense based on a detection mechanism that
makes RGB-D systems more robust against adversarial examples. We empirically
show that this defense improves the performances of RGB-D systems against
adversarial examples even when they are computed ad-hoc to circumvent this
detection mechanism, and that is also more effective than adversarial training.


http://arxiv.org/abs/2309.07197
Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environments. (99%)
Simon Queyrut; Valerio Schiavoni; Pascal Felber
  The main premise of federated learning (FL) is that machine learning model
updates are computed locally to preserve user data privacy. This approach
avoids by design user data to ever leave the perimeter of their device. Once
the updates aggregated, the model is broadcast to all nodes in the federation.
However, without proper defenses, compromised nodes can probe the model inside
their local memory in search for adversarial examples, which can lead to
dangerous real-world scenarios. For instance, in image-based applications,
adversarial examples consist of images slightly perturbed to the human eye
getting misclassified by the local model. These adversarial images are then
later presented to a victim node's counterpart model to replay the attack.
Typical examples harness dissemination strategies such as altered traffic signs
(patch attacks) no longer recognized by autonomous vehicles or seemingly
unaltered samples that poison the local dataset of the FL scheme to undermine
its robustness. Pelta is a novel shielding mechanism leveraging Trusted
Execution Environments (TEEs) that reduce the ability of attackers to craft
adversarial samples. Pelta masks inside the TEE the first part of the
back-propagation chain rule, typically exploited by attackers to craft the
malicious samples. We evaluate Pelta on state-of-the-art accurate models using
three well-established datasets: CIFAR-10, CIFAR-100 and ImageNet. We show the
effectiveness of Pelta in mitigating six white-box state-of-the-art adversarial
attacks, such as Projected Gradient Descent, Momentum Iterative Method, Auto
Projected Gradient Descent, the Carlini & Wagner attack. In particular, Pelta
constitutes the first attempt at defending an ensemble model against the
Self-Attention Gradient attack to the best of our knowledge. Our code is
available to the research community at https://github.com/queyrusi/Pelta.


http://arxiv.org/abs/2309.06960
PhantomSound: Black-Box, Query-Efficient Audio Adversarial Attack via Split-Second Phoneme Injection. (99%)
Hanqing Guo; Guangjing Wang; Yuanda Wang; Bocheng Chen; Qiben Yan; Li Xiao
  In this paper, we propose PhantomSound, a query-efficient black-box attack
toward voice assistants. Existing black-box adversarial attacks on voice
assistants either apply substitution models or leverage the intermediate model
output to estimate the gradients for crafting adversarial audio samples.
However, these attack approaches require a significant amount of queries with a
lengthy training stage. PhantomSound leverages the decision-based attack to
produce effective adversarial audios, and reduces the number of queries by
optimizing the gradient estimation. In the experiments, we perform our attack
against 4 different speech-to-text APIs under 3 real-world scenarios to
demonstrate the real-time attack impact. The results show that PhantomSound is
practical and robust in attacking 5 popular commercial voice controllable
devices over the air, and is able to bypass 3 liveness detection mechanisms
with >95% success rate. The benchmark result shows that PhantomSound can
generate adversarial examples and launch the attack in a few minutes. We
significantly enhance the query efficiency and reduce the cost of a successful
untargeted and targeted adversarial attack by 93.1% and 65.5% compared with the
state-of-the-art black-box attacks, using merely ~300 queries (~5 minutes) and
~1,500 queries (~25 minutes), respectively.


http://arxiv.org/abs/2309.07026
APICom: Automatic API Completion via Prompt Learning and Adversarial Training-based Data Augmentation. (92%)
Yafeng Gu; Yiheng Shen; Xiang Chen; Shaoyu Yang; Yiling Huang; Zhixiang Cao
  Based on developer needs and usage scenarios, API (Application Programming
Interface) recommendation is the process of assisting developers in finding the
required API among numerous candidate APIs. Previous studies mainly modeled API
recommendation as the recommendation task, which can recommend multiple
candidate APIs for the given query, and developers may not yet be able to find
what they need. Motivated by the neural machine translation research domain, we
can model this problem as the generation task, which aims to directly generate
the required API for the developer query. After our preliminary investigation,
we find the performance of this intuitive approach is not promising. The reason
is that there exists an error when generating the prefixes of the API. However,
developers may know certain API prefix information during actual development in
most cases. Therefore, we model this problem as the automatic completion task
and propose a novel approach APICom based on prompt learning, which can
generate API related to the query according to the prompts (i.e., API prefix
information). Moreover, the effectiveness of APICom highly depends on the
quality of the training dataset. In this study, we further design a novel
gradient-based adversarial training method {\atpart} for data augmentation,
which can improve the normalized stability when generating adversarial
examples. To evaluate the effectiveness of APICom, we consider a corpus of 33k
developer queries and corresponding APIs. Compared with the state-of-the-art
baselines, our experimental results show that APICom can outperform all
baselines by at least 40.02\%, 13.20\%, and 16.31\% in terms of the performance
measures EM@1, MRR, and MAP. Finally, our ablation studies confirm the
effectiveness of our component setting (such as our designed adversarial
training method, our used pre-trained model, and prompt learning) in APICom.


http://arxiv.org/abs/2309.07124
RAIN: Your Language Models Can Align Themselves without Finetuning. (83%)
Yuhui Li; Fangyun Wei; Jinjing Zhao; Chao Zhang; Hongyang Zhang
  Large language models (LLMs) often demonstrate inconsistencies with human
preferences. Previous research gathered human preference data and then aligned
the pre-trained models using reinforcement learning or instruction tuning, the
so-called finetuning step. In contrast, aligning frozen LLMs without any extra
data is more appealing. This work explores the potential of the latter setting.
We discover that by integrating self-evaluation and rewind mechanisms,
unaligned LLMs can directly produce responses consistent with human preferences
via self-boosting. We introduce a novel inference method, Rewindable
Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate
their own generation and use the evaluation results to guide backward rewind
and forward generation for AI safety. Notably, RAIN operates without the need
of extra data for model alignment and abstains from any training, gradient
computation, or parameter updates; during the self-evaluation phase, the model
receives guidance on which human preference to align with through a
fixed-template prompt, eliminating the need to modify the initial prompt.
Experimental results evaluated by GPT-4 and humans demonstrate the
effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate
of LLaMA 30B over vanilla inference from 82% to 97%, while maintaining the
helpfulness rate. Under the leading adversarial attack llm-attacks on Vicuna
33B, RAIN establishes a new defense baseline by reducing the attack success
rate from 94% to 19%.


http://arxiv.org/abs/2309.06978
Differentiable JPEG: The Devil is in the Details. (70%)
Christoph Reich; Biplob Debnath; Deep Patel; Srimat Chakradhar
  JPEG remains one of the most widespread lossy image coding methods. However,
the non-differentiable nature of JPEG restricts the application in deep
learning pipelines. Several differentiable approximations of JPEG have recently
been proposed to address this issue. This paper conducts a comprehensive review
of existing diff. JPEG approaches and identifies critical details that have
been missed by previous methods. To this end, we propose a novel diff. JPEG
approach, overcoming previous limitations. Our approach is differentiable
w.r.t. the input image, the JPEG quality, the quantization tables, and the
color conversion parameters. We evaluate the forward and backward performance
of our diff. JPEG approach against existing methods. Additionally, extensive
ablations are performed to evaluate crucial design choices. Our proposed diff.
JPEG resembles the (non-diff.) reference implementation best, significantly
surpassing the recent-best diff. approach by $3.47$dB (PSNR) on average. For
strong compression rates, we can even improve PSNR by $9.51$dB. Strong
adversarial attack results are yielded by our diff. JPEG, demonstrating the
effective gradient approximation. Our code is available at
https://github.com/necla-ml/Diff-JPEG.


http://arxiv.org/abs/2309.06724
Deep Nonparametric Convexified Filtering for Computational Photography, Image Synthesis and Adversarial Defense. (41%)
Jianqiao Wangni
  We aim to provide a general framework of for computational photography that
recovers the real scene from imperfect images, via the Deep Nonparametric
Convexified Filtering (DNCF). It is consists of a nonparametric deep network to
resemble the physical equations behind the image formation, such as denoising,
super-resolution, inpainting, and flash. DNCF has no parameterization dependent
on training data, therefore has a strong generalization and robustness to
adversarial image manipulation. During inference, we also encourage the network
parameters to be nonnegative and create a bi-convex function on the input and
parameters, and this adapts to second-order optimization algorithms with
insufficient running time, having 10X acceleration over Deep Image Prior. With
these tools, we empirically verify its capability to defend image
classification deep networks against adversary attack algorithms in real-time.


http://arxiv.org/abs/2309.06981
MASTERKEY: Practical Backdoor Attack Against Speaker Verification Systems. (38%)
Hanqing Guo; Xun Chen; Junfeng Guo; Li Xiao; Qiben Yan
  Speaker Verification (SV) is widely deployed in mobile systems to
authenticate legitimate users by using their voice traits. In this work, we
propose a backdoor attack MASTERKEY, to compromise the SV models. Different
from previous attacks, we focus on a real-world practical setting where the
attacker possesses no knowledge of the intended victim. To design MASTERKEY, we
investigate the limitation of existing poisoning attacks against unseen
targets. Then, we optimize a universal backdoor that is capable of attacking
arbitrary targets. Next, we embed the speaker's characteristics and semantics
information into the backdoor, making it imperceptible. Finally, we estimate
the channel distortion and integrate it into the backdoor. We validate our
attack on 6 popular SV models. Specifically, we poison a total of 53 models and
use our trigger to attack 16,430 enrolled speakers, composed of 310 target
speakers enrolled in 53 poisoned models. Our attack achieves 100% attack
success rate with a 15% poison rate. By decreasing the poison rate to 3%, the
attack success rate remains around 50%. We validate our attack in 3 real-world
scenarios and successfully demonstrate the attack through both over-the-air and
over-the-telephony-line scenarios.


http://arxiv.org/abs/2309.07415
Client-side Gradient Inversion Against Federated Learning from Poisoning. (22%)
Jiaheng Wei; Yanjun Zhang; Leo Yu Zhang; Chao Chen; Shirui Pan; Kok-Leong Ong; Jun Zhang; Yang Xiang
  Federated Learning (FL) enables distributed participants (e.g., mobile
devices) to train a global model without sharing data directly to a central
server. Recent studies have revealed that FL is vulnerable to gradient
inversion attack (GIA), which aims to reconstruct the original training samples
and poses high risk against the privacy of clients in FL. However, most
existing GIAs necessitate control over the server and rely on strong prior
knowledge including batch normalization and data distribution information. In
this work, we propose Client-side poisoning Gradient Inversion (CGI), which is
a novel attack method that can be launched from clients. For the first time, we
show the feasibility of a client-side adversary with limited knowledge being
able to recover the training samples from the aggregated global model. We take
a distinct approach in which the adversary utilizes a malicious model that
amplifies the loss of a specific targeted class of interest. When honest
clients employ the poisoned global model, the gradients of samples belonging to
the targeted class are magnified, making them the dominant factor in the
aggregated update. This enables the adversary to effectively reconstruct the
private input belonging to other clients using the aggregated update. In
addition, our CGI also features its ability to remain stealthy against
Byzantine-robust aggregation rules (AGRs). By optimizing malicious updates and
blending benign updates with a malicious replacement vector, our method remains
undetected by these defense mechanisms. To evaluate the performance of CGI, we
conduct experiments on various benchmark datasets, considering representative
Byzantine-robust AGRs, and exploring diverse FL settings with different levels
of adversary knowledge about the data. Our results demonstrate that CGI
consistently and successfully extracts training input in all tested scenarios.


http://arxiv.org/abs/2309.06835
Safe Reinforcement Learning with Dual Robustness. (1%)
Zeyang Li; Chuxiong Hu; Yunan Wang; Yujie Yang; Shengbo Eben Li
  Reinforcement learning (RL) agents are vulnerable to adversarial
disturbances, which can deteriorate task performance or compromise safety
specifications. Existing methods either address safety requirements under the
assumption of no adversary (e.g., safe RL) or only focus on robustness against
performance adversaries (e.g., robust RL). Learning one policy that is both
safe and robust remains a challenging open problem. The difficulty is how to
tackle two intertwined aspects in the worst cases: feasibility and optimality.
Optimality is only valid inside a feasible region, while identification of
maximal feasible region must rely on learning the optimal policy. To address
this issue, we propose a systematic framework to unify safe RL and robust RL,
including problem formulation, iteration scheme, convergence analysis and
practical algorithm design. This unification is built upon constrained
two-player zero-sum Markov games. A dual policy iteration scheme is proposed,
which simultaneously optimizes a task policy and a safety policy. The
convergence of this iteration scheme is proved. Furthermore, we design a deep
RL algorithm for practical implementation, called dually robust actor-critic
(DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC
achieves high performance and persistent safety under all scenarios (no
adversary, safety adversary, performance adversary), outperforming all
baselines significantly.


http://arxiv.org/abs/2309.06359
Using Reed-Muller Codes for Classification with Rejection and Recovery. (99%)
Daniel University of Birmingham Fentham; David University of Oxford Parker; Mark University of Birmingham Ryan
  When deploying classifiers in the real world, users expect them to respond to
inputs appropriately. However, traditional classifiers are not equipped to
handle inputs which lie far from the distribution they were trained on.
Malicious actors can exploit this defect by making adversarial perturbations
designed to cause the classifier to give an incorrect output.
Classification-with-rejection methods attempt to solve this problem by allowing
networks to refuse to classify an input in which they have low confidence. This
works well for strongly adversarial examples, but also leads to the rejection
of weakly perturbed images, which intuitively could be correctly classified. To
address these issues, we propose Reed-Muller Aggregation Networks (RMAggNet), a
classifier inspired by Reed-Muller error-correction codes which can correct and
reject inputs. This paper shows that RMAggNet can minimise incorrectness while
maintaining good correctness over multiple adversarial attacks at different
perturbation budgets by leveraging the ability to correct errors in the
classification process. This provides an alternative
classification-with-rejection method which can reduce the amount of additional
processing in situations where a small number of incorrect classifications are
permissible.


http://arxiv.org/abs/2309.06166
Certified Robust Models with Slack Control and Large Lipschitz Constants. (98%)
Max Losch; David Stutz; Bernt Schiele; Mario Fritz
  Despite recent success, state-of-the-art learning-based models remain highly
vulnerable to input changes such as adversarial examples. In order to obtain
certifiable robustness against such perturbations, recent work considers
Lipschitz-based regularizers or constraints while at the same time increasing
prediction margin. Unfortunately, this comes at the cost of significantly
decreased accuracy. In this paper, we propose a Calibrated Lipschitz-Margin
Loss (CLL) that addresses this issue and improves certified robustness by
tackling two problems: Firstly, commonly used margin losses do not adjust the
penalties to the shrinking output distribution; caused by minimizing the
Lipschitz constant $K$. Secondly, and most importantly, we observe that
minimization of $K$ can lead to overly smooth decision functions. This limits
the model's complexity and thus reduces accuracy. Our CLL addresses these
issues by explicitly calibrating the loss w.r.t. margin and Lipschitz constant,
thereby establishing full control over slack and improving robustness
certificates even with larger Lipschitz constants. On CIFAR-10, CIFAR-100 and
Tiny-ImageNet, our models consistently outperform losses that leave the
constant unattended. On CIFAR-100 and Tiny-ImageNet, CLL improves upon
state-of-the-art deterministic $L_2$ robust accuracies. In contrast to current
trends, we unlock potential of much smaller models without $K=1$ constraints.


http://arxiv.org/abs/2309.06438
Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks. (98%)
Jindong Gu; Fangyun Wei; Philip Torr; Han Hu
  Deep Neural Networks can be easily fooled by small and imperceptible
perturbations. The query-based black-box attack (QBBA) is able to create the
perturbations using model output probabilities of image queries requiring no
access to the underlying models. QBBA poses realistic threats to real-world
applications. Recently, various types of robustness have been explored to
defend against QBBA. In this work, we first taxonomize the stochastic defense
strategies against QBBA. Following our taxonomy, we propose to explore
non-additive randomness in models to defend against QBBA. Specifically, we
focus on underexplored Vision Transformers based on their flexible
architectures. Extensive experiments show that the proposed defense approach
achieves effective defense, without much sacrifice in performance.


http://arxiv.org/abs/2309.06055
Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review. (61%)
Pengzhou Cheng; Zongru Wu; Wei Du; Gongshen Liu
  Deep Neural Networks (DNNs) have led to unprecedented progress in various
natural language processing (NLP) tasks. Owing to limited data and computation
resources, using third-party data and models has become a new paradigm for
adapting various tasks. However, research shows that it has some potential
security vulnerabilities because attackers can manipulate the training process
and data source. Such a way can set specific triggers, making the model exhibit
expected behaviors that have little inferior influence on the model's
performance for primitive tasks, called backdoor attacks. Hence, it could have
dire consequences, especially considering that the backdoor attack surfaces are
broad.
  To get a precise grasp and understanding of this problem, a systematic and
comprehensive review is required to confront various security challenges from
different phases and attack purposes. Additionally, there is a dearth of
analysis and comparison of the various emerging backdoor countermeasures in
this situation. In this paper, we conduct a timely review of backdoor attacks
and countermeasures to sound the red alarm for the NLP security community.
According to the affected stage of the machine learning pipeline, the attack
surfaces are recognized to be wide and then formalized into three
categorizations: attacking pre-trained model with fine-tuning (APMF) or
prompt-tuning (APMP), and attacking final model with training (AFMT), where
AFMT can be subdivided into different attack aims. Thus, attacks under each
categorization are combed. The countermeasures are categorized into two general
classes: sample inspection and model inspection. Overall, the research on the
defense side is far behind the attack side, and there is no single defense that
can prevent all types of backdoor attacks. An attacker can intelligently bypass
existing defenses with a more invisible attack. ......


http://arxiv.org/abs/2309.05978
CToMP: A Cycle-task-oriented Memory Protection Scheme for Unmanned Systems. (8%)
Chengyan Ma; Ning Xi; Di Lu; Yebo Feng; Jianfeng Ma
  Memory corruption attacks (MCAs) refer to malicious behaviors of system
intruders that modify the contents of a memory location to disrupt the normal
operation of computing systems, causing leakage of sensitive data or
perturbations to ongoing processes. Unlike general-purpose systems, unmanned
systems cannot deploy complete security protection schemes, due to their
limitations in size, cost and performance. MCAs in unmanned systems are
particularly difficult to defend against. Furthermore, MCAs have diverse and
unpredictable attack interfaces in unmanned systems, severely impacting digital
and physical sectors. In this paper, we first generalize, model and taxonomize
MCAs found in unmanned systems currently, laying the foundation for designing a
portable and general defense approach. According to different attack
mechanisms, we found that MCAs are mainly categorized into two
types--return2libc and return2shellcode. To tackle return2libc attacks, we
model the erratic operation of unmanned systems with cycles and then propose a
cycle-task-oriented memory protection (CToMP) approach to protect control flows
from tampering. To defend against return2shellcode attacks, we introduce a
secure process stack with a randomized memory address by leveraging the memory
pool to prevent Shellcode from being executed. Moreover, we discuss the
mechanism by which CToMP resists the ROP attack, a novel variant of return2libc
attacks. Finally, we implement CToMP on CUAV V5+ with Ardupilot and Crazyflie.
The evaluation and security analysis results demonstrate that the proposed
approach CToMP is resilient to various MCAs in unmanned systems with low
footprints and system overhead.


http://arxiv.org/abs/2309.05950
Language Models as Black-Box Optimizers for Vision-Language Models. (4%)
Shihong Liu; Samuel Yu; Zhiqiu Lin; Deepak Pathak; Deva Ramanan
  Vision-language models (VLMs) pre-trained on web-scale datasets have
demonstrated remarkable capabilities across a variety of vision and multimodal
tasks. Currently, fine-tuning methods for VLMs mainly operate in a white-box
setting, requiring access to model parameters for backpropagation. However,
many VLMs rely on proprietary data and are not open-source, which restricts the
use of white-box approaches for fine-tuning. Given that popular private large
language models (LLMs) like ChatGPT still offer a language-based user
interface, we aim to develop a novel fine-tuning approach for VLMs through
natural language prompts, thereby avoiding the need to access model parameters,
feature embeddings, or output logits. In this setup, we propose employing
chat-based LLMs as black-box optimizers to search for the best text prompt on
the illustrative task of few-shot image classification using CLIP.
Specifically, we adopt an automatic "hill-climbing" procedure that converges on
an effective prompt by evaluating the accuracy of current prompts and asking
LLMs to refine them based on textual feedback, all within a conversational
process without human-in-the-loop. In a challenging 1-shot learning setup, our
simple approach surpasses the white-box continuous prompting method (CoOp) by
an average of 1.5% across 11 datasets including ImageNet. Our approach also
outperforms OpenAI's manually crafted prompts. Additionally, we highlight the
advantage of conversational feedback that incorporates both positive and
negative prompts, suggesting that LLMs can utilize the implicit "gradient"
direction in textual feedback for a more efficient search. Lastly, we find that
the text prompts generated through our strategy are not only more interpretable
but also transfer well across different CLIP architectures in a black-box
manner.


http://arxiv.org/abs/2309.06223
Unveiling Signle-Bit-Flip Attacks on DNN Executables. (1%)
Yanzuo The Hong Kong University of Science and Technology Chen; Zhibo The Hong Kong University of Science and Technology Liu; Yuanyuan The Hong Kong University of Science and Technology Yuan; Sihang Huawei Technologies Hu; Tianxiang Huawei Technologies Li; Shuai The Hong Kong University of Science and Technology Wang
  Recent research has shown that bit-flip attacks (BFAs) can manipulate deep
neural networks (DNNs) via DRAM Rowhammer exploitations. Existing attacks are
primarily launched over high-level DNN frameworks like PyTorch and flip bits in
model weight files. Nevertheless, DNNs are frequently compiled into low-level
executables by deep learning (DL) compilers to fully leverage low-level
hardware primitives. The compiled code is usually high-speed and manifests
dramatically distinct execution paradigms from high-level DNN frameworks.
  In this paper, we launch the first systematic study on the attack surface of
BFA specifically for DNN executables compiled by DL compilers. We design an
automated search tool to identify vulnerable bits in DNN executables and
identify practical attack vectors that exploit the model structure in DNN
executables with BFAs (whereas prior works make likely strong assumptions to
attack model weights). DNN executables appear more "opaque" than models in
high-level DNN frameworks. Nevertheless, we find that DNN executables contain
extensive, severe (e.g., single-bit flip), and transferrable attack surfaces
that are not present in high-level DNN models and can be exploited to deplete
full model intelligence and control output labels. Our finding calls for
incorporating security mechanisms in future DNN compilation toolchains.


http://arxiv.org/abs/2309.05879
Generalized Attacks on Face Verification Systems. (88%)
Ehsan Nazari; Paula Branco; Guy-Vincent Jourdan
  Face verification (FV) using deep neural network models has made tremendous
progress in recent years, surpassing human accuracy and seeing deployment in
various applications such as border control and smartphone unlocking. However,
FV systems are vulnerable to Adversarial Attacks, which manipulate input images
to deceive these systems in ways usually unnoticeable to humans. This paper
provides an in-depth study of attacks on FV systems. We introduce the
DodgePersonation Attack that formulates the creation of face images that
impersonate a set of given identities while avoiding being identified as any of
the identities in a separate, disjoint set. A taxonomy is proposed to provide a
unified view of different types of Adversarial Attacks against FV systems,
including Dodging Attacks, Impersonation Attacks, and Master Face Attacks.
Finally, we propose the ''One Face to Rule Them All'' Attack which implements
the DodgePersonation Attack with state-of-the-art performance on a well-known
scenario (Master Face Attack) and which can also be used for the new scenarios
introduced in this paper. While the state-of-the-art Master Face Attack can
produce a set of 9 images to cover 43.82% of the identities in their test
database, with 9 images our attack can cover 57.27% to 58.5% of these
identifies while giving the attacker the choice of the identity to use to
create the impersonation. Moreover, the 9 generated attack images appear
identical to a casual observer.


http://arxiv.org/abs/2309.05900
Adversarial Attacks Assessment of Salient Object Detection via Symbolic Learning. (76%)
Gustavo Olague; Roberto Pineda; Gerardo Ibarra-Vazquez; Matthieu Olague; Axel Martinez; Sambit Bakshi; Jonathan Vargas; Isnardo Reducindo
  Machine learning is at the center of mainstream technology and outperforms
classical approaches to handcrafted feature design. Aside from its learning
process for artificial feature extraction, it has an end-to-end paradigm from
input to output, reaching outstandingly accurate results. However, security
concerns about its robustness to malicious and imperceptible perturbations have
drawn attention since its prediction can be changed entirely. Salient object
detection is a research area where deep convolutional neural networks have
proven effective but whose trustworthiness represents a significant issue
requiring analysis and solutions to hackers' attacks. Brain programming is a
kind of symbolic learning in the vein of good old-fashioned artificial
intelligence. This work provides evidence that symbolic learning robustness is
crucial in designing reliable visual attention systems since it can withstand
even the most intense perturbations. We test this evolutionary computation
methodology against several adversarial attacks and noise perturbations using
standard databases and a real-world problem of a shorebird called the Snowy
Plover portraying a visual attention task. We compare our methodology with five
different deep learning approaches, proving that they do not match the symbolic
paradigm regarding robustness. All neural networks suffer significant
performance losses, while brain programming stands its ground and remains
unaffected. Also, by studying the Snowy Plover, we remark on the importance of
security in surveillance activities regarding wildlife protection and
conservation.


http://arxiv.org/abs/2310.10659
Backdoor Attack through Machine Unlearning. (67%)
Peixin Zhang; Jun Sun; Mingtian Tan; Xinyu Wang
  In recent years, the security issues of artificial intelligence have become
increasingly prominent due to the rapid development of deep learning research
and applications. Backdoor attack is an attack targeting the vulnerability of
deep learning models, where hidden backdoors are activated by triggers embedded
by the attacker, thereby outputting malicious predictions that may not align
with the intended output for a given input. In this work, we propose a novel
black-box backdoor attack based on machine unlearning. The attacker first
augments the training set with carefully designed samples, including poison and
mitigation data, to train a 'benign' model. Then, the attacker posts unlearning
requests for the mitigation samples to remove the impact of relevant data on
the model, gradually activating the hidden backdoor. Since backdoors are
implanted during the iterative unlearning process, it significantly increases
the computational overhead of existing defense methods for backdoor detection
or mitigation. To address this new security threat, we propose two methods for
detecting or mitigating such malicious unlearning requests. We conduct the
experiment in both naive unlearning and SISA settings. Experimental results
show that: 1) our attack can successfully implant backdoor into the model, and
sharding increases the difficulty of attack; 2) our detection algorithms are
effective in identifying the mitigation samples, while sharding reduces the
effectiveness of our detection algorithms.


http://arxiv.org/abs/2309.05610
Privacy Side Channels in Machine Learning Systems. (10%)
Edoardo Debenedetti; Giorgio Severi; Nicholas Carlini; Christopher A. Choquette-Choo; Matthew Jagielski; Milad Nasr; Eric Wallace; Florian Tramèr
  Most current approaches for protecting privacy in machine learning (ML)
assume that models exist in a vacuum, when in reality, ML models are part of
larger systems that include components for training data filtering, output
monitoring, and more. In this work, we introduce privacy side channels: attacks
that exploit these system-level components to extract private information at
far higher rates than is otherwise possible for standalone models. We propose
four categories of side channels that span the entire ML lifecycle (training
data filtering, input preprocessing, output post-processing, and query
filtering) and allow for either enhanced membership inference attacks or even
novel threats such as extracting users' test queries. For example, we show that
deduplicating training data before applying differentially-private training
creates a side-channel that completely invalidates any provable privacy
guarantees. Moreover, we show that systems which block language models from
regenerating training data can be exploited to allow exact reconstruction of
private keys contained in the training set -- even if the model did not
memorize these keys. Taken together, our results demonstrate the need for a
holistic, end-to-end privacy analysis of machine learning.


http://arxiv.org/abs/2309.05809
Divergences in Color Perception between Deep Neural Networks and Humans. (4%)
Ethan O. Nadler; Elise Darragh-Ford; Bhargav Srinivasa Desikan; Christian Conaway; Mark Chu; Tasker Hull; Douglas Guilbeault
  Deep neural networks (DNNs) are increasingly proposed as models of human
vision, bolstered by their impressive performance on image classification and
object recognition tasks. Yet, the extent to which DNNs capture fundamental
aspects of human vision such as color perception remains unclear. Here, we
develop novel experiments for evaluating the perceptual coherence of color
embeddings in DNNs, and we assess how well these algorithms predict human color
similarity judgments collected via an online survey. We find that
state-of-the-art DNN architectures $-$ including convolutional neural networks
and vision transformers $-$ provide color similarity judgments that strikingly
diverge from human color judgments of (i) images with controlled color
properties, (ii) images generated from online searches, and (iii) real-world
images from the canonical CIFAR-10 dataset. We compare DNN performance against
an interpretable and cognitively plausible model of color perception based on
wavelet decomposition, inspired by foundational theories in computational
neuroscience. While one deep learning model $-$ a convolutional DNN trained on
a style transfer task $-$ captures some aspects of human color perception, our
wavelet algorithm provides more coherent color embeddings that better predict
human color judgments compared to all DNNs we examine. These results hold when
altering the high-level visual task used to train similar DNN architectures
(e.g., image classification versus image segmentation), as well as when
examining the color embeddings of different layers in a given DNN architecture.
These findings break new ground in the effort to analyze the perceptual
representations of machine learning algorithms and to improve their ability to
serve as cognitively plausible models of human vision. Implications for machine
learning, human perception, and embodied cognition are discussed.


http://arxiv.org/abs/2309.05940
Catch You Everything Everywhere: Guarding Textual Inversion via Concept Watermarking. (1%)
Weitao Feng; Jiyan He; Jie Zhang; Tianwei Zhang; Wenbo Zhou; Weiming Zhang; Nenghai Yu
  AIGC (AI-Generated Content) has achieved tremendous success in many
applications such as text-to-image tasks, where the model can generate
high-quality images with diverse prompts, namely, different descriptions in
natural languages. More surprisingly, the emerging personalization techniques
even succeed in describing unseen concepts with only a few personal images as
references, and there have been some commercial platforms for sharing the
valuable personalized concept. However, such an advanced technique also
introduces a severe threat, where malicious users can misuse the target concept
to generate highly-realistic illegal images. Therefore, it becomes necessary
for the platform to trace malicious users and hold them accountable.
  In this paper, we focus on guarding the most popular lightweight
personalization model, ie, Textual Inversion (TI). To achieve it, we propose
the novel concept watermarking, where watermark information is embedded into
the target concept and then extracted from generated images based on the
watermarked concept. Specifically, we jointly train a watermark encoder and a
watermark decoder with the sampler in the loop.
  It shows great resilience to different diffusion sampling processes possibly
chosen by malicious users, meanwhile preserving utility for normal use. In
practice, the concept owner can upload his concept with different watermarks
(ie, serial numbers) to the platform, and the platform allocates different
users with different serial numbers for subsequent tracing and forensics.


http://arxiv.org/abs/2309.05516
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs. (1%)
Wenhua Cheng; Weiwei Zhang; Haihao Shen; Yiyang Cai; Xin He; Kaokao Lv
  Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound competes impressively against recent
methods without introducing additional inference overhead. The source code will
be publicly available at \url{https://github.com/intel/neural-compressor} soon.


http://arxiv.org/abs/2309.05145
Outlier Robust Adversarial Training. (98%)
Shu Hu; Zhenhuan Yang; Xin Wang; Yiming Ying; Siwei Lyu
  Supervised learning models are challenged by the intrinsic complexities of
training data such as outliers and minority subpopulations and intentional
attacks at inference time with adversarial samples. While traditional robust
learning methods and the recent adversarial training approaches are designed to
handle each of the two challenges, to date, no work has been done to develop
models that are robust with regard to the low-quality training data and the
potential adversarial attack at inference time simultaneously. It is for this
reason that we introduce Outlier Robust Adversarial Training (ORAT) in this
work. ORAT is based on a bi-level optimization formulation of adversarial
training with a robust rank-based loss function. Theoretically, we show that
the learning objective of ORAT satisfies the $\mathcal{H}$-consistency in
binary classification, which establishes it as a proper surrogate to
adversarial 0/1 loss. Furthermore, we analyze its generalization ability and
provide uniform convergence rates in high probability. ORAT can be optimized
with a simple algorithm. Experimental evaluations on three benchmark datasets
demonstrate the effectiveness and robustness of ORAT in handling outliers and
adversarial attacks. Our code is available at
https://github.com/discovershu/ORAT.


http://arxiv.org/abs/2309.05132
DAD++: Improved Data-free Test Time Adversarial Defense. (98%)
Gaurav Kumar Nayak; Inder Khatri; Shubham Randive; Ruchit Rawal; Anirban Chakraborty
  With the increasing deployment of deep neural networks in safety-critical
applications such as self-driving cars, medical imaging, anomaly detection,
etc., adversarial robustness has become a crucial concern in the reliability of
these networks in real-world scenarios. A plethora of works based on
adversarial training and regularization-based techniques have been proposed to
make these deep networks robust against adversarial attacks. However, these
methods require either retraining models or training them from scratch, making
them infeasible to defend pre-trained models when access to training data is
restricted. To address this problem, we propose a test time Data-free
Adversarial Defense (DAD) containing detection and correction frameworks.
Moreover, to further improve the efficacy of the correction framework in cases
when the detector is under-confident, we propose a soft-detection scheme
(dubbed as "DAD++"). We conduct a wide range of experiments and ablations on
several datasets and network architectures to show the efficacy of our proposed
approach. Furthermore, we demonstrate the applicability of our approach in
imparting adversarial defense at test time under data-free (or data-efficient)
applications/setups, such as Data-free Knowledge Distillation and Source-free
Unsupervised Domain Adaptation, as well as Semi-supervised classification
frameworks. We observe that in all the experiments and applications, our DAD++
gives an impressive performance against various adversarial attacks with a
minimal drop in clean accuracy. The source code is available at:
https://github.com/vcl-iisc/Improved-Data-free-Test-Time-Adversarial-Defense


http://arxiv.org/abs/2309.06527
Machine Translation Models Stand Strong in the Face of Adversarial Attacks. (86%)
Pavel Burnyshev; Elizaveta Kostenok; Alexey Zaytsev
  Adversarial attacks expose vulnerabilities of deep learning models by
introducing minor perturbations to the input, which lead to substantial
alterations in the output. Our research focuses on the impact of such
adversarial attacks on sequence-to-sequence (seq2seq) models, specifically
machine translation models. We introduce algorithms that incorporate basic text
perturbation heuristics and more advanced strategies, such as the
gradient-based attack, which utilizes a differentiable approximation of the
inherently non-differentiable translation metric. Through our investigation, we
provide evidence that machine translation models display robustness displayed
robustness against best performed known adversarial attacks, as the degree of
perturbation in the output is directly proportional to the perturbation in the
input. However, among underdogs, our attacks outperform alternatives, providing
the best relative performance. Another strong candidate is an attack based on
mixing of individual characters.


http://arxiv.org/abs/2309.05075
Secure Set-Based State Estimation for Linear Systems under Adversarial Attacks on Sensors. (3%)
Muhammad Umar B. Niazi; Michelle S. Chong; Amr Alanwar; Karl H. Johansson
  When a strategic adversary can attack multiple sensors of a system and freely
choose a different set of sensors at different times, how can we ensure that
the state estimate remains uncorrupted by the attacker? The existing literature
addressing this problem mandates that the adversary can only corrupt less than
half of the total number of sensors. This limitation is fundamental to all
point-based secure state estimators because of their dependence on algorithms
that rely on majority voting among sensors. However, in reality, an adversary
with ample resources may not be limited to attacking less than half of the
total number of sensors. This paper avoids the above-mentioned fundamental
limitation by proposing a set-based approach that allows attacks on all but one
sensor at any given time. We guarantee that the true state is always contained
in the estimated set, which is represented by a collection of constrained
zonotopes, provided that the system is bounded-input-bounded-state stable and
redundantly observable via every combination of sensor subsets with size equal
to the number of uncompromised sensors. Additionally, we show that the
estimated set is secure and stable irrespective of the attack signals if the
process and measurement noises are bounded. To detect the set of attacked
sensors at each time, we propose a simple attack detection technique. However,
we acknowledge that intelligently designed stealthy attacks may not be detected
and, in the worst-case scenario, could even result in exponential growth in the
algorithm's complexity. We alleviate this shortcoming by presenting a range of
strategies that offer different levels of trade-offs between estimation
performance and complexity.


http://arxiv.org/abs/2309.04777
Towards Robust Model Watermark via Reducing Parametric Vulnerability. (8%)
Guanhao Gan; Yiming Li; Dongxian Wu; Shu-Tao Xia
  Deep neural networks are valuable assets considering their commercial
benefits and huge demands for costly annotation and computation resources. To
protect the copyright of DNNs, backdoor-based ownership verification becomes
popular recently, in which the model owner can watermark the model by embedding
a specific backdoor behavior before releasing it. The defenders (usually the
model owners) can identify whether a suspicious third-party model is ``stolen''
from them based on the presence of the behavior. Unfortunately, these
watermarks are proven to be vulnerable to removal attacks even like
fine-tuning. To further explore this vulnerability, we investigate the
parameter space and find there exist many watermark-removed models in the
vicinity of the watermarked one, which may be easily used by removal attacks.
Inspired by this finding, we propose a mini-max formulation to find these
watermark-removed models and recover their watermark behavior. Extensive
experiments demonstrate that our method improves the robustness of the model
watermarking against parametric changes and numerous watermark-removal attacks.
The codes for reproducing our main experiments are available at
\url{https://github.com/GuanhaoGan/robust-model-watermarking}.


http://arxiv.org/abs/2309.04884
RecAD: Towards A Unified Library for Recommender Attack and Defense. (1%)
Changsheng Wang; Jianbai Ye; Wenjie Wang; Chongming Gao; Fuli Feng; Xiangnan He
  In recent years, recommender systems have become a ubiquitous part of our
daily lives, while they suffer from a high risk of being attacked due to the
growing commercial and social values. Despite significant research progress in
recommender attack and defense, there is a lack of a widely-recognized
benchmarking standard in the field, leading to unfair performance comparison
and limited credibility of experiments. To address this, we propose RecAD, a
unified library aiming at establishing an open benchmark for recommender attack
and defense. RecAD takes an initial step to set up a unified benchmarking
pipeline for reproducible research by integrating diverse datasets, standard
source codes, hyper-parameter settings, running logs, attack knowledge, attack
budget, and evaluation results. The benchmark is designed to be comprehensive
and sustainable, covering both attack, defense, and evaluation tasks, enabling
more researchers to easily follow and contribute to this promising field. RecAD
will drive more solid and reproducible research on recommender systems attack
and defense, reduce the redundant efforts of researchers, and ultimately
increase the credibility and practical value of recommender attack and defense.
The project is released at https://github.com/gusye1234/recad.


http://arxiv.org/abs/2309.04650
Exploring Robust Features for Improving Adversarial Robustness. (99%)
Hong Wang; Yuefan Deng; Shinjae Yoo; Yuewei Lin
  While deep neural networks (DNNs) have revolutionized many fields, their
fragility to carefully designed adversarial attacks impedes the usage of DNNs
in safety-critical applications. In this paper, we strive to explore the robust
features which are not affected by the adversarial perturbations, i.e.,
invariant to the clean image and its adversarial examples, to improve the
model's adversarial robustness. Specifically, we propose a feature
disentanglement model to segregate the robust features from non-robust features
and domain specific features. The extensive experiments on four widely used
datasets with different attacks demonstrate that robust features obtained from
our model improve the model's adversarial robustness compared to the
state-of-the-art approaches. Moreover, the trained domain discriminator is able
to identify the domain specific features from the clean images and adversarial
examples almost perfectly. This enables adversarial example detection without
incurring additional computational costs. With that, we can also specify
different classifiers for clean images and adversarial examples, thereby
avoiding any drop in clean image accuracy.


http://arxiv.org/abs/2309.04386
ARRTOC: Adversarially Robust Real-Time Optimization and Control. (2%)
Akhil Ahmed; Rio-Chanona Ehecatl Antonio del; Mehmet Mercangoz
  Real-Time Optimization (RTO) plays a crucial role in the process operation
hierarchy by determining optimal set-points for the lower-level controllers.
However, these optimal set-points can become inoperable due to implementation
errors, such as disturbances and noise, at the control layers. To address this
challenge, in this paper, we present the Adversarially Robust Real-Time
Optimization and Control (ARRTOC) algorithm. ARRTOC draws inspiration from
adversarial machine learning, offering an online constrained Adversarially
Robust Optimization (ARO) solution applied to the RTO layer. This approach
identifies set-points that are both optimal and inherently robust to control
layer perturbations. By integrating controller design with RTO, ARRTOC enhances
overall system performance and robustness. Importantly, ARRTOC maintains
versatility through a loose coupling between the RTO and control layers,
ensuring compatibility with various controller architectures and RTO
algorithms. To validate our claims, we present three case studies: an
illustrative example, a bioreactor case study, and a multi-loop evaporator
process. Our results demonstrate the effectiveness of ARRTOC in achieving the
delicate balance between optimality and operability in RTO and control.


http://arxiv.org/abs/2309.06377
Adversarial attacks on hybrid classical-quantum Deep Learning models for Histopathological Cancer Detection. (1%)
Biswaraj Baral; Reek Majumdar; Bhavika Bhalgamiya; Taposh Dutta Roy
  We present an effective application of quantum machine learning in
histopathological cancer detection. The study here emphasizes two primary
applications of hybrid classical-quantum Deep Learning models. The first
application is to build a classification model for histopathological cancer
detection using the quantum transfer learning strategy. The second application
is to test the performance of this model for various adversarial attacks.
Rather than using a single transfer learning model, the hybrid
classical-quantum models are tested using multiple transfer learning models,
especially ResNet18, VGG-16, Inception-v3, and AlexNet as feature extractors
and integrate it with several quantum circuit-based variational quantum
circuits (VQC) with high expressibility. As a result, we provide a comparative
analysis of classical models and hybrid classical-quantum transfer learning
models for histopathological cancer detection under several adversarial
attacks. We compared the performance accuracy of the classical model with the
hybrid classical-quantum model using pennylane default quantum simulator. We
also observed that for histopathological cancer detection under several
adversarial attacks, Hybrid Classical-Quantum (HCQ) models provided better
accuracy than classical image classification models.


http://arxiv.org/abs/2309.04211
Counterfactual Explanations via Locally-guided Sequential Algorithmic Recourse. (1%)
Edward A. Small; Jeffrey N. Clark; Christopher J. McWilliams; Kacper Sokol; Jeffrey Chan; Flora D. Salim; Raul Santos-Rodriguez
  Counterfactuals operationalised through algorithmic recourse have become a
powerful tool to make artificial intelligence systems explainable.
Conceptually, given an individual classified as y -- the factual -- we seek
actions such that their prediction becomes the desired class y' -- the
counterfactual. This process offers algorithmic recourse that is (1) easy to
customise and interpret, and (2) directly aligned with the goals of each
individual. However, the properties of a "good" counterfactual are still
largely debated; it remains an open challenge to effectively locate a
counterfactual along with its corresponding recourse. Some strategies use
gradient-driven methods, but these offer no guarantees on the feasibility of
the recourse and are open to adversarial attacks on carefully created
manifolds. This can lead to unfairness and lack of robustness. Other methods
are data-driven, which mostly addresses the feasibility problem at the expense
of privacy, security and secrecy as they require access to the entire training
data set. Here, we introduce LocalFACE, a model-agnostic technique that
composes feasible and actionable counterfactual explanations using
locally-acquired information at each step of the algorithmic recourse. Our
explainer preserves the privacy of users by only leveraging data that it
specifically requires to construct actionable algorithmic recourse, and
protects the model by offering transparency solely in the regions deemed
necessary for the intervention.


http://arxiv.org/abs/2309.03665
How adversarial attacks can disrupt seemingly stable accurate classifiers. (99%)
Oliver J. Sutton; Qinghua Zhou; Ivan Y. Tyukin; Alexander N. Gorban; Alexander Bastounis; Desmond J. Higham
  Adversarial attacks dramatically change the output of an otherwise accurate
learning system using a seemingly inconsequential modification to a piece of
input data. Paradoxically, empirical evidence indicates that even systems which
are robust to large random perturbations of the input data remain susceptible
to small, easily constructed, adversarial perturbations of their inputs. Here,
we show that this may be seen as a fundamental feature of classifiers working
with high dimensional input data. We introduce a simple generic and
generalisable framework for which key behaviours observed in practical systems
arise with high probability -- notably the simultaneous susceptibility of the
(otherwise accurate) model to easily constructed adversarial attacks, and
robustness to random perturbations of the input data. We confirm that the same
phenomena are directly observed in practical neural networks trained on
standard image classification problems, where even large additive random noise
fails to trigger the adversarial instability of the network. A surprising
takeaway is that even small margins separating a classifier's decision surface
from training and testing data can hide adversarial susceptibility from being
detected using randomly sampled perturbations. Counterintuitively, using
additive noise during training or testing is therefore inefficient for
eradicating or detecting adversarial examples, and more demanding adversarial
training is required.


http://arxiv.org/abs/2309.03844
Experimental Study of Adversarial Attacks on ML-based xApps in O-RAN. (99%)
Naveen Naik Sapavath; Brian Kim; Kaushik Chowdhury; Vijay K Shah
  Open Radio Access Network (O-RAN) is considered as a major step in the
evolution of next-generation cellular networks given its support for open
interfaces and utilization of artificial intelligence (AI) into the deployment,
operation, and maintenance of RAN. However, due to the openness of the O-RAN
architecture, such AI models are inherently vulnerable to various adversarial
machine learning (ML) attacks, i.e., adversarial attacks which correspond to
slight manipulation of the input to the ML model. In this work, we showcase the
vulnerability of an example ML model used in O-RAN, and experimentally deploy
it in the near-real time (near-RT) RAN intelligent controller (RIC). Our
ML-based interference classifier xApp (extensible application in near-RT RIC)
tries to classify the type of interference to mitigate the interference effect
on the O-RAN system. We demonstrate the first-ever scenario of how such an xApp
can be impacted through an adversarial attack by manipulating the data stored
in a shared database inside the near-RT RIC. Through a rigorous performance
analysis deployed on a laboratory O-RAN testbed, we evaluate the performance in
terms of capacity and the prediction accuracy of the interference classifier
xApp using both clean and perturbed data. We show that even small adversarial
attacks can significantly decrease the accuracy of ML application in near-RT
RIC, which can directly impact the performance of the entire O-RAN deployment.


http://arxiv.org/abs/2309.03791
Adversarially Robust Deep Learning with Optimal-Transport-Regularized Divergences. (95%)
Jeremiah Birrell; Mohammadreza Ebrahimi
  We introduce the $ARMOR_D$ methods as novel approaches to enhancing the
adversarial robustness of deep learning models. These methods are based on a
new class of optimal-transport-regularized divergences, constructed via an
infimal convolution between an information divergence and an optimal-transport
(OT) cost. We use these as tools to enhance adversarial robustness by
maximizing the expected loss over a neighborhood of distributions, a technique
known as distributionally robust optimization. Viewed as a tool for
constructing adversarial samples, our method allows samples to be both
transported, according to the OT cost, and re-weighted, according to the
information divergence. We demonstrate the effectiveness of our method on
malware detection and image recognition applications and find that, to our
knowledge, it outperforms existing methods at enhancing the robustness against
adversarial attacks. $ARMOR_D$ yields the robustified accuracy of $98.29\%$
against $FGSM$ and $98.18\%$ against $PGD^{40}$ on the MNIST dataset, reducing
the error rate by more than $19.7\%$ and $37.2\%$ respectively compared to
prior methods. Similarly, in malware detection, a discrete (binary) data
domain, $ARMOR_D$ improves the robustified accuracy under $rFGSM^{50}$ attack
compared to the previous best-performing adversarial training methods by
$37.0\%$ while lowering false negative and false positive rates by $51.1\%$ and
$57.53\%$, respectively.


http://arxiv.org/abs/2309.03702
DiffDefense: Defending against Adversarial Attacks via Diffusion Models. (80%)
Hondamunige Prasanna Silva; Lorenzo Seidenari; Bimbo Alberto Del
  This paper presents a novel reconstruction method that leverages Diffusion
Models to protect machine learning classifiers against adversarial attacks, all
without requiring any modifications to the classifiers themselves. The
susceptibility of machine learning models to minor input perturbations renders
them vulnerable to adversarial attacks. While diffusion-based methods are
typically disregarded for adversarial defense due to their slow reverse
process, this paper demonstrates that our proposed method offers robustness
against adversarial threats while preserving clean accuracy, speed, and
plug-and-play compatibility. Code at:
https://github.com/HondamunigePrasannaSilva/DiffDefence.


http://arxiv.org/abs/2309.04036
One-to-Multiple Clean-Label Image Camouflage (OmClic) based Backdoor Attack on Deep Learning. (73%)
Guohong Wang; Hua Ma; Yansong Gao; Alsharif Abuadbba; Zhi Zhang; Wei Kang; Said F. Al-Sarawib; Gongxuan Zhang; Derek Abbott
  Image camouflage has been utilized to create clean-label poisoned images for
implanting backdoor into a DL model. But there exists a crucial limitation that
one attack/poisoned image can only fit a single input size of the DL model,
which greatly increases its attack budget when attacking multiple commonly
adopted input sizes of DL models. This work proposes to constructively craft an
attack image through camouflaging but can fit multiple DL models' input sizes
simultaneously, namely OmClic. Thus, through OmClic, we are able to always
implant a backdoor regardless of which common input size is chosen by the user
to train the DL model given the same attack budget (i.e., a fraction of the
poisoning rate). With our camouflaging algorithm formulated as a
multi-objective optimization, M=5 input sizes can be concurrently targeted with
one attack image, which artifact is retained to be almost visually
imperceptible at the same time. Extensive evaluations validate the proposed
OmClic can reliably succeed in various settings using diverse types of images.
Further experiments on OmClic based backdoor insertion to DL models show that
high backdoor performances (i.e., attack success rate and clean data accuracy)
are achievable no matter which common input size is randomly chosen by the user
to train the model. So that the OmClic based backdoor attack budget is reduced
by M$\times$ compared to the state-of-the-art camouflage based backdoor attack
as a baseline. Significantly, the same set of OmClic based poisonous attack
images is transferable to different model architectures for backdoor implant.


http://arxiv.org/abs/2309.03648
Promoting Fairness in GNNs: A Characterization of Stability. (1%)
Yaning Jia; Chunhui Zhang
  The Lipschitz bound, a technique from robust statistics, can limit the
maximum changes in the output concerning the input, taking into account
associated irrelevant biased factors. It is an efficient and provable method
for examining the output stability of machine learning models without incurring
additional computation costs. Recently, Graph Neural Networks (GNNs), which
operate on non-Euclidean data, have gained significant attention. However, no
previous research has investigated the GNN Lipschitz bounds to shed light on
stabilizing model outputs, especially when working on non-Euclidean data with
inherent biases. Given the inherent biases in common graph data used for GNN
training, it poses a serious challenge to constraining the GNN output
perturbations induced by input biases, thereby safeguarding fairness during
training. Recently, despite the Lipschitz constant's use in controlling the
stability of Euclideanneural networks, the calculation of the precise Lipschitz
constant remains elusive for non-Euclidean neural networks like GNNs,
especially within fairness contexts. To narrow this gap, we begin with the
general GNNs operating on an attributed graph, and formulate a Lipschitz bound
to limit the changes in the output regarding biases associated with the input.
Additionally, we theoretically analyze how the Lipschitz constant of a GNN
model could constrain the output perturbations induced by biases learned from
data for fairness training. We experimentally validate the Lipschitz bound's
effectiveness in limiting biases of the model output. Finally, from a training
dynamics perspective, we demonstrate why the theoretical Lipschitz bound can
effectively guide the GNN training to better trade-off between accuracy and
fairness.


http://arxiv.org/abs/2309.02705
Certifying LLM Safety against Adversarial Prompting. (86%)
Aounon Kumar; Chirag Agarwal; Suraj Srinivas; Soheil Feizi; Hima Lakkaraju
  Large language models (LLMs) released for public use incorporate guardrails
to ensure their output is safe, often referred to as "model alignment." An
aligned language model should decline a user's request to produce harmful
content. However, such safety measures are vulnerable to adversarial prompts,
which contain maliciously designed token sequences to circumvent the model's
safety guards and cause it to produce harmful content. In this work, we
introduce erase-and-check, the first framework to defend against adversarial
prompts with verifiable safety guarantees. We erase tokens individually and
inspect the resulting subsequences using a safety filter. Our procedure labels
the input prompt as harmful if any subsequences or the input prompt are
detected as harmful by the filter. This guarantees that any adversarial
modification of a harmful prompt up to a certain size is also labeled harmful.
We defend against three attack modes: i) adversarial suffix, which appends an
adversarial sequence at the end of the prompt; ii) adversarial insertion, where
the adversarial sequence is inserted anywhere in the middle of the prompt; and
iii) adversarial infusion, where adversarial tokens are inserted at arbitrary
positions in the prompt, not necessarily as a contiguous block. Empirical
results demonstrate that our technique obtains strong certified safety
guarantees on harmful prompts while maintaining good performance on safe
prompts. For example, against adversarial suffixes of length 20, it certifiably
detects 93% of the harmful prompts and labels 94% of the safe prompts as safe
using the open source language model Llama 2 as the safety filter.


http://arxiv.org/abs/2309.02752
SWAP: Exploiting Second-Ranked Logits for Adversarial Attacks on Time Series. (84%)
Chang George Dong; Liangwei Nathan Zheng; Weitong Chen; Wei Emma Zhang; Lin Yue
  Time series classification (TSC) has emerged as a critical task in various
domains, and deep neural models have shown superior performance in TSC tasks.
However, these models are vulnerable to adversarial attacks, where subtle
perturbations can significantly impact the prediction results. Existing
adversarial methods often suffer from over-parameterization or random logit
perturbation, hindering their effectiveness. Additionally, increasing the
attack success rate (ASR) typically involves generating more noise, making the
attack more easily detectable. To address these limitations, we propose SWAP, a
novel attacking method for TSC models. SWAP focuses on enhancing the confidence
of the second-ranked logits while minimizing the manipulation of other logits.
This is achieved by minimizing the Kullback-Leibler divergence between the
target logit distribution and the predictive logit distribution. Experimental
results demonstrate that SWAP achieves state-of-the-art performance, with an
ASR exceeding 50% and an 18% increase compared to existing methods.


http://arxiv.org/abs/2309.03437
Byzantine-Robust Federated Learning with Variance Reduction and Differential Privacy. (68%)
Zikai Zhang; Rui Hu
  Federated learning (FL) is designed to preserve data privacy during model
training, where the data remains on the client side (i.e., IoT devices), and
only model updates of clients are shared iteratively for collaborative
learning. However, this process is vulnerable to privacy attacks and Byzantine
attacks: the local model updates shared throughout the FL network will leak
private information about the local training data, and they can also be
maliciously crafted by Byzantine attackers to disturb the learning. In this
paper, we propose a new FL scheme that guarantees rigorous privacy and
simultaneously enhances system robustness against Byzantine attacks. Our
approach introduces sparsification- and momentum-driven variance reduction into
the client-level differential privacy (DP) mechanism, to defend against
Byzantine attackers. The security design does not violate the privacy guarantee
of the client-level DP mechanism; hence, our approach achieves the same
client-level DP guarantee as the state-of-the-art. We conduct extensive
experiments on both IID and non-IID datasets and different tasks and evaluate
the performance of our approach against different Byzantine attacks by
comparing it with state-of-the-art defense methods. The results of our
experiments show the efficacy of our framework and demonstrate its ability to
improve system robustness against Byzantine attacks while achieving a strong
privacy guarantee.


http://arxiv.org/abs/2309.03164
J-Guard: Journalism Guided Adversarially Robust Detection of AI-generated News. (38%)
Tharindu Kumarage; Amrita Bhattacharjee; Djordje Padejski; Kristy Roschke; Dan Gillmor; Scott Ruston; Huan Liu; Joshua Garland
  The rapid proliferation of AI-generated text online is profoundly reshaping
the information landscape. Among various types of AI-generated text,
AI-generated news presents a significant threat as it can be a prominent source
of misinformation online. While several recent efforts have focused on
detecting AI-generated text in general, these methods require enhanced
reliability, given concerns about their vulnerability to simple adversarial
attacks. Furthermore, due to the eccentricities of news writing, applying these
detection methods for AI-generated news can produce false positives,
potentially damaging the reputation of news organizations. To address these
challenges, we leverage the expertise of an interdisciplinary team to develop a
framework, J-Guard, capable of steering existing supervised AI text detectors
for detecting AI-generated news while boosting adversarial robustness. By
incorporating stylistic cues inspired by the unique journalistic attributes,
J-Guard effectively distinguishes between real-world journalism and
AI-generated news articles. Our experiments on news articles generated by a
vast array of AI models, including ChatGPT (GPT3.5), demonstrate the
effectiveness of J-Guard in enhancing detection capabilities while maintaining
an average performance decrease of as low as 7% when faced with adversarial
attacks.


http://arxiv.org/abs/2309.03466
MIRA: Cracking Black-box Watermarking on Deep Neural Networks via Model Inversion-based Removal Attacks. (22%)
Yifan Lu; Wenxuan Li; Mi Zhang; Xudong Pan; Min Yang
  To protect the intellectual property of well-trained deep neural networks
(DNNs), black-box DNN watermarks, which are embedded into the prediction
behavior of DNN models on a set of specially-crafted samples, have gained
increasing popularity in both academy and industry. Watermark robustness is
usually implemented against attackers who steal the protected model and
obfuscate its parameters for watermark removal. Recent studies empirically
prove the robustness of most black-box watermarking schemes against known
removal attempts.
  In this paper, we propose a novel Model Inversion-based Removal Attack
(\textsc{Mira}), which is watermark-agnostic and effective against most of
mainstream black-box DNN watermarking schemes. In general, our attack pipeline
exploits the internals of the protected model to recover and unlearn the
watermark message. We further design target class detection and recovered
sample splitting algorithms to reduce the utility loss caused by \textsc{Mira}
and achieve data-free watermark removal on half of the watermarking schemes. We
conduct comprehensive evaluation of \textsc{Mira} against ten mainstream
black-box watermarks on three benchmark datasets and DNN architectures.
Compared with six baseline removal attacks, \textsc{Mira} achieves strong
watermark removal effects on the covered watermarks, preserving at least $90\%$
of the stolen model utility, under more relaxed or even no assumptions on the
dataset availability.


http://arxiv.org/abs/2309.03198
My Art My Choice: Adversarial Protection Against Unruly AI. (2%)
Anthony Rhodes; Ram Bhagat; Umur Aybars Ciftci; Ilke Demir
  Generative AI is on the rise, enabling everyone to produce realistic content
via publicly available interfaces. Especially for guided image generation,
diffusion models are changing the creator economy by producing high quality low
cost content. In parallel, artists are rising against unruly AI, since their
artwork are leveraged, distributed, and dissimulated by large generative
models. Our approach, My Art My Choice (MAMC), aims to empower content owners
by protecting their copyrighted materials from being utilized by diffusion
models in an adversarial fashion. MAMC learns to generate adversarially
perturbed "protected" versions of images which can in turn "break" diffusion
models. The perturbation amount is decided by the artist to balance distortion
vs. protection of the content. MAMC is designed with a simple UNet-based
generator, attacking black box diffusion models, combining several losses to
create adversarial twins of the original artwork. We experiment on three
datasets for various image-to-image tasks, with different user control values.
Both protected image and diffusion output results are evaluated in visual,
noise, structure, pixel, and generative spaces to validate our claims. We
believe that MAMC is a crucial step for preserving ownership information for AI
generated content in a flawless, based-on-need, and human-centric way.


http://arxiv.org/abs/2310.10656
VeriDIP: Verifying Ownership of Deep Neural Networks through Privacy Leakage Fingerprints. (1%)
Aoting Hu; Zhigang Lu; Renjie Xie; Minhui Xue
  Deploying Machine Learning as a Service gives rise to model plagiarism,
leading to copyright infringement. Ownership testing techniques are designed to
identify model fingerprints for verifying plagiarism. However, previous works
often rely on overfitting or robustness features as fingerprints, lacking
theoretical guarantees and exhibiting under-performance on generalized models.
In this paper, we propose a novel ownership testing method called VeriDIP,
which verifies a DNN model's intellectual property. VeriDIP makes two major
contributions. (1) It utilizes membership inference attacks to estimate the
lower bound of privacy leakage, which reflects the fingerprint of a given
model. The privacy leakage fingerprints highlight the unique patterns through
which the models memorize sensitive training datasets. (2) We introduce a novel
approach using less private samples to enhance the performance of ownership
testing.
  Extensive experimental results confirm that VeriDIP is effective and
efficient in validating the ownership of deep learning models trained on both
image and tabular datasets. VeriDIP achieves comparable performance to
state-of-the-art methods on image datasets while significantly reducing
computation and communication costs. Enhanced VeriDIP demonstrates superior
verification performance on generalized deep learning models, particularly on
table-trained models. Additionally, VeriDIP exhibits similar effectiveness on
utility-preserving differentially private models compared to non-differentially
private baselines.


http://arxiv.org/abs/2309.03004
A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness. (1%)
Ze Peng; Lei Qi; Yinghuan Shi; Yang Gao
  A recent empirical observation (Li et al., 2022b) of activation sparsity in
MLP blocks offers an opportunity to drastically reduce computation costs for
free. Although having attributed it to training dynamics, existing theoretical
explanations of activation sparsity are restricted to shallow networks, small
training steps and special training, despite its emergence in deep models
standardly trained for a large number of steps. To fill these gaps, we propose
the notion of gradient sparsity as one source of activation sparsity and a
theoretical explanation based on it that sees sparsity a necessary step to
adversarial robustness w.r.t. hidden features and parameters, which is
approximately the flatness of minima for well-learned models. The theory
applies to standardly trained LayerNorm-ed MLPs, and further to Transformers or
other architectures trained with weight noises. Eliminating other sources of
flatness except for sparsity, we discover the phenomenon that the ratio between
the largest and smallest non-zero singular values of weight matrices is small.
When discussing the emergence of this spectral concentration, we use random
matrix theory (RMT) as a powerful tool to analyze stochastic gradient noises.
Validational experiments are conducted to verify our gradient-sparsity-based
explanation. We propose two plug-and-play modules for both training and
finetuning for sparsity. Experiments on ImageNet-1k and C4 demonstrate their
50% sparsity improvements, indicating further potential cost reduction in both
training and inference.


http://arxiv.org/abs/2309.02159
The Adversarial Implications of Variable-Time Inference. (99%)
Dudi Biton; Aditi Misra; Efrat Levy; Jaidip Kotak; Ron Bitton; Roei Schuster; Nicolas Papernot; Yuval Elovici; Ben Nassi
  Machine learning (ML) models are known to be vulnerable to a number of
attacks that target the integrity of their predictions or the privacy of their
training data. To carry out these attacks, a black-box adversary must typically
possess the ability to query the model and observe its outputs (e.g., labels).
In this work, we demonstrate, for the first time, the ability to enhance such
decision-based attacks. To accomplish this, we present an approach that
exploits a novel side channel in which the adversary simply measures the
execution time of the algorithm used to post-process the predictions of the ML
model under attack. The leakage of inference-state elements into algorithmic
timing side channels has never been studied before, and we have found that it
can contain rich information that facilitates superior timing attacks that
significantly outperform attacks based solely on label outputs. In a case
study, we investigate leakage from the non-maximum suppression (NMS) algorithm,
which plays a crucial role in the operation of object detectors. In our
examination of the timing side-channel vulnerabilities associated with this
algorithm, we identified the potential to enhance decision-based attacks. We
demonstrate attacks against the YOLOv3 detector, leveraging the timing leakage
to successfully evade object detection using adversarial examples, and perform
dataset inference. Our experiments show that our adversarial examples exhibit
superior perturbation quality compared to a decision-based attack. In addition,
we present a new threat model in which dataset inference based solely on timing
leakage is performed. To address the timing leakage vulnerability inherent in
the NMS algorithm, we explore the potential and limitations of implementing
constant-time inference passes as a mitigation strategy.


http://arxiv.org/abs/2309.02528
Adaptive Adversarial Training Does Not Increase Recourse Costs. (92%)
Ian Hardy; Jayanth Yetukuri; Yang Liu
  Recent work has connected adversarial attack methods and algorithmic recourse
methods: both seek minimal changes to an input instance which alter a model's
classification decision. It has been shown that traditional adversarial
training, which seeks to minimize a classifier's susceptibility to malicious
perturbations, increases the cost of generated recourse; with larger
adversarial training radii correlating with higher recourse costs. From the
perspective of algorithmic recourse, however, the appropriate adversarial
training radius has always been unknown. Another recent line of work has
motivated adversarial training with adaptive training radii to address the
issue of instance-wise variable adversarial vulnerability, showing success in
domains with unknown attack radii. This work studies the effects of adaptive
adversarial training on algorithmic recourse costs. We establish that the
improvements in model robustness induced by adaptive adversarial training show
little effect on algorithmic recourse costs, providing a potential avenue for
affordable robustness in domains where recoursability is critical.


http://arxiv.org/abs/2309.02396
Black-Box Attacks against Signed Graph Analysis via Balance Poisoning. (87%)
Jialong Zhou; Yuni Lai; Jian Ren; Kai Zhou
  Signed graphs are well-suited for modeling social networks as they capture
both positive and negative relationships. Signed graph neural networks (SGNNs)
are commonly employed to predict link signs (i.e., positive and negative) in
such graphs due to their ability to handle the unique structure of signed
graphs. However, real-world signed graphs are vulnerable to malicious attacks
by manipulating edge relationships, and existing adversarial graph attack
methods do not consider the specific structure of signed graphs. SGNNs often
incorporate balance theory to effectively model the positive and negative
links. Surprisingly, we find that the balance theory that they rely on can
ironically be exploited as a black-box attack. In this paper, we propose a
novel black-box attack called balance-attack that aims to decrease the balance
degree of the signed graphs. We present an efficient heuristic algorithm to
solve this NP-hard optimization problem. We conduct extensive experiments on
five popular SGNN models and four real-world datasets to demonstrate the
effectiveness and wide applicability of our proposed attack method. By
addressing these challenges, our research contributes to a better understanding
of the limitations and resilience of robust models when facing attacks on
SGNNs. This work contributes to enhancing the security and reliability of
signed graph analysis in social network modeling. Our PyTorch implementation of
the attack is publicly available on GitHub:
https://github.com/JialongZhou666/Balance-Attack.git.


http://arxiv.org/abs/2310.06845
RobustEdge: Low Power Adversarial Detection for Cloud-Edge Systems. (83%)
Abhishek Moitra; Abhiroop Bhattacharjee; Youngeun Kim; Priyadarshini Panda
  In practical cloud-edge scenarios, where a resource constrained edge performs
data acquisition and a cloud system (having sufficient resources) performs
inference tasks with a deep neural network (DNN), adversarial robustness is
critical for reliability and ubiquitous deployment. Adversarial detection is a
prime adversarial defence technique used in prior literature. However, in prior
detection works, the detector is attached to the classifier model and both
detector and classifier work in tandem to perform adversarial detection that
requires a high computational overhead which is not available at the low-power
edge. Therefore, prior works can only perform adversarial detection at the
cloud and not at the edge. This means that in case of adversarial attacks, the
unfavourable adversarial samples must be communicated to the cloud which leads
to energy wastage at the edge device. Therefore, a low-power edge-friendly
adversarial detection method is required to improve the energy efficiency of
the edge and robustness of the cloud-based classifier. To this end, RobustEdge
proposes Quantization-enabled Energy Separation (QES) training with "early
detection and exit" to perform edge-based low cost adversarial detection. The
QES-trained detector implemented at the edge blocks adversarial data
transmission to the classifier model, thereby improving adversarial robustness
and energy-efficiency of the Cloud-Edge system.


http://arxiv.org/abs/2309.02429
Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach. (4%)
Vimal K B; Saketh Bachu; Tanmay Garg; Niveditha Lakshmi Narasimhan; Raghavan Konuru; Vineeth N Balasubramanian
  Estimating the transferability of publicly available pretrained models to a
target task has assumed an important place for transfer learning tasks in
recent years. Existing efforts propose metrics that allow a user to choose one
model from a pool of pre-trained models without having to fine-tune each model
individually and identify one explicitly. With the growth in the number of
available pre-trained models and the popularity of model ensembles, it also
becomes essential to study the transferability of multiple-source models for a
given target task. The few existing efforts study transferability in such
multi-source ensemble settings using just the outputs of the classification
layer and neglect possible domain or task mismatch. Moreover, they overlook the
most important factor while selecting the source models, viz., the cohesiveness
factor between them, which can impact the performance and confidence in the
prediction of the ensemble. To address these gaps, we propose a novel Optimal
tranSport-based suBmOdular tRaNsferability metric (OSBORN) to estimate the
transferability of an ensemble of models to a downstream task. OSBORN
collectively accounts for image domain difference, task difference, and
cohesiveness of models in the ensemble to provide reliable estimates of
transferability. We gauge the performance of OSBORN on both image
classification and semantic segmentation tasks. Our setup includes 28 source
datasets, 11 target datasets, 5 model architectures, and 2 pre-training
methods. We benchmark our method against current state-of-the-art metrics
MS-LEEP and E-LEEP, and outperform them consistently using the proposed
approach.


http://arxiv.org/abs/2309.02057
Robust Recommender System: A Survey and Future Directions. (2%)
Kaike Zhang; Qi Cao; Fei Sun; Yunfan Wu; Shuchang Tao; Huawei Shen; Xueqi Cheng
  With the rapid growth of information, recommender systems have become
integral for providing personalized suggestions and overcoming information
overload. However, their practical deployment often encounters "dirty" data,
where noise or malicious information can lead to abnormal recommendations.
Research on improving recommender systems' robustness against such dirty data
has thus gained significant attention. This survey provides a comprehensive
review of recent work on recommender systems' robustness. We first present a
taxonomy to organize current techniques for withstanding malicious attacks and
natural noise. We then explore state-of-the-art methods in each category,
including fraudster detection, adversarial training, certifiable robust
training against malicious attacks, and regularization, purification,
self-supervised learning against natural noise. Additionally, we summarize
evaluation metrics and common datasets used to assess robustness. We discuss
robustness across varying recommendation scenarios and its interplay with other
properties like accuracy, interpretability, privacy, and fairness. Finally, we
delve into open issues and future research directions in this emerging field.
Our goal is to equip readers with a holistic understanding of robust
recommender systems and spotlight pathways for future research and development.


http://arxiv.org/abs/2309.02088
Dual Adversarial Alignment for Realistic Support-Query Shift Few-shot Learning. (1%)
Siyang Jiang; Rui Fang; Hsi-Wen Chen; Wei Ding; Ming-Syan Chen
  Support-query shift few-shot learning aims to classify unseen examples (query
set) to labeled data (support set) based on the learned embedding in a
low-dimensional space under a distribution shift between the support set and
the query set. However, in real-world scenarios the shifts are usually unknown
and varied, making it difficult to estimate in advance. Therefore, in this
paper, we propose a novel but more difficult challenge, RSQS, focusing on
Realistic Support-Query Shift few-shot learning. The key feature of RSQS is
that the individual samples in a meta-task are subjected to multiple
distribution shifts in each meta-task. In addition, we propose a unified
adversarial feature alignment method called DUal adversarial ALignment
framework (DuaL) to relieve RSQS from two aspects, i.e., inter-domain bias and
intra-domain variance. On the one hand, for the inter-domain bias, we corrupt
the original data in advance and use the synthesized perturbed inputs to train
the repairer network by minimizing distance in the feature level. On the other
hand, for intra-domain variance, we proposed a generator network to synthesize
hard, i.e., less similar, examples from the support set in a self-supervised
manner and introduce regularized optimal transportation to derive a smooth
optimal transportation plan. Lastly, a benchmark of RSQS is built with several
state-of-the-art baselines among three datasets (CIFAR100, mini-ImageNet, and
Tiered-Imagenet). Experiment results show that DuaL significantly outperforms
the state-of-the-art methods in our benchmark.


http://arxiv.org/abs/2309.01620
Hindering Adversarial Attacks with Multiple Encrypted Patch Embeddings. (99%)
AprilPyone MaungMaung; Isao Echizen; Hitoshi Kiya
  In this paper, we propose a new key-based defense focusing on both efficiency
and robustness. Although the previous key-based defense seems effective in
defending against adversarial examples, carefully designed adaptive attacks can
bypass the previous defense, and it is difficult to train the previous defense
on large datasets like ImageNet. We build upon the previous defense with two
major improvements: (1) efficient training and (2) optional randomization. The
proposed defense utilizes one or more secret patch embeddings and classifier
heads with a pre-trained isotropic network. When more than one secret
embeddings are used, the proposed defense enables randomization on inference.
Experiments were carried out on the ImageNet dataset, and the proposed defense
was evaluated against an arsenal of state-of-the-art attacks, including
adaptive ones. The results show that the proposed defense achieves a high
robust accuracy and a comparable clean accuracy compared to the previous
key-based defense.


http://arxiv.org/abs/2309.01582
Improving Visual Quality and Transferability of Adversarial Attacks on Face Recognition Simultaneously with Adversarial Restoration. (99%)
Fengfan Zhou; Hefei Ling; Yuxuan Shi; Jiazhong Chen; Ping Li
  Adversarial face examples possess two critical properties: Visual Quality and
Transferability. However, existing approaches rarely address these properties
simultaneously, leading to subpar results. To address this issue, we propose a
novel adversarial attack technique known as Adversarial Restoration
(AdvRestore), which enhances both visual quality and transferability of
adversarial face examples by leveraging a face restoration prior. In our
approach, we initially train a Restoration Latent Diffusion Model (RLDM)
designed for face restoration. Subsequently, we employ the inference process of
RLDM to generate adversarial face examples. The adversarial perturbations are
applied to the intermediate features of RLDM. Additionally, by treating RLDM
face restoration as a sibling task, the transferability of the generated
adversarial face examples is further improved. Our experimental results
validate the effectiveness of the proposed attack method.


http://arxiv.org/abs/2309.01351
Adv3D: Generating 3D Adversarial Examples in Driving Scenarios with NeRF. (99%)
Leheng Li; Qing Lian; Ying-Cong Chen
  Deep neural networks (DNNs) have been proven extremely susceptible to
adversarial examples, which raises special safety-critical concerns for
DNN-based autonomous driving stacks (i.e., 3D object detection). Although there
are extensive works on image-level attacks, most are restricted to 2D pixel
spaces, and such attacks are not always physically realistic in our 3D world.
Here we present Adv3D, the first exploration of modeling adversarial examples
as Neural Radiance Fields (NeRFs). Advances in NeRF provide photorealistic
appearances and 3D accurate generation, yielding a more realistic and
realizable adversarial example. We train our adversarial NeRF by minimizing the
surrounding objects' confidence predicted by 3D detectors on the training set.
Then we evaluate Adv3D on the unseen validation set and show that it can cause
a large performance reduction when rendering NeRF in any sampled pose. To
generate physically realizable adversarial examples, we propose primitive-aware
sampling and semantic-guided regularization that enable 3D patch attacks with
camouflage adversarial texture. Experimental results demonstrate that the
trained adversarial NeRF generalizes well to different poses, scenes, and 3D
detectors. Finally, we provide a defense method to our attacks that involves
adversarial training through data augmentation. Project page:
https://len-li.github.io/adv3d-web


http://arxiv.org/abs/2309.01452
Toward Defensive Letter Design. (98%)
Rentaro Kataoka; Akisato Kimura; Seiichi Uchida
  A major approach for defending against adversarial attacks aims at
controlling only image classifiers to be more resilient, and it does not care
about visual objects, such as pandas and cars, in images. This means that
visual objects themselves cannot take any defensive actions, and they are still
vulnerable to adversarial attacks. In contrast, letters are artificial symbols,
and we can freely control their appearance unless losing their readability. In
other words, we can make the letters more defensive to the attacks. This paper
poses three research questions related to the adversarial vulnerability of
letter images: (1) How defensive are the letters against adversarial attacks?
(2) Can we estimate how defensive a given letter image is before attacks? (3)
Can we control the letter images to be more defensive against adversarial
attacks? For answering the first and second questions, we measure the
defensibility of letters by employing Iterative Fast Gradient Sign Method
(I-FGSM) and then build a deep regression model for estimating the
defensibility of each letter image. We also propose a two-step method based on
a generative adversarial network (GAN) for generating character images with
higher defensibility, which solves the third research question.


http://arxiv.org/abs/2309.01686
MathAttack: Attacking Large Language Models Towards Math Solving Ability. (97%)
Zihao Zhou; Qiufeng Wang; Mingyu Jin; Jie Yao; Jianan Ye; Wei Liu; Wei Wang; Xiaowei Huang; Kaizhu Huang
  With the boom of Large Language Models (LLMs), the research of solving Math
Word Problem (MWP) has recently made great progress. However, there are few
studies to examine the security of LLMs in math solving ability. Instead of
attacking prompts in the use of LLMs, we propose a MathAttack model to attack
MWP samples which are closer to the essence of security in solving math
problems. Compared to traditional text adversarial attack, it is essential to
preserve the mathematical logic of original MWPs during the attacking. To this
end, we propose logical entity recognition to identify logical entries which
are then frozen. Subsequently, the remaining text are attacked by adopting a
word-level attacker. Furthermore, we propose a new dataset RobustMath to
evaluate the robustness of LLMs in math solving ability. Extensive experiments
on our RobustMath and two another math benchmark datasets GSM8K and MultiAirth
show that MathAttack could effectively attack the math solving ability of LLMs.
In the experiments, we observe that (1) Our adversarial samples from
higher-accuracy LLMs are also effective for attacking LLMs with lower accuracy
(e.g., transfer from larger to smaller-size LLMs, or from few-shot to zero-shot
prompts); (2) Complex MWPs (such as more solving steps, longer text, more
numbers) are more vulnerable to attack; (3) We can improve the robustness of
LLMs by using our adversarial samples in few-shot prompts. Finally, we hope our
practice and observation can serve as an important attempt towards enhancing
the robustness of LLMs in math solving ability. We will release our code and
dataset.


http://arxiv.org/abs/2309.01838
Efficient Defense Against Model Stealing Attacks on Convolutional Neural Networks. (93%)
Kacem Khaled; Mouna Dhaouadi; Magalhães Felipe Gohring de; Gabriela Nicolescu
  Model stealing attacks have become a serious concern for deep learning
models, where an attacker can steal a trained model by querying its black-box
API. This can lead to intellectual property theft and other security and
privacy risks. The current state-of-the-art defenses against model stealing
attacks suggest adding perturbations to the prediction probabilities. However,
they suffer from heavy computations and make impracticable assumptions about
the adversary. They often require the training of auxiliary models. This can be
time-consuming and resource-intensive which hinders the deployment of these
defenses in real-world applications. In this paper, we propose a simple yet
effective and efficient defense alternative. We introduce a heuristic approach
to perturb the output probabilities. The proposed defense can be easily
integrated into models without additional training. We show that our defense is
effective in defending against three state-of-the-art stealing attacks. We
evaluate our approach on large and quantized (i.e., compressed) Convolutional
Neural Networks (CNNs) trained on several vision datasets. Our technique
outperforms the state-of-the-art defenses with a $\times37$ faster inference
latency without requiring any additional model and with a low impact on the
model's performance. We validate that our defense is also effective for
quantized CNNs targeting edge devices.


http://arxiv.org/abs/2309.01866
Efficient Query-Based Attack against ML-Based Android Malware Detection under Zero Knowledge Setting. (92%)
Ping He; Yifan Xia; Xuhong Zhang; Shouling Ji
  The widespread adoption of the Android operating system has made malicious
Android applications an appealing target for attackers. Machine learning-based
(ML-based) Android malware detection (AMD) methods are crucial in addressing
this problem; however, their vulnerability to adversarial examples raises
concerns. Current attacks against ML-based AMD methods demonstrate remarkable
performance but rely on strong assumptions that may not be realistic in
real-world scenarios, e.g., the knowledge requirements about feature space,
model parameters, and training dataset. To address this limitation, we
introduce AdvDroidZero, an efficient query-based attack framework against
ML-based AMD methods that operates under the zero knowledge setting. Our
extensive evaluation shows that AdvDroidZero is effective against various
mainstream ML-based AMD methods, in particular, state-of-the-art such methods
and real-world antivirus solutions.


http://arxiv.org/abs/2309.01786
Safe and Robust Watermark Injection with a Single OoD Image. (8%)
Shuyang Yu; Junyuan Hong; Haobo Zhang; Haotao Wang; Zhangyang Wang; Jiayu Zhou
  Training a high-performance deep neural network requires large amounts of
data and computational resources. Protecting the intellectual property (IP) and
commercial ownership of a deep model is challenging yet increasingly crucial. A
major stream of watermarking strategies implants verifiable backdoor triggers
by poisoning training samples, but these are often unrealistic due to data
privacy and safety concerns and are vulnerable to minor model changes such as
fine-tuning. To overcome these challenges, we propose a safe and robust
backdoor-based watermark injection technique that leverages the diverse
knowledge from a single out-of-distribution (OoD) image, which serves as a
secret key for IP verification. The independence of training data makes it
agnostic to third-party promises of IP security. We induce robustness via
random perturbation of model parameters during watermark injection to defend
against common watermark removal attacks, including fine-tuning, pruning, and
model extraction. Our experimental results demonstrate that the proposed
watermarking approach is not only time- and sample-efficient without training
data, but also robust against the watermark removal attacks above.


http://arxiv.org/abs/2309.01614
Dropout Attacks. (2%)
Andrew Yuan; Alina Oprea; Cheng Tan
  Dropout is a common operator in deep learning, aiming to prevent overfitting
by randomly dropping neurons during training. This paper introduces a new
family of poisoning attacks against neural networks named DROPOUTATTACK.
DROPOUTATTACK attacks the dropout operator by manipulating the selection of
neurons to drop instead of selecting them uniformly at random. We design,
implement, and evaluate four DROPOUTATTACK variants that cover a broad range of
scenarios. These attacks can slow or stop training, destroy prediction accuracy
of target classes, and sabotage either precision or recall of a target class.
In our experiments of training a VGG-16 model on CIFAR-100, our attack can
reduce the precision of the victim class by 34.6% (from 81.7% to 47.1%) without
incurring any degradation in model accuracy


http://arxiv.org/abs/2309.01850
Uncertainty in AI: Evaluating Deep Neural Networks on Out-of-Distribution Images. (2%)
Jamiu Idowu; Ahmed Almasoud
  As AI models are increasingly deployed in critical applications, ensuring the
consistent performance of models when exposed to unusual situations such as
out-of-distribution (OOD) or perturbed data, is important. Therefore, this
paper investigates the uncertainty of various deep neural networks, including
ResNet-50, VGG16, DenseNet121, AlexNet, and GoogleNet, when dealing with such
data. Our approach includes three experiments. First, we used the pretrained
models to classify OOD images generated via DALL-E to assess their performance.
Second, we built an ensemble from the models' predictions using probabilistic
averaging for consensus due to its advantages over plurality or majority
voting. The ensemble's uncertainty was quantified using average probabilities,
variance, and entropy metrics. Our results showed that while ResNet-50 was the
most accurate single model for OOD images, the ensemble performed even better,
correctly classifying all images. Third, we tested model robustness by adding
perturbations (filters, rotations, etc.) to new epistemic images from DALL-E or
real-world captures. ResNet-50 was chosen for this being the best performing
model. While it classified 4 out of 5 unperturbed images correctly, it
misclassified all of them post-perturbation, indicating a significant
vulnerability. These misclassifications, which are clear to human observers,
highlight AI models' limitations. Using saliency maps, we identified regions of
the images that the model considered important for their decisions.


http://arxiv.org/abs/2310.05947
Robust and Efficient Interference Neural Networks for Defending Against Adversarial Attacks in ImageNet. (99%)
Yunuo Xiong; Shujuan Liu; Hongwei Xiong
  The existence of adversarial images has seriously affected the task of image
recognition and practical application of deep learning, it is also a key
scientific problem that deep learning urgently needs to solve. By far the most
effective approach is to train the neural network with a large number of
adversarial examples. However, this adversarial training method requires a huge
amount of computing resources when applied to ImageNet, and has not yet
achieved satisfactory results for high-intensity adversarial attacks. In this
paper, we construct an interference neural network by applying additional
background images and corresponding labels, and use pre-trained ResNet-152 to
efficiently complete the training. Compared with the state-of-the-art results
under the PGD attack, it has a better defense effect with much smaller
computing resources. This work provides new ideas for academic research and
practical applications of effective defense against adversarial attacks.


http://arxiv.org/abs/2309.01104
Turn Fake into Real: Adversarial Head Turn Attacks Against Deepfake Detection. (98%)
Weijie Wang; Zhengyu Zhao; Nicu Sebe; Bruno Lepri
  Malicious use of deepfakes leads to serious public concerns and reduces
people's trust in digital media. Although effective deepfake detectors have
been proposed, they are substantially vulnerable to adversarial attacks. To
evaluate the detector's robustness, recent studies have explored various
attacks. However, all existing attacks are limited to 2D image perturbations,
which are hard to translate into real-world facial changes. In this paper, we
propose adversarial head turn (AdvHeat), the first attempt at 3D adversarial
face views against deepfake detectors, based on face view synthesis from a
single-view fake image. Extensive experiments validate the vulnerability of
various detectors to AdvHeat in realistic, black-box scenarios. For example,
AdvHeat based on a simple random search yields a high attack success rate of
96.8% with 360 searching steps. When additional query access is allowed, we can
further reduce the step budget to 50. Additional analyses demonstrate that
AdvHeat is better than conventional attacks on both the cross-detector
transferability and robustness to defenses. The adversarial images generated by
AdvHeat are also shown to have natural looks. Our code, including that for
generating a multi-view dataset consisting of 360 synthetic views for each of
1000 IDs from FaceForensics++, is available at
https://github.com/twowwj/AdvHeaT.


http://arxiv.org/abs/2309.01106
AdvMono3D: Advanced Monocular 3D Object Detection with Depth-Aware Robust Adversarial Training. (98%)
Xingyuan Li; Jinyuan Liu; Long Ma; Xin Fan; Risheng Liu
  Monocular 3D object detection plays a pivotal role in the field of autonomous
driving and numerous deep learning-based methods have made significant
breakthroughs in this area. Despite the advancements in detection accuracy and
efficiency, these models tend to fail when faced with such attacks, rendering
them ineffective. Therefore, bolstering the adversarial robustness of 3D
detection models has become a crucial issue that demands immediate attention
and innovative solutions. To mitigate this issue, we propose a depth-aware
robust adversarial training method for monocular 3D object detection, dubbed
DART3D. Specifically, we first design an adversarial attack that iteratively
degrades the 2D and 3D perception capabilities of 3D object detection
models(IDP), serves as the foundation for our subsequent defense mechanism. In
response to this attack, we propose an uncertainty-based residual learning
method for adversarial training. Our adversarial training approach capitalizes
on the inherent uncertainty, enabling the model to significantly improve its
robustness against adversarial attacks. We conducted extensive experiments on
the KITTI 3D datasets, demonstrating that DART3D surpasses direct adversarial
training (the most popular approach) under attacks in 3D object detection
$AP_{R40}$ of car category for the Easy, Moderate, and Hard settings, with
improvements of 4.415%, 4.112%, and 3.195%, respectively.


http://arxiv.org/abs/2309.01077
Robust Adversarial Defense by Tensor Factorization. (89%)
Manish Bhattarai; Mehmet Cagri Kaymak; Ryan Barron; Ben Nebgen; Kim Rasmussen; Boian Alexandrov
  As machine learning techniques become increasingly prevalent in data
analysis, the threat of adversarial attacks has surged, necessitating robust
defense mechanisms. Among these defenses, methods exploiting low-rank
approximations for input data preprocessing and neural network (NN) parameter
factorization have shown potential. Our work advances this field further by
integrating the tensorization of input data with low-rank decomposition and
tensorization of NN parameters to enhance adversarial defense. The proposed
approach demonstrates significant defense capabilities, maintaining robust
accuracy even when subjected to the strongest known auto-attacks. Evaluations
against leading-edge robust performance benchmarks reveal that our results not
only hold their ground against the best defensive methods available but also
exceed all current defense strategies that rely on tensor factorizations. This
study underscores the potential of integrating tensorization and low-rank
decomposition as a robust defense against adversarial attacks in machine
learning.


http://arxiv.org/abs/2309.01102
Dual Adversarial Resilience for Collaborating Robust Underwater Image Enhancement and Perception. (13%)
Zengxi Zhang; Zhiying Jiang; Zeru Shi; Jinyuan Liu; Risheng Liu
  Due to the uneven scattering and absorption of different light wavelengths in
aquatic environments, underwater images suffer from low visibility and clear
color deviations. With the advancement of autonomous underwater vehicles,
extensive research has been conducted on learning-based underwater enhancement
algorithms. These works can generate visually pleasing enhanced images and
mitigate the adverse effects of degraded images on subsequent perception tasks.
However, learning-based methods are susceptible to the inherent fragility of
adversarial attacks, causing significant disruption in results. In this work,
we introduce a collaborative adversarial resilience network, dubbed CARNet, for
underwater image enhancement and subsequent detection tasks. Concretely, we
first introduce an invertible network with strong perturbation-perceptual
abilities to isolate attacks from underwater images, preventing interference
with image enhancement and perceptual tasks. Furthermore, we propose a
synchronized attack training strategy with both visual-driven and
perception-driven attacks enabling the network to discern and remove various
types of attacks. Additionally, we incorporate an attack pattern discriminator
to heighten the robustness of the network against different attacks. Extensive
experiments demonstrate that the proposed method outputs visually appealing
enhancement images and perform averagely 6.71% higher detection mAP than
state-of-the-art methods.


http://arxiv.org/abs/2309.00879
Towards Certified Probabilistic Robustness with High Accuracy. (98%)
Ruihan Zhang; Peixin Zhang; Jun Sun
  Adversarial examples pose a security threat to many critical systems built on
neural networks (such as face recognition systems, and self-driving cars).
While many methods have been proposed to build robust models, how to build
certifiably robust yet accurate neural network models remains an open problem.
For example, adversarial training improves empirical robustness, but they do
not provide certification of the model's robustness. On the other hand,
certified training provides certified robustness but at the cost of a
significant accuracy drop. In this work, we propose a novel approach that aims
to achieve both high accuracy and certified probabilistic robustness. Our
method has two parts, i.e., a probabilistic robust training method with an
additional goal of minimizing variance in terms of divergence and a runtime
inference method for certified probabilistic robustness of the prediction. The
latter enables efficient certification of the model's probabilistic robustness
at runtime with statistical guarantees. This is supported by our training
objective, which minimizes the variance of the model's predictions in a given
vicinity, derived from a general definition of model robustness. Our approach
works for a variety of perturbations and is reasonably efficient. Our
experiments on multiple models trained on different datasets demonstrate that
our approach significantly outperforms existing approaches in terms of both
certification rate and accuracy.


http://arxiv.org/abs/2309.00929
Timbre-reserved Adversarial Attack in Speaker Identification. (98%)
Qing Wang; Jixun Yao; Li Zhang; Pengcheng Guo; Lei Xie
  As a type of biometric identification, a speaker identification (SID) system
is confronted with various kinds of attacks. The spoofing attacks typically
imitate the timbre of the target speakers, while the adversarial attacks
confuse the SID system by adding a well-designed adversarial perturbation to an
arbitrary speech. Although the spoofing attack copies a similar timbre as the
victim, it does not exploit the vulnerability of the SID model and may not make
the SID system give the attacker's desired decision. As for the adversarial
attack, despite the SID system can be led to a designated decision, it cannot
meet the specified text or speaker timbre requirements for the specific attack
scenarios. In this study, to make the attack in SID not only leverage the
vulnerability of the SID model but also reserve the timbre of the target
speaker, we propose a timbre-reserved adversarial attack in the speaker
identification. We generate the timbre-reserved adversarial audios by adding an
adversarial constraint during the different training stages of the voice
conversion (VC) model. Specifically, the adversarial constraint is using the
target speaker label to optimize the adversarial perturbation added to the VC
model representations and is implemented by a speaker classifier joining in the
VC model training. The adversarial constraint can help to control the VC model
to generate the speaker-wised audio. Eventually, the inference of the VC model
is the ideal adversarial fake audio, which is timbre-reserved and can fool the
SID system.


http://arxiv.org/abs/2309.00894
Regularly Truncated M-estimators for Learning with Noisy Labels. (1%)
Xiaobo Xia; Pengqian Lu; Chen Gong; Bo Han; Jun Yu; Jun Yu; Tongliang Liu
  The sample selection approach is very popular in learning with noisy labels.
As deep networks learn pattern first, prior methods built on sample selection
share a similar training procedure: the small-loss examples can be regarded as
clean examples and used for helping generalization, while the large-loss
examples are treated as mislabeled ones and excluded from network parameter
updates. However, such a procedure is arguably debatable from two folds: (a) it
does not consider the bad influence of noisy labels in selected small-loss
examples; (b) it does not make good use of the discarded large-loss examples,
which may be clean or have meaningful information for generalization. In this
paper, we propose regularly truncated M-estimators (RTME) to address the above
two issues simultaneously. Specifically, RTME can alternately switch modes
between truncated M-estimators and original M-estimators. The former can
adaptively select small-losses examples without knowing the noise rate and
reduce the side-effects of noisy labels in them. The latter makes the possibly
clean examples but with large losses involved to help generalization.
Theoretically, we demonstrate that our strategies are label-noise-tolerant.
Empirically, comprehensive experimental results show that our method can
outperform multiple baselines and is robust to broad noise types and levels.


http://arxiv.org/abs/2309.00614
Baseline Defenses for Adversarial Attacks Against Aligned Language Models. (99%)
Neel Jain; Avi Schwarzschild; Yuxin Wen; Gowthami Somepalli; John Kirchenbauer; Ping-yeh Chiang; Micah Goldblum; Aniruddha Saha; Jonas Geiping; Tom Goldstein
  As Large Language Models quickly become ubiquitous, their security
vulnerabilities are critical to understand. Recent work shows that text
optimizers can produce jailbreaking prompts that bypass moderation and
alignment. Drawing from the rich body of work on adversarial machine learning,
we approach these attacks with three questions: What threat models are
practically useful in this domain? How do baseline defense techniques perform
in this new domain? How does LLM security differ from computer vision?
  We evaluate several baseline defense strategies against leading adversarial
attacks on LLMs, discussing the various settings in which each is feasible and
effective. Particularly, we look at three types of defenses: detection
(perplexity based), input preprocessing (paraphrase and retokenization), and
adversarial training. We discuss white-box and gray-box settings and discuss
the robustness-performance trade-off for each of the defenses considered.
Surprisingly, we find much more success with filtering and preprocessing than
we would expect from other domains, such as vision, providing a first
indication that the relative strengths of these defenses may be weighed
differently in these domains.


http://arxiv.org/abs/2309.00543
Curating Naturally Adversarial Datasets for Trustworthy AI in Healthcare. (99%)
Sydney Pugh; Ivan Ruchkin; Insup Lee; James Weimer
  Deep learning models have shown promising predictive accuracy for time-series
healthcare applications. However, ensuring the robustness of these models is
vital for building trustworthy AI systems. Existing research predominantly
focuses on robustness to synthetic adversarial examples, crafted by adding
imperceptible perturbations to clean input data. However, these synthetic
adversarial examples do not accurately reflect the most challenging real-world
scenarios, especially in the context of healthcare data. Consequently,
robustness to synthetic adversarial examples may not necessarily translate to
robustness against naturally occurring adversarial examples, which is highly
desirable for trustworthy AI. We propose a method to curate datasets comprised
of natural adversarial examples to evaluate model robustness. The method relies
on probabilistic labels obtained from automated weakly-supervised labeling that
combines noisy and cheap-to-obtain labeling heuristics. Based on these labels,
our method adversarially orders the input data and uses this ordering to
construct a sequence of increasingly adversarial datasets. Our evaluation on
six medical case studies and three non-medical case studies demonstrates the
efficacy and statistical validity of our approach to generating naturally
adversarial datasets


http://arxiv.org/abs/2309.00771
Non-Asymptotic Bounds for Adversarial Excess Risk under Misspecified Models. (89%)
Changyu Liu; Yuling Jiao; Junhui Wang; Jian Huang
  We propose a general approach to evaluating the performance of robust
estimators based on adversarial losses under misspecified models. We first show
that adversarial risk is equivalent to the risk induced by a distributional
adversarial attack under certain smoothness conditions. This ensures that the
adversarial training procedure is well-defined. To evaluate the generalization
performance of the adversarial estimator, we study the adversarial excess risk.
Our proposed analysis method includes investigations on both generalization
error and approximation error. We then establish non-asymptotic upper bounds
for the adversarial excess risk associated with Lipschitz loss functions. In
addition, we apply our general results to adversarial training for
classification and regression problems. For the quadratic loss in nonparametric
regression, we show that the adversarial excess risk bound can be improved over
those for a general loss.


http://arxiv.org/abs/2309.00254
Why do universal adversarial attacks work on large language models?: Geometry might be the answer. (83%)
Varshini Subhash; Anna Bialas; Weiwei Pan; Finale Doshi-Velez
  Transformer based large language models with emergent capabilities are
becoming increasingly ubiquitous in society. However, the task of understanding
and interpreting their internal workings, in the context of adversarial
attacks, remains largely unsolved. Gradient-based universal adversarial attacks
have been shown to be highly effective on large language models and potentially
dangerous due to their input-agnostic nature. This work presents a novel
geometric perspective explaining universal adversarial attacks on large
language models. By attacking the 117M parameter GPT-2 model, we find evidence
indicating that universal adversarial triggers could be embedding vectors which
merely approximate the semantic information in their adversarial training
region. This hypothesis is supported by white-box model analysis comprising
dimensionality reduction and similarity measurement of hidden representations.
We believe this new geometric perspective on the underlying mechanism driving
universal attacks could help us gain deeper insight into the internal workings
and failure modes of LLMs, thus enabling their mitigation.


http://arxiv.org/abs/2309.00810
RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model. (1%)
Fengxiang Bie; Yibo Yang; Zhongzhu Zhou; Adam Ghanem; Minjia Zhang; Zhewei Yao; Xiaoxia Wu; Connor Holmes; Pareesa Golnari; David A. Clifton; Yuxiong He; Dacheng Tao; Shuaiwen Leon Song
  Text-to-image generation (TTI) refers to the usage of models that could
process text input and generate high fidelity images based on text
descriptions. Text-to-image generation using neural networks could be traced
back to the emergence of Generative Adversial Network (GAN), followed by the
autoregressive Transformer. Diffusion models are one prominent type of
generative model used for the generation of images through the systematic
introduction of noises with repeating steps. As an effect of the impressive
results of diffusion models on image synthesis, it has been cemented as the
major image decoder used by text-to-image models and brought text-to-image
generation to the forefront of machine-learning (ML) research. In the era of
large models, scaling up model size and the integration with large language
models have further improved the performance of TTI models, resulting the
generation result nearly indistinguishable from real-world images,
revolutionizing the way we retrieval images. Our explorative study has
incentivised us to think that there are further ways of scaling text-to-image
models with the combination of innovative model architectures and prediction
enhancement techniques. We have divided the work of this survey into five main
sections wherein we detail the frameworks of major literature in order to delve
into the different types of text-to-image generation methods. Following this we
provide a detailed comparison and critique of these methods and offer possible
pathways of improvement for future work. In the future work, we argue that TTI
development could yield impressive productivity improvements for creation,
particularly in the context of the AIGC era, and could be extended to more
complex tasks such as video generation and 3D generation.


http://arxiv.org/abs/2309.00733
Learned Visual Features to Textual Explanations. (1%)
Saeid Asgari Taghanaki; Aliasghar Khani; Amir Khasahmadi; Aditya Sanghi; Karl D. D. Willis; Ali Mahdavi-Amiri
  Interpreting the learned features of vision models has posed a longstanding
challenge in the field of machine learning. To address this issue, we propose a
novel method that leverages the capabilities of large language models (LLMs) to
interpret the learned features of pre-trained image classifiers. Our method,
called TExplain, tackles this task by training a neural network to establish a
connection between the feature space of image classifiers and LLMs. Then,
during inference, our approach generates a vast number of sentences to explain
the features learned by the classifier for a given image. These sentences are
then used to extract the most frequent words, providing a comprehensive
understanding of the learned features and patterns within the classifier. Our
method, for the first time, utilizes these frequent words corresponding to a
visual representation to provide insights into the decision-making process of
the independently trained classifier, enabling the detection of spurious
correlations, biases, and a deeper comprehension of its behavior. To validate
the effectiveness of our approach, we conduct experiments on diverse datasets,
including ImageNet-9L and Waterbirds. The results demonstrate the potential of
our method to enhance the interpretability and robustness of image classifiers.


http://arxiv.org/abs/2308.16454
Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoff. (98%)
Satoshi Suzuki; Shin'ya Yamaguchi; Shoichiro Takeda; Sekitoshi Kanai; Naoki Makishima; Atsushi Ando; Ryo Masumura
  This paper addresses the tradeoff between standard accuracy on clean examples
and robustness against adversarial examples in deep neural networks (DNNs).
Although adversarial training (AT) improves robustness, it degrades the
standard accuracy, thus yielding the tradeoff. To mitigate this tradeoff, we
propose a novel AT method called ARREST, which comprises three components: (i)
adversarial finetuning (AFT), (ii) representation-guided knowledge distillation
(RGKD), and (iii) noisy replay (NR). AFT trains a DNN on adversarial examples
by initializing its parameters with a DNN that is standardly pretrained on
clean examples. RGKD and NR respectively entail a regularization term and an
algorithm to preserve latent representations of clean examples during AFT. RGKD
penalizes the distance between the representations of the standardly pretrained
and AFT DNNs. NR switches input adversarial examples to nonadversarial ones
when the representation changes significantly during AFT. By combining these
components, ARREST achieves both high standard accuracy and robustness.
Experimental results demonstrate that ARREST mitigates the tradeoff more
effectively than previous AT-based methods do.


http://arxiv.org/abs/2309.00236
Image Hijacking: Adversarial Images can Control Generative Models at Runtime. (98%)
Luke Bailey; Euan Ong; Stuart Russell; Scott Emmons
  Are foundation models secure from malicious actors? In this work, we focus on
the image input to a vision-language model (VLM). We discover image hijacks,
adversarial images that control generative models at runtime. We introduce
Behavior Matching, a general method for creating image hijacks, and we use it
to explore three types of attacks. Specific string attacks generate arbitrary
output of the adversary's choosing. Leak context attacks leak information from
the context window into the output. Jailbreak attacks circumvent a model's
safety training. We study these attacks against LLaVA-2, a state-of-the-art VLM
based on CLIP and LLaMA-2, and find that all our attack types have above a 90\%
success rate. Moreover, our attacks are automated and require only small image
perturbations. These findings raise serious concerns about the security of
foundation models. If image hijacks are as difficult to defend against as
adversarial examples in CIFAR-10, then it might be many years before a solution
is found -- if it even exists.


http://arxiv.org/abs/2308.16562
The Power of MEME: Adversarial Malware Creation with Model-Based Reinforcement Learning. (93%)
Maria Rigaki; Sebastian Garcia
  Due to the proliferation of malware, defenders are increasingly turning to
automation and machine learning as part of the malware detection tool-chain.
However, machine learning models are susceptible to adversarial attacks,
requiring the testing of model and product robustness. Meanwhile, attackers
also seek to automate malware generation and evasion of antivirus systems, and
defenders try to gain insight into their methods. This work proposes a new
algorithm that combines Malware Evasion and Model Extraction (MEME) attacks.
MEME uses model-based reinforcement learning to adversarially modify Windows
executable binary samples while simultaneously training a surrogate model with
a high agreement with the target model to evade. To evaluate this method, we
compare it with two state-of-the-art attacks in adversarial malware creation,
using three well-known published models and one antivirus product as targets.
Results show that MEME outperforms the state-of-the-art methods in terms of
evasion capabilities in almost all cases, producing evasive malware with an
evasion rate in the range of 32-73%. It also produces surrogate models with a
prediction label agreement with the respective target models between 97-99%.
The surrogate could be used to fine-tune and improve the evasion rate in the
future.


http://arxiv.org/abs/2308.16703
Fault Injection and Safe-Error Attack for Extraction of Embedded Neural Network Models. (75%)
Kevin Hector; Pierre-Alain Moellic; Mathieu Dumont; Jean-Max Dutertre
  Model extraction emerges as a critical security threat with attack vectors
exploiting both algorithmic and implementation-based approaches. The main goal
of an attacker is to steal as much information as possible about a protected
victim model, so that he can mimic it with a substitute model, even with a
limited access to similar training data. Recently, physical attacks such as
fault injection have shown worrying efficiency against the integrity and
confidentiality of embedded models. We focus on embedded deep neural network
models on 32-bit microcontrollers, a widespread family of hardware platforms in
IoT, and the use of a standard fault injection strategy - Safe Error Attack
(SEA) - to perform a model extraction attack with an adversary having a limited
access to training data. Since the attack strongly depends on the input
queries, we propose a black-box approach to craft a successful attack set. For
a classical convolutional neural network, we successfully recover at least 90%
of the most significant bits with about 1500 crafted inputs. These information
enable to efficiently train a substitute model, with only 8% of the training
dataset, that reaches high fidelity and near identical accuracy level than the
victim model.


http://arxiv.org/abs/2308.16684
Everyone Can Attack: Repurpose Lossy Compression as a Natural Backdoor Attack. (75%)
Sze Jue Yang; Quang Nguyen; Chee Seng Chan; Khoa D. Doan
  The vulnerabilities to backdoor attacks have recently threatened the
trustworthiness of machine learning models in practical applications.
Conventional wisdom suggests that not everyone can be an attacker since the
process of designing the trigger generation algorithm often involves
significant effort and extensive experimentation to ensure the attack's
stealthiness and effectiveness. Alternatively, this paper shows that there
exists a more severe backdoor threat: anyone can exploit an easily-accessible
algorithm for silent backdoor attacks. Specifically, this attacker can employ
the widely-used lossy image compression from a plethora of compression tools to
effortlessly inject a trigger pattern into an image without leaving any
noticeable trace; i.e., the generated triggers are natural artifacts. One does
not require extensive knowledge to click on the "convert" or "save as" button
while using tools for lossy image compression. Via this attack, the adversary
does not need to design a trigger generator as seen in prior works and only
requires poisoning the data. Empirically, the proposed attack consistently
achieves 100% attack success rate in several benchmark datasets such as MNIST,
CIFAR-10, GTSRB and CelebA. More significantly, the proposed attack can still
achieve almost 100% attack success rate with very small (approximately 10%)
poisoning rates in the clean label setting. The generated trigger of the
proposed attack using one lossy compression algorithm is also transferable
across other related compression algorithms, exacerbating the severity of this
backdoor threat. This work takes another crucial step toward understanding the
extensive risks of backdoor attacks in practice, urging practitioners to
investigate similar attacks and relevant backdoor mitigation methods.


http://arxiv.org/abs/2309.00127
FTA: Stealthy and Robust Backdoor Attack with Flexible Trigger on Federated Learning. (45%)
Yanqi Qiao; Congwen Chen; Rui Wang; Kaitai Liang
  Current backdoor attacks against federated learning (FL) strongly rely on
universal triggers or semantic patterns, which can be easily detected and
filtered by certain defense mechanisms such as norm clipping, comparing
parameter divergences among local updates. In this work, we propose a new
stealthy and robust backdoor attack with flexible triggers against FL defenses.
To achieve this, we build a generative trigger function that can learn to
manipulate the benign samples with an imperceptible flexible trigger pattern
and simultaneously make the trigger pattern include the most significant hidden
features of the attacker-chosen label. Moreover, our trigger generator can keep
learning and adapt across different rounds, allowing it to adjust to changes in
the global model. By filling the distinguishable difference (the mapping
between the trigger pattern and target label), we make our attack naturally
stealthy. Extensive experiments on real-world datasets verify the effectiveness
and stealthiness of our attack compared to prior attacks on decentralized
learning framework with eight well-studied defenses.


http://arxiv.org/abs/2309.03215
Explainable and Trustworthy Traffic Sign Detection for Safe Autonomous Driving: An Inductive Logic Programming Approach. (98%)
Zahra University of Surrey Chaghazardi; Saber University of Surrey Fallah; Alireza University of Surrey Tamaddoni-Nezhad
  Traffic sign detection is a critical task in the operation of Autonomous
Vehicles (AV), as it ensures the safety of all road users. Current DNN-based
sign classification systems rely on pixel-level features to detect traffic
signs and can be susceptible to adversarial attacks. These attacks involve
small, imperceptible changes to a sign that can cause traditional classifiers
to misidentify the sign. We propose an Inductive Logic Programming (ILP) based
approach for stop sign detection in AVs to address this issue. This method
utilises high-level features of a sign, such as its shape, colour, and text, to
detect categories of traffic signs. This approach is more robust against
adversarial attacks, as it mimics human-like perception and is less susceptible
to the limitations of current DNN classifiers. We consider two adversarial
attacking methods to evaluate our approach: Robust Physical Perturbation (PR2)
and Adversarial Camouflage (AdvCam). These attacks are able to deceive DNN
classifiers, causing them to misidentify stop signs as other signs with high
confidence. The results show that the proposed ILP-based technique is able to
correctly identify all targeted stop signs, even in the presence of PR2 and
ADvCam attacks. The proposed learning method is also efficient as it requires
minimal training data. Moreover, it is fully explainable, making it possible to
debug AVs.


http://arxiv.org/abs/2308.16258
Robust Principles: Architectural Design Principles for Adversarially Robust CNNs. (11%)
ShengYun Peng; Weilin Xu; Cory Cornelius; Matthew Hull; Kevin Li; Rahul Duggal; Mansi Phute; Jason Martin; Duen Horng Chau
  Our research aims to unify existing works' diverging opinions on how
architectural components affect the adversarial robustness of CNNs. To
accomplish our goal, we synthesize a suite of three generalizable robust
architectural design principles: (a) optimal range for depth and width
configurations, (b) preferring convolutional over patchify stem stage, and (c)
robust residual block design through adopting squeeze and excitation blocks and
non-parametric smooth activation functions. Through extensive experiments
across a wide spectrum of dataset scales, adversarial training methods, model
parameters, and network design spaces, our principles consistently and markedly
improve AutoAttack accuracy: 1-3 percentage points (pp) on CIFAR-10 and
CIFAR-100, and 4-9 pp on ImageNet. The code is publicly available at
https://github.com/poloclub/robust-principles.


http://arxiv.org/abs/2308.15663
Adaptive Attack Detection in Text Classification: Leveraging Space Exploration Features for Text Sentiment Classification. (99%)
Atefeh Mahdavi; Neda Keivandarian; Marco Carvalho
  Adversarial example detection plays a vital role in adaptive cyber defense,
especially in the face of rapidly evolving attacks. In adaptive cyber defense,
the nature and characteristics of attacks continuously change, making it
crucial to have robust mechanisms in place to detect and counter these threats
effectively. By incorporating adversarial example detection techniques,
adaptive cyber defense systems can enhance their ability to identify and
mitigate attacks that attempt to exploit vulnerabilities in machine learning
models or other systems. Adversarial examples are inputs that are crafted by
applying intentional perturbations to natural inputs that result in incorrect
classification. In this paper, we propose a novel approach that leverages the
power of BERT (Bidirectional Encoder Representations from Transformers) and
introduces the concept of Space Exploration Features. We utilize the feature
vectors obtained from the BERT model's output to capture a new representation
of feature space to improve the density estimation method.


http://arxiv.org/abs/2308.15072
Advancing Adversarial Robustness Through Adversarial Logit Update. (99%)
Hao Xuan; Peican Zhu; Xingyu Li
  Deep Neural Networks are susceptible to adversarial perturbations.
Adversarial training and adversarial purification are among the most widely
recognized defense strategies. Although these methods have different underlying
logic, both rely on absolute logit values to generate label predictions. In
this study, we theoretically analyze the logit difference around successful
adversarial attacks from a theoretical point of view and propose a new
principle, namely Adversarial Logit Update (ALU), to infer adversarial sample's
labels. Based on ALU, we introduce a new classification paradigm that utilizes
pre- and post-purification logit differences for model's adversarial robustness
boost. Without requiring adversarial or additional data for model training, our
clean data synthesis model can be easily applied to various pre-trained models
for both adversarial sample detection and ALU-based data classification.
Extensive experiments on both CIFAR-10, CIFAR-100, and tiny-ImageNet datasets
show that even with simple components, the proposed solution achieves superior
robustness performance compared to state-of-the-art methods against a wide
range of adversarial attacks. Our python implementation is submitted in our
Supplementary document and will be published upon the paper's acceptance.


http://arxiv.org/abs/2308.15344
Imperceptible Adversarial Attack on Deep Neural Networks from Image Boundary. (99%)
Fahad Alrasheedi; Xin Zhong
  Although Deep Neural Networks (DNNs), such as the convolutional neural
networks (CNN) and Vision Transformers (ViTs), have been successfully applied
in the field of computer vision, they are demonstrated to be vulnerable to
well-sought Adversarial Examples (AEs) that can easily fool the DNNs. The
research in AEs has been active, and many adversarial attacks and explanations
have been proposed since they were discovered in 2014. The mystery of the AE's
existence is still an open question, and many studies suggest that DNN training
algorithms have blind spots. The salient objects usually do not overlap with
boundaries; hence, the boundaries are not the DNN model's attention.
Nevertheless, recent studies show that the boundaries can dominate the behavior
of the DNN models. Hence, this study aims to look at the AEs from a different
perspective and proposes an imperceptible adversarial attack that systemically
attacks the input image boundary for finding the AEs. The experimental results
have shown that the proposed boundary attacking method effectively attacks six
CNN models and the ViT using only 32% of the input image content (from the
boundaries) with an average success rate (SR) of 95.2% and an average peak
signal-to-noise ratio of 41.37 dB. Correlation analyses are conducted,
including the relation between the adversarial boundary's width and the SR and
how the adversarial boundary changes the DNN model's attention. This paper's
discoveries can potentially advance the understanding of AEs and provide a
different perspective on how AEs can be constructed.


http://arxiv.org/abs/2308.15246
A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation. (99%)
Sahar Sadrizadeh; Ljiljana Dolamic; Pascal Frossard
  Neural Machine Translation (NMT) models have been shown to be vulnerable to
adversarial attacks, wherein carefully crafted perturbations of the input can
mislead the target model. In this paper, we introduce ACT, a novel adversarial
attack framework against NMT systems guided by a classifier. In our attack, the
adversary aims to craft meaning-preserving adversarial examples whose
translations by the NMT model belong to a different class than the original
translations in the target language. Unlike previous attacks, our new approach
has a more substantial effect on the translation by altering the overall
meaning, which leads to a different class determined by a classifier. To
evaluate the robustness of NMT models to this attack, we propose enhancements
to existing black-box word-replacement-based attacks by incorporating output
translations of the target NMT model and the output logits of a classifier
within the attack process. Extensive experiments in various settings, including
a comparison with existing untargeted attacks, demonstrate that the proposed
attack is considerably more successful in altering the class of the output
translation and has more effect on the translation. This new paradigm can show
the vulnerabilities of NMT systems by focusing on the class of translation
rather than the mere translation quality as studied traditionally.


http://arxiv.org/abs/2308.15673
MDTD: A Multi Domain Trojan Detector for Deep Neural Networks. (97%)
Arezoo Rajabi; Surudhi Asokraj; Fengqing Jiang; Luyao Niu; Bhaskar Ramasubramanian; Jim Ritcey; Radha Poovendran
  Machine learning models that use deep neural networks (DNNs) are vulnerable
to backdoor attacks. An adversary carrying out a backdoor attack embeds a
predefined perturbation called a trigger into a small subset of input samples
and trains the DNN such that the presence of the trigger in the input results
in an adversary-desired output class. Such adversarial retraining however needs
to ensure that outputs for inputs without the trigger remain unaffected and
provide high classification accuracy on clean samples. In this paper, we
propose MDTD, a Multi-Domain Trojan Detector for DNNs, which detects inputs
containing a Trojan trigger at testing time. MDTD does not require knowledge of
trigger-embedding strategy of the attacker and can be applied to a pre-trained
DNN model with image, audio, or graph-based inputs. MDTD leverages an insight
that input samples containing a Trojan trigger are located relatively farther
away from a decision boundary than clean samples. MDTD estimates the distance
to a decision boundary using adversarial learning methods and uses this
distance to infer whether a test-time input sample is Trojaned or not. We
evaluate MDTD against state-of-the-art Trojan detection methods across five
widely used image-based datasets: CIFAR100, CIFAR10, GTSRB, SVHN, and
Flowers102; four graph-based datasets: AIDS, WinMal, Toxicant, and COLLAB; and
the SpeechCommand audio dataset. MDTD effectively identifies samples that
contain different types of Trojan triggers. We evaluate MDTD against adaptive
attacks where an adversary trains a robust DNN to increase (decrease) distance
of benign (Trojan) inputs from a decision boundary.


http://arxiv.org/abs/2308.15479
3D Adversarial Augmentations for Robust Out-of-Domain Predictions. (87%)
Alexander Lehner; Stefano Gasperini; Alvaro Marcos-Ramiro; Michael Schmidt; Nassir Navab; Benjamin Busam; Federico Tombari
  Since real-world training datasets cannot properly sample the long tail of
the underlying data distribution, corner cases and rare out-of-domain samples
can severely hinder the performance of state-of-the-art models. This problem
becomes even more severe for dense tasks, such as 3D semantic segmentation,
where points of non-standard objects can be confidently associated to the wrong
class. In this work, we focus on improving the generalization to out-of-domain
data. We achieve this by augmenting the training set with adversarial examples.
First, we learn a set of vectors that deform the objects in an adversarial
fashion. To prevent the adversarial examples from being too far from the
existing data distribution, we preserve their plausibility through a series of
constraints, ensuring sensor-awareness and shapes smoothness. Then, we perform
adversarial augmentation by applying the learned sample-independent vectors to
the available objects when training a model. We conduct extensive experiments
across a variety of scenarios on data from KITTI, Waymo, and CrashD for 3D
object detection, and on data from SemanticKITTI, Waymo, and nuScenes for 3D
semantic segmentation. Despite training on a standard single dataset, our
approach substantially improves the robustness and generalization of both 3D
object detection and 3D semantic segmentation methods to out-of-domain data.


http://arxiv.org/abs/2308.15614
Everything Perturbed All at Once: Enabling Differentiable Graph Attacks. (84%)
Haoran Liu; Bokun Wang; Jianling Wang; Xiangjue Dong; Tianbao Yang; James Caverlee
  As powerful tools for representation learning on graphs, graph neural
networks (GNNs) have played an important role in applications including social
networks, recommendation systems, and online web services. However, GNNs have
been shown to be vulnerable to adversarial attacks, which can significantly
degrade their effectiveness. Recent state-of-the-art approaches in adversarial
attacks rely on gradient-based meta-learning to selectively perturb a single
edge with the highest attack score until they reach the budget constraint.
While effective in identifying vulnerable links, these methods are plagued by
high computational costs. By leveraging continuous relaxation and
parameterization of the graph structure, we propose a novel attack method
called Differentiable Graph Attack (DGA) to efficiently generate effective
attacks and meanwhile eliminate the need for costly retraining. Compared to the
state-of-the-art, DGA achieves nearly equivalent attack performance with 6
times less training time and 11 times smaller GPU memory footprint on different
benchmark datasets. Additionally, we provide extensive experimental analyses of
the transferability of the DGA among different graph models, as well as its
robustness against widely-used defense mechanisms.


http://arxiv.org/abs/2308.15736
Adaversarial Issue of Machine Learning Approaches Applied in Smart Grid: A Survey. (70%)
Zhenyong Zhang; Mengxiang Liu
  The machine learning (ML) sees an increasing prevalence of being used in the
internet-of-things enabled smart grid. However, the trustworthiness of ML is a
severe issue that must be addressed to accommodate the trend of ML-based smart
grid applications (MLsgAPPs). The adversarial distortion injected into the
power signal will greatly affect the system's normal control and operation.
Therefore, it is imperative to conduct vulnerability assessment for MLsgAPPs
applied in the context of safety-critical power systems. In this paper, we
provide a comprehensive review of the recent progress in designing attack and
defense methods for MLsgAPPs. Unlike the traditional survey about ML security,
this is the first review work about the security of MLsgAPPs that focuses on
the characteristics of power systems. The survey is organized from the aspects
of adversarial assumptions, targeted applications, evaluation metrics,
defending approaches, physics-related constraints, and applied datasets. We
also highlight future directions on this topic to encourage more researchers to
conduct further research on adversarial attacks and defending approaches for
MLsgAPPs.


http://arxiv.org/abs/2308.15692
Intriguing Properties of Diffusion Models: A Large-Scale Dataset for Evaluating Natural Attack Capability in Text-to-Image Generative Models. (67%)
Takami Sato; Justin Yue; Nanze Chen; Ningfei Wang; Qi Alfred Chen
  Denoising probabilistic diffusion models have shown breakthrough performance
that can generate more photo-realistic images or human-level illustrations than
the prior models such as GANs. This high image-generation capability has
stimulated the creation of many downstream applications in various areas.
However, we find that this technology is indeed a double-edged sword: We
identify a new type of attack, called the Natural Denoising Diffusion (NDD)
attack based on the finding that state-of-the-art deep neural network (DNN)
models still hold their prediction even if we intentionally remove their robust
features, which are essential to the human visual system (HVS), by text
prompts. The NDD attack can generate low-cost, model-agnostic, and
transferrable adversarial attacks by exploiting the natural attack capability
in diffusion models. Motivated by the finding, we construct a large-scale
dataset, Natural Denoising Diffusion Attack (NDDA) dataset, to systematically
evaluate the risk of the natural attack capability of diffusion models with
state-of-the-art text-to-image diffusion models. We evaluate the natural attack
capability by answering 6 research questions. Through a user study to confirm
the validity of the NDD attack, we find that the NDD attack can achieve an 88%
detection rate while being stealthy to 93% of human subjects. We also find that
the non-robust features embedded by diffusion models contribute to the natural
attack capability. To confirm the model-agnostic and transferrable attack
capability, we perform the NDD attack against an AD vehicle and find that 73%
of the physically printed attacks can be detected as a stop sign. We hope that
our study and dataset can help our community to be aware of the risk of
diffusion models and facilitate further research toward robust DNN models.


http://arxiv.org/abs/2308.15092
Can We Rely on AI? (50%)
Desmond J. Higham
  Over the last decade, adversarial attack algorithms have revealed
instabilities in deep learning tools. These algorithms raise issues regarding
safety, reliability and interpretability in artificial intelligence; especially
in high risk settings. From a practical perspective, there has been a war of
escalation between those developing attack and defence strategies. At a more
theoretical level, researchers have also studied bigger picture questions
concerning the existence and computability of attacks. Here we give a brief
overview of the topic, focusing on aspects that are likely to be of interest to
researchers in applied and computational mathematics.


http://arxiv.org/abs/2308.15141
Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images. (1%)
Tareen Dawood; Chen Chen; Baldeep S. Sidhua; Bram Ruijsink; Justin Goulda; Bradley Porter; Mark K. Elliott; Vishal Mehta; Christopher A. Rinaldi; Esther Puyol-Anton; Reza Razavi; Andrew P. King
  Quantifying uncertainty of predictions has been identified as one way to
develop more trustworthy artificial intelligence (AI) models beyond
conventional reporting of performance metrics. When considering their role in a
clinical decision support setting, AI classification models should ideally
avoid confident wrong predictions and maximise the confidence of correct
predictions. Models that do this are said to be well-calibrated with regard to
confidence. However, relatively little attention has been paid to how to
improve calibration when training these models, i.e., to make the training
strategy uncertainty-aware. In this work we evaluate three novel
uncertainty-aware training strategies comparing against two state-of-the-art
approaches. We analyse performance on two different clinical applications:
cardiac resynchronisation therapy (CRT) response prediction and coronary artery
disease (CAD) diagnosis from cardiac magnetic resonance (CMR) images. The
best-performing model in terms of both classification accuracy and the most
common calibration measure, expected calibration error (ECE) was the Confidence
Weight method, a novel approach that weights the loss of samples to explicitly
penalise confident incorrect predictions. The method reduced the ECE by 17% for
CRT response prediction and by 22% for CAD diagnosis when compared to a
baseline classifier in which no uncertainty-aware strategy was included. In
both applications, as well as reducing the ECE there was a slight increase in
accuracy from 69% to 70% and 70% to 72% for CRT response prediction and CAD
diagnosis respectively. However, our analysis showed a lack of consistency in
terms of optimal models when using different calibration measures. This
indicates the need for careful consideration of performance metrics when
training and selecting models for complex high-risk applications in healthcare.


http://arxiv.org/abs/2308.14597
Adversarial Attacks on Foundational Vision Models. (80%)
Nathan Inkawhich; Gwendolyn McDonald; Ryan Luley
  Rapid progress is being made in developing large, pretrained, task-agnostic
foundational vision models such as CLIP, ALIGN, DINOv2, etc. In fact, we are
approaching the point where these models do not have to be finetuned
downstream, and can simply be used in zero-shot or with a lightweight probing
head. Critically, given the complexity of working at this scale, there is a
bottleneck where relatively few organizations in the world are executing the
training then sharing the models on centralized platforms such as HuggingFace
and torch.hub. The goal of this work is to identify several key adversarial
vulnerabilities of these models in an effort to make future designs more
robust. Intuitively, our attacks manipulate deep feature representations to
fool an out-of-distribution (OOD) detector which will be required when using
these open-world-aware models to solve closed-set downstream tasks. Our methods
reliably make in-distribution (ID) images (w.r.t. a downstream task) be
predicted as OOD and vice versa while existing in extremely
low-knowledge-assumption threat models. We show our attacks to be potent in
whitebox and blackbox settings, as well as when transferred across foundational
model types (e.g., attack DINOv2 with CLIP)! This work is only just the
beginning of a long journey towards adversarially robust foundational vision
models.


http://arxiv.org/abs/2308.14333
DiffSmooth: Certifiably Robust Learning via Diffusion Models and Local Smoothing. (45%)
Jiawei Zhang; Zhongzhu Chen; Huan Zhang; Chaowei Xiao; Bo Li
  Diffusion models have been leveraged to perform adversarial purification and
thus provide both empirical and certified robustness for a standard model. On
the other hand, different robustly trained smoothed models have been studied to
improve the certified robustness. Thus, it raises a natural question: Can
diffusion model be used to achieve improved certified robustness on those
robustly trained smoothed models? In this work, we first theoretically show
that recovered instances by diffusion models are in the bounded neighborhood of
the original instance with high probability; and the "one-shot" denoising
diffusion probabilistic models (DDPM) can approximate the mean of the generated
distribution of a continuous-time diffusion model, which approximates the
original instance under mild conditions. Inspired by our analysis, we propose a
certifiably robust pipeline DiffSmooth, which first performs adversarial
purification via diffusion models and then maps the purified instances to a
common region via a simple yet effective local smoothing strategy. We conduct
extensive experiments on different datasets and show that DiffSmooth achieves
SOTA-certified robustness compared with eight baselines. For instance,
DiffSmooth improves the SOTA-certified accuracy from $36.0\%$ to $53.0\%$ under
$\ell_2$ radius $1.5$ on ImageNet. The code is available at
[https://github.com/javyduck/DiffSmooth].


http://arxiv.org/abs/2308.14840
Identifying and Mitigating the Security Risks of Generative AI. (45%)
Clark Barrett; Brad Boyd; Elie Burzstein; Nicholas Carlini; Brad Chen; Jihye Choi; Amrita Roy Chowdhury; Mihai Christodorescu; Anupam Datta; Soheil Feizi; Kathleen Fisher; Tatsunori Hashimoto; Dan Hendrycks; Somesh Jha; Daniel Kang; Florian Kerschbaum; Eric Mitchell; John Mitchell; Zulfikar Ramzan; Khawaja Shams; Dawn Song; Ankur Taly; Diyi Yang
  Every major technical invention resurfaces the dual-use dilemma -- the new
technology has the potential to be used for good as well as for harm.
Generative AI (GenAI) techniques, such as large language models (LLMs) and
diffusion models, have shown remarkable capabilities (e.g., in-context
learning, code-completion, and text-to-image generation and editing). However,
GenAI can be used just as well by attackers to generate new attacks and
increase the velocity and efficacy of existing attacks.
  This paper reports the findings of a workshop held at Google (co-organized by
Stanford University and the University of Wisconsin-Madison) on the dual-use
dilemma posed by GenAI. This paper is not meant to be comprehensive, but is
rather an attempt to synthesize some of the interesting findings from the
workshop. We discuss short-term and long-term goals for the community on this
topic. We hope this paper provides both a launching point for a discussion on
this important topic as well as interesting problems that the research
community can work to address.


http://arxiv.org/abs/2308.14550
ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events. (2%)
Aizaz Sharif; Dusica Marijan
  Autonomous vehicles are advanced driving systems that are well known for
being vulnerable to various adversarial attacks, compromising the vehicle's
safety, and posing danger to other road users. Rather than actively training
complex adversaries by interacting with the environment, there is a need to
first intelligently find and reduce the search space to only those states where
autonomous vehicles are found less confident. In this paper, we propose a
blackbox testing framework ReMAV using offline trajectories first to analyze
the existing behavior of autonomous vehicles and determine appropriate
thresholds for finding the probability of failure events. Our reward modeling
technique helps in creating a behavior representation that allows us to
highlight regions of likely uncertain behavior even when the baseline
autonomous vehicle is performing well. This approach allows for more efficient
testing without the need for computational and inefficient active adversarial
learning techniques. We perform our experiments in a high-fidelity urban
driving environment using three different driving scenarios containing single
and multi-agent interactions. Our experiment shows 35%, 23%, 48%, and 50%
increase in occurrences of vehicle collision, road objects collision,
pedestrian collision, and offroad steering events respectively by the
autonomous vehicle under test, demonstrating a significant increase in failure
events. We also perform a comparative analysis with prior testing frameworks
and show that they underperform in terms of training-testing efficiency,
finding total infractions, and simulation steps to identify the first failure
compared to our approach. The results show that the proposed framework can be
used to understand existing weaknesses of the autonomous vehicles under test in
order to only attack those regions, starting with the simplistic perturbation
models.


http://arxiv.org/abs/2308.14553
Rep2wav: Noise Robust text-to-speech Using self-supervised representations. (1%)
Qiushi Zhu; Yu Gu; Rilin Chen; Chao Weng; Yuchen Hu; Lirong Dai; Jie Zhang
  Benefiting from the development of deep learning, text-to-speech (TTS)
techniques using clean speech have achieved significant performance
improvements. The data collected from real scenes often contains noise and
generally needs to be denoised by speech enhancement models. Noise-robust TTS
models are often trained using the enhanced speech, which thus suffer from
speech distortion and background noise that affect the quality of the
synthesized speech. Meanwhile, it was shown that self-supervised pre-trained
models exhibit excellent noise robustness on many speech tasks, implying that
the learned representation has a better tolerance for noise perturbations. In
this work, we therefore explore pre-trained models to improve the noise
robustness of TTS models. Based on HiFi-GAN, we first propose a
representation-to-waveform vocoder, which aims to learn to map the
representation of pre-trained models to the waveform. We then propose a
text-to-representation FastSpeech2 model, which aims to learn to map text to
pre-trained model representations. Experimental results on the LJSpeech and
LibriTTS datasets show that our method outperforms those using speech
enhancement methods in both subjective and objective metrics. Audio samples are
available at: https://zqs01.github.io/rep2wav.


http://arxiv.org/abs/2308.14376
Are Existing Out-Of-Distribution Techniques Suitable for Network Intrusion Detection? (1%)
Andrea Corsini; Shanchieh Jay Yang
  Machine learning (ML) has become increasingly popular in network intrusion
detection. However, ML-based solutions always respond regardless of whether the
input data reflects known patterns, a common issue across safety-critical
applications. While several proposals exist for detecting Out-Of-Distribution
(OOD) in other fields, it remains unclear whether these approaches can
effectively identify new forms of intrusions for network security. New attacks,
not necessarily affecting overall distributions, are not guaranteed to be
clearly OOD as instead, images depicting new classes are in computer vision. In
this work, we investigate whether existing OOD detectors from other fields
allow the identification of unknown malicious traffic. We also explore whether
more discriminative and semantically richer embedding spaces within models,
such as those created with contrastive learning and multi-class tasks, benefit
detection. Our investigation covers a set of six OOD techniques that employ
different detection strategies. These techniques are applied to models trained
in various ways and subsequently exposed to unknown malicious traffic from the
same and different datasets (network environments). Our findings suggest that
existing detectors can identify a consistent portion of new malicious traffic,
and that improved embedding spaces enhance detection. We also demonstrate that
simple combinations of certain detectors can identify almost 100% of malicious
traffic in our tested scenarios.


http://arxiv.org/abs/2308.14132
Detecting Language Model Attacks with Perplexity. (1%)
Gabriel Alon; Michael Kamfonas
  A novel hack involving Large Language Models (LLMs) has emerged, leveraging
adversarial suffixes to trick models into generating perilous responses. This
method has garnered considerable attention from reputable media outlets such as
the New York Times and Wired, thereby influencing public perception regarding
the security and safety of LLMs. In this study, we advocate the utilization of
perplexity as one of the means to recognize such potential attacks. The
underlying concept behind these hacks revolves around appending an unusually
constructed string of text to a harmful query that would otherwise be blocked.
This maneuver confuses the protective mechanisms and tricks the model into
generating a forbidden response. Such scenarios could result in providing
detailed instructions to a malicious user for constructing explosives or
orchestrating a bank heist. Our investigation demonstrates the feasibility of
employing perplexity, a prevalent natural language processing metric, to detect
these adversarial tactics before generating a forbidden response. By evaluating
the perplexity of queries with and without such adversarial suffixes using an
open-source LLM, we discovered that nearly 90 percent were above a perplexity
of 1000. This contrast underscores the efficacy of perplexity for detecting
this type of exploit.


http://arxiv.org/abs/2308.12636
Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning. (99%)
Youze Wang; Wenbo Hu; Yinpeng Dong; Richang Hong
  Vision-language pre-training models (VLP) are vulnerable, especially to
multimodal adversarial samples, which can be crafted by adding imperceptible
perturbations on both original images and texts. However, under the black-box
setting, there have been no works to explore the transferability of multimodal
adversarial attacks against the VLP models. In this work, we take CLIP as the
surrogate model and propose a gradient-based multimodal attack method to
generate transferable adversarial samples against the VLP models. By applying
the gradient to optimize the adversarial images and adversarial texts
simultaneously, our method can better search for and attack the vulnerable
images and text information pairs. To improve the transferability of the
attack, we utilize contrastive learning including image-text contrastive
learning and intra-modal contrastive learning to have a more generalized
understanding of the underlying data distribution and mitigate the overfitting
of the surrogate model so that the generated multimodal adversarial samples
have a higher transferability for VLP models. Extensive experiments validate
the effectiveness of the proposed method.


http://arxiv.org/abs/2308.12661
Don't Look into the Sun: Adversarial Solarization Attacks on Image Classifiers. (92%)
Paul Gavrikov; Janis Keuper
  Assessing the robustness of deep neural networks against out-of-distribution
inputs is crucial, especially in safety-critical domains like autonomous
driving, but also in safety systems where malicious actors can digitally alter
inputs to circumvent safety guards. However, designing effective
out-of-distribution tests that encompass all possible scenarios while
preserving accurate label information is a challenging task. Existing
methodologies often entail a compromise between variety and constraint levels
for attacks and sometimes even both. In a first step towards a more holistic
robustness evaluation of image classification models, we introduce an attack
method based on image solarization that is conceptually straightforward yet
avoids jeopardizing the global structure of natural images independent of the
intensity. Through comprehensive evaluations of multiple ImageNet models, we
demonstrate the attack's capacity to degrade accuracy significantly, provided
it is not integrated into the training augmentations. Interestingly, even then,
no full immunity to accuracy deterioration is achieved. In other settings, the
attack can often be simplified into a black-box attack with model-independent
parameters. Defenses against other corruptions do not consistently extend to be
effective against our specific attack.
  Project website: https://github.com/paulgavrikov/adversarial_solarization


http://arxiv.org/abs/2308.12918
Evaluating the Vulnerabilities in ML systems in terms of adversarial attacks. (82%)
John Harshith; Mantej Singh Gill; Madhan Jothimani
  There have been recent adversarial attacks that are difficult to find. These
new adversarial attacks methods may pose challenges to current deep learning
cyber defense systems and could influence the future defense of cyberattacks.
The authors focus on this domain in this research paper. They explore the
consequences of vulnerabilities in AI systems. This includes discussing how
they might arise, differences between randomized and adversarial examples and
also potential ethical implications of vulnerabilities. Moreover, it is
important to train the AI systems appropriately when they are in testing phase
and getting them ready for broader use.


http://arxiv.org/abs/2308.12857
Fast Adversarial Training with Smooth Convergence. (3%)
Mengnan Zhao; Lihe Zhang; Yuqiu Kong; Baocai Yin
  Fast adversarial training (FAT) is beneficial for improving the adversarial
robustness of neural networks. However, previous FAT work has encountered a
significant issue known as catastrophic overfitting when dealing with large
perturbation budgets, \ie the adversarial robustness of models declines to near
zero during training.
  To address this, we analyze the training process of prior FAT work and
observe that catastrophic overfitting is accompanied by the appearance of loss
convergence outliers.
  Therefore, we argue a moderately smooth loss convergence process will be a
stable FAT process that solves catastrophic overfitting.
  To obtain a smooth loss convergence process, we propose a novel oscillatory
constraint (dubbed ConvergeSmooth) to limit the loss difference between
adjacent epochs. The convergence stride of ConvergeSmooth is introduced to
balance convergence and smoothing. Likewise, we design weight centralization
without introducing additional hyperparameters other than the loss balance
coefficient.
  Our proposed methods are attack-agnostic and thus can improve the training
stability of various FAT techniques.
  Extensive experiments on popular datasets show that the proposed methods
efficiently avoid catastrophic overfitting and outperform all previous FAT
methods. Code is available at \url{https://github.com/FAT-CS/ConvergeSmooth}.


http://arxiv.org/abs/2308.12770
WavMark: Watermarking for Audio Generation. (2%)
Guangyu Chen; Yu Wu; Shujie Liu; Tao Liu; Xiaoyong Du; Furu Wei
  Recent breakthroughs in zero-shot voice synthesis have enabled imitating a
speaker's voice using just a few seconds of recording while maintaining a high
level of realism. Alongside its potential benefits, this powerful technology
introduces notable risks, including voice fraud and speaker impersonation.
Unlike the conventional approach of solely relying on passive methods for
detecting synthetic data, watermarking presents a proactive and robust defence
mechanism against these looming risks. This paper introduces an innovative
audio watermarking framework that encodes up to 32 bits of watermark within a
mere 1-second audio snippet. The watermark is imperceptible to human senses and
exhibits strong resilience against various attacks. It can serve as an
effective identifier for synthesized voices and holds potential for broader
applications in audio copyright protection. Moreover, this framework boasts
high flexibility, allowing for the combination of multiple watermark segments
to achieve heightened robustness and expanded capacity. Utilizing 10 to
20-second audio as the host, our approach demonstrates an average Bit Error
Rate (BER) of 0.48\% across ten common attacks, a remarkable reduction of over
2800\% in BER compared to the state-of-the-art watermarking tool. See
https://aka.ms/wavmark for demos of our work.


http://arxiv.org/abs/2308.12279
On-Manifold Projected Gradient Descent. (99%)
Aaron Mahler; Tyrus Berry; Tom Stephens; Harbir Antil; Michael Merritt; Jeanie Schreiber; Ioannis Kevrekidis
  This work provides a computable, direct, and mathematically rigorous
approximation to the differential geometry of class manifolds for
high-dimensional data, along with nonlinear projections from input space onto
these class manifolds. The tools are applied to the setting of neural network
image classifiers, where we generate novel, on-manifold data samples, and
implement a projected gradient descent algorithm for on-manifold adversarial
training. The susceptibility of neural networks (NNs) to adversarial attack
highlights the brittle nature of NN decision boundaries in input space.
Introducing adversarial examples during training has been shown to reduce the
susceptibility of NNs to adversarial attack; however, it has also been shown to
reduce the accuracy of the classifier if the examples are not valid examples
for that class. Realistic "on-manifold" examples have been previously generated
from class manifolds in the latent of an autoencoder. Our work explores these
phenomena in a geometric and computational setting that is much closer to the
raw, high-dimensional input space than can be provided by VAE or other black
box dimensionality reductions. We employ conformally invariant diffusion maps
(CIDM) to approximate class manifolds in diffusion coordinates, and develop the
Nystr\"{o}m projection to project novel points onto class manifolds in this
setting. On top of the manifold approximation, we leverage the spectral
exterior calculus (SEC) to determine geometric quantities such as tangent
vectors of the manifold. We use these tools to obtain adversarial examples that
reside on a class manifold, yet fool a classifier. These misclassifications
then become explainable in terms of human-understandable manipulations within
the data, by expressing the on-manifold adversary in the semantic basis on the
manifold.


http://arxiv.org/abs/2308.12054
Sample Complexity of Robust Learning against Evasion Attacks. (98%)
Pascale Gourdeau
  It is becoming increasingly important to understand the vulnerability of
machine learning models to adversarial attacks. One of the fundamental problems
in adversarial machine learning is to quantify how much training data is needed
in the presence of evasion attacks, where data is corrupted at test time. In
this thesis, we work with the exact-in-the-ball notion of robustness and study
the feasibility of adversarially robust learning from the perspective of
learning theory, considering sample complexity.
  We first explore the setting where the learner has access to random examples
only, and show that distributional assumptions are essential. We then focus on
learning problems with distributions on the input data that satisfy a Lipschitz
condition and show that robustly learning monotone conjunctions has sample
complexity at least exponential in the adversary's budget (the maximum number
of bits it can perturb on each input). However, if the adversary is restricted
to perturbing $O(\log n)$ bits, then one can robustly learn conjunctions and
decision lists w.r.t. log-Lipschitz distributions.
  We then study learning models where the learner is given more power. We first
consider local membership queries, where the learner can query the label of
points near the training sample. We show that, under the uniform distribution,
the exponential dependence on the adversary's budget to robustly learn
conjunctions remains inevitable. We then introduce a local equivalence query
oracle, which returns whether the hypothesis and target concept agree in a
given region around a point in the training sample, and a counterexample if it
exists. We show that if the query radius is equal to the adversary's budget, we
can develop robust empirical risk minimization algorithms in the
distribution-free setting. We give general query complexity upper and lower
bounds, as well as for concrete concept classes.


http://arxiv.org/abs/2308.12882
LCANets++: Robust Audio Classification using Multi-layer Neural Networks with Lateral Competition. (92%)
Sayanton V. Dibbo; Juston S. Moore; Garrett T. Kenyon; Michael A. Teti
  Audio classification aims at recognizing audio signals, including speech
commands or sound events. However, current audio classifiers are susceptible to
perturbations and adversarial attacks. In addition, real-world audio
classification tasks often suffer from limited labeled data. To help bridge
these gaps, previous work developed neuro-inspired convolutional neural
networks (CNNs) with sparse coding via the Locally Competitive Algorithm (LCA)
in the first layer (i.e., LCANets) for computer vision. LCANets learn in a
combination of supervised and unsupervised learning, reducing dependency on
labeled samples. Motivated by the fact that auditory cortex is also sparse, we
extend LCANets to audio recognition tasks and introduce LCANets++, which are
CNNs that perform sparse coding in multiple layers via LCA. We demonstrate that
LCANets++ are more robust than standard CNNs and LCANets against perturbations,
e.g., background noise, as well as black-box and white-box attacks, e.g.,
evasion and fast gradient sign (FGSM) attacks.


http://arxiv.org/abs/2308.12439
BaDExpert: Extracting Backdoor Functionality for Accurate Backdoor Input Detection. (74%)
Tinghao Xie; Xiangyu Qi; Ping He; Yiming Li; Jiachen T. Wang; Prateek Mittal
  We present a novel defense, against backdoor attacks on Deep Neural Networks
(DNNs), wherein adversaries covertly implant malicious behaviors (backdoors)
into DNNs. Our defense falls within the category of post-development defenses
that operate independently of how the model was generated. The proposed defense
is built upon a novel reverse engineering approach that can directly extract
backdoor functionality of a given backdoored model to a backdoor expert model.
The approach is straightforward -- finetuning the backdoored model over a small
set of intentionally mislabeled clean samples, such that it unlearns the normal
functionality while still preserving the backdoor functionality, and thus
resulting in a model (dubbed a backdoor expert model) that can only recognize
backdoor inputs. Based on the extracted backdoor expert model, we show the
feasibility of devising highly accurate backdoor input detectors that filter
out the backdoor inputs during model inference. Further augmented by an
ensemble strategy with a finetuned auxiliary model, our defense, BaDExpert
(Backdoor Input Detection with Backdoor Expert), effectively mitigates 17 SOTA
backdoor attacks while minimally impacting clean utility. The effectiveness of
BaDExpert has been verified on multiple datasets (CIFAR10, GTSRB and ImageNet)
across various model architectures (ResNet, VGG, MobileNetV2 and Vision
Transformer).


http://arxiv.org/abs/2308.12319
RemovalNet: DNN Fingerprint Removal Attacks. (69%)
Hongwei Yao; Zheng Li; Kunzhe Huang; Jian Lou; Zhan Qin; Kui Ren
  With the performance of deep neural networks (DNNs) remarkably improving,
DNNs have been widely used in many areas. Consequently, the DNN model has
become a valuable asset, and its intellectual property is safeguarded by
ownership verification techniques (e.g., DNN fingerprinting). However, the
feasibility of the DNN fingerprint removal attack and its potential influence
remains an open problem. In this paper, we perform the first comprehensive
investigation of DNN fingerprint removal attacks. Generally, the knowledge
contained in a DNN model can be categorized into general semantic and
fingerprint-specific knowledge. To this end, we propose a min-max bilevel
optimization-based DNN fingerprint removal attack named RemovalNet, to evade
model ownership verification. The lower-level optimization is designed to
remove fingerprint-specific knowledge. While in the upper-level optimization,
we distill the victim model's general semantic knowledge to maintain the
surrogate model's performance. We conduct extensive experiments to evaluate the
fidelity, effectiveness, and efficiency of the RemovalNet against four advanced
defense methods on six metrics. The empirical results demonstrate that (1) the
RemovalNet is effective. After our DNN fingerprint removal attack, the model
distance between the target and surrogate models is x100 times higher than that
of the baseline attacks, (2) the RemovalNet is efficient. It uses only 0.2%
(400 samples) of the substitute dataset and 1,000 iterations to conduct our
attack. Besides, compared with advanced model stealing attacks, the RemovalNet
saves nearly 85% of computational resources at most, (3) the RemovalNet
achieves high fidelity that the created surrogate model maintains high accuracy
after the DNN fingerprint removal process. Our code is available at:
https://github.com/grasses/RemovalNet.


http://arxiv.org/abs/2310.02164
Graph Unlearning: A Review. (2%)
Anwar Said; Tyler Derr; Mudassir Shabbir; Waseem Abbas; Xenofon Koutsoukos
  Graph unlearning emerges as a crucial advancement in the pursuit of
responsible AI, providing the means to remove sensitive data traces from
trained models, thereby upholding the right to be forgotten. It is evident that
graph machine learning exhibits sensitivity to data privacy and adversarial
attacks, necessitating the application of graph unlearning techniques to
address these concerns effectively. In this comprehensive survey paper, we
present the first systematic review of graph unlearning approaches,
encompassing a diverse array of methodologies and offering a detailed taxonomy
and up-to-date literature overview to facilitate the understanding of
researchers new to this field. Additionally, we establish the vital connections
between graph unlearning and differential privacy, augmenting our understanding
of the relevance of privacy-preserving techniques in this context. To ensure
clarity, we provide lucid explanations of the fundamental concepts and
evaluation measures used in graph unlearning, catering to a broader audience
with varying levels of expertise. Delving into potential applications, we
explore the versatility of graph unlearning across various domains, including
but not limited to social networks, adversarial settings, and
resource-constrained environments like the Internet of Things (IoT),
illustrating its potential impact in safeguarding data privacy and enhancing AI
systems' robustness. Finally, we shed light on promising research directions,
encouraging further progress and innovation within the domain of graph
unlearning. By laying a solid foundation and fostering continued progress, this
survey seeks to inspire researchers to further advance the field of graph
unlearning, thereby instilling confidence in the ethical growth of AI systems
and reinforcing the responsible application of machine learning techniques in
various domains.


http://arxiv.org/abs/2308.12065
Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers. (1%)
Tommaso Zoppi; Andrea Ceccarelli; Andrea Bondavalli
  Machine Learning (ML) algorithms that perform classification may predict the
wrong class, experiencing misclassifications. It is well-known that
misclassifications may have cascading effects on the encompassing system,
possibly resulting in critical failures. This paper proposes SPROUT, a Safety
wraPper thROugh ensembles of UncertainTy measures, which suspects
misclassifications by computing uncertainty measures on the inputs and outputs
of a black-box classifier. If a misclassification is detected, SPROUT blocks
the propagation of the output of the classifier to the encompassing system. The
resulting impact on safety is that SPROUT transforms erratic outputs
(misclassifications) into data omission failures, which can be easily managed
at the system level. SPROUT has a broad range of applications as it fits binary
and multi-class classification, comprising image and tabular datasets. We
experimentally show that SPROUT always identifies a huge fraction of the
misclassifications of supervised classifiers, and it is able to detect all
misclassifications in specific cases. SPROUT implementation contains
pre-trained wrappers, it is publicly available and ready to be deployed with
minimal effort.


http://arxiv.org/abs/2308.12141
Aparecium: Revealing Secrets from Physical Photographs. (1%)
Zhe Lei; Jie Zhang; Jingtao Li; Weiming Zhang; Nenghai Yu
  Watermarking is a crucial tool for safeguarding copyrights and can serve as a
more aesthetically pleasing alternative to QR codes. In recent years,
watermarking methods based on deep learning have proved superior robustness
against complex physical distortions than traditional watermarking methods.
However, they have certain limitations that render them less effective in
practice. For instance, current solutions necessitate physical photographs to
be rectangular for accurate localization, cannot handle physical bending or
folding, and require the hidden area to be completely captured at a close
distance and small angle. To overcome these challenges, we propose a novel deep
watermarking framework dubbed \textit{Aparecium}. Specifically, we preprocess
secrets (i.e., watermarks) into a pattern and then embed it into the cover
image, which is symmetrical to the final decoding-then-extracting process. To
capture the watermarked region from complex physical scenarios, a locator is
also introduced. Besides, we adopt a three-stage training strategy for training
convergence. Extensive experiments demonstrate that \textit{Aparecium} is not
only robust against different digital distortions, but also can resist various
physical distortions, such as screen-shooting and printing-shooting, even in
severe cases including different shapes, curvature, folding, incompleteness,
long distances, and big angles while maintaining high visual quality.
Furthermore, some ablation studies are also conducted to verify our design.


http://arxiv.org/abs/2308.11845
SEA: Shareable and Explainable Attribution for Query-based Black-box Attacks. (99%)
Yue Gao; Ilia Shumailov; Kassem Fawaz
  Machine Learning (ML) systems are vulnerable to adversarial examples,
particularly those from query-based black-box attacks. Despite various efforts
to detect and prevent such attacks, there is a need for a more comprehensive
approach to logging, analyzing, and sharing evidence of attacks. While classic
security benefits from well-established forensics and intelligence sharing,
Machine Learning is yet to find a way to profile its attackers and share
information about them. In response, this paper introduces SEA, a novel ML
security system to characterize black-box attacks on ML systems for forensic
purposes and to facilitate human-explainable intelligence sharing. SEA
leverages the Hidden Markov Models framework to attribute the observed query
sequence to known attacks. It thus understands the attack's progression rather
than just focusing on the final adversarial examples. Our evaluations reveal
that SEA is effective at attack attribution, even on their second occurrence,
and is robust to adaptive strategies designed to evade forensics analysis.
Interestingly, SEA's explanations of the attack behavior allow us even to
fingerprint specific minor implementation bugs in attack libraries. For
example, we discover that the SignOPT and Square attacks implementation in ART
v1.14 sends over 50% specific zero difference queries. We thoroughly evaluate
SEA on a variety of settings and demonstrate that it can recognize the same
attack's second occurrence with 90+% Top-1 and 95+% Top-3 accuracy.


http://arxiv.org/abs/2308.11754
Multi-Instance Adversarial Attack on GNN-Based Malicious Domain Detection. (99%)
Mahmoud Nazzal; Issa Khalil; Abdallah Khreishah; NhatHai Phan; Yao Ma
  Malicious domain detection (MDD) is an open security challenge that aims to
detect if an Internet domain is associated with cyber-attacks. Among many
approaches to this problem, graph neural networks (GNNs) are deemed highly
effective. GNN-based MDD uses DNS logs to represent Internet domains as nodes
in a maliciousness graph (DMG) and trains a GNN to infer their maliciousness by
leveraging identified malicious domains. Since this method relies on accessible
DNS logs to construct DMGs, it exposes a vulnerability for adversaries to
manipulate their domain nodes' features and connections within DMGs. Existing
research mainly concentrates on threat models that manipulate individual
attacker nodes. However, adversaries commonly generate multiple domains to
achieve their goals economically and avoid detection. Their objective is to
evade discovery across as many domains as feasible. In this work, we call the
attack that manipulates several nodes in the DMG concurrently a multi-instance
evasion attack. We present theoretical and empirical evidence that the existing
single-instance evasion techniques for are inadequate to launch multi-instance
evasion attacks against GNN-based MDDs. Therefore, we introduce MintA, an
inference-time multi-instance adversarial attack on GNN-based MDDs. MintA
enhances node and neighborhood evasiveness through optimized perturbations and
operates successfully with only black-box access to the target model,
eliminating the need for knowledge about the model's specifics or non-adversary
nodes. We formulate an optimization challenge for MintA, achieving an
approximate solution. Evaluating MintA on a leading GNN-based MDD technique
with real-world data showcases an attack success rate exceeding 80%. These
findings act as a warning for security experts, underscoring GNN-based MDDs'
susceptibility to practical attacks that can undermine their effectiveness and
benefits.


http://arxiv.org/abs/2308.11894
Does Physical Adversarial Example Really Matter to Autonomous Driving? Towards System-Level Effect of Adversarial Object Evasion Attack. (98%)
Ningfei Wang; Yunpeng Luo; Takami Sato; Kaidi Xu; Qi Alfred Chen
  In autonomous driving (AD), accurate perception is indispensable to achieving
safe and secure driving. Due to its safety-criticality, the security of AD
perception has been widely studied. Among different attacks on AD perception,
the physical adversarial object evasion attacks are especially severe. However,
we find that all existing literature only evaluates their attack effect at the
targeted AI component level but not at the system level, i.e., with the entire
system semantics and context such as the full AD pipeline. Thereby, this raises
a critical research question: can these existing researches effectively achieve
system-level attack effects (e.g., traffic rule violations) in the real-world
AD context? In this work, we conduct the first measurement study on whether and
how effectively the existing designs can lead to system-level effects,
especially for the STOP sign-evasion attacks due to their popularity and
severity. Our evaluation results show that all the representative prior works
cannot achieve any system-level effects. We observe two design limitations in
the prior works: 1) physical model-inconsistent object size distribution in
pixel sampling and 2) lack of vehicle plant model and AD system model
consideration. Then, we propose SysAdv, a novel system-driven attack design in
the AD context and our evaluation results show that the system-level effects
can be significantly improved, i.e., the violation rate increases by around
70%.


http://arxiv.org/abs/2308.11333
Protect Federated Learning Against Backdoor Attacks via Data-Free Trigger Generation. (86%)
Yanxin Yang; Ming Hu; Yue Cao; Jun Xia; Yihao Huang; Yang Liu; Mingsong Chen
  As a distributed machine learning paradigm, Federated Learning (FL) enables
large-scale clients to collaboratively train a model without sharing their raw
data. However, due to the lack of data auditing for untrusted clients, FL is
vulnerable to poisoning attacks, especially backdoor attacks. By using poisoned
data for local training or directly changing the model parameters, attackers
can easily inject backdoors into the model, which can trigger the model to make
misclassification of targeted patterns in images. To address these issues, we
propose a novel data-free trigger-generation-based defense approach based on
the two characteristics of backdoor attacks: i) triggers are learned faster
than normal knowledge, and ii) trigger patterns have a greater effect on image
classification than normal class patterns. Our approach generates the images
with newly learned knowledge by identifying the differences between the old and
new global models, and filters trigger images by evaluating the effect of these
generated images. By using these trigger images, our approach eliminates
poisoned models to ensure the updated global model is benign. Comprehensive
experiments demonstrate that our approach can defend against almost all the
existing types of backdoor attacks and outperform all the seven
state-of-the-art defense methods with both IID and non-IID scenarios.
Especially, our approach can successfully defend against the backdoor attack
even when 80\% of the clients are malicious.


http://arxiv.org/abs/2308.11443
Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging. (76%)
Xiaojun Jia; Yuefeng Chen; Xiaofeng Mao; Ranjie Duan; Jindong Gu; Rong Zhang; Hui Xue; Xiaochun Cao
  Fast Adversarial Training (FAT) not only improves the model robustness but
also reduces the training cost of standard adversarial training. However, fast
adversarial training often suffers from Catastrophic Overfitting (CO), which
results in poor robustness performance. Catastrophic Overfitting describes the
phenomenon of a sudden and significant decrease in robust accuracy during the
training of fast adversarial training. Many effective techniques have been
developed to prevent Catastrophic Overfitting and improve the model robustness
from different perspectives. However, these techniques adopt inconsistent
training settings and require different training costs, i.e, training time and
memory costs, leading to unfair comparisons. In this paper, we conduct a
comprehensive study of over 10 fast adversarial training methods in terms of
adversarial robustness and training costs. We revisit the effectiveness and
efficiency of fast adversarial training techniques in preventing Catastrophic
Overfitting from the perspective of model local nonlinearity and propose an
effective Lipschitz regularization method for fast adversarial training.
Furthermore, we explore the effect of data augmentation and weight averaging in
fast adversarial training and propose a simple yet effective auto weight
averaging method to improve robustness further. By assembling these techniques,
we propose a FGSM-based fast adversarial training method equipped with
Lipschitz regularization and Auto Weight averaging, abbreviated as FGSM-LAW.
Experimental evaluations on four benchmark databases demonstrate the
superiority of the proposed method over state-of-the-art fast adversarial
training methods and the advanced standard adversarial training methods.


http://arxiv.org/abs/2308.11406
Designing an attack-defense game: how to increase robustness of financial transaction models via a competition. (75%)
Alexey Zaytsev; Alex Natekin; Evgeni Vorsin; Valerii Smirnov; Oleg Sidorshin; Alexander Senin; Alexander Dudin; Dmitry Berestnev
  Given the escalating risks of malicious attacks in the finance sector and the
consequential severe damage, a thorough understanding of adversarial strategies
and robust defense mechanisms for machine learning models is critical. The
threat becomes even more severe with the increased adoption in banks more
accurate, but potentially fragile neural networks. We aim to investigate the
current state and dynamics of adversarial attacks and defenses for neural
network models that use sequential financial data as the input.
  To achieve this goal, we have designed a competition that allows realistic
and detailed investigation of problems in modern financial transaction data.
The participants compete directly against each other, so possible attacks and
defenses are examined in close-to-real-life conditions. Our main contributions
are the analysis of the competition dynamics that answers the questions on how
important it is to conceal a model from malicious users, how long does it take
to break it, and what techniques one should use to make it more robust, and
introduction additional way to attack models or increase their robustness.
  Our analysis continues with a meta-study on the used approaches with their
power, numerical experiments, and accompanied ablations studies. We show that
the developed attacks and defenses outperform existing alternatives from the
literature while being practical in terms of execution, proving the validity of
the competition as a tool for uncovering vulnerabilities of machine learning
models and mitigating them in various domains.


http://arxiv.org/abs/2308.11881
Adversarial Training Using Feedback Loops. (74%)
Ali Haisam Muhammad Rafid; Adrian Sandu
  Deep neural networks (DNN) have found wide applicability in numerous fields
due to their ability to accurately learn very complex input-output relations.
Despite their accuracy and extensive use, DNNs are highly susceptible to
adversarial attacks due to limited generalizability. For future progress in the
field, it is essential to build DNNs that are robust to any kind of
perturbations to the data points. In the past, many techniques have been
proposed to robustify DNNs using first-order derivative information of the
network.
  This paper proposes a new robustification approach based on control theory. A
neural network architecture that incorporates feedback control, named Feedback
Neural Networks, is proposed. The controller is itself a neural network, which
is trained using regular and adversarial data such as to stabilize the system
outputs. The novel adversarial training approach based on the feedback control
architecture is called Feedback Looped Adversarial Training (FLAT). Numerical
results on standard test problems empirically show that our FLAT method is more
effective than the state-of-the-art to guard against adversarial attacks.


http://arxiv.org/abs/2308.11284
LEAP: Efficient and Automated Test Method for NLP Software. (31%)
Mingxuan Xiao; Yan Xiao; Hai Dong; Shunhui Ji; Pengcheng Zhang
  The widespread adoption of DNNs in NLP software has highlighted the need for
robustness. Researchers proposed various automatic testing techniques for
adversarial test cases. However, existing methods suffer from two limitations:
weak error-discovering capabilities, with success rates ranging from 0% to
24.6% for BERT-based NLP software, and time inefficiency, taking 177.8s to
205.28s per test case, making them challenging for time-constrained scenarios.
To address these issues, this paper proposes LEAP, an automated test method
that uses LEvy flight-based Adaptive Particle swarm optimization integrated
with textual features to generate adversarial test cases. Specifically, we
adopt Levy flight for population initialization to increase the diversity of
generated test cases. We also design an inertial weight adaptive update
operator to improve the efficiency of LEAP's global optimization of
high-dimensional text examples and a mutation operator based on the greedy
strategy to reduce the search time. We conducted a series of experiments to
validate LEAP's ability to test NLP software and found that the average success
rate of LEAP in generating adversarial test cases is 79.1%, which is 6.1%
higher than the next best approach (PSOattack). While ensuring high success
rates, LEAP significantly reduces time overhead by up to 147.6s compared to
other heuristic-based methods. Additionally, the experimental results
demonstrate that LEAP can generate more transferable test cases and
significantly enhance the robustness of DNN-based systems.


http://arxiv.org/abs/2308.11822
PatchBackdoor: Backdoor Attack against Deep Neural Networks without Model Modification. (16%)
Yizhen Institute for AI Industry Research Yuan; Rui Shanghai Jiao Tong University, Shanghai, China Kong; Shenghao Wuhan University, Wuhan, China Xie; Yuanchun Institute for AI Industry Research Shanghai AI Laboratory, Shanghai, China Li; Yunxin Institute for AI Industry Research Shanghai AI Laboratory, Shanghai, China Liu
  Backdoor attack is a major threat to deep learning systems in safety-critical
scenarios, which aims to trigger misbehavior of neural network models under
attacker-controlled conditions. However, most backdoor attacks have to modify
the neural network models through training with poisoned data and/or direct
model editing, which leads to a common but false belief that backdoor attack
can be easily avoided by properly protecting the model. In this paper, we show
that backdoor attacks can be achieved without any model modification. Instead
of injecting backdoor logic into the training data or the model, we propose to
place a carefully-designed patch (namely backdoor patch) in front of the
camera, which is fed into the model together with the input images. The patch
can be trained to behave normally at most of the time, while producing wrong
prediction when the input image contains an attacker-controlled trigger object.
Our main techniques include an effective training method to generate the
backdoor patch and a digital-physical transformation modeling method to enhance
the feasibility of the patch in real deployments. Extensive experiments show
that PatchBackdoor can be applied to common deep learning models (VGG,
MobileNet, ResNet) with an attack success rate of 93% to 99% on classification
tasks. Moreover, we implement PatchBackdoor in real-world scenarios and show
that the attack is still threatening.


http://arxiv.org/abs/2308.10779
Spear and Shield: Adversarial Attacks and Defense Methods for Model-Based Link Prediction on Continuous-Time Dynamic Graphs. (99%)
Dongjin Lee; Juho Lee; Kijung Shin
  Real-world graphs are dynamic, constantly evolving with new interactions,
such as financial transactions in financial networks. Temporal Graph Neural
Networks (TGNNs) have been developed to effectively capture the evolving
patterns in dynamic graphs. While these models have demonstrated their
superiority, being widely adopted in various important fields, their
vulnerabilities against adversarial attacks remain largely unexplored. In this
paper, we propose T-SPEAR, a simple and effective adversarial attack method for
link prediction on continuous-time dynamic graphs, focusing on investigating
the vulnerabilities of TGNNs. Specifically, before the training procedure of a
victim model, which is a TGNN for link prediction, we inject edge perturbations
to the data that are unnoticeable in terms of the four constraints we propose,
and yet effective enough to cause malfunction of the victim model. Moreover, we
propose a robust training approach T-SHIELD to mitigate the impact of
adversarial attacks. By using edge filtering and enforcing temporal smoothness
to node embeddings, we enhance the robustness of the victim model. Our
experimental study shows that T-SPEAR significantly degrades the victim model's
performance on link prediction tasks, and even more, our attacks are
transferable to other TGNNs, which differ from the victim model assumed by the
attacker. Moreover, we demonstrate that T-SHIELD effectively filters out
adversarial edges and exhibits robustness against adversarial attacks,
surpassing the link prediction performance of the naive TGNN by up to 11.2%
under T-SPEAR.


http://arxiv.org/abs/2308.10601
Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer. (99%)
Zhijin Ge; Fanhua Shang; Hongying Liu; Yuanyuan Liu; Liang Wan; Wei Feng; Xiaosen Wang
  Deep neural networks are vulnerable to adversarial examples crafted by
applying human-imperceptible perturbations on clean inputs. Although many
attack methods can achieve high success rates in the white-box setting, they
also exhibit weak transferability in the black-box setting. Recently, various
methods have been proposed to improve adversarial transferability, in which the
input transformation is one of the most effective methods. In this work, we
notice that existing input transformation-based works mainly adopt the
transformed data in the same domain for augmentation. Inspired by domain
generalization, we aim to further improve the transferability using the data
augmented from different domains. Specifically, a style transfer network can
alter the distribution of low-level visual features in an image while
preserving semantic content for humans. Hence, we propose a novel attack method
named Style Transfer Method (STM) that utilizes a proposed arbitrary style
transfer network to transform the images into different domains. To avoid
inconsistent semantic information of stylized images for the classification
network, we fine-tune the style transfer network and mix up the generated
images added by random noise with the original images to maintain semantic
consistency and boost input diversity. Extensive experimental results on the
ImageNet-compatible dataset show that our proposed method can significantly
improve the adversarial transferability on either normally trained models or
adversarially trained models than state-of-the-art input transformation-based
attacks. Code is available at: https://github.com/Zhijin-Ge/STM.


http://arxiv.org/abs/2308.10743
Enhancing Adversarial Attacks: The Similar Target Method. (99%)
Shuo Zhang; Ziruo Wang; Zikai Zhou; Huanran Chen
  Deep neural networks are vulnerable to adversarial examples, posing a threat
to the models' applications and raising security concerns. An intriguing
property of adversarial examples is their strong transferability. Several
methods have been proposed to enhance transferability, including ensemble
attacks which have demonstrated their efficacy. However, prior approaches
simply average logits, probabilities, or losses for model ensembling, lacking a
comprehensive analysis of how and why model ensembling significantly improves
transferability. In this paper, we propose a similar targeted attack method
named Similar Target~(ST). By promoting cosine similarity between the gradients
of each model, our method regularizes the optimization direction to
simultaneously attack all surrogate models. This strategy has been proven to
enhance generalization ability. Experimental results on ImageNet validate the
effectiveness of our approach in improving adversarial transferability. Our
method outperforms state-of-the-art attackers on 18 discriminative classifiers
and adversarially trained models.


http://arxiv.org/abs/2308.11161
Adversarial Attacks on Code Models with Discriminative Graph Patterns. (96%)
Thanh-Dat Pick Nguyen; Yang Pick Zhou; Xuan Bach D. Pick Le; Pick Patanamon; Thongtanunam; David Lo
  Pre-trained language models of code are now widely used in various software
engineering tasks such as code generation, code completion, vulnerability
detection, etc. This, in turn, poses security and reliability risks to these
models. One of the important threats is \textit{adversarial attacks}, which can
lead to erroneous predictions and largely affect model performance on
downstream tasks. Current adversarial attacks on code models usually adopt
fixed sets of program transformations, such as variable renaming and dead code
insertion, leading to limited attack effectiveness. To address the
aforementioned challenges, we propose a novel adversarial attack framework,
GraphCodeAttack, to better evaluate the robustness of code models. Given a
target code model, GraphCodeAttack automatically mines important code patterns,
which can influence the model's decisions, to perturb the structure of input
code to the model. To do so, GraphCodeAttack uses a set of input source codes
to probe the model's outputs and identifies the \textit{discriminative} ASTs
patterns that can influence the model decisions. GraphCodeAttack then selects
appropriate AST patterns, concretizes the selected patterns as attacks, and
inserts them as dead code into the model's input program. To effectively
synthesize attacks from AST patterns, GraphCodeAttack uses a separate
pre-trained code model to fill in the ASTs with concrete code snippets. We
evaluate the robustness of two popular code models (e.g., CodeBERT and
GraphCodeBERT) against our proposed approach on three tasks: Authorship
Attribution, Vulnerability Prediction, and Clone Detection. The experimental
results suggest that our proposed approach significantly outperforms
state-of-the-art approaches in attacking code models such as CARROT and ALERT.


http://arxiv.org/abs/2308.11070
Temporal-Distributed Backdoor Attack Against Video Based Action Recognition. (88%)
Xi Li; Songhe Wang; Ruiquan Huang; Mahanth Gowda; George Kesidis
  Deep neural networks (DNNs) have achieved tremendous success in various
applications including video action recognition, yet remain vulnerable to
backdoor attacks (Trojans). The backdoor-compromised model will mis-classify to
the target class chosen by the attacker when a test instance (from a non-target
class) is embedded with a specific trigger, while maintaining high accuracy on
attack-free instances. Although there are extensive studies on backdoor attacks
against image data, the susceptibility of video-based systems under backdoor
attacks remains largely unexplored. Current studies are direct extensions of
approaches proposed for image data, e.g., the triggers are independently
embedded within the frames, which tend to be detectable by existing defenses.
In this paper, we introduce a simple yet effective backdoor attack against
video data. Our proposed attack, adding perturbations in a transformed domain,
plants an imperceptible, temporally distributed trigger across the video
frames, and is shown to be resilient to existing defensive strategies. The
effectiveness of the proposed attack is demonstrated by extensive experiments
with various well-known models on two video recognition benchmarks, UCF101 and
HMDB51, and a sign language recognition benchmark, Greek Sign Language (GSL)
dataset. We delve into the impact of several influential factors on our
proposed attack and identify an intriguing effect termed "collateral damage"
through extensive studies.


http://arxiv.org/abs/2308.10708
Measuring the Effect of Causal Disentanglement on the Adversarial Robustness of Neural Network Models. (76%)
Preben M. Ness; Dusica Marijan; Sunanda Bose
  Causal Neural Network models have shown high levels of robustness to
adversarial attacks as well as an increased capacity for generalisation tasks
such as few-shot learning and rare-context classification compared to
traditional Neural Networks. This robustness is argued to stem from the
disentanglement of causal and confounder input signals. However, no
quantitative study has yet measured the level of disentanglement achieved by
these types of causal models or assessed how this relates to their adversarial
robustness.
  Existing causal disentanglement metrics are not applicable to deterministic
models trained on real-world datasets. We, therefore, utilise metrics of
content/style disentanglement from the field of Computer Vision to measure
different aspects of the causal disentanglement for four state-of-the-art
causal Neural Network models. By re-implementing these models with a common
ResNet18 architecture we are able to fairly measure their adversarial
robustness on three standard image classification benchmarking datasets under
seven common white-box attacks. We find a strong association (r=0.820, p=0.001)
between the degree to which models decorrelate causal and confounder signals
and their adversarial robustness. Additionally, we find a moderate negative
association between the pixel-level information content of the confounder
signal and adversarial robustness (r=-0.597, p=0.040).


http://arxiv.org/abs/2308.10467
Single-User Injection for Invisible Shilling Attack against Recommender Systems. (62%)
Chengzhi Huang; Hui Li
  Recommendation systems (RS) are crucial for alleviating the information
overload problem. Due to its pivotal role in guiding users to make decisions,
unscrupulous parties are lured to launch attacks against RS to affect the
decisions of normal users and gain illegal profits. Among various types of
attacks, shilling attack is one of the most subsistent and profitable attacks.
In shilling attack, an adversarial party injects a number of well-designed fake
user profiles into the system to mislead RS so that the attack goal can be
achieved. Although existing shilling attack methods have achieved promising
results, they all adopt the attack paradigm of multi-user injection, where some
fake user profiles are required. This paper provides the first study of
shilling attack in an extremely limited scenario: only one fake user profile is
injected into the victim RS to launch shilling attacks (i.e., single-user
injection). We propose a novel single-user injection method SUI-Attack for
invisible shilling attack. SUI-Attack is a graph based attack method that
models shilling attack as a node generation task over the user-item bipartite
graph of the victim RS, and it constructs the fake user profile by generating
user features and edges that link the fake user to items. Extensive experiments
demonstrate that SUI-Attack can achieve promising attack results in single-user
injection. In addition to its attack power, SUI-Attack increases the
stealthiness of shilling attack and reduces the risk of being detected. We
provide our implementation at: https://github.com/KDEGroup/SUI-Attack.


http://arxiv.org/abs/2308.10741
On the Adversarial Robustness of Multi-Modal Foundation Models. (4%)
Christian Schlarmann; Matthias Hein
  Multi-modal foundation models combining vision and language models such as
Flamingo or GPT-4 have recently gained enormous interest. Alignment of
foundation models is used to prevent models from providing toxic or harmful
output. While malicious users have successfully tried to jailbreak foundation
models, an equally important question is if honest users could be harmed by
malicious third-party content. In this paper we show that imperceivable attacks
on images in order to change the caption output of a multi-modal foundation
model can be used by malicious content providers to harm honest users e.g. by
guiding them to malicious websites or broadcast fake information. This
indicates that countermeasures to adversarial attacks should be used by any
deployed multi-modal foundation model.


http://arxiv.org/abs/2308.10888
Unlocking Accuracy and Fairness in Differentially Private Image Classification. (2%)
Leonard Berrada; Soham De; Judy Hanwen Shen; Jamie Hayes; Robert Stanforth; David Stutz; Pushmeet Kohli; Samuel L. Smith; Borja Balle
  Privacy-preserving machine learning aims to train models on private data
without leaking sensitive information. Differential privacy (DP) is considered
the gold standard framework for privacy-preserving training, as it provides
formal privacy guarantees. However, compared to their non-private counterparts,
models trained with DP often have significantly reduced accuracy. Private
classifiers are also believed to exhibit larger performance disparities across
subpopulations, raising fairness concerns. The poor performance of classifiers
trained with DP has prevented the widespread adoption of privacy preserving
machine learning in industry. Here we show that pre-trained foundation models
fine-tuned with DP can achieve similar accuracy to non-private classifiers,
even in the presence of significant distribution shifts between pre-training
data and downstream tasks. We achieve private accuracies within a few percent
of the non-private state of the art across four datasets, including two medical
imaging benchmarks. Furthermore, our private medical classifiers do not exhibit
larger performance disparities across demographic groups than non-private
models. This milestone to make DP training a practical and reliable technology
has the potential to widely enable machine learning practitioners to train
safely on sensitive datasets while protecting individuals' privacy.


http://arxiv.org/abs/2308.10299
Boosting Adversarial Transferability by Block Shuffle and Rotation. (99%)
Kunyu Wang; Xuanran He; Wenxuan Wang; Xiaosen Wang
  Adversarial examples mislead deep neural networks with imperceptible
perturbations and have brought significant threats to deep learning. An
important aspect is their transferability, which refers to their ability to
deceive other models, thus enabling attacks in the black-box setting. Though
various methods have been proposed to boost transferability, the performance
still falls short compared with white-box attacks. In this work, we observe
that existing input transformation based attacks, one of the mainstream
transfer-based attacks, result in different attention heatmaps on various
models, which might limit the transferability. We also find that breaking the
intrinsic relation of the image can disrupt the attention heatmap of the
original image. Based on this finding, we propose a novel input transformation
based attack called block shuffle and rotation (BSR). Specifically, BSR splits
the input image into several blocks, then randomly shuffles and rotates these
blocks to construct a set of new images for gradient calculation. Empirical
evaluations on the ImageNet dataset demonstrate that BSR could achieve
significantly better transferability than the existing input transformation
based methods under single-model and ensemble-model settings. Combining BSR
with the current input transformation method can further improve the
transferability, which significantly outperforms the state-of-the-art methods.


http://arxiv.org/abs/2308.10315
Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting. (96%)
Qidong Huang; Xiaoyi Dong; Dongdong Chen; Yinpeng Chen; Lu Yuan; Gang Hua; Weiming Zhang; Nenghai Yu
  In this paper, we investigate the adversarial robustness of vision
transformers that are equipped with BERT pretraining (e.g., BEiT, MAE). A
surprising observation is that MAE has significantly worse adversarial
robustness than other BERT pretraining methods. This observation drives us to
rethink the basic differences between these BERT pretraining methods and how
these differences affect the robustness against adversarial perturbations. Our
empirical analysis reveals that the adversarial robustness of BERT pretraining
is highly related to the reconstruction target, i.e., predicting the raw pixels
of masked image patches will degrade more adversarial robustness of the model
than predicting the semantic context, since it guides the model to concentrate
more on medium-/high-frequency components of images. Based on our analysis, we
provide a simple yet effective way to boost the adversarial robustness of MAE.
The basic idea is using the dataset-extracted domain knowledge to occupy the
medium-/high-frequency of images, thus narrowing the optimization space of
adversarial perturbations. Specifically, we group the distribution of
pretraining data and optimize a set of cluster-specific visual prompts on
frequency domain. These prompts are incorporated with input images through
prototype-based prompt selection during test period. Extensive evaluation shows
that our method clearly boost MAE's adversarial robustness while maintaining
its clean performance on ImageNet-1k classification. Our code is available at:
https://github.com/shikiw/RobustMAE.


http://arxiv.org/abs/2308.10373
HoSNN: Adversarially-Robust Homeostatic Spiking Neural Networks with Adaptive Firing Thresholds. (96%)
Hejia Geng; Peng Li
  Spiking neural networks (SNNs) offer promise for efficient and powerful
neurally inspired computation. Common to other types of neural networks,
however, SNNs face the severe issue of vulnerability to adversarial attacks. We
present the first study that draws inspiration from neural homeostasis to
develop a bio-inspired solution that counters the susceptibilities of SNNs to
adversarial onslaughts. At the heart of our approach is a novel
threshold-adapting leaky integrate-and-fire (TA-LIF) neuron model, which we
adopt to construct the proposed adversarially robust homeostatic SNN (HoSNN).
Distinct from traditional LIF models, our TA-LIF model incorporates a
self-stabilizing dynamic thresholding mechanism, curtailing adversarial noise
propagation and safeguarding the robustness of HoSNNs in an unsupervised
manner. Theoretical analysis is presented to shed light on the stability and
convergence properties of the TA-LIF neurons, underscoring their superior
dynamic robustness under input distributional shifts over traditional LIF
neurons. Remarkably, without explicit adversarial training, our HoSNNs
demonstrate inherent robustness on CIFAR-10, with accuracy improvements to
72.6% and 54.19% against FGSM and PGD attacks, up from 20.97% and 0.6%,
respectively. Furthermore, with minimal FGSM adversarial training, our HoSNNs
surpass previous models by 29.99% under FGSM and 47.83% under PGD attacks on
CIFAR-10. Our findings offer a new perspective on harnessing biological
principles for bolstering SNNs adversarial robustness and defense, paving the
way to more resilient neuromorphic computing.


http://arxiv.org/abs/2308.10201
Hiding Backdoors within Event Sequence Data via Poisoning Attacks. (95%)
Elizaveta Kovtun; Alina Ermilova; Dmitry Berestnev; Alexey Zaytsev
  The financial industry relies on deep learning models for making important
decisions. This adoption brings new danger, as deep black-box models are known
to be vulnerable to adversarial attacks. In computer vision, one can shape the
output during inference by performing an adversarial attack called poisoning
via introducing a backdoor into the model during training. For sequences of
financial transactions of a customer, insertion of a backdoor is harder to
perform, as models operate over a more complex discrete space of sequences, and
systematic checks for insecurities occur. We provide a method to introduce
concealed backdoors, creating vulnerabilities without altering their
functionality for uncontaminated data. To achieve this, we replace a clean
model with a poisoned one that is aware of the availability of a backdoor and
utilize this knowledge. Our most difficult for uncovering attacks include
either additional supervised detection step of poisoned data activated during
the test or well-hidden model weight modifications. The experimental study
provides insights into how these effects vary across different datasets,
architectures, and model components. Alternative methods and baselines, such as
distillation-type regularization, are also explored but found to be less
efficient. Conducted on three open transaction datasets and architectures,
including LSTM, CNN, and Transformer, our findings not only illuminate the
vulnerabilities in contemporary models but also can drive the construction of
more robust systems.


http://arxiv.org/abs/2308.13541
Adversarial Collaborative Filtering for Free. (61%)
Huiyuan Chen; Xiaoting Li; Vivian Lai; Chin-Chia Michael Yeh; Yujie Fan; Yan Zheng; Mahashweta Das; Hao Yang
  Collaborative Filtering (CF) has been successfully used to help users
discover the items of interest. Nevertheless, existing CF methods suffer from
noisy data issue, which negatively impacts the quality of recommendation. To
tackle this problem, many prior studies leverage adversarial learning to
regularize the representations of users/items, which improves both
generalizability and robustness. Those methods often learn adversarial
perturbations and model parameters under min-max optimization framework.
However, there still have two major drawbacks: 1) Existing methods lack
theoretical guarantees of why adding perturbations improve the model
generalizability and robustness; 2) Solving min-max optimization is
time-consuming. In addition to updating the model parameters, each iteration
requires additional computations to update the perturbations, making them not
scalable for industry-scale datasets.
  In this paper, we present Sharpness-aware Collaborative Filtering (SharpCF),
a simple yet effective method that conducts adversarial training without extra
computational cost over the base optimizer. To achieve this goal, we first
revisit the existing adversarial collaborative filtering and discuss its
connection with recent Sharpness-aware Minimization. This analysis shows that
adversarial training actually seeks model parameters that lie in neighborhoods
around the optimal model parameters having uniformly low loss values, resulting
in better generalizability. To reduce the computational overhead, SharpCF
introduces a novel trajectory loss to measure the alignment between current
weights and past weights. Experimental results on real-world datasets
demonstrate that our SharpCF achieves superior performance with almost zero
additional computational cost comparing to adversarial training.


http://arxiv.org/abs/2308.10438
Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks. (1%)
Kaixin Xu; Zhe Wang; Xue Geng; Jie Lin; Min Wu; Xiaoli Li; Weisi Lin
  In this paper, we propose a novel layer-adaptive weight-pruning approach for
Deep Neural Networks (DNNs) that addresses the challenge of optimizing the
output distortion minimization while adhering to a target pruning ratio
constraint. Our approach takes into account the collective influence of all
layers to design a layer-adaptive pruning scheme. We discover and utilize a
very important additivity property of output distortion caused by pruning
weights on multiple layers. This property enables us to formulate the pruning
as a combinatorial optimization problem and efficiently solve it through
dynamic programming. By decomposing the problem into sub-problems, we achieve
linear time complexity, making our optimization algorithm fast and feasible to
run on CPUs. Our extensive experiments demonstrate the superiority of our
approach over existing methods on the ImageNet and CIFAR-10 datasets. On
CIFAR-10, our method achieves remarkable improvements, outperforming others by
up to 1.0% for ResNet-32, 0.5% for VGG-16, and 0.7% for DenseNet-121 in terms
of top-1 accuracy. On ImageNet, we achieve up to 4.7% and 4.6% higher top-1
accuracy compared to other methods for VGG-16 and ResNet-50, respectively.
These results highlight the effectiveness and practicality of our approach for
enhancing DNN performance through layer-adaptive weight pruning. Code will be
available on https://github.com/Akimoto-Cris/RD_VIT_PRUNE.


http://arxiv.org/abs/2308.10335
A Study on Robustness and Reliability of Large Language Model Code Generation. (1%)
Li Zhong; Zilong Wang
  Recently, the large language models (LLMs) have shown extraordinary ability
in understanding natural language and generating programming code. It has been
a common practice of software engineers to consult LLMs when encountering
coding questions. Although efforts have been made to avoid syntax errors and
align the code with the intended semantics, the reliability and robustness of
the code generationfrom LLMs have not yet been thoroughly studied. The
executable code is not equivalent to the reliable and robust code, especially
in the context of real-world software development. The misuse of APIs in the
generated code could lead to severe problem, such as resource leaks, program
crashes. To make things worse, the users of LLM code generation services are
actually the developers that are most vulnerable to these code that seems right
-- They are always novice developers that are not familiar with the APIs that
LLMs generate code for them. Therefore, they could hardly tell the misuse in
the code generated by LLMs, which further facilitates the incorrect code
applied in real-world software. Existing code evaluation benchmark and datasets
focus on crafting small tasks such as programming questions in coding
interviews, which however deviates from the problem that developers would ask
LLM for real-world coding help. To fill the missing piece, in this work, we
propose a dataset RobustAPI for evaluating the reliability and robustness of
code generated by LLMs. We collect 1208 coding questions from StackOverflow on
24 representative Java APIs. We summarize thecommon misuse patterns of these
APIs and evaluate them oncurrent popular LLMs. The evaluation results show that
evenfor GPT-4, 62% of the generated code contains API misuses,which would cause
unexpected consequences if the code isintroduced into real-world software.


http://arxiv.org/abs/2308.09958
A Comparison of Adversarial Learning Techniques for Malware Detection. (99%)
Pavla Louthánová; Matouš Kozák; Martin Jureček; Mark Stamp
  Machine learning has proven to be a useful tool for automated malware
detection, but machine learning models have also been shown to be vulnerable to
adversarial attacks. This article addresses the problem of generating
adversarial malware samples, specifically malicious Windows Portable Executable
files. We summarize and compare work that has focused on adversarial machine
learning for malware detection. We use gradient-based, evolutionary
algorithm-based, and reinforcement-based methods to generate adversarial
samples, and then test the generated samples against selected antivirus
products. We compare the selected methods in terms of accuracy and practical
applicability. The results show that applying optimized modifications to
previously detected malware can lead to incorrect classification of the file as
benign. It is also known that generated malware samples can be successfully
used against detection models other than those used to generate them and that
using combinations of generators can create new samples that evade detection.
Experiments show that the Gym-malware generator, which uses a reinforcement
learning approach, has the greatest practical potential. This generator
achieved an average sample generation time of 5.73 seconds and the highest
average evasion rate of 44.11%. Using the Gym-malware generator in combination
with itself improved the evasion rate to 58.35%.


http://arxiv.org/abs/2308.10110
Robust Mixture-of-Expert Training for Convolutional Neural Networks. (83%)
Yihua Zhang; Ruisi Cai; Tianlong Chen; Guanhua Zhang; Huan Zhang; Pin-Yu Chen; Shiyu Chang; Zhangyang Wang; Sijia Liu
  Sparsely-gated Mixture of Expert (MoE), an emerging deep model architecture,
has demonstrated a great promise to enable high-accuracy and ultra-efficient
model inference. Despite the growing popularity of MoE, little work
investigated its potential to advance convolutional neural networks (CNNs),
especially in the plane of adversarial robustness. Since the lack of robustness
has become one of the main hurdles for CNNs, in this paper we ask: How to
adversarially robustify a CNN-based MoE model? Can we robustly train it like an
ordinary CNN model? Our pilot study shows that the conventional adversarial
training (AT) mechanism (developed for vanilla CNNs) no longer remains
effective to robustify an MoE-CNN. To better understand this phenomenon, we
dissect the robustness of an MoE-CNN into two dimensions: Robustness of routers
(i.e., gating functions to select data-specific experts) and robustness of
experts (i.e., the router-guided pathways defined by the subnetworks of the
backbone CNN). Our analyses show that routers and experts are hard to adapt to
each other in the vanilla AT. Thus, we propose a new router-expert alternating
Adversarial training framework for MoE, termed AdvMoE. The effectiveness of our
proposal is justified across 4 commonly-used CNN model architectures over 4
benchmark datasets. We find that AdvMoE achieves 1% ~ 4% adversarial robustness
improvement over the original dense CNN, and enjoys the efficiency merit of
sparsity-gated MoE, leading to more than 50% inference cost reduction. Codes
are available at https://github.com/OPTML-Group/Robust-MoE-CNN.


http://arxiv.org/abs/2308.09861
Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method. (99%)
Yu-An Liu; Ruqing Zhang; Jiafeng Guo; Rijke Maarten de; Wei Chen; Yixing Fan; Xueqi Cheng
  Neural ranking models (NRMs) and dense retrieval (DR) models have given rise
to substantial improvements in overall retrieval performance. In addition to
their effectiveness, and motivated by the proven lack of robustness of deep
learning-based approaches in other areas, there is growing interest in the
robustness of deep learning-based approaches to the core retrieval problem.
Adversarial attack methods that have so far been developed mainly focus on
attacking NRMs, with very little attention being paid to the robustness of DR
models. In this paper, we introduce the adversarial retrieval attack (AREA)
task. The AREA task is meant to trick DR models into retrieving a target
document that is outside the initial set of candidate documents retrieved by
the DR model in response to a query. We consider the decision-based black-box
adversarial setting, which is realistic in real-world search engines. To
address the AREA task, we first employ existing adversarial attack methods
designed for NRMs. We find that the promising results that have previously been
reported on attacking NRMs, do not generalize to DR models: these methods
underperform a simple term spamming method. We attribute the observed lack of
generalizability to the interaction-focused architecture of NRMs, which
emphasizes fine-grained relevance matching. DR models follow a different
representation-focused architecture that prioritizes coarse-grained
representations. We propose to formalize attacks on DR models as a contrastive
learning problem in a multi-view representation space. The core idea is to
encourage the consistency between each view representation of the target
document and its corresponding viewer via view-wise supervision signals.
Experimental results demonstrate that the proposed method can significantly
outperform existing attack strategies in misleading the DR model with small
indiscernible text perturbations.


http://arxiv.org/abs/2308.09392
Attacking logo-based phishing website detectors with adversarial perturbations. (99%)
Jehyun Lee; Zhe Xin; Melanie Ng Pei See; Kanav Sabharwal; Giovanni Apruzzese; Dinil Mon Divakaran
  Recent times have witnessed the rise of anti-phishing schemes powered by deep
learning (DL). In particular, logo-based phishing detectors rely on DL models
from Computer Vision to identify logos of well-known brands on webpages, to
detect malicious webpages that imitate a given brand. For instance, Siamese
networks have demonstrated notable performance for these tasks, enabling the
corresponding anti-phishing solutions to detect even "zero-day" phishing
webpages. In this work, we take the next step of studying the robustness of
logo-based phishing detectors against adversarial ML attacks. We propose a
novel attack exploiting generative adversarial perturbations to craft
"adversarial logos" that evade phishing detectors. We evaluate our attacks
through: (i) experiments on datasets containing real logos, to evaluate the
robustness of state-of-the-art phishing detectors; and (ii) user studies to
gauge whether our adversarial logos can deceive human eyes. The results show
that our proposed attack is capable of crafting perturbed logos subtle enough
to evade various DL models-achieving an evasion rate of up to 95%. Moreover,
users are not able to spot significant differences between generated
adversarial logos and original ones.


http://arxiv.org/abs/2308.09546
Compensating Removed Frequency Components: Thwarting Voice Spectrum Reduction Attacks. (92%)
Shu Wang; Kun Sun; Qi Li
  Automatic speech recognition (ASR) provides diverse audio-to-text services
for humans to communicate with machines. However, recent research reveals ASR
systems are vulnerable to various malicious audio attacks. In particular, by
removing the non-essential frequency components, a new spectrum reduction
attack can generate adversarial audios that can be perceived by humans but
cannot be correctly interpreted by ASR systems. It raises a new challenge for
content moderation solutions to detect harmful content in audio and video
available on social media platforms. In this paper, we propose an acoustic
compensation system named ACE to counter the spectrum reduction attacks over
ASR systems. Our system design is based on two observations, namely, frequency
component dependencies and perturbation sensitivity. First, since the Discrete
Fourier Transform computation inevitably introduces spectral leakage and
aliasing effects to the audio frequency spectrum, the frequency components with
similar frequencies will have a high correlation. Thus, considering the
intrinsic dependencies between neighboring frequency components, it is possible
to recover more of the original audio by compensating for the removed
components based on the remaining ones. Second, since the removed components in
the spectrum reduction attacks can be regarded as an inverse of adversarial
noise, the attack success rate will decrease when the adversarial audio is
replayed in an over-the-air scenario. Hence, we can model the acoustic
propagation process to add over-the-air perturbations into the attacked audio.
We implement a prototype of ACE and the experiments show ACE can effectively
reduce up to 87.9% of ASR inference errors caused by spectrum reduction
attacks. Also, by analyzing residual errors, we summarize six general types of
ASR inference errors and investigate the error causes and potential mitigation
solutions.


http://arxiv.org/abs/2308.09487
Poison Dart Frog: A Clean-Label Attack with Low Poisoning Rate and High Attack Success Rate in the Absence of Training Data. (54%)
Binhao Ma; Jiahui Wang; Dejun Wang; Bo Meng
  To successfully launch backdoor attacks, injected data needs to be correctly
labeled; otherwise, they can be easily detected by even basic data filters.
Hence, the concept of clean-label attacks was introduced, which is more
dangerous as it doesn't require changing the labels of injected data. To the
best of our knowledge, the existing clean-label backdoor attacks largely relies
on an understanding of the entire training set or a portion of it. However, in
practice, it is very difficult for attackers to have it because of training
datasets often collected from multiple independent sources. Unlike all current
clean-label attacks, we propose a novel clean label method called 'Poison Dart
Frog'. Poison Dart Frog does not require access to any training data; it only
necessitates knowledge of the target class for the attack, such as 'frog'. On
CIFAR10, Tiny-ImageNet, and TSRD, with a mere 0.1\%, 0.025\%, and 0.4\%
poisoning rate of the training set size, respectively, Poison Dart Frog
achieves a high Attack Success Rate compared to LC, HTBA, BadNets, and Blend.
Furthermore, compared to the state-of-the-art attack, NARCISSUS, Poison Dart
Frog achieves similar attack success rates without any training data. Finally,
we demonstrate that four typical backdoor defense algorithms struggle to
counter Poison Dart Frog.


http://arxiv.org/abs/2308.09850
Backdoor Mitigation by Correcting the Distribution of Neural Activations. (11%)
Xi Li; Zhen Xiang; David J. Miller; George Kesidis
  Backdoor (Trojan) attacks are an important type of adversarial exploit
against deep neural networks (DNNs), wherein a test instance is (mis)classified
to the attacker's target class whenever the attacker's backdoor trigger is
present. In this paper, we reveal and analyze an important property of backdoor
attacks: a successful attack causes an alteration in the distribution of
internal layer activations for backdoor-trigger instances, compared to that for
clean instances. Even more importantly, we find that instances with the
backdoor trigger will be correctly classified to their original source classes
if this distribution alteration is corrected. Based on our observations, we
propose an efficient and effective method that achieves post-training backdoor
mitigation by correcting the distribution alteration using reverse-engineered
triggers. Notably, our method does not change any trainable parameters of the
DNN, but achieves generally better mitigation performance than existing methods
that do require intensive DNN parameter tuning. It also efficiently detects
test instances with the trigger, which may help to catch adversarial entities
in the act of exploiting the backdoor.


http://arxiv.org/abs/2308.09381
On Gradient-like Explanation under a Black-box Setting: When Black-box Explanations Become as Good as White-box. (9%)
Yi Cai; Gerhard Wunder
  Attribution methods shed light on the explainability of data-driven
approaches such as deep learning models by revealing the most contributing
features to decisions that have been made. A widely accepted way of deriving
feature attributions is to analyze the gradients of the target function with
respect to input features. Analysis of gradients requires full access to the
target system, meaning that solutions of this kind treat the target system as a
white-box. However, the white-box assumption may be untenable due to security
and safety concerns, thus limiting their practical applications. As an answer
to the limited flexibility, this paper presents GEEX (gradient-estimation-based
explanation), an explanation method that delivers gradient-like explanations
under a black-box setting. Furthermore, we integrate the proposed method with a
path method. The resulting approach iGEEX (integrated GEEX) satisfies the four
fundamental axioms of attribution methods: sensitivity, insensitivity,
implementation invariance, and linearity. With a focus on image data, the
exhaustive experiments empirically show that the proposed methods outperform
state-of-the-art black-box methods and achieve competitive performance compared
to the ones with full access.


http://arxiv.org/abs/2308.09318
Towards Attack-tolerant Federated Learning via Critical Parameter Analysis. (9%)
Sungwon Han; Sungwon Park; Fangzhao Wu; Sundong Kim; Bin Zhu; Xing Xie; Meeyoung Cha
  Federated learning is used to train a shared model in a decentralized way
without clients sharing private data with each other. Federated learning
systems are susceptible to poisoning attacks when malicious clients send false
updates to the central server. Existing defense strategies are ineffective
under non-IID data settings. This paper proposes a new defense strategy, FedCPA
(Federated learning with Critical Parameter Analysis). Our attack-tolerant
aggregation method is based on the observation that benign local models have
similar sets of top-k and bottom-k critical parameters, whereas poisoned local
models do not. Experiments with different attack scenarios on multiple datasets
demonstrate that our model outperforms existing defense strategies in defending
against poisoning attacks.


http://arxiv.org/abs/2308.09448
Defending Label Inference Attacks in Split Learning under Regression Setting. (4%)
Haoze Qiu; Fei Zheng; Chaochao Chen; Xiaolin Zheng
  As a privacy-preserving method for implementing Vertical Federated Learning,
Split Learning has been extensively researched. However, numerous studies have
indicated that the privacy-preserving capability of Split Learning is
insufficient. In this paper, we primarily focus on label inference attacks in
Split Learning under regression setting, which are mainly implemented through
the gradient inversion method. To defend against label inference attacks, we
propose Random Label Extension (RLE), where labels are extended to obfuscate
the label information contained in the gradients, thereby preventing the
attacker from utilizing gradients to train an attack model that can infer the
original labels. To further minimize the impact on the original task, we
propose Model-based adaptive Label Extension (MLE), where original labels are
preserved in the extended labels and dominate the training process. The
experimental results show that compared to the basic defense methods, our
proposed defense methods can significantly reduce the attack model's
performance while preserving the original task's performance.


http://arxiv.org/abs/2308.09810
An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software. (1%)
Wenxuan Wang; Jingyuan Huang; Jen-tse Huang; Chang Chen; Jiazhen Gu; Pinjia He; Michael R. Lyu
  The exponential growth of social media platforms has brought about a
revolution in communication and content dissemination in human society.
Nevertheless, these platforms are being increasingly misused to spread toxic
content, including hate speech, malicious advertising, and pornography, leading
to severe negative consequences such as harm to teenagers' mental health.
Despite tremendous efforts in developing and deploying textual and image
content moderation methods, malicious users can evade moderation by embedding
texts into images, such as screenshots of the text, usually with some
interference. We find that modern content moderation software's performance
against such malicious inputs remains underexplored. In this work, we propose
OASIS, a metamorphic testing framework for content moderation software. OASIS
employs 21 transform rules summarized from our pilot study on 5,000 real-world
toxic contents collected from 4 popular social media applications, including
Twitter, Instagram, Sina Weibo, and Baidu Tieba. Given toxic textual contents,
OASIS can generate image test cases, which preserve the toxicity yet are likely
to bypass moderation. In the evaluation, we employ OASIS to test five
commercial textual content moderation software from famous companies (i.e.,
Google Cloud, Microsoft Azure, Baidu Cloud, Alibaba Cloud and Tencent Cloud),
as well as a state-of-the-art moderation research model. The results show that
OASIS achieves up to 100% error finding rates. Moreover, through retraining the
models with the test cases generated by OASIS, the robustness of the moderation
model can be improved without performance degradation.


http://arxiv.org/abs/2308.09520
Proceedings of the 2nd International Workshop on Adaptive Cyber Defense. (1%)
Marco Carvalho; Damian Marriott; Mark Bilinski; Ahmad Ridley
  The 2nd International Workshop on Adaptive Cyber Defense was held at the
Florida Institute of Technology, Florida. This workshop was organized to share
research that explores unique applications of Artificial Intelligence (AI) and
Machine Learning (ML) as foundational capabilities for the pursuit of adaptive
cyber defense. The cyber domain cannot currently be reliably and effectively
defended without extensive reliance on human experts. Skilled cyber defenders
are in short supply and often cannot respond fast enough to cyber threats.
  Building on recent advances in AI and ML the Cyber defense research community
has been motivated to develop new dynamic and sustainable defenses through the
adoption of AI and ML techniques to cyber settings. Bridging critical gaps
between AI and Cyber researchers and practitioners can accelerate efforts to
create semi-autonomous cyber defenses that can learn to recognize and respond
to cyber attacks or discover and mitigate weaknesses in cooperation with other
cyber operation systems and human experts. Furthermore, these defenses are
expected to be adaptive and able to evolve over time to thwart changes in
attacker behavior, changes in the system health and readiness, and natural
shifts in user behavior over time.
  The workshop was comprised of invited keynote talks, technical presentations
and a panel discussion about how AI/ML can enable autonomous mitigation of
current and future cyber attacks. Workshop submissions were peer reviewed by a
panel of domain experts with a proceedings consisting of six technical articles
exploring challenging problems of critical importance to national and global
security. Participation in this workshop offered new opportunities to stimulate
research and innovation in the emerging domain of adaptive and autonomous cyber
defense.


http://arxiv.org/abs/2308.08906
Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing. (99%)
Daniel Gibert; Giulio Zizzo; Quan Le
  Malware detectors based on deep learning (DL) have been shown to be
susceptible to malware examples that have been deliberately manipulated in
order to evade detection, a.k.a. adversarial malware examples. More
specifically, it has been show that deep learning detectors are vulnerable to
small changes on the input file. Given this vulnerability of deep learning
detectors, we propose a practical defense against adversarial malware examples
inspired by randomized smoothing. In our work, instead of employing Gaussian or
Laplace noise when randomizing inputs, we propose a randomized ablation-based
smoothing scheme that ablates a percentage of the bytes within an executable.
During training, our randomized ablation-based smoothing scheme trains a base
classifier based on ablated versions of the executable files. At test time, the
final classification for a given input executable is taken as the class most
commonly predicted by the classifier on a set of ablated versions of the
original executable. To demonstrate the suitability of our approach we have
empirically evaluated the proposed ablation-based model against various
state-of-the-art evasion attacks on the BODMAS dataset. Results show greater
robustness and generalization capabilities to adversarial malware examples in
comparison to a non-smoothed classifier.


http://arxiv.org/abs/2309.16706
AIR: Threats of Adversarial Attacks on Deep Learning-Based Information Recovery. (99%)
Jinyin Chen; Jie Ge; Shilian Zheng; Linhui Ye; Haibin Zheng; Weiguo Shen; Keqiang Yue; Xiaoniu Yang
  A wireless communications system usually consists of a transmitter which
transmits the information and a receiver which recovers the original
information from the received distorted signal. Deep learning (DL) has been
used to improve the performance of the receiver in complicated channel
environments and state-of-the-art (SOTA) performance has been achieved.
However, its robustness has not been investigated. In order to evaluate the
robustness of DL-based information recovery models under adversarial
circumstances, we investigate adversarial attacks on the SOTA DL-based
information recovery model, i.e., DeepReceiver. We formulate the problem as an
optimization problem with power and peak-to-average power ratio (PAPR)
constraints. We design different adversarial attack methods according to the
adversary's knowledge of DeepReceiver's model and/or testing samples. Extensive
experiments show that the DeepReceiver is vulnerable to the designed attack
methods in all of the considered scenarios. Even in the scenario of both model
and test sample restricted, the adversary can attack the DeepReceiver and
increase its bit error rate (BER) above 10%. It can also be found that the
DeepReceiver is vulnerable to adversarial perturbations even with very low
power and limited PAPR. These results suggest that defense measures should be
taken to enhance the robustness of DeepReceiver.


http://arxiv.org/abs/2308.08925
A White-Box False Positive Adversarial Attack Method on Contrastive Loss-Based Offline Handwritten Signature Verification Models. (98%)
Zhongliang Guo; Yifei Qian; Ognjen Arandjelović; Lei Fang
  In this paper, we tackle the challenge of white-box false positive
adversarial attacks on contrastive loss-based offline handwritten signature
verification models. We propose a novel attack method that treats the attack as
a style transfer between closely related but distinct writing styles. To guide
the generation of deceptive images, we introduce two new loss functions that
enhance the attack success rate by perturbing the Euclidean distance between
the embedding vectors of the original and synthesized samples, while ensuring
minimal perturbations by reducing the difference between the generated image
and the original image. Our method demonstrates state-of-the-art performance in
white-box attacks on contrastive loss-based offline handwritten signature
verification models, as evidenced by our experiments. The key contributions of
this paper include a novel false positive attack method, two new loss
functions, effective style transfer in handwriting styles, and superior
performance in white-box false positive attacks compared to other white-box
attack methods.


http://arxiv.org/abs/2308.08938
Causal Adversarial Perturbations for Individual Fairness and Robustness in Heterogeneous Data Spaces. (16%)
Ahmad-Reza Ehyaei; Kiarash Mohammadi; Amir-Hossein Karimi; Samira Samadi; Golnoosh Farnadi
  As responsible AI gains importance in machine learning algorithms, properties
such as fairness, adversarial robustness, and causality have received
considerable attention in recent years. However, despite their individual
significance, there remains a critical gap in simultaneously exploring and
integrating these properties. In this paper, we propose a novel approach that
examines the relationship between individual fairness, adversarial robustness,
and structural causal models in heterogeneous data spaces, particularly when
dealing with discrete sensitive attributes. We use causal structural models and
sensitive attributes to create a fair metric and apply it to measure semantic
similarity among individuals. By introducing a novel causal adversarial
perturbation and applying adversarial training, we create a new regularizer
that combines individual fairness, causality, and robustness in the classifier.
Our method is evaluated on both real-world and synthetic datasets,
demonstrating its effectiveness in achieving an accurate classifier that
simultaneously exhibits fairness, adversarial robustness, and causal awareness.


http://arxiv.org/abs/2308.09146
That Doesn't Go There: Attacks on Shared State in Multi-User Augmented Reality Applications. (10%)
Carter Slocum; Yicheng Zhang; Erfan Shayegani; Pedram Zaree; Nael Abu-Ghazaleh; Jiasi Chen
  Augmented Reality (AR) is expected to become a pervasive component in
enabling shared virtual experiences. In order to facilitate collaboration among
multiple users, it is crucial for multi-user AR applications to establish a
consensus on the "shared state" of the virtual world and its augmentations,
through which they interact within augmented reality spaces. Current methods to
create and access shared state collect sensor data from devices (e.g., camera
images), process them, and integrate them into the shared state. However, this
process introduces new vulnerabilities and opportunities for attacks.
Maliciously writing false data to "poison" the shared state is a major concern
for the security of the downstream victims that depend on it. Another type of
vulnerability arises when reading the shared state; by providing false inputs,
an attacker can view hologram augmentations at locations they are not allowed
to access. In this work, we demonstrate a series of novel attacks on multiple
AR frameworks with shared states, focusing on three publicly-accessible
frameworks. We show that these frameworks, while using different underlying
implementations, scopes, and mechanisms to read from and write to the shared
state, have shared vulnerability to a unified threat model. Our evaluation of
these state-of-art AR applications demonstrates reliable attacks both on
updating and accessing shared state across the different systems. To defend
against such threats, we discuss a number of potential mitigation strategies
that can help enhance the security of multi-user AR applications.


http://arxiv.org/abs/2308.10819
Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection. (10%)
Zekun Li; Baolin Peng; Pengcheng He; Xifeng Yan
  Large Language Models (LLMs) have shown remarkable proficiency in following
instructions, making them valuable in customer-facing applications. However,
their impressive capabilities also raise concerns about the amplification of
risks posed by adversarial instructions, which can be injected into the model
input by third-party attackers to manipulate LLMs' original instructions and
prompt unintended actions and content. Therefore, it is crucial to understand
LLMs' ability to accurately discern which instructions to follow to ensure
their safe deployment in real-world scenarios. In this paper, we propose a
pioneering benchmark for automatically evaluating the robustness of
instruction-following LLMs against adversarial instructions injected in the
prompt. The objective of this benchmark is to quantify the extent to which LLMs
are influenced by injected adversarial instructions and assess their ability to
differentiate between these injected adversarial instructions and original user
instructions. Through experiments conducted with state-of-the-art
instruction-following LLMs, we uncover significant limitations in their
robustness against adversarial instruction injection attacks. Furthermore, our
findings indicate that prevalent instruction-tuned models are prone to being
``overfitted'' to follow any instruction phrase in the prompt without truly
understanding which instructions should be followed. This highlights the need
to address the challenge of training models to comprehend prompts instead of
merely following instruction phrases and completing the text. The data and code
can be found at \url{https://github.com/Leezekun/Adv-Instruct-Eval}.


http://arxiv.org/abs/2309.16710
General Lipschitz: Certified Robustness Against Resolvable Semantic Transformations via Transformation-Dependent Randomized Smoothing. (3%)
Dmitrii Korzh; Mikhail Pautov; Olga Tsymboi; Ivan Oseledets
  Randomized smoothing is the state-of-the-art approach to construct image
classifiers that are provably robust against additive adversarial perturbations
of bounded magnitude. However, it is more complicated to construct reasonable
certificates against semantic transformation (e.g., image blurring,
translation, gamma correction) and their compositions. In this work, we propose
\emph{General Lipschitz (GL),} a new framework to certify neural networks
against composable resolvable semantic perturbations. Within the framework, we
analyze transformation-dependent Lipschitz-continuity of smoothed classifiers
w.r.t. transformation parameters and derive corresponding robustness
certificates. Our method performs comparably to state-of-the-art approaches on
the ImageNet dataset.


http://arxiv.org/abs/2308.08160
Benchmarking Adversarial Robustness of Compressed Deep Learning Models. (81%)
Brijesh Vora; Kartik Patwari; Syed Mahbub Hafiz; Zubair Shafiq; Chen-Nee Chuah
  The increasing size of Deep Neural Networks (DNNs) poses a pressing need for
model compression, particularly when employed on resource constrained devices.
Concurrently, the susceptibility of DNNs to adversarial attacks presents
another significant hurdle. Despite substantial research on both model
compression and adversarial robustness, their joint examination remains
underexplored. Our study bridges this gap, seeking to understand the effect of
adversarial inputs crafted for base models on their pruned versions. To examine
this relationship, we have developed a comprehensive benchmark across diverse
adversarial attacks and popular DNN models. We uniquely focus on models not
previously exposed to adversarial training and apply pruning schemes optimized
for accuracy and performance. Our findings reveal that while the benefits of
pruning enhanced generalizability, compression, and faster inference times are
preserved, adversarial robustness remains comparable to the base model. This
suggests that model compression while offering its unique advantages, does not
undermine adversarial robustness.


http://arxiv.org/abs/2308.08505
Test-Time Poisoning Attacks Against Test-Time Adaptation Models. (73%)
Tianshuo Cong; Xinlei He; Yun Shen; Yang Zhang
  Deploying machine learning (ML) models in the wild is challenging as it
suffers from distribution shifts, where the model trained on an original domain
cannot generalize well to unforeseen diverse transfer domains. To address this
challenge, several test-time adaptation (TTA) methods have been proposed to
improve the generalization ability of the target pre-trained models under test
data to cope with the shifted distribution. The success of TTA can be credited
to the continuous fine-tuning of the target model according to the
distributional hint from the test samples during test time. Despite being
powerful, it also opens a new attack surface, i.e., test-time poisoning
attacks, which are substantially different from previous poisoning attacks that
occur during the training time of ML models (i.e., adversaries cannot intervene
in the training process). In this paper, we perform the first test-time
poisoning attack against four mainstream TTA methods, including TTT, DUA, TENT,
and RPL. Concretely, we generate poisoned samples based on the surrogate models
and feed them to the target TTA models. Experimental results show that the TTA
methods are generally vulnerable to test-time poisoning attacks. For instance,
the adversary can feed as few as 10 poisoned samples to degrade the performance
of the target model from 76.20% to 41.83%. Our results demonstrate that TTA
algorithms lacking a rigorous security assessment are unsuitable for deployment
in real-life scenarios. As such, we advocate for the integration of defenses
against test-time poisoning attacks into the design of TTA methods.


http://arxiv.org/abs/2308.11521
Self-Deception: Reverse Penetrating the Semantic Firewall of Large Language Models. (67%)
Zhenhua Wang; Wei Xie; Kai Chen; Baosheng Wang; Zhiwen Gui; Enze Wang
  Large language models (LLMs), such as ChatGPT, have emerged with astonishing
capabilities approaching artificial general intelligence. While providing
convenience for various societal needs, LLMs have also lowered the cost of
generating harmful content. Consequently, LLM developers have deployed
semantic-level defenses to recognize and reject prompts that may lead to
inappropriate content. Unfortunately, these defenses are not foolproof, and
some attackers have crafted "jailbreak" prompts that temporarily hypnotize the
LLM into forgetting content defense rules and answering any improper questions.
To date, there is no clear explanation of the principles behind these
semantic-level attacks and defenses in both industry and academia.
  This paper investigates the LLM jailbreak problem and proposes an automatic
jailbreak method for the first time. We propose the concept of a semantic
firewall and provide three technical implementation approaches. Inspired by the
attack that penetrates traditional firewalls through reverse tunnels, we
introduce a "self-deception" attack that can bypass the semantic firewall by
inducing LLM to generate prompts that facilitate jailbreak. We generated a
total of 2,520 attack payloads in six languages (English, Russian, French,
Spanish, Chinese, and Arabic) across seven virtual scenarios, targeting the
three most common types of violations: violence, hate, and pornography. The
experiment was conducted on two models, namely the GPT-3.5-Turbo and GPT-4. The
success rates on the two models were 86.2% and 67%, while the failure rates
were 4.7% and 2.2%, respectively. This highlighted the effectiveness of the
proposed attack method. All experimental code and raw data will be released as
open-source to inspire future research. We believe that manipulating AI
behavior through carefully crafted prompts will become an important research
direction in the future.


http://arxiv.org/abs/2308.08709
Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks. (61%)
Mirazul Haque; Wei Yang
  Deep Neural Networks (DNNs) have been used to solve different day-to-day
problems. Recently, DNNs have been deployed in real-time systems, and lowering
the energy consumption and response time has become the need of the hour. To
address this scenario, researchers have proposed incorporating dynamic
mechanism to static DNNs (SDNN) to create Dynamic Neural Networks (DyNNs)
performing dynamic amounts of computation based on the input complexity.
Although incorporating dynamic mechanism into SDNNs would be preferable in
real-time systems, it also becomes important to evaluate how the introduction
of dynamic mechanism impacts the robustness of the models. However, there has
not been a significant number of works focusing on the robustness trade-off
between SDNNs and DyNNs. To address this issue, we propose to investigate the
robustness of dynamic mechanism in DyNNs and how dynamic mechanism design
impacts the robustness of DyNNs. For that purpose, we evaluate three research
questions. These evaluations are performed on three models and two datasets.
Through the studies, we find that attack transferability from DyNNs to SDNNs is
higher than attack transferability from SDNNs to DyNNs. Also, we find that
DyNNs can be used to generate adversarial samples more efficiently than SDNNs.
Then, through research studies, we provide insight into the design choices that
can increase robustness of DyNNs against the attack generated using static
model. Finally, we propose a novel attack to understand the additional attack
surface introduced by the dynamic mechanism and provide design choices to
improve robustness against the attack.


http://arxiv.org/abs/2308.08173
Expressivity of Graph Neural Networks Through the Lens of Adversarial Robustness. (33%)
Francesco Campi; Lukas Gosch; Tom Wollschläger; Yan Scholten; Stephan Günnemann
  We perform the first adversarial robustness study into Graph Neural Networks
(GNNs) that are provably more powerful than traditional Message Passing Neural
Networks (MPNNs). In particular, we use adversarial robustness as a tool to
uncover a significant gap between their theoretically possible and empirically
achieved expressive power. To do so, we focus on the ability of GNNs to count
specific subgraph patterns, which is an established measure of expressivity,
and extend the concept of adversarial robustness to this task. Based on this,
we develop efficient adversarial attacks for subgraph counting and show that
more powerful GNNs fail to generalize even to small perturbations to the
graph's structure. Expanding on this, we show that such architectures also fail
to count substructures on out-of-distribution graphs.


http://arxiv.org/abs/2308.07874
SEDA: Self-Ensembling ViT with Defensive Distillation and Adversarial Training for robust Chest X-rays Classification. (99%)
Raza Imam; Ibrahim Almakky; Salma Alrashdi; Baketah Alrashdi; Mohammad Yaqub
  Deep Learning methods have recently seen increased adoption in medical
imaging applications. However, elevated vulnerabilities have been explored in
recent Deep Learning solutions, which can hinder future adoption. Particularly,
the vulnerability of Vision Transformer (ViT) to adversarial, privacy, and
confidentiality attacks raise serious concerns about their reliability in
medical settings. This work aims to enhance the robustness of self-ensembling
ViTs for the tuberculosis chest x-ray classification task. We propose
Self-Ensembling ViT with defensive Distillation and Adversarial training
(SEDA). SEDA utilizes efficient CNN blocks to learn spatial features with
various levels of abstraction from feature representations extracted from
intermediate ViT blocks, that are largely unaffected by adversarial
perturbations. Furthermore, SEDA leverages adversarial training in combination
with defensive distillation for improved robustness against adversaries.
Training using adversarial examples leads to better model generalizability and
improves its ability to handle perturbations. Distillation using soft
probabilities introduces uncertainty and variation into the output
probabilities, making it more difficult for adversarial and privacy attacks.
Extensive experiments performed with the proposed architecture and training
paradigm on publicly available Tuberculosis x-ray dataset shows SOTA efficacy
of SEDA compared to SEViT in terms of computational efficiency with 70x times
lighter framework and enhanced robustness of +9%.


http://arxiv.org/abs/2308.07625
Backpropagation Path Search On Adversarial Transferability. (99%)
Zhuoer Xu; Zhangxuan Gu; Jianping Zhang; Shiwen Cui; Changhua Meng; Weiqiang Wang
  Deep neural networks are vulnerable to adversarial examples, dictating the
imperativeness to test the model's robustness before deployment. Transfer-based
attackers craft adversarial examples against surrogate models and transfer them
to victim models deployed in the black-box situation. To enhance the
adversarial transferability, structure-based attackers adjust the
backpropagation path to avoid the attack from overfitting the surrogate model.
However, existing structure-based attackers fail to explore the convolution
module in CNNs and modify the backpropagation graph heuristically, leading to
limited effectiveness. In this paper, we propose backPropagation pAth Search
(PAS), solving the aforementioned two problems. We first propose SkipConv to
adjust the backpropagation path of convolution by structural
reparameterization. To overcome the drawback of heuristically designed
backpropagation paths, we further construct a DAG-based search space, utilize
one-step approximation for path evaluation and employ Bayesian Optimization to
search for the optimal path. We conduct comprehensive experiments in a wide
range of transfer settings, showing that PAS improves the attack success rate
by a huge margin for both normally trained and defense models.


http://arxiv.org/abs/2308.07673
A Review of Adversarial Attacks in Computer Vision. (99%)
Yutong Zhang; Yao Li; Yin Li; Zhichang Guo
  Deep neural networks have been widely used in various downstream tasks,
especially those safety-critical scenario such as autonomous driving, but deep
networks are often threatened by adversarial samples. Such adversarial attacks
can be invisible to human eyes, but can lead to DNN misclassification, and
often exhibits transferability between deep learning and machine learning
models and real-world achievability. Adversarial attacks can be divided into
white-box attacks, for which the attacker knows the parameters and gradient of
the model, and black-box attacks, for the latter, the attacker can only obtain
the input and output of the model. In terms of the attacker's purpose, it can
be divided into targeted attacks and non-targeted attacks, which means that the
attacker wants the model to misclassify the original sample into the specified
class, which is more practical, while the non-targeted attack just needs to
make the model misclassify the sample. The black box setting is a scenario we
will encounter in practice.


http://arxiv.org/abs/2308.07847
Robustness Over Time: Understanding Adversarial Examples' Effectiveness on Longitudinal Versions of Large Language Models. (95%)
Yugeng Liu; Tianshuo Cong; Zhengyu Zhao; Michael Backes; Yun Shen; Yang Zhang
  Large Language Models (LLMs) have led to significant improvements in many
tasks across various domains, such as code interpretation, response generation,
and ambiguity handling. These LLMs, however, when upgrading, primarily
prioritize enhancing user experience while neglecting security, privacy, and
safety implications. Consequently, unintended vulnerabilities or biases can be
introduced. Previous studies have predominantly focused on specific versions of
the models and disregard the potential emergence of new attack vectors
targeting the updated versions. Through the lens of adversarial examples within
the in-context learning framework, this longitudinal study addresses this gap
by conducting a comprehensive assessment of the robustness of successive
versions of LLMs, vis-\`a-vis GPT-3.5. We conduct extensive experiments to
analyze and understand the impact of the robustness in two distinct learning
categories: zero-shot learning and few-shot learning. Our findings indicate
that, in comparison to earlier versions of LLMs, the updated versions do not
exhibit the anticipated level of robustness against adversarial attacks. In
addition, our study emphasizes the increased effectiveness of synergized
adversarial queries in most zero-shot learning and few-shot learning cases. We
hope that our study can lead to a more refined assessment of the robustness of
LLMs over time and provide valuable insights of these models for both
developers and users.


http://arxiv.org/abs/2308.07834
Simple and Efficient Partial Graph Adversarial Attack: A New Perspective. (93%)
Guanghui Zhu; Mengyu Chen; Chunfeng Yuan; Yihua Huang
  As the study of graph neural networks becomes more intensive and
comprehensive, their robustness and security have received great research
interest. The existing global attack methods treat all nodes in the graph as
their attack targets. Although existing methods have achieved excellent
results, there is still considerable space for improvement. The key problem is
that the current approaches rigidly follow the definition of global attacks.
They ignore an important issue, i.e., different nodes have different robustness
and are not equally resilient to attacks. From a global attacker's view, we
should arrange the attack budget wisely, rather than wasting them on highly
robust nodes. To this end, we propose a totally new method named partial graph
attack (PGA), which selects the vulnerable nodes as attack targets. First, to
select the vulnerable items, we propose a hierarchical target selection policy,
which allows attackers to only focus on easy-to-attack nodes. Then, we propose
a cost-effective anchor-picking policy to pick the most promising anchors for
adding or removing edges, and a more aggressive iterative greedy-based attack
method to perform more efficient attacks. Extensive experimental results
demonstrate that PGA can achieve significant improvements in both attack effect
and attack efficiency compared to other existing graph global attack methods.


http://arxiv.org/abs/2308.07546
3DHacker: Spectrum-based Decision Boundary Generation for Hard-label 3D Point Cloud Attack. (99%)
Yunbo Tao; Daizong Liu; Pan Zhou; Yulai Xie; Wei Du; Wei Hu
  With the maturity of depth sensors, the vulnerability of 3D point cloud
models has received increasing attention in various applications such as
autonomous driving and robot navigation. Previous 3D adversarial attackers
either follow the white-box setting to iteratively update the coordinate
perturbations based on gradients, or utilize the output model logits to
estimate noisy gradients in the black-box setting. However, these attack
methods are hard to be deployed in real-world scenarios since realistic 3D
applications will not share any model details to users. Therefore, we explore a
more challenging yet practical 3D attack setting, \textit{i.e.}, attacking
point clouds with black-box hard labels, in which the attacker can only have
access to the prediction label of the input. To tackle this setting, we propose
a novel 3D attack method, termed \textbf{3D} \textbf{H}ard-label
att\textbf{acker} (\textbf{3DHacker}), based on the developed decision boundary
algorithm to generate adversarial samples solely with the knowledge of class
labels. Specifically, to construct the class-aware model decision boundary,
3DHacker first randomly fuses two point clouds of different classes in the
spectral domain to craft their intermediate sample with high imperceptibility,
then projects it onto the decision boundary via binary search. To restrict the
final perturbation size, 3DHacker further introduces an iterative optimization
strategy to move the intermediate sample along the decision boundary for
generating adversarial point clouds with smallest trivial perturbations.
Extensive evaluations show that, even in the challenging hard-label setting,
3DHacker still competitively outperforms existing 3D attacks regarding the
attack performance as well as adversary quality.


http://arxiv.org/abs/2308.07433
White-Box Adversarial Attacks on Deep Learning-Based Radio Frequency Fingerprint Identification. (99%)
Jie Ma; Junqing Zhang; Guanxiong Shen; Alan Marshall; Chip-Hong Chang
  Radio frequency fingerprint identification (RFFI) is an emerging technique
for the lightweight authentication of wireless Internet of things (IoT)
devices. RFFI exploits unique hardware impairments as device identifiers, and
deep learning is widely deployed as the feature extractor and classifier for
RFFI. However, deep learning is vulnerable to adversarial attacks, where
adversarial examples are generated by adding perturbation to clean data for
causing the classifier to make wrong predictions. Deep learning-based RFFI has
been shown to be vulnerable to such attacks, however, there is currently no
exploration of effective adversarial attacks against a diversity of RFFI
classifiers. In this paper, we report on investigations into white-box attacks
(non-targeted and targeted) using two approaches, namely the fast gradient sign
method (FGSM) and projected gradient descent (PGD). A LoRa testbed was built
and real datasets were collected. These adversarial examples have been
experimentally demonstrated to be effective against convolutional neural
networks (CNNs), long short-term memory (LSTM) networks, and gated recurrent
units (GRU).


http://arxiv.org/abs/2308.07026
AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning. (99%)
Ziqi Zhou; Shengshan Hu; Minghui Li; Hangtao Zhang; Yechao Zhang; Hai Jin
  Multimodal contrastive learning aims to train a general-purpose feature
extractor, such as CLIP, on vast amounts of raw, unlabeled paired image-text
data. This can greatly benefit various complex downstream tasks, including
cross-modal image-text retrieval and image classification. Despite its
promising prospect, the security issue of cross-modal pre-trained encoder has
not been fully explored yet, especially when the pre-trained encoder is
publicly available for commercial use.
  In this work, we propose AdvCLIP, the first attack framework for generating
downstream-agnostic adversarial examples based on cross-modal pre-trained
encoders. AdvCLIP aims to construct a universal adversarial patch for a set of
natural images that can fool all the downstream tasks inheriting the victim
cross-modal pre-trained encoder. To address the challenges of heterogeneity
between different modalities and unknown downstream tasks, we first build a
topological graph structure to capture the relevant positions between target
samples and their neighbors. Then, we design a topology-deviation based
generative adversarial network to generate a universal adversarial patch. By
adding the patch to images, we minimize their embeddings similarity to
different modality and perturb the sample distribution in the feature space,
achieving unviersal non-targeted attacks. Our results demonstrate the excellent
attack performance of AdvCLIP on two types of downstream tasks across eight
datasets. We also tailor three popular defenses to mitigate AdvCLIP,
highlighting the need for new defense mechanisms to defend cross-modal
pre-trained encoders.


http://arxiv.org/abs/2308.07553
Enhancing the Antidote: Improved Pointwise Certifications against Poisoning Attacks. (68%)
Shijie Liu; Andrew C. Cullen; Paul Montague; Sarah M. Erfani; Benjamin I. P. Rubinstein
  Poisoning attacks can disproportionately influence model behaviour by making
small changes to the training corpus. While defences against specific poisoning
attacks do exist, they in general do not provide any guarantees, leaving them
potentially countered by novel attacks. In contrast, by examining worst-case
behaviours Certified Defences make it possible to provide guarantees of the
robustness of a sample against adversarial attacks modifying a finite number of
training samples, known as pointwise certification. We achieve this by
exploiting both Differential Privacy and the Sampled Gaussian Mechanism to
ensure the invariance of prediction for each testing instance against finite
numbers of poisoned examples. In doing so, our model provides guarantees of
adversarial robustness that are more than twice as large as those provided by
prior certifications.


http://arxiv.org/abs/2308.07308
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked. (54%)
Alec Helbling; Mansi Phute; Matthew Hull; Duen Horng Chau
  Large language models (LLMs) have skyrocketed in popularity in recent years
due to their ability to generate high-quality text in response to human
prompting. However, these models have been shown to have the potential to
generate harmful content in response to user prompting (e.g., giving users
instructions on how to commit crimes). There has been a focus in the literature
on mitigating these risks, through methods like aligning models with human
values through reinforcement learning. However, it has been shown that even
aligned language models are susceptible to adversarial attacks that bypass
their restrictions on generating harmful text. We propose a simple approach to
defending against these attacks by having a large language model filter its own
responses. Our current results show that even if a model is not fine-tuned to
be aligned with human values, it is possible to stop it from presenting harmful
content to users by validating the content using a language model.


http://arxiv.org/abs/2308.07387
DISBELIEVE: Distance Between Client Models is Very Essential for Effective Local Model Poisoning Attacks. (13%)
Indu Joshi; Priyank Upadhya; Gaurav Kumar Nayak; Peter Schüffler; Nassir Navab
  Federated learning is a promising direction to tackle the privacy issues
related to sharing patients' sensitive data. Often, federated systems in the
medical image analysis domain assume that the participating local clients are
\textit{honest}. Several studies report mechanisms through which a set of
malicious clients can be introduced that can poison the federated setup,
hampering the performance of the global model. To overcome this, robust
aggregation methods have been proposed that defend against those attacks. We
observe that most of the state-of-the-art robust aggregation methods are
heavily dependent on the distance between the parameters or gradients of
malicious clients and benign clients, which makes them prone to local model
poisoning attacks when the parameters or gradients of malicious and benign
clients are close. Leveraging this, we introduce DISBELIEVE, a local model
poisoning attack that creates malicious parameters or gradients such that their
distance to benign clients' parameters or gradients is low respectively but at
the same time their adverse effect on the global model's performance is high.
Experiments on three publicly available medical image datasets demonstrate the
efficacy of the proposed DISBELIEVE attack as it significantly lowers the
performance of the state-of-the-art \textit{robust aggregation} methods for
medical image analysis. Furthermore, compared to state-of-the-art local model
poisoning attacks, DISBELIEVE attack is also effective on natural images where
we observe a severe drop in classification performance of the global model for
multi-class classification on benchmark dataset CIFAR-10.


http://arxiv.org/abs/2308.07009
ACTIVE: Towards Highly Transferable 3D Physical Camouflage for Universal and Robust Vehicle Evasion. (10%)
Naufal Suryanto; Yongsu Kim; Harashta Tatimma Larasati; Hyoeun Kang; Thi-Thu-Huong Le; Yoonyoung Hong; Hunmin Yang; Se-Yoon Oh; Howon Kim
  Adversarial camouflage has garnered attention for its ability to attack
object detectors from any viewpoint by covering the entire object's surface.
However, universality and robustness in existing methods often fall short as
the transferability aspect is often overlooked, thus restricting their
application only to a specific target with limited performance. To address
these challenges, we present Adversarial Camouflage for Transferable and
Intensive Vehicle Evasion (ACTIVE), a state-of-the-art physical camouflage
attack framework designed to generate universal and robust adversarial
camouflage capable of concealing any 3D vehicle from detectors. Our framework
incorporates innovative techniques to enhance universality and robustness: a
refined texture rendering that enables common texture application to different
vehicles without being constrained to a specific texture map, a novel stealth
loss that renders the vehicle undetectable, and a smooth and camouflage loss to
enhance the naturalness of the adversarial camouflage. Our extensive
experiments on 15 different models show that ACTIVE consistently outperforms
existing works on various public detectors, including the latest YOLOv7.
Notably, our universality evaluations reveal promising transferability to other
vehicle classes, tasks (segmentation models), and the real world, not just
other vehicles.


http://arxiv.org/abs/2308.07156
SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation. (1%)
An Wang; Mobarakol Islam; Mengya Xu; Yang Zhang; Hongliang Ren
  The Segment Anything Model (SAM) serves as a fundamental model for semantic
segmentation and demonstrates remarkable generalization capabilities across a
wide range of downstream scenarios. In this empirical study, we examine SAM's
robustness and zero-shot generalizability in the field of robotic surgery. We
comprehensively explore different scenarios, including prompted and unprompted
situations, bounding box and points-based prompt approaches, as well as the
ability to generalize under corruptions and perturbations at five severity
levels. Additionally, we compare the performance of SAM with state-of-the-art
supervised models. We conduct all the experiments with two well-known robotic
instrument segmentation datasets from MICCAI EndoVis 2017 and 2018 challenges.
Our extensive evaluation results reveal that although SAM shows remarkable
zero-shot generalization ability with bounding box prompts, it struggles to
segment the whole instrument with point-based prompts and unprompted settings.
Furthermore, our qualitative figures demonstrate that the model either failed
to predict certain parts of the instrument mask (e.g., jaws, wrist) or
predicted parts of the instrument as wrong classes in the scenario of
overlapping instruments within the same bounding box or with the point-based
prompt. In fact, SAM struggles to identify instruments in complex surgical
scenarios characterized by the presence of blood, reflection, blur, and shade.
Additionally, SAM is insufficiently robust to maintain high performance when
subjected to various forms of data corruption. We also attempt to fine-tune SAM
using Low-rank Adaptation (LoRA) and propose SurgicalSAM, which shows the
capability in class-wise mask prediction without prompt. Therefore, we can
argue that, without further domain-specific fine-tuning, SAM is not ready for
downstream surgical tasks.


http://arxiv.org/abs/2308.06819
SoK: Realistic Adversarial Attacks and Defenses for Intelligent Network Intrusion Detection. (99%)
João Vitorino; Isabel Praça; Eva Maia
  Machine Learning (ML) can be incredibly valuable to automate anomaly
detection and cyber-attack classification, improving the way that Network
Intrusion Detection (NID) is performed. However, despite the benefits of ML
models, they are highly susceptible to adversarial cyber-attack examples
specifically crafted to exploit them. A wide range of adversarial attacks have
been created and researchers have worked on various defense strategies to
safeguard ML models, but most were not intended for the specific constraints of
a communication network and its communication protocols, so they may lead to
unrealistic examples in the NID domain. This Systematization of Knowledge (SoK)
consolidates and summarizes the state-of-the-art adversarial learning
approaches that can generate realistic examples and could be used in real ML
development and deployment scenarios with real network traffic flows. This SoK
also describes the open challenges regarding the use of adversarial ML in the
NID domain, defines the fundamental properties that are required for an
adversarial example to be realistic, and provides guidelines for researchers to
ensure that their future experiments are adequate for a real communication
network.


http://arxiv.org/abs/2308.06703
Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods. (45%)
Avery Ma; Yangchen Pan; Amir-massoud Farahmand
  Stochastic gradient descent (SGD) and adaptive gradient methods, such as Adam
and RMSProp, have been widely used in training deep neural networks. We
empirically show that while the difference between the standard generalization
performance of models trained using these methods is small, those trained using
SGD exhibit far greater robustness under input perturbations. Notably, our
investigation demonstrates the presence of irrelevant frequencies in natural
datasets, where alterations do not affect models' generalization performance.
However, models trained with adaptive methods show sensitivity to these
changes, suggesting that their use of irrelevant frequencies can lead to
solutions sensitive to perturbations. To better understand this difference, we
study the learning dynamics of gradient descent (GD) and sign gradient descent
(signGD) on a synthetic dataset that mirrors natural signals. With a
three-dimensional input space, the models optimized with GD and signGD have
standard risks close to zero but vary in their adversarial risks. Our result
shows that linear models' robustness to $\ell_2$-norm bounded changes is
inversely proportional to the model parameters' weight norm: a smaller weight
norm implies better robustness. In the context of deep learning, our
experiments show that SGD-trained neural networks show smaller Lipschitz
constants, explaining the better robustness to input perturbations than those
trained with adaptive gradient methods.


http://arxiv.org/abs/2308.06767
A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations. (1%)
Hongrong Cheng; Miao Zhang; Javen Qinfeng Shi
  Modern deep neural networks, particularly recent large language models, come
with massive model sizes that require significant computational and storage
resources. To enable the deployment of modern models on resource-constrained
environments and accelerate inference time, researchers have increasingly
explored pruning techniques as a popular research direction in neural network
compression. However, there is a dearth of up-to-date comprehensive review
papers on pruning. To address this issue, in this survey, we provide a
comprehensive review of existing research works on deep neural network pruning
in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to
prune, and 4) fusion of pruning and other compression techniques. We then
provide a thorough comparative analysis of seven pairs of contrast settings for
pruning (e.g., unstructured/structured) and explore emerging topics, including
post-training pruning, different levels of supervision for pruning, and broader
applications (e.g., adversarial robustness) to shed light on the commonalities
and differences of existing methods and lay the foundation for further method
development. To facilitate future research, we build a curated collection of
datasets, networks, and evaluations on different applications. Finally, we
provide some valuable recommendations on selecting pruning methods and prospect
promising research directions. We build a repository at
https://github.com/hrcheng1066/awesome-pruning.


http://arxiv.org/abs/2308.06887
Robustified ANNs Reveal Wormholes Between Human Category Percepts. (1%)
Guy Gaziv; Michael J. Lee; James J. DiCarlo
  The visual object category reports of artificial neural networks (ANNs) are
notoriously sensitive to tiny, adversarial image perturbations. Because human
category reports (aka human percepts) are thought to be insensitive to those
same small-norm perturbations -- and locally stable in general -- this argues
that ANNs are incomplete scientific models of human visual perception.
Consistent with this, we show that when small-norm image perturbations are
generated by standard ANN models, human object category percepts are indeed
highly stable. However, in this very same "human-presumed-stable" regime, we
find that robustified ANNs reliably discover low-norm image perturbations that
strongly disrupt human percepts. These previously undetectable human perceptual
disruptions are massive in amplitude, approaching the same level of sensitivity
seen in robustified ANNs. Further, we show that robustified ANNs support
precise perceptual state interventions: they guide the construction of low-norm
image perturbations that strongly alter human category percepts toward specific
prescribed percepts. These observations suggest that for arbitrary starting
points in image space, there exists a set of nearby "wormholes", each leading
the subject from their current category perceptual state into a semantically
very different state. Moreover, contemporary ANN models of biological visual
processing are now accurate enough to consistently guide us to those portals.


http://arxiv.org/abs/2308.06795
Faithful to Whom? Questioning Interpretability Measures in NLP. (1%)
Evan Crothers; Herna Viktor; Nathalie Japkowicz
  A common approach to quantifying model interpretability is to calculate
faithfulness metrics based on iteratively masking input tokens and measuring
how much the predicted label changes as a result. However, we show that such
metrics are generally not suitable for comparing the interpretability of
different neural text classifiers as the response to masked inputs is highly
model-specific. We demonstrate that iterative masking can produce large
variation in faithfulness scores between comparable models, and show that
masked samples are frequently outside the distribution seen during training. We
further investigate the impact of adversarial attacks and adversarial training
on faithfulness scores, and demonstrate the relevance of faithfulness measures
for analyzing feature salience in text adversarial attacks. Our findings
provide new insights into the limitations of current faithfulness metrics and
key considerations to utilize them appropriately.


http://arxiv.org/abs/2308.06467
Not So Robust After All: Evaluating the Robustness of Deep Neural Networks to Unseen Adversarial Attacks. (99%)
Roman Garaev; Bader Rasheed; Adil Khan
  Deep neural networks (DNNs) have gained prominence in various applications,
such as classification, recognition, and prediction, prompting increased
scrutiny of their properties. A fundamental attribute of traditional DNNs is
their vulnerability to modifications in input data, which has resulted in the
investigation of adversarial attacks. These attacks manipulate the data in
order to mislead a DNN. This study aims to challenge the efficacy and
generalization of contemporary defense mechanisms against adversarial attacks.
Specifically, we explore the hypothesis proposed by Ilyas et. al, which posits
that DNN image features can be either robust or non-robust, with adversarial
attacks targeting the latter. This hypothesis suggests that training a DNN on a
dataset consisting solely of robust features should produce a model resistant
to adversarial attacks. However, our experiments demonstrate that this is not
universally true. To gain further insights into our findings, we analyze the
impact of adversarial attack norms on DNN representations, focusing on samples
subjected to $L_2$ and $L_{\infty}$ norm attacks. Further, we employ canonical
correlation analysis, visualize the representations, and calculate the mean
distance between these representations and various DNN decision boundaries. Our
results reveal a significant difference between $L_2$ and $L_{\infty}$ norms,
which could provide insights into the potential dangers posed by $L_{\infty}$
norm attacks, previously underestimated by the research community.


http://arxiv.org/abs/2308.07934
One-bit Flip is All You Need: When Bit-flip Attack Meets Model Training. (13%)
Jianshuo Dong; Han Qiu; Yiming Li; Tianwei Zhang; Yuanjie Li; Zeqi Lai; Chao Zhang; Shu-Tao Xia
  Deep neural networks (DNNs) are widely deployed on real-world devices.
Concerns regarding their security have gained great attention from researchers.
Recently, a new weight modification attack called bit flip attack (BFA) was
proposed, which exploits memory fault inject techniques such as row hammer to
attack quantized models in the deployment stage. With only a few bit flips, the
target model can be rendered useless as a random guesser or even be implanted
with malicious functionalities. In this work, we seek to further reduce the
number of bit flips. We propose a training-assisted bit flip attack, in which
the adversary is involved in the training stage to build a high-risk model to
release. This high-risk model, obtained coupled with a corresponding malicious
model, behaves normally and can escape various detection methods. The results
on benchmark datasets show that an adversary can easily convert this high-risk
but normal model to a malicious one on victim's side by \textbf{flipping only
one critical bit} on average in the deployment stage. Moreover, our attack
still poses a significant threat even when defenses are employed. The codes for
reproducing main experiments are available at
\url{https://github.com/jianshuod/TBA}.


http://arxiv.org/abs/2308.06015
Enhancing Generalization of Universal Adversarial Perturbation through Gradient Aggregation. (98%)
Xuannan Liu; Yaoyao Zhong; Yuhang Zhang; Lixiong Qin; Weihong Deng
  Deep neural networks are vulnerable to universal adversarial perturbation
(UAP), an instance-agnostic perturbation capable of fooling the target model
for most samples. Compared to instance-specific adversarial examples, UAP is
more challenging as it needs to generalize across various samples and models.
In this paper, we examine the serious dilemma of UAP generation methods from a
generalization perspective -- the gradient vanishing problem using small-batch
stochastic gradient optimization and the local optima problem using large-batch
optimization. To address these problems, we propose a simple and effective
method called Stochastic Gradient Aggregation (SGA), which alleviates the
gradient vanishing and escapes from poor local optima at the same time.
Specifically, SGA employs the small-batch training to perform multiple
iterations of inner pre-search. Then, all the inner gradients are aggregated as
a one-step gradient estimation to enhance the gradient stability and reduce
quantization errors. Extensive experiments on the standard ImageNet dataset
demonstrate that our method significantly enhances the generalization ability
of UAP and outperforms other state-of-the-art methods. The code is available at
https://github.com/liuxuannan/Stochastic-Gradient-Aggregation.


http://arxiv.org/abs/2308.06173
Physical Adversarial Attacks For Camera-based Smart Systems: Current Trends, Categorization, Applications, Research Challenges, and Future Outlook. (98%)
Amira Guesmi; Muhammad Abdullah Hanif; Bassem Ouni; Muhammed Shafique
  In this paper, we present a comprehensive survey of the current trends
focusing specifically on physical adversarial attacks. We aim to provide a
thorough understanding of the concept of physical adversarial attacks,
analyzing their key characteristics and distinguishing features. Furthermore,
we explore the specific requirements and challenges associated with executing
attacks in the physical world. Our article delves into various physical
adversarial attack methods, categorized according to their target tasks in
different applications, including classification, detection, face recognition,
semantic segmentation and depth estimation. We assess the performance of these
attack methods in terms of their effectiveness, stealthiness, and robustness.
We examine how each technique strives to ensure the successful manipulation of
DNNs while mitigating the risk of detection and withstanding real-world
distortions. Lastly, we discuss the current challenges and outline potential
future research directions in the field of physical adversarial attacks. We
highlight the need for enhanced defense mechanisms, the exploration of novel
attack strategies, the evaluation of attacks in different application domains,
and the establishment of standardized benchmarks and evaluation criteria for
physical adversarial attacks. Through this comprehensive survey, we aim to
provide a valuable resource for researchers, practitioners, and policymakers to
gain a holistic understanding of physical adversarial attacks in computer
vision and facilitate the development of robust and secure DNN-based systems.


http://arxiv.org/abs/2308.05983
Face Encryption via Frequency-Restricted Identity-Agnostic Attacks. (96%)
Xin Dong; Rui Wang; Siyuan Liang; Aishan Liu; Lihua Jing
  Billions of people are sharing their daily live images on social media
everyday. However, malicious collectors use deep face recognition systems to
easily steal their biometric information (e.g., faces) from these images. Some
studies are being conducted to generate encrypted face photos using adversarial
attacks by introducing imperceptible perturbations to reduce face information
leakage. However, existing studies need stronger black-box scenario feasibility
and more natural visual appearances, which challenge the feasibility of privacy
protection. To address these problems, we propose a frequency-restricted
identity-agnostic (FRIA) framework to encrypt face images from unauthorized
face recognition without access to personal information. As for the weak
black-box scenario feasibility, we obverse that representations of the average
feature in multiple face recognition models are similar, thus we propose to
utilize the average feature via the crawled dataset from the Internet as the
target to guide the generation, which is also agnostic to identities of unknown
face recognition systems; in nature, the low-frequency perturbations are more
visually perceptible by the human vision system. Inspired by this, we restrict
the perturbation in the low-frequency facial regions by discrete cosine
transform to achieve the visual naturalness guarantee. Extensive experiments on
several face recognition models demonstrate that our FRIA outperforms other
state-of-the-art methods in generating more natural encrypted faces while
attaining high black-box attack success rates of 96%. In addition, we validate
the efficacy of FRIA using real-world black-box commercial API, which reveals
the potential of FRIA in practice. Our codes can be found in
https://github.com/XinDong10/FRIA.


http://arxiv.org/abs/2308.06405
White-box Membership Inference Attacks against Diffusion Models. (68%)
Yan Pang; Tianhao Wang; Xuhui Kang; Mengdi Huai; Yang Zhang
  Diffusion models have begun to overshadow GANs and other generative models in
industrial applications due to their superior image generation performance. The
complex architecture of these models furnishes an extensive array of attack
features. In light of this, we aim to design membership inference attacks
(MIAs) catered to diffusion models. We first conduct an exhaustive analysis of
existing MIAs on diffusion models, taking into account factors such as
black-box/white-box models and the selection of attack features. We found that
white-box attacks are highly applicable in real-world scenarios, and the most
effective attacks presently are white-box. Departing from earlier research,
which employs model loss as the attack feature for white-box MIAs, we employ
model gradients in our attack, leveraging the fact that these gradients provide
a more profound understanding of model responses to various samples. We subject
these models to rigorous testing across a range of parameters, including
training steps, sampling frequency, diffusion steps, and data variance. Across
all experimental settings, our method consistently demonstrated near-flawless
attack performance, with attack success rate approaching $100\%$ and attack
AUCROC near $1.0$. We also evaluate our attack against common defense
mechanisms, and observe our attacks continue to exhibit commendable
performance.


http://arxiv.org/abs/2308.06107
Test-Time Adaptation for Backdoor Defense. (10%)
Jiyang Guan; Jian Liang; Ran He
  Deep neural networks have played a crucial part in many critical domains,
such as autonomous driving, face recognition, and medical diagnosis. However,
deep neural networks are facing security threats from backdoor attacks and can
be manipulated into attacker-decided behaviors by the backdoor attacker. To
defend the backdoor, prior research has focused on using clean data to remove
backdoor attacks before model deployment. In this paper, we investigate the
possibility of defending against backdoor attacks at test time by utilizing
partially poisoned data to remove the backdoor from the model. To address the
problem, a two-stage method Test-Time Backdoor Defense (TTBD) is proposed. In
the first stage, we propose two backdoor sample detection methods, namely DDP
and TeCo, to identify poisoned samples from a batch of mixed, partially
poisoned samples. Once the poisoned samples are detected, we employ Shapley
estimation to calculate the contribution of each neuron's significance in the
network, locate the poisoned neurons, and prune them to remove backdoor in the
models. Our experiments demonstrate that TTBD removes the backdoor successfully
with only a batch of partially poisoned data across different model
architectures and datasets against different types of backdoor attacks.


http://arxiv.org/abs/2308.06217
Continual Face Forgery Detection via Historical Distribution Preserving. (2%)
Ke Sun; Shen Chen; Taiping Yao; Xiaoshuai Sun; Shouhong Ding; Rongrong Ji
  Face forgery techniques have advanced rapidly and pose serious security
threats. Existing face forgery detection methods try to learn generalizable
features, but they still fall short of practical application. Additionally,
finetuning these methods on historical training data is resource-intensive in
terms of time and storage. In this paper, we focus on a novel and challenging
problem: Continual Face Forgery Detection (CFFD), which aims to efficiently
learn from new forgery attacks without forgetting previous ones. Specifically,
we propose a Historical Distribution Preserving (HDP) framework that reserves
and preserves the distributions of historical faces. To achieve this, we use
universal adversarial perturbation (UAP) to simulate historical forgery
distribution, and knowledge distillation to maintain the distribution variation
of real faces across different models. We also construct a new benchmark for
CFFD with three evaluation protocols. Our extensive experiments on the
benchmarks show that our method outperforms the state-of-the-art competitors.


http://arxiv.org/abs/2308.05986
Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance. (1%)
Huiwen Xu; U Kang
  Given a set of pre-trained models, how can we quickly and accurately find the
most useful pre-trained model for a downstream task? Transferability
measurement is to quantify how transferable is a pre-trained model learned on a
source task to a target task. It is used for quickly ranking pre-trained models
for a given task and thus becomes a crucial step for transfer learning.
Existing methods measure transferability as the discrimination ability of a
source model for a target data before transfer learning, which cannot
accurately estimate the fine-tuning performance. Some of them restrict the
application of transferability measurement in selecting the best supervised
pre-trained models that have classifiers. It is important to have a general
method for measuring transferability that can be applied in a variety of
situations, such as selecting the best self-supervised pre-trained models that
do not have classifiers, and selecting the best transferring layer for a target
task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH
INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure
transferability. We view transferability as the generalization of a pre-trained
model on a target task by measuring intra-class feature variance. Intra-class
variance evaluates the adaptability of the model to a new task, which measures
how transferable the model is. Compared to previous studies that estimate how
discriminative the models are, intra-class variance is more accurate than those
as it does not require an optimal feature extractor and classifier. Extensive
experiments on real-world datasets show that TMI outperforms competitors for
selecting the top-5 best models, and exhibits consistently better correlation
in 13 out of 17 cases.


http://arxiv.org/abs/2308.05681
Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient. (99%)
Zhengzhi Lu; He Wang; Ziyi Chang; Guoan Yang; Hubert P. H. Shum
  Recently, methods for skeleton-based human activity recognition have been
shown to be vulnerable to adversarial attacks. However, these attack methods
require either the full knowledge of the victim (i.e. white-box attacks),
access to training data (i.e. transfer-based attacks) or frequent model queries
(i.e. black-box attacks). All their requirements are highly restrictive,
raising the question of how detrimental the vulnerability is. In this paper, we
show that the vulnerability indeed exists. To this end, we consider a new
attack task: the attacker has no access to the victim model or the training
data or labels, where we coin the term hard no-box attack. Specifically, we
first learn a motion manifold where we define an adversarial loss to compute a
new gradient for the attack, named skeleton-motion-informed (SMI) gradient. Our
gradient contains information of the motion dynamics, which is different from
existing gradient-based attack methods that compute the loss gradient assuming
each dimension in the data is independent. The SMI gradient can augment many
gradient-based attack methods, leading to a new family of no-box attack
methods. Extensive evaluation and comparison show that our method imposes a
real threat to existing classifiers. They also show that the SMI gradient
improves the transferability and imperceptibility of adversarial samples in
both no-box and transfer-based black-box settings.


http://arxiv.org/abs/2308.05575
Symmetry Defense Against XGBoost Adversarial Perturbation Attacks. (96%)
Blerta Lindqvist
  We examine whether symmetry can be used to defend tree-based ensemble
classifiers such as gradient-boosting decision trees (GBDTs) against
adversarial perturbation attacks. The idea is based on a recent symmetry
defense for convolutional neural network classifiers (CNNs) that utilizes CNNs'
lack of invariance with respect to symmetries. CNNs lack invariance because
they can classify a symmetric sample, such as a horizontally flipped image,
differently from the original sample. CNNs' lack of invariance also means that
CNNs can classify symmetric adversarial samples differently from the incorrect
classification of adversarial samples. Using CNNs' lack of invariance, the
recent CNN symmetry defense has shown that the classification of symmetric
adversarial samples reverts to the correct sample classification. In order to
apply the same symmetry defense to GBDTs, we examine GBDT invariance and are
the first to show that GBDTs also lack invariance with respect to symmetries.
We apply and evaluate the GBDT symmetry defense for nine datasets against six
perturbation attacks with a threat model that ranges from zero-knowledge to
perfect-knowledge adversaries. Using the feature inversion symmetry against
zero-knowledge adversaries, we achieve up to 100% accuracy on adversarial
samples even when default and robust classifiers have 0% accuracy. Using the
feature inversion and horizontal flip symmetries against perfect-knowledge
adversaries, we achieve up to over 95% accuracy on adversarial samples for the
GBDT classifier of the F-MNIST dataset even when default and robust classifiers
have 0% accuracy.


http://arxiv.org/abs/2308.05498
Complex Network Effects on the Robustness of Graph Convolutional Networks. (92%)
Benjamin A. Miller; Kevin Chan; Tina Eliassi-Rad
  Vertex classification -- the problem of identifying the class labels of nodes
in a graph -- has applicability in a wide variety of domains. Examples include
classifying subject areas of papers in citation networks or roles of machines
in a computer network. Vertex classification using graph convolutional networks
is susceptible to targeted poisoning attacks, in which both graph structure and
node attributes can be changed in an attempt to misclassify a target node. This
vulnerability decreases users' confidence in the learning method and can
prevent adoption in high-stakes contexts. Defenses have also been proposed,
focused on filtering edges before creating the model or aggregating information
from neighbors more robustly. This paper considers an alternative: we leverage
network characteristics in the training data selection process to improve
robustness of vertex classifiers.
  We propose two alternative methods of selecting training data: (1) to select
the highest-degree nodes and (2) to iteratively select the node with the most
neighbors minimally connected to the training set. In the datasets on which the
original attack was demonstrated, we show that changing the training set can
make the network much harder to attack. To maintain a given probability of
attack success, the adversary must use far more perturbations; often a factor
of 2--4 over the random training baseline. These training set selection methods
often work in conjunction with the best recently published defenses to provide
even greater robustness. While increasing the amount of randomly selected
training data sometimes results in a more robust classifier, the proposed
methods increase robustness substantially more. We also run a simulation study
in which we demonstrate conditions under which each of the two methods
outperforms the other, controlling for the graph topology, homophily of the
labels, and node attributes.


http://arxiv.org/abs/2310.10789
State Machine Frameworks for Website Fingerprinting Defenses: Maybe Not. (61%)
Ethan Witwer
  Tor is an anonymity network used by millions of people every day to evade
censorship and protect their browsing activity from privacy threats such as
mass surveillance. Unfortunately, Tor has been shown to be vulnerable to
website fingerprinting attacks, in which an adversary observes the connection
between a user and the Tor network and uses features of the encrypted traffic,
such as the timing and volume of packets, to identify the websites that are
being visited. In response, researchers have proposed a number of defenses
against website fingerprinting attacks, and a "circuit padding framework" has
been added to the Tor software which supports the implementation of defenses.
However, many proposed defenses are not supported by this framework, and no
defenses are currently present in Tor.
  As Arti, a reimplementation of Tor in Rust, is being developed, the issue
arises of whether a new state machine framework should be included or if
alternative models should instead be considered for future defense
implementation. We address this question by using an improved Rust-based state
machine framework, Maybenot, to implement three state-of-the-art website
fingerprinting defenses. Through our evaluation, we demonstrate the potential
of state machine frameworks to support effective defenses, and we highlight
important features that they should contain to do so. However, our evaluation
also raises uncertainty about the long-term feasibility of state machine
frameworks for defense implementation. We recommend enhancements to Maybenot
and substantial further evaluation, along with consideration of alternative
designs, before any decision is made regarding a mechanism for implementing
website fingerprinting defenses in Arti.


http://arxiv.org/abs/2308.05832
FLShield: A Validation Based Federated Learning Framework to Defend Against Poisoning Attacks. (45%)
Ehsanul Kabir; Zeyu Song; Md Rafi Ur Rashid; Shagufta Mehnaz
  Federated learning (FL) is revolutionizing how we learn from data. With its
growing popularity, it is now being used in many safety-critical domains such
as autonomous vehicles and healthcare. Since thousands of participants can
contribute in this collaborative setting, it is, however, challenging to ensure
security and reliability of such systems. This highlights the need to design FL
systems that are secure and robust against malicious participants' actions
while also ensuring high utility, privacy of local data, and efficiency. In
this paper, we propose a novel FL framework dubbed as FLShield that utilizes
benign data from FL participants to validate the local models before taking
them into account for generating the global model. This is in stark contrast
with existing defenses relying on server's access to clean datasets -- an
assumption often impractical in real-life scenarios and conflicting with the
fundamentals of FL. We conduct extensive experiments to evaluate our FLShield
framework in different settings and demonstrate its effectiveness in thwarting
various types of poisoning and backdoor attacks including a defense-aware one.
FLShield also preserves privacy of local data against gradient inversion
attacks.


http://arxiv.org/abs/2308.05525
Critical Points ++: An Agile Point Cloud Importance Measure for Robust Classification, Adversarial Defense and Explainable AI. (5%)
Meir Yossef Levi; Guy Gilboa
  The ability to cope accurately and fast with Out-Of-Distribution (OOD)
samples is crucial in real-world safety demanding applications. In this work we
first study the interplay between critical points of 3D point clouds and OOD
samples. Our findings are that common corruptions and outliers are often
interpreted as critical points. We generalize the notion of critical points
into importance measures. We show that training a classification network based
only on less important points dramatically improves robustness, at a cost of
minor performance loss on the clean set. We observe that normalized entropy is
highly informative for corruption analysis. An adaptive threshold based on
normalized entropy is suggested for selecting the set of uncritical points. Our
proposed importance measure is extremely fast to compute. We show it can be
used for a variety of applications, such as Explainable AI (XAI), Outlier
Removal, Uncertainty Estimation, Robust Classification and Adversarial Defense.
We reach SOTA results on the two latter tasks.


http://arxiv.org/abs/2308.08012
Comprehensive Analysis of Network Robustness Evaluation Based on Convolutional Neural Networks with Spatial Pyramid Pooling. (1%)
Wenjun Jiang; Tianlong Fan; Changhao Li; Chuanfu Zhang; Tao Zhang; Zong-fu Luo
  Connectivity robustness, a crucial aspect for understanding, optimizing, and
repairing complex networks, has traditionally been evaluated through
time-consuming and often impractical simulations. Fortunately, machine learning
provides a new avenue for addressing this challenge. However, several key
issues remain unresolved, including the performance in more general edge
removal scenarios, capturing robustness through attack curves instead of
directly training for robustness, scalability of predictive tasks, and
transferability of predictive capabilities. In this paper, we address these
challenges by designing a convolutional neural networks (CNN) model with
spatial pyramid pooling networks (SPP-net), adapting existing evaluation
metrics, redesigning the attack modes, introducing appropriate filtering rules,
and incorporating the value of robustness as training data. The results
demonstrate the thoroughness of the proposed CNN framework in addressing the
challenges of high computational time across various network types, failure
component types and failure scenarios. However, the performance of the proposed
CNN model varies: for evaluation tasks that are consistent with the trained
network type, the proposed CNN model consistently achieves accurate evaluations
of both attack curves and robustness values across all removal scenarios. When
the predicted network type differs from the trained network, the CNN model
still demonstrates favorable performance in the scenario of random node
failure, showcasing its scalability and performance transferability.
Nevertheless, the performance falls short of expectations in other removal
scenarios. This observed scenario-sensitivity in the evaluation of network
features has been overlooked in previous studies and necessitates further
attention and optimization. Lastly, we discuss important unresolved questions
and further investigation.


http://arxiv.org/abs/2308.05320
Adv-Inpainting: Generating Natural and Transferable Adversarial Patch via Attention-guided Feature Fusion. (98%)
Yanjie Li; Mingxing Duan; Bin Xiao
  The rudimentary adversarial attacks utilize additive noise to attack facial
recognition (FR) models. However, because manipulating the total face is
impractical in the physical setting, most real-world FR attacks are based on
adversarial patches, which limit perturbations to a small area. Previous
adversarial patch attacks often resulted in unnatural patterns and clear
boundaries that were easily noticeable. In this paper, we argue that generating
adversarial patches with plausible content can result in stronger
transferability than using additive noise or directly sampling from the latent
space. To generate natural-looking and highly transferable adversarial patches,
we propose an innovative two-stage coarse-to-fine attack framework called
Adv-Inpainting. In the first stage, we propose an attention-guided StyleGAN
(Att-StyleGAN) that adaptively combines texture and identity features based on
the attention map to generate high-transferable and natural adversarial
patches. In the second stage, we design a refinement network with a new
boundary variance loss to further improve the coherence between the patch and
its surrounding area. Experiment results demonstrate that Adv-Inpainting is
stealthy and can produce adversarial patches with stronger transferability and
improved visual quality than previous adversarial patch attacks.


http://arxiv.org/abs/2308.04964
Adversarial ModSec