http://arxiv.org/abs/2302.03657
Toward Face Biometric De-identification using Adversarial Examples. (98%)
Mahdi Ghafourian; Julian Fierrez; Luis Felipe Gomez; Ruben Vera-Rodriguez; Aythami Morales; Zohra Rezgui; Raymond Veldhuis
  The remarkable success of face recognition (FR) has endangered the privacy of
internet users particularly in social media. Recently, researchers turned to
use adversarial examples as a countermeasure. In this paper, we assess the
effectiveness of using two widely known adversarial methods (BIM and ILLC) for
de-identifying personal images. We discovered, unlike previous claims in the
literature, that it is not easy to get a high protection success rate
(suppressing identification rate) with imperceptible adversarial perturbation
to the human visual system. Finally, we found out that the transferability of
adversarial examples is highly affected by the training parameters of the
network with which they are generated.


http://arxiv.org/abs/2302.03322
Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influence. (83%)
Simin Li; Jun Guo; Jingqiao Xiu; Pu Feng; Xin Yu; Jiakai Wang; Aishan Liu; Wenjun Wu; Xianglong Liu
  Cooperative multi-agent reinforcement learning (c-MARL) offers a general
paradigm for a group of agents to achieve a shared goal by taking individual
decisions, yet is found to be vulnerable to adversarial attacks. Though
harmful, adversarial attacks also play a critical role in evaluating the
robustness and finding blind spots of c-MARL algorithms. However, existing
attacks are not sufficiently strong and practical, which is mainly due to the
ignorance of complex influence between agents and cooperative nature of victims
in c-MARL.
  In this paper, we propose adversarial minority influence (AMI), the first
practical attack against c-MARL by introducing an adversarial agent. AMI
addresses the aforementioned problems by unilaterally influencing other
cooperative victims to a targeted worst-case cooperation. Technically, to
maximally deviate victim policy under complex agent-wise influence, our
unilateral attack characterize and maximize the influence from adversary to
victims. This is done by adapting a unilateral agent-wise relation metric
derived from mutual information, which filters out the detrimental influence
from victims to adversary. To fool victims into a jointly worst-case failure,
our targeted attack influence victims to a long-term, cooperatively worst case
by distracting each victim to a specific target. Such target is learned by a
reinforcement learning agent in a trial-and-error process. Extensive
experiments in simulation environments, including discrete control (SMAC),
continuous control (MAMujoco) and real-world robot swarm control demonstrate
the superiority of our AMI approach. Our codes are available in
https://anonymous.4open.science/r/AMI.


http://arxiv.org/abs/2302.03262
Membership Inference Attacks against Diffusion Models. (64%)
Tomoya Matsumoto; Takayuki Miura; Naoto Yanai
  Diffusion models have attracted attention in recent years as innovative
generative models. In this paper, we investigate whether a diffusion model is
resistant to a membership inference attack, which evaluates the privacy leakage
of a machine learning model. We primarily discuss the diffusion model from the
standpoints of comparison with a generative adversarial network (GAN) as
conventional models and hyperparameters unique to the diffusion model, i.e.,
time steps, sampling steps, and sampling variances. We conduct extensive
experiments with DDIM as a diffusion model and DCGAN as a GAN on the CelebA and
CIFAR-10 datasets in both white-box and black-box settings and then confirm if
the diffusion model is comparably resistant to a membership inference attack as
GAN. Next, we demonstrate that the impact of time steps is significant and
intermediate steps in a noise schedule are the most vulnerable to the attack.
We also found two key insights through further analysis. First, we identify
that DDIM is vulnerable to the attack for small sample sizes instead of
achieving a lower FID. Second, sampling steps in hyperparameters are important
for resistance to the attack, whereas the impact of sampling variances is quite
limited.


http://arxiv.org/abs/2302.03684
Temporal Robustness against Data Poisoning. (12%)
Wenxiao Wang; Soheil Feizi
  Data poisoning considers cases when an adversary maliciously inserts and
removes training data to manipulate the behavior of machine learning
algorithms. Traditional threat models of data poisoning center around a single
metric, the number of poisoned samples. In consequence, existing defenses are
essentially vulnerable in practice when poisoning more samples remains a
feasible option for attackers. To address this issue, we leverage timestamps
denoting the birth dates of data, which are often available but neglected in
the past. Benefiting from these timestamps, we propose a temporal threat model
of data poisoning and derive two novel metrics, earliness and duration, which
respectively measure how long an attack started in advance and how long an
attack lasted. With these metrics, we define the notions of temporal robustness
against data poisoning, providing a meaningful sense of protection even with
unbounded amounts of poisoned samples. We present a benchmark with an
evaluation protocol simulating continuous data collection and periodic
deployments of updated models, thus enabling empirical evaluation of temporal
robustness. Lastly, we develop and also empirically verify a baseline defense,
namely temporal aggregation, offering provable temporal robustness and
highlighting the potential of our temporal modeling of data poisoning.


http://arxiv.org/abs/2302.03465
Robustness Implies Fairness in Casual Algorithmic Recourse. (2%)
Ahmad-Reza Ehyaei; Amir-Hossein Karimi; Bernhard Schölkopf; Setareh Maghsudi
  Algorithmic recourse aims to disclose the inner workings of the black-box
decision process in situations where decisions have significant consequences,
by providing recommendations to empower beneficiaries to achieve a more
favorable outcome. To ensure an effective remedy, suggested interventions must
not only be low-cost but also robust and fair. This goal is accomplished by
providing similar explanations to individuals who are alike. This study
explores the concept of individual fairness and adversarial robustness in
causal algorithmic recourse and addresses the challenge of achieving both. To
resolve the challenges, we propose a new framework for defining adversarially
robust recourse. The new setting views the protected feature as a pseudometric
and demonstrates that individual fairness is a special case of adversarial
robustness. Finally, we introduce the fair robust recourse problem to achieve
both desirable properties and show how it can be satisfied both theoretically
and empirically.


http://arxiv.org/abs/2302.03335
Low-Latency Communication using Delay-Aware Relays Against Reactive Adversaries. (1%)
Vivek Chaudhary; J. Harshan
  This work addresses a reactive jamming attack on the low-latency messages of
a victim, wherein the jammer deploys countermeasure detection mechanisms to
change its strategy. We highlight that the existing schemes against reactive
jammers use relays with instantaneous full-duplex (FD) radios to evade the
attack. However, due to the limitation of the radio architecture of the FD
helper, instantaneous forwarding may not be possible in practice, thereby
leading to increased decoding complexity at the destination and a high
detection probability at the adversary. Pointing at this drawback, we propose a
delay-aware cooperative framework wherein the victim seeks assistance from a
delay-aware FD helper to forward its messages to the destination within the
latency constraints. In particular, we first model the processing delay at the
helper based on its hardware architecture, and then propose two low-complexity
mitigation schemes, wherein the victim and the helper share their uplink
frequencies using appropriate energy-splitting factors. For both the schemes,
we solve the optimization problems of computing the near-optimal
energy-splitting factors that minimize the joint error rates at the
destination. Finally, through analytical and simulation results, we show that
the proposed schemes facilitate the victim in evading the jamming attack whilst
deceiving the reactive adversary.


http://arxiv.org/abs/2302.02568
Less is More: Understanding Word-level Textual Adversarial Attack via n-gram Frequency Descend. (99%)
Ning Lu; Shengcai Liu; Zhirui Zhang; Qi Wang; Haifeng Liu; Ke Tang
  Word-level textual adversarial attacks have achieved striking performance in
fooling natural language processing models. However, the fundamental questions
of why these attacks are effective, and the intrinsic properties of the
adversarial examples (AEs), are still not well understood. This work attempts
to interpret textual attacks through the lens of $n$-gram frequency.
Specifically, it is revealed that existing word-level attacks exhibit a strong
tendency toward generation of examples with $n$-gram frequency descend
($n$-FD). Intuitively, this finding suggests a natural way to improve model
robustness by training the model on the $n$-FD examples. To verify this idea,
we devise a model-agnostic and gradient-free AE generation approach that relies
solely on the $n$-gram frequency information, and further integrate it into the
recently proposed convex hull framework for adversarial training. Surprisingly,
the resultant method performs quite similarly to the original gradient-based
method in terms of model robustness. These findings provide a
human-understandable perspective for interpreting word-level textual
adversarial attacks, and a new direction to improve model robustness.


http://arxiv.org/abs/2302.03251
SCALE-UP: An Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled Prediction Consistency. (92%)
Junfeng Guo; Yiming Li; Xun Chen; Hanqing Guo; Lichao Sun; Cong Liu
  Deep neural networks (DNNs) are vulnerable to backdoor attacks, where
adversaries embed a hidden backdoor trigger during the training process for
malicious prediction manipulation. These attacks pose great threats to the
applications of DNNs under the real-world machine learning as a service (MLaaS)
setting, where the deployed model is fully black-box while the users can only
query and obtain its predictions. Currently, there are many existing defenses
to reduce backdoor threats. However, almost all of them cannot be adopted in
MLaaS scenarios since they require getting access to or even modifying the
suspicious models. In this paper, we propose a simple yet effective black-box
input-level backdoor detection, called SCALE-UP, which requires only the
predicted labels to alleviate this problem. Specifically, we identify and
filter malicious testing samples by analyzing their prediction consistency
during the pixel-wise amplification process. Our defense is motivated by an
intriguing observation (dubbed scaled prediction consistency) that the
predictions of poisoned samples are significantly more consistent compared to
those of benign ones when amplifying all pixel values. Besides, we also provide
theoretical foundations to explain this phenomenon. Extensive experiments are
conducted on benchmark datasets, verifying the effectiveness and efficiency of
our defense and its resistance to potential adaptive attacks. Our codes are
available at https://github.com/JunfengGo/SCALE-UP.


http://arxiv.org/abs/2302.03015
Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustness. (87%)
Yuancheng Xu; Yanchao Sun; Micah Goldblum; Tom Goldstein; Furong Huang
  The robustness of a deep classifier can be characterized by its margins: the
decision boundary's distances to natural data points. However, it is unclear
whether existing robust training methods effectively increase the margin for
each vulnerable point during training. To understand this, we propose a
continuous-time framework for quantifying the relative speed of the decision
boundary with respect to each individual point. Through visualizing the moving
speed of the decision boundary under Adversarial Training, one of the most
effective robust training algorithms, a surprising moving-behavior is revealed:
the decision boundary moves away from some vulnerable points but simultaneously
moves closer to others, decreasing their margins. To alleviate these
conflicting dynamics of the decision boundary, we propose Dynamics-aware Robust
Training (DyART), which encourages the decision boundary to engage in movement
that prioritizes increasing smaller margins. In contrast to prior works, DyART
directly operates on the margins rather than their indirect approximations,
allowing for more targeted and effective robustness improvement. Experiments on
the CIFAR-10 and Tiny-ImageNet datasets verify that DyART alleviates the
conflicting dynamics of the decision boundary and obtains improved robustness
under various perturbation sizes compared to the state-of-the-art defenses. Our
code is available at
https://github.com/Yuancheng-Xu/Dynamics-Aware-Robust-Training.


http://arxiv.org/abs/2302.02829
Collective Robustness Certificates: Exploiting Interdependence in Graph Neural Networks. (75%)
Jan Schuchardt; Aleksandar Bojchevski; Johannes Gasteiger; Stephan Günnemann
  In tasks like node classification, image segmentation, and named-entity
recognition we have a classifier that simultaneously outputs multiple
predictions (a vector of labels) based on a single input, i.e. a single graph,
image, or document respectively. Existing adversarial robustness certificates
consider each prediction independently and are thus overly pessimistic for such
tasks. They implicitly assume that an adversary can use different perturbed
inputs to attack different predictions, ignoring the fact that we have a single
shared input. We propose the first collective robustness certificate which
computes the number of predictions that are simultaneously guaranteed to remain
stable under perturbation, i.e. cannot be attacked. We focus on Graph Neural
Networks and leverage their locality property - perturbations only affect the
predictions in a close neighborhood - to fuse multiple single-node certificates
into a drastically stronger collective certificate. For example, on the
Citeseer dataset our collective certificate for node classification increases
the average number of certifiable feature perturbations from $7$ to $351$.


http://arxiv.org/abs/2302.02907
GAT: Guided Adversarial Training with Pareto-optimal Auxiliary Tasks. (67%)
Salah Ghamizi; Jingfeng Zhang; Maxime Cordy; Mike Papadakis; Masashi Sugiyama; Yves Le Traon
  While leveraging additional training data is well established to improve
adversarial robustness, it incurs the unavoidable cost of data collection and
the heavy computation to train models. To mitigate the costs, we propose
\textit{Guided Adversarial Training } (GAT), a novel adversarial training
technique that exploits auxiliary tasks under a limited set of training data.
Our approach extends single-task models into multi-task models during the
min-max optimization of adversarial training, and drives the loss optimization
with a regularization of the gradient curvature across multiple tasks. GAT
leverages two types of auxiliary tasks: self-supervised tasks, where the labels
are generated automatically, and domain-knowledge tasks, where human experts
provide additional labels. Experimentally, under limited data, GAT increases
the robust accuracy on CIFAR-10 up to four times (from 11% to 42% robust
accuracy) and the robust AUC of CheXpert medical imaging dataset from 50\% to
83\%. On the full CIFAR-10 dataset, GAT outperforms eight state-of-the-art
adversarial training strategies.
  Our large study across five datasets and six tasks demonstrates that task
augmentation is an efficient alternative to data augmentation, and can be key
to achieving both clean and robust performances.


http://arxiv.org/abs/2302.02607
Target-based Surrogates for Stochastic Optimization. (1%)
Jonathan Wilder Lavington; Sharan Vaswani; Reza Babanezhad; Mark Schmidt; Nicolas Le Roux
  We consider minimizing functions for which it is expensive to compute the
(possibly stochastic) gradient. Such functions are prevalent in reinforcement
learning, imitation learning and adversarial training. Our target optimization
framework uses the (expensive) gradient computation to construct surrogate
functions in a target space (e.g. the logits output by a linear model for
classification) that can be minimized efficiently. This allows for multiple
parameter updates to the model, amortizing the cost of gradient computation. In
the full-batch setting, we prove that our surrogate is a global upper-bound on
the loss, and can be (locally) minimized using a black-box optimization
algorithm. We prove that the resulting majorization-minimization algorithm
ensures convergence to a stationary point of the loss. Next, we instantiate our
framework in the stochastic setting and propose the $SSO$ algorithm, which can
be viewed as projected stochastic gradient descent in the target space. This
connection enables us to prove theoretical guarantees for $SSO$ when minimizing
convex functions. Our framework allows the use of standard stochastic
optimization algorithms to construct surrogates which can be minimized by any
deterministic optimization method. To evaluate our framework, we consider a
suite of supervised learning and imitation learning problems. Our experiments
indicate the benefits of target optimization and the effectiveness of $SSO$.


http://arxiv.org/abs/2302.02924
Dropout Injection at Test Time for Post Hoc Uncertainty Quantification in Neural Networks. (1%)
Emanuele Ledda; Giorgio Fumera; Fabio Roli
  Among Bayesian methods, Monte-Carlo dropout provides principled tools for
evaluating the epistemic uncertainty of neural networks. Its popularity
recently led to seminal works that proposed activating the dropout layers only
during inference for evaluating uncertainty. This approach, which we call
dropout injection, provides clear benefits over its traditional counterpart
(which we call embedded dropout) since it allows one to obtain a post hoc
uncertainty measure for any existing network previously trained without
dropout, avoiding an additional, time-consuming training process.
Unfortunately, no previous work compared injected and embedded dropout;
therefore, we provide the first thorough investigation, focusing on regression
problems. The main contribution of our work is to provide guidelines on the
effective use of injected dropout so that it can be a practical alternative to
the current use of embedded dropout. In particular, we show that its
effectiveness strongly relies on a suitable scaling of the corresponding
uncertainty measure, and we discuss the trade-off between negative
log-likelihood and calibration error as a function of the scale factor.
Experimental results on UCI data sets and crowd counting benchmarks support our
claim that dropout injection can effectively behave as a competitive post hoc
uncertainty quantification technique.


http://arxiv.org/abs/2302.02502
On the Role of Contrastive Representation Learning in Adversarial Robustness: An Empirical Study. (54%)
Fatemeh Ghofrani; Mehdi Yaghouti; Pooyan Jamshidi
  Self-supervised contrastive learning has solved one of the significant
obstacles in deep learning by alleviating the annotation cost. This advantage
comes with the price of false negative-pair selection without any label
information. Supervised contrastive learning has emerged as an extension of
contrastive learning to eliminate this issue. However, aside from accuracy,
there is a lack of understanding about the impacts of adversarial training on
the representations learned by these learning schemes. In this work, we utilize
supervised learning as a baseline to comprehensively study the robustness of
contrastive and supervised contrastive learning under different adversarial
training scenarios. Then, we begin by looking at how adversarial training
affects the learned representations in hidden layers, discovering more
redundant representations between layers of the model. Our results on CIFAR-10
and CIFAR-100 image classification benchmarks demonstrate that this redundancy
is highly reduced by adversarial fine-tuning applied to the contrastive
learning scheme, leading to more robust representations. However, adversarial
fine-tuning is not very effective for supervised contrastive learning and
supervised learning schemes. Our code is released at
https://github.com/softsys4ai/CL-Robustness.


http://arxiv.org/abs/2302.02213
CosPGD: a unified white-box adversarial attack for pixel-wise prediction tasks. (98%)
Shashank Agnihotri; Margret Keuper
  While neural networks allow highly accurate predictions in many tasks, their
lack in robustness towards even slight input perturbations hampers their
deployment in many real-world applications. Recent research towards evaluating
the robustness of neural networks such as the seminal \emph{projected gradient
descent} (PGD) attack and subsequent works and benchmarks have therefore drawn
significant attention. Yet, such methods focus predominantly on classification
tasks, while only a few approaches specifically address the analysis of
pixel-wise prediction tasks such as semantic segmentation, optical flow, or
disparity estimation. One notable exception is the recently proposed SegPGD
attack, which could showcase the importance of pixel-wise attacks for
evaluating semantic segmentation. While SegPGD is limited to pixel-wise
classification (i.e. segmentation), in this work, we propose CosPGD, a novel
white-box adversarial attack that allows to optimize dedicated attacks for any
pixel-wise prediction task in a unified setting. It leverages the cosine
similarity between the predictions and ground truth to extend directly from
classification tasks to regression settings. Further, we empirically show the
superior performance of CosPGD for semantic segmentation as well as for optical
flow and disparity estimation.


http://arxiv.org/abs/2302.02216
A Minimax Approach Against Multi-Armed Adversarial Attacks Detection. (86%)
Federica Granese; Marco Romanelli; Siddharth Garg; Pablo Piantanida
  Multi-armed adversarial attacks, in which multiple algorithms and objective
loss functions are simultaneously used at evaluation time, have been shown to
be highly successful in fooling state-of-the-art adversarial examples detectors
while requiring no specific side information about the detection mechanism. By
formalizing the problem at hand, we can propose a solution that aggregates the
soft-probability outputs of multiple pre-trained detectors according to a
minimax approach. The proposed framework is mathematically sound, easy to
implement, and modular, allowing for integrating existing or future detectors.
Through extensive evaluation on popular datasets (e.g., CIFAR10 and SVHN), we
show that our aggregation consistently outperforms individual state-of-the-art
detectors against multi-armed adversarial attacks, making it an effective
solution to improve the resilience of available methods.


http://arxiv.org/abs/2302.02300
Run-Off Election: Improved Provable Defense against Data Poisoning Attacks. (81%)
Keivan Rezaei; Kiarash Banihashem; Atoosa Chegini; Soheil Feizi
  In data poisoning attacks, an adversary tries to change a model's prediction
by adding, modifying, or removing samples in the training data. Recently,
ensemble-based approaches for obtaining provable defenses against data
poisoning have been proposed where predictions are done by taking a majority
vote across multiple base models. In this work, we show that merely considering
the majority vote in ensemble defenses is wasteful as it does not effectively
utilize available information in the logits layers of the base models. Instead,
we propose Run-Off Election (ROE), a novel aggregation method based on a
two-round election across the base models: In the first round, models vote for
their preferred class and then a second, Run-Off election is held between the
top two classes in the first round. Based on this approach, we propose DPA+ROE
and FA+ROE defense methods based on Deep Partition Aggregation (DPA) and Finite
Aggregation (FA) approaches from prior work. We show how to obtain robustness
for these methods using ideas inspired by dynamic programming and duality. We
evaluate our methods on MNIST, CIFAR-10, and GTSRB and obtain improvements in
certified accuracy by up to 4.73%, 3.63%, and 3.54%, respectively, establishing
a new state-of-the-art in (pointwise) certified robustness against data
poisoning. In many cases, our approach outperforms the state-of-the-art, even
when using 32 times less computational power.


http://arxiv.org/abs/2302.02208
Certified Robust Control under Adversarial Perturbations. (78%)
Jinghan Yang; Hunmin Kim; Wenbin Wan; Naira Hovakimyan; Yevgeniy Vorobeychik
  Autonomous systems increasingly rely on machine learning techniques to
transform high-dimensional raw inputs into predictions that are then used for
decision-making and control. However, it is often easy to maliciously
manipulate such inputs and, as a result, predictions. While effective
techniques have been proposed to certify the robustness of predictions to
adversarial input perturbations, such techniques have been disembodied from
control systems that make downstream use of the predictions. We propose the
first approach for composing robustness certification of predictions with
respect to raw input perturbations with robust control to obtain certified
robustness of control to adversarial input perturbations. We use a case study
of adaptive vehicle control to illustrate our approach and show the value of
the resulting end-to-end certificates through extensive experiments.


http://arxiv.org/abs/2302.02162
AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Decision Tree Models. (1%)
Abdullah Caglar Oksuz; Anisa Halimi; Erman Ayday
  Model extraction attack is one of the most prominent adversarial techniques
to target machine learning models along with membership inference attack and
model inversion attack. On the other hand, Explainable Artificial Intelligence
(XAI) is a set of techniques and procedures to explain the decision making
process behind AI. XAI is a great tool to understand the reasoning behind AI
models but the data provided for such revelation creates security and privacy
vulnerabilities. In this poster, we propose AUTOLYCUS, a model extraction
attack that exploits the explanations provided by LIME to infer the decision
boundaries of decision tree models and create extracted surrogate models that
behave similar to a target model.


http://arxiv.org/abs/2302.02023
TextShield: Beyond Successfully Detecting Adversarial Sentences in Text Classification. (96%)
Lingfeng Shen; Ze Zhang; Haiyun Jiang; Ying Chen
  Adversarial attack serves as a major challenge for neural network models in
NLP, which precludes the model's deployment in safety-critical applications. A
recent line of work, detection-based defense, aims to distinguish adversarial
sentences from benign ones. However, {the core limitation of previous detection
methods is being incapable of giving correct predictions on adversarial
sentences unlike defense methods from other paradigms.} To solve this issue,
this paper proposes TextShield: (1) we discover a link between text attack and
saliency information, and then we propose a saliency-based detector, which can
effectively detect whether an input sentence is adversarial or not. (2) We
design a saliency-based corrector, which converts the detected adversary
sentences to benign ones. By combining the saliency-based detector and
corrector, TextShield extends the detection-only paradigm to a
detection-correction paradigm, thus filling the gap in the existing
detection-based defense. Comprehensive experiments show that (a) TextShield
consistently achieves higher or comparable performance than state-of-the-art
defense methods across various attacks on different benchmarks. (b) our
saliency-based detector outperforms existing detectors for detecting
adversarial sentences.


http://arxiv.org/abs/2302.02012
DeTorrent: An Adversarial Padding-only Traffic Analysis Defense. (73%)
James K Holland; Jason Carpenter; Se Eun Oh; Nicholas Hopper
  While anonymity networks like Tor aim to protect the privacy of their users,
they are vulnerable to traffic analysis attacks such as Website Fingerprinting
(WF) and Flow Correlation (FC). Recent implementations of WF and FC attacks,
such as Tik-Tok and DeepCoFFEA, have shown that the attacks can be effectively
carried out, threatening user privacy. Consequently, there is a need for
effective traffic analysis defense.
  There are a variety of existing defenses, but most are either ineffective,
incur high latency and bandwidth overhead, or require additional
infrastructure. As a result, we aim to design a traffic analysis defense that
is efficient and highly resistant to both WF and FC attacks. We propose
DeTorrent, which uses competing neural networks to generate and evaluate
traffic analysis defenses that insert 'dummy' traffic into real traffic flows.
DeTorrent operates with moderate overhead and without delaying traffic. In a
closed-world WF setting, it reduces an attacker's accuracy by 60.5%, a
reduction 9.5% better than the next-best padding-only defense. Against the
state-of-the-art FC attacker, DeTorrent reduces the true positive rate for a
$10^{-4}$ false positive rate to about .30, which is less than half that of the
next-best defense. We also demonstrate DeTorrent's practicality by deploying it
alongside the Tor network and find that it maintains its performance when
applied to live traffic.


http://arxiv.org/abs/2302.01740
A Systematic Evaluation of Backdoor Trigger Characteristics in Image Classification. (56%)
Gorka Abad; Jing Xu; Stefanos Koffas; Behrad Tajalli; Stjepan Picek
  Deep learning achieves outstanding results in many machine learning tasks.
Nevertheless, it is vulnerable to backdoor attacks that modify the training set
to embed a secret functionality in the trained model. The modified training
samples have a secret property, i.e., a trigger. At inference time, the secret
functionality is activated when the input contains the trigger, while the model
functions correctly in other cases. While there are many known backdoor attacks
(and defenses), deploying a stealthy attack is still far from trivial.
Successfully creating backdoor triggers heavily depends on numerous parameters.
Unfortunately, research has not yet determined which parameters contribute most
to the attack performance.
  This paper systematically analyzes the most relevant parameters for the
backdoor attacks, i.e., trigger size, position, color, and poisoning rate.
Using transfer learning, which is very common in computer vision, we evaluate
the attack on numerous state-of-the-art models (ResNet, VGG, AlexNet, and
GoogLeNet) and datasets (MNIST, CIFAR10, and TinyImageNet). Our attacks cover
the majority of backdoor settings in research, providing concrete directions
for future works. Our code is publicly available to facilitate the
reproducibility of our results.


http://arxiv.org/abs/2302.01629
Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels. (15%)
Simone Bombari; Shayan Kiyani; Marco Mondelli
  Machine learning models are vulnerable to adversarial perturbations, and a
thought-provoking paper by Bubeck and Sellke has analyzed this phenomenon
through the lens of over-parameterization: interpolating smoothly the data
requires significantly more parameters than simply memorizing it. However, this
"universal" law provides only a necessary condition for robustness, and it is
unable to discriminate between models. In this paper, we address these gaps by
focusing on empirical risk minimization in two prototypical settings, namely,
random features and the neural tangent kernel (NTK). We prove that, for random
features, the model is not robust for any degree of over-parameterization, even
when the necessary condition coming from the universal law of robustness is
satisfied. In contrast, for even activations, the NTK model meets the universal
lower bound, and it is robust as soon as the necessary condition on
over-parameterization is fulfilled. This also addresses a conjecture in prior
work by Bubeck, Li and Nagaraj. Our analysis decouples the effect of the kernel
of the model from an "interaction matrix", which describes the interaction with
the test data and captures the effect of the activation. Our theoretical
results are corroborated by numerical evidence on both synthetic and standard
datasets (MNIST, CIFAR-10).


http://arxiv.org/abs/2302.01961
Asymmetric Certified Robustness via Feature-Convex Neural Networks. (8%)
Samuel Pfrommer; Brendon G. Anderson; Julien Piet; Somayeh Sojoudi
  Recent works have introduced input-convex neural networks (ICNNs) as learning
models with advantageous training, inference, and generalization properties
linked to their convex structure. In this paper, we propose a novel
feature-convex neural network architecture as the composition of an ICNN with a
Lipschitz feature map in order to achieve adversarial robustness. We consider
the asymmetric binary classification setting with one "sensitive" class, and
for this class we prove deterministic, closed-form, and easily-computable
certified robust radii for arbitrary $\ell_p$-norms. We theoretically justify
the use of these models by characterizing their decision region geometry,
extending the universal approximation theorem for ICNN regression to the
classification setting, and proving a lower bound on the probability that such
models perfectly fit even unstructured uniformly distributed data in
sufficiently high dimensions. Experiments on Malimg malware classification and
subsets of MNIST, Fashion-MNIST, and CIFAR-10 datasets show that feature-convex
classifiers attain state-of-the-art certified $\ell_1$-radii as well as
substantial $\ell_2$- and $\ell_{\infty}$-radii while being far more
computationally efficient than any competitive baseline.


http://arxiv.org/abs/2302.02042
BarrierBypass: Out-of-Sight Clean Voice Command Injection Attacks through Physical Barriers. (2%)
Payton Walker; Tianfang Zhang; Cong Shi; Nitesh Saxena; Yingying Chen
  The growing adoption of voice-enabled devices (e.g., smart speakers),
particularly in smart home environments, has introduced many security
vulnerabilities that pose significant threats to users' privacy and safety.
When multiple devices are connected to a voice assistant, an attacker can cause
serious damage if they can gain control of these devices. We ask where and how
can an attacker issue clean voice commands stealthily across a physical
barrier, and perform the first academic measurement study of this nature on the
command injection attack. We present the BarrierBypass attack that can be
launched against three different barrier-based scenarios termed across-door,
across-window, and across-wall. We conduct a broad set of experiments to
observe the command injection attack success rates for multiple speaker samples
(TTS and live human recorded) at different command audio volumes (65, 75, 85
dB), and smart speaker locations (0.1-4.0m from barrier). Against Amazon Echo
Dot 2, BarrierBypass is able to achieve 100% wake word and command injection
success for the across-wall and across-window attacks, and for the across-door
attack (up to 2 meters). At 4 meters for the across-door attack, BarrierBypass
can achieve 90% and 80% injection accuracy for the wake word and command,
respectively. Against Google Home mini BarrierBypass is able to achieve 100%
wake word injection accuracy for all attack scenarios. For command injection
BarrierBypass can achieve 100% accuracy for all the three barrier settings (up
to 2 meters). For the across-door attack at 4 meters, BarrierBypass can achieve
80% command injection accuracy. Further, our demonstration using drones yielded
high command injection success, up to 100%. Overall, our results demonstrate
the potentially devastating nature of this vulnerability to control a user's
device from outside of the device's physical space.


http://arxiv.org/abs/2302.01855
From Robustness to Privacy and Back. (2%)
Hilal Asi; Jonathan Ullman; Lydia Zakynthinou
  We study the relationship between two desiderata of algorithms in statistical
inference and machine learning: differential privacy and robustness to
adversarial data corruptions. Their conceptual similarity was first observed by
Dwork and Lei (STOC 2009), who observed that private algorithms satisfy
robustness, and gave a general method for converting robust algorithms to
private ones. However, all general methods for transforming robust algorithms
into private ones lead to suboptimal error rates. Our work gives the first
black-box transformation that converts any adversarially robust algorithm into
one that satisfies pure differential privacy. Moreover, we show that for any
low-dimensional estimation task, applying our transformation to an optimal
robust estimator results in an optimal private estimator. Thus, we conclude
that for any low-dimensional task, the optimal error rate for
$\varepsilon$-differentially private estimators is essentially the same as the
optimal error rate for estimators that are robust to adversarially corrupting
$1/\varepsilon$ training samples. We apply our transformation to obtain new
optimal private estimators for several high-dimensional tasks, including
Gaussian (sparse) linear regression and PCA. Finally, we present an extension
of our transformation that leads to approximate differentially private
algorithms whose error does not depend on the range of the output space, which
is impossible under pure differential privacy.


http://arxiv.org/abs/2302.02031
Augmenting Rule-based DNS Censorship Detection at Scale with Machine Learning. (1%)
Jacob Alexander Markson Brown; Xi Jiang; Van Tran; Arjun Nitin Bhagoji; Nguyen Phong Hoang; Nick Feamster; Prateek Mittal; Vinod Yegneswaran
  The proliferation of global censorship has led to the development of a
plethora of measurement platforms to monitor and expose it. Censorship of the
domain name system (DNS) is a key mechanism used across different countries. It
is currently detected by applying heuristics to samples of DNS queries and
responses (probes) for specific destinations. These heuristics, however, are
both platform-specific and have been found to be brittle when censors change
their blocking behavior, necessitating a more reliable automated process for
detecting censorship. In this paper, we explore how machine learning (ML)
models can (1) help streamline the detection process, (2) improve the usability
of large-scale datasets for censorship detection, and (3) discover new
censorship instances and blocking signatures missed by existing heuristic
methods. Our study shows that supervised models, trained using expert-derived
labels on instances of known anomalies and possible censorship, can learn the
detection heuristics employed by different measurement platforms. More
crucially, we find that unsupervised models, trained solely on uncensored
instances, can identify new instances and variations of censorship missed by
existing heuristics. Moreover, both methods demonstrate the capability to
uncover a substantial number of new DNS blocking signatures, i.e., injected
fake IP addresses overlooked by existing heuristics. These results are
underpinned by an important methodological finding: comparing the outputs of
models trained using the same probes but with labels arising from independent
processes allows us to more reliably detect cases of censorship in the absence
of ground-truth labels of censorship.


http://arxiv.org/abs/2302.01056
Beyond Pretrained Features: Noisy Image Modeling Provides Adversarial Defense. (99%)
Zunzhi You; Daochang Liu; Chang Xu
  Masked Image Modeling (MIM) has been a prevailing framework for
self-supervised visual representation learning. Within the
pretraining-finetuning paradigm, the MIM framework trains an encoder by
reconstructing masked image patches with the help of a decoder which would be
abandoned when the encoder is used for finetuning. Despite its state-of-the-art
performance on clean images, MIM models are vulnerable to adversarial attacks,
limiting its real-world application, and few studies have focused on this
issue. In this paper, we have discovered that noisy image modeling (NIM), a
variant of MIM that uses denoising as the pre-text task, provides not only good
pretrained visual features, but also effective adversarial defense for
downstream models. To achieve a better accuracy-robustness trade-off, we
further propose to sample the hyperparameter that controls the reconstruction
difficulty from random distributions instead of setting it globally, and
fine-tune downstream networks with denoised images. Experimental results
demonstrate that our pre-trained denoising autoencoders are effective against
different white-box, gray-box, and black-box attacks without being trained with
adversarial images, while not harming the clean accuracy of fine-tuned models.
Source code and models will be made available.


http://arxiv.org/abs/2302.00944
TransFool: An Adversarial Attack against Neural Machine Translation Models. (99%)
Sahar Sadrizadeh; Ljiljana Dolamic; Pascal Frossard
  Deep neural networks have been shown to be vulnerable to small perturbations
of their inputs, known as adversarial attacks. In this paper, we investigate
the vulnerability of Neural Machine Translation (NMT) models to adversarial
attacks and propose a new attack algorithm called TransFool. To fool NMT
models, TransFool builds on a multi-term optimization problem and a gradient
projection step. By integrating the embedding representation of a language
model, we generate fluent adversarial examples in the source language that
maintain a high level of semantic similarity with the clean samples.
Experimental results demonstrate that, for different translation tasks and NMT
architectures, our white-box attack can severely degrade the translation
quality while the semantic similarity between the original and the adversarial
sentences stays high. Moreover, we show that TransFool is transferable to
unknown target models. Finally, based on automatic and human evaluations,
TransFool leads to improvement in terms of success rate, semantic similarity,
and fluency compared to the existing attacks both in white-box and black-box
settings. Thus, TransFool permits us to better characterize the vulnerability
of NMT models and outlines the necessity to design strong defense mechanisms
and more robust NMT systems for real-life applications.


http://arxiv.org/abs/2302.01375
On the Robustness of Randomized Ensembles to Adversarial Perturbations. (75%)
Hassan Dbouk; Naresh R. Shanbhag
  Randomized ensemble classifiers (RECs), where one classifier is randomly
selected during inference, have emerged as an attractive alternative to
traditional ensembling methods for realizing adversarially robust classifiers
with limited compute requirements. However, recent works have shown that
existing methods for constructing RECs are more vulnerable than initially
claimed, casting major doubts on their efficacy and prompting fundamental
questions such as: "When are RECs useful?", "What are their limits?", and "How
do we train them?". In this work, we first demystify RECs as we derive
fundamental results regarding their theoretical limits, necessary and
sufficient conditions for them to be useful, and more. Leveraging this new
understanding, we propose a new boosting algorithm (BARRE) for training robust
RECs, and empirically demonstrate its effectiveness at defending against strong
$\ell_\infty$ norm-bounded adversaries across various network architectures and
datasets.


http://arxiv.org/abs/2302.01459
A sliced-Wasserstein distance-based approach for out-of-class-distribution detection. (62%)
Mohammad Shifat E Rabbi; Abu Hasnat Mohammad Rubaiyat; Yan Zhuang; Gustavo K Rohde
  There exist growing interests in intelligent systems for numerous medical
imaging, image processing, and computer vision applications, such as face
recognition, medical diagnosis, character recognition, and self-driving cars,
among others. These applications usually require solving complex classification
problems involving complex images with unknown data generative processes. In
addition to recent successes of the current classification approaches relying
on feature engineering and deep learning, several shortcomings of them, such as
the lack of robustness, generalizability, and interpretability, have also been
observed. These methods often require extensive training data, are
computationally expensive, and are vulnerable to out-of-distribution samples,
e.g., adversarial attacks. Recently, an accurate, data-efficient,
computationally efficient, and robust transport-based classification approach
has been proposed, which describes a generative model-based problem formulation
and closed-form solution for a specific category of classification problems.
However, all these approaches lack mechanisms to detect test samples outside
the class distributions used during training. In real-world settings, where the
collected training samples are unable to exhaust or cover all classes, the
traditional classification schemes are unable to handle the unseen classes
effectively, which is especially an important issue for safety-critical
systems, such as self-driving and medical imaging diagnosis. In this work, we
propose a method for detecting out-of-class distributions based on the
distribution of sliced-Wasserstein distance from the Radon Cumulative
Distribution Transform (R-CDT) subspace. We tested our method on the MNIST and
two medical image datasets and reported better accuracy than the
state-of-the-art methods without an out-of-class distribution detection
procedure.


http://arxiv.org/abs/2302.01381
Effective Robustness against Natural Distribution Shifts for Models with Different Training Data. (13%)
Zhouxing Shi; Nicholas Carlini; Ananth Balashankar; Ludwig Schmidt; Cho-Jui Hsieh; Alex Beutel; Yao Qin
  ``Effective robustness'' measures the extra out-of-distribution (OOD)
robustness beyond what can be predicted from the in-distribution (ID)
performance. Existing effective robustness evaluations typically use a single
test set such as ImageNet to evaluate ID accuracy. This becomes problematic
when evaluating models trained on different data distributions, e.g., comparing
models trained on ImageNet vs. zero-shot language-image pre-trained models
trained on LAION. In this paper, we propose a new effective robustness
evaluation metric to compare the effective robustness of models trained on
different data distributions. To do this we control for the accuracy on
multiple ID test sets that cover the training distributions for all the
evaluated models. Our new evaluation metric provides a better estimate of the
effectiveness robustness and explains the surprising effective robustness gains
of zero-shot CLIP-like models exhibited when considering only one ID dataset,
while the gains diminish under our evaluation.


http://arxiv.org/abs/2302.00947
SPECWANDS: An Efficient Priority-based Scheduler Against Speculation Contention Attacks. (10%)
Bowen Tang; Chenggang Wu; Pen-Chung Yew; Yinqian Zhang; Mengyao Xie; Yuanming Lai; Yan Kang; Wei Wang; Qiang Wei; Zhe Wang
  Transient Execution Attacks (TEAs) have gradually become a major security
threat to modern high-performance processors. They exploit the vulnerability of
speculative execution to illegally access private data, and transmit them
through timing-based covert channels. While new vulnerabilities are discovered
continuously, the covert channels can be categorised to two types: 1)
Persistent Type, in which covert channels are based on the layout changes of
buffering, e.g. through caches or TLBs; 2) Volatile Type, in which covert
channels are based on the contention of sharing resources, e.g. through
execution units or issuing ports. The defenses against the persistent-type
covert channels have been well addressed, while those for the volatile-type are
still rather inadequate. Existing mitigation schemes for the volatile type such
as Speculative Compression and Time-Division-Multiplexing will introduce
significant overhead due to the need to stall the pipeline or to disallow
resource sharing. In this paper, we look into such attacks and defenses with a
new perspective, and propose a scheduling-based mitigation scheme, called
SPECWANDS. It consists of three priority-based scheduling policies to prevent
an attacker from transmitting the secret in different contention situations.
SPECWANDS not only can defend against both inter-thread and intra-thread based
attacks, but also can keep most of the performance benefit from speculative
execution and resource-sharing. We evaluate its runtime overhead on SPEC
2006/2017 benchmarks. The experimental results show that SPECWANDS has a
significant performance advantage (3%-5%) over the other two representative
schemes.


http://arxiv.org/abs/2302.01404
Provably Bounding Neural Network Preimages. (8%)
Suhas Dj Kotha; Christopher Dj Brix; Zico Dj Kolter; Dj Krishnamurthy; Dvijotham; Huan Zhang
  Most work on the formal verification of neural networks has focused on
bounding forward images of neural networks, i.e., the set of outputs of a
neural network that correspond to a given set of inputs (for example, bounded
perturbations of a nominal input). However, many use cases of neural network
verification require solving the inverse problem, i.e, over-approximating the
set of inputs that lead to certain outputs. In this work, we present the first
efficient bound propagation algorithm, INVPROP, for verifying properties over
the preimage of a linearly constrained output set of a neural network, which
can be combined with branch-and-bound to achieve completeness. Our efficient
algorithm allows multiple passes of intermediate bound refinements, which are
crucial for tight inverse verification because the bounds of an intermediate
layer depend on relaxations both before and after this layer. We demonstrate
our algorithm on applications related to quantifying safe control regions for a
dynamical system and detecting out-of-distribution inputs to a neural network.
Our results show that in certain settings, we can find over-approximations that
are over 2500 times tighter than prior work while being 2.5 times faster on the
same hardware.


http://arxiv.org/abs/2302.01474
Defensive ML: Defending Architectural Side-channels with Adversarial Obfuscation. (2%)
Hyoungwook Nam; Raghavendra Pradyumna Pothukuchi; Bo Li; Nam Sung Kim; Josep Torrellas
  Side-channel attacks that use machine learning (ML) for signal analysis have
become prominent threats to computer security, as ML models easily find
patterns in signals. To address this problem, this paper explores using
Adversarial Machine Learning (AML) methods as a defense at the computer
architecture layer to obfuscate side channels. We call this approach Defensive
ML, and the generator to obfuscate signals, defender. Defensive ML is a
workflow to design, implement, train, and deploy defenders for different
environments. First, we design a defender architecture given the physical
characteristics and hardware constraints of the side-channel. Next, we use our
DefenderGAN structure to train the defender. Finally, we apply defensive ML to
thwart two side-channel attacks: one based on memory contention and the other
on application power. The former uses a hardware defender with ns-level
response time that attains a high level of security with half the performance
impact of a traditional scheme; the latter uses a software defender with
ms-level response time that provides better security than a traditional scheme
with only 70% of its power overhead.


http://arxiv.org/abs/2302.01440
Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applications. (1%)
Chengyu Dong
  Deep neural networks have seen enormous success in various real-world
applications. Beyond their predictions as point estimates, increasing attention
has been focused on quantifying the uncertainty of their predictions. In this
review, we show that the uncertainty of deep neural networks is not only
important in a sense of interpretability and transparency, but also crucial in
further advancing their performance, particularly in learning systems seeking
robustness and efficiency. We will generalize the definition of the uncertainty
of deep neural networks to any number or vector that is associated with an
input or an input-label pair, and catalog existing methods on ``mining'' such
uncertainty from a deep model. We will include those methods from the classic
field of uncertainty quantification as well as those methods that are specific
to deep neural networks. We then show a wide spectrum of applications of such
generalized uncertainty in realistic learning tasks including robust learning
such as noisy learning, adversarially robust learning; data-efficient learning
such as semi-supervised and weakly-supervised learning; and model-efficient
learning such as model compression and knowledge distillation.


http://arxiv.org/abs/2302.01428
Dataset Distillation Fixes Dataset Reconstruction Attacks. (1%)
Noel Loo; Ramin Hasani; Mathias Lechner; Daniela Rus
  Modern deep learning requires large volumes of data, which could contain
sensitive or private information which cannot be leaked. Recent work has shown
for homogeneous neural networks a large portion of this training data could be
reconstructed with only access to the trained network parameters. While the
attack was shown to work empirically, there exists little formal understanding
of its effectiveness regime, and ways to defend against it. In this work, we
first build a stronger version of the dataset reconstruction attack and show
how it can provably recover its entire training set in the infinite width
regime. We then empirically study the characteristics of this attack on
two-layer networks and reveal that its success heavily depends on deviations
from the frozen infinite-width Neural Tangent Kernel limit. More importantly,
we formally show for the first time that dataset reconstruction attacks are a
variation of dataset distillation. This key theoretical result on the
unification of dataset reconstruction and distillation not only sheds more
light on the characteristics of the attack but enables us to design defense
mechanisms against them via distillation algorithms.


http://arxiv.org/abs/2302.00747
Universal Soldier: Using Universal Adversarial Perturbations for Detecting Backdoor Attacks. (99%)
Xiaoyun Xu; Oguzhan Ersoy; Stjepan Picek
  Deep learning models achieve excellent performance in numerous machine
learning tasks. Yet, they suffer from security-related issues such as
adversarial examples and poisoning (backdoor) attacks. A deep learning model
may be poisoned by training with backdoored data or by modifying inner network
parameters. Then, a backdoored model performs as expected when receiving a
clean input, but it misclassifies when receiving a backdoored input stamped
with a pre-designed pattern called "trigger". Unfortunately, it is difficult to
distinguish between clean and backdoored models without prior knowledge of the
trigger. This paper proposes a backdoor detection method by utilizing a special
type of adversarial attack, universal adversarial perturbation (UAP), and its
similarities with a backdoor trigger. We observe an intuitive phenomenon: UAPs
generated from backdoored models need fewer perturbations to mislead the model
than UAPs from clean models. UAPs of backdoored models tend to exploit the
shortcut from all classes to the target class, built by the backdoor trigger.
We propose a novel method called Universal Soldier for Backdoor detection (USB)
and reverse engineering potential backdoor triggers via UAPs. Experiments on
345 models trained on several datasets show that USB effectively detects the
injected backdoor and provides comparable or better results than
state-of-the-art methods.


http://arxiv.org/abs/2302.00537
Effectiveness of Moving Target Defenses for Adversarial Attacks in ML-based Malware Detection. (92%)
Aqib Rashid; Jose Such
  Several moving target defenses (MTDs) to counter adversarial ML attacks have
been proposed in recent years. MTDs claim to increase the difficulty for the
attacker in conducting attacks by regularly changing certain elements of the
defense, such as cycling through configurations. To examine these claims, we
study for the first time the effectiveness of several recent MTDs for
adversarial ML attacks applied to the malware detection domain. Under different
threat models, we show that transferability and query attack strategies can
achieve high levels of evasion against these defenses through existing and
novel attack strategies across Android and Windows. We also show that
fingerprinting and reconnaissance are possible and demonstrate how attackers
may obtain critical defense hyperparameters as well as information about how
predictions are produced. Based on our findings, we present key recommendations
for future work on the development of effective MTDs for adversarial attacks in
ML-based malware detection.


http://arxiv.org/abs/2302.00509
Exploring Semantic Perturbations on Grover. (56%)
Pranav Kulkarni; Ziqing Ji; Yan Xu; Marko Neskovic; Kevin Nolan
  With news and information being as easy to access as they currently are, it
is more important than ever to ensure that people are not mislead by what they
read. Recently, the rise of neural fake news (AI-generated fake news) and its
demonstrated effectiveness at fooling humans has prompted the development of
models to detect it. One such model is the Grover model, which can both detect
neural fake news to prevent it, and generate it to demonstrate how a model
could be misused to fool human readers. In this work we explore the Grover
model's fake news detection capabilities by performing targeted attacks through
perturbations on input news articles. Through this we test Grover's resilience
to these adversarial attacks and expose some potential vulnerabilities which
should be addressed in further iterations to ensure it can detect all types of
fake news accurately.


http://arxiv.org/abs/2302.01762
BackdoorBox: A Python Toolbox for Backdoor Learning. (10%)
Yiming Li; Mengxi Ya; Yang Bai; Yong Jiang; Shu-Tao Xia
  Third-party resources ($e.g.$, samples, backbones, and pre-trained models)
are usually involved in the training of deep neural networks (DNNs), which
brings backdoor attacks as a new training-phase threat. In general, backdoor
attackers intend to implant hidden backdoor in DNNs, so that the attacked DNNs
behave normally on benign samples whereas their predictions will be maliciously
changed to a pre-defined target label if hidden backdoors are activated by
attacker-specified trigger patterns. To facilitate the research and development
of more secure training schemes and defenses, we design an open-sourced Python
toolbox that implements representative and advanced backdoor attacks and
defenses under a unified and flexible framework. Our toolbox has four important
and promising characteristics, including consistency, simplicity, flexibility,
and co-development. It allows researchers and developers to easily implement
and compare different methods on benchmark or their local datasets. This Python
toolbox, namely \texttt{BackdoorBox}, is available at
\url{https://github.com/THUYimingLi/BackdoorBox}.


http://arxiv.org/abs/2301.13869
Reverse engineering adversarial attacks with fingerprints from adversarial examples. (99%)
David Aaron Embedded Intelligence Nicholson; Vincent Embedded Intelligence Emanuele
  In spite of intense research efforts, deep neural networks remain vulnerable
to adversarial examples: an input that forces the network to confidently
produce incorrect outputs. Adversarial examples are typically generated by an
attack algorithm that optimizes a perturbation added to a benign input. Many
such algorithms have been developed. If it were possible to reverse engineer
attack algorithms from adversarial examples, this could deter bad actors
because of the possibility of attribution. Here we formulate reverse
engineering as a supervised learning problem where the goal is to assign an
adversarial example to a class that represents the algorithm and parameters
used. To our knowledge it has not been previously shown whether this is even
possible. We first test whether we can classify the perturbations added to
images by attacks on undefended single-label image classification models.
Taking a ``fight fire with fire'' approach, we leverage the sensitivity of deep
neural networks to adversarial examples, training them to classify these
perturbations. On a 17-class dataset (5 attacks, 4 bounded with 4 epsilon
values each), we achieve an accuracy of 99.4\% with a ResNet50 model trained on
the perturbations. We then ask whether we can perform this task without access
to the perturbations, obtaining an estimate of them with signal processing
algorithms, an approach we call ``fingerprinting''. We find the JPEG algorithm
serves as a simple yet effective fingerprinter (85.05\% accuracy), providing a
strong baseline for future work. We discuss how our approach can be extended to
attack agnostic, learnable fingerprints, and to open-world scenarios with
unknown attacks.


http://arxiv.org/abs/2302.00094
The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models. (97%)
Son Quoc Tran; Phong Nguyen-Thuan Do; Uyen Le; Matt Kretchmar
  Pretrained language models have achieved super-human performances on many
Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative
inability to defend against adversarial attacks has spurred skepticism about
their natural language understanding. In this paper, we ask whether training
with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC
models against adversarial attacks. To explore that question, we fine-tune
three state-of-the-art language models on either SQuAD 1.1 or SQuAD 2.0 and
then evaluate their robustness under adversarial attacks. Our experiments
reveal that current models fine-tuned on SQuAD 2.0 do not initially appear to
be any more robust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure
of hidden robustness that can be leveraged to realize actual performance gains.
Furthermore, we find that the robustness of models fine-tuned on SQuAD 2.0
extends to additional out-of-domain datasets. Finally, we introduce a new
adversarial attack to reveal artifacts of SQuAD 2.0 that current MRC models are
learning.


http://arxiv.org/abs/2301.13694
Are Defenses for Graph Neural Networks Robust? (80%)
Felix Mujkanovic; Simon Geisler; Stephan Günnemann; Aleksandar Bojchevski
  A cursory reading of the literature suggests that we have made a lot of
progress in designing effective adversarial defenses for Graph Neural Networks
(GNNs). Yet, the standard methodology has a serious flaw - virtually all of the
defenses are evaluated against non-adaptive attacks leading to overly
optimistic robustness estimates. We perform a thorough robustness analysis of 7
of the most popular defenses spanning the entire spectrum of strategies, i.e.,
aimed at improving the graph, the architecture, or the training. The results
are sobering - most defenses show no or only marginal improvement compared to
an undefended baseline. We advocate using custom adaptive attacks as a gold
standard and we outline the lessons we learned from successfully designing such
attacks. Moreover, our diverse collection of perturbed graphs forms a
(black-box) unit test offering a first glance at a model's robustness.


http://arxiv.org/abs/2301.13487
Adversarial Training of Self-supervised Monocular Depth Estimation against Physical-World Attacks. (75%)
Zhiyuan Cheng; James Liang; Guanhong Tao; Dongfang Liu; Xiangyu Zhang
  Monocular Depth Estimation (MDE) is a critical component in applications such
as autonomous driving. There are various attacks against MDE networks. These
attacks, especially the physical ones, pose a great threat to the security of
such systems. Traditional adversarial training method requires ground-truth
labels hence cannot be directly applied to self-supervised MDE that does not
have ground-truth depth. Some self-supervised model hardening techniques (e.g.,
contrastive learning) ignore the domain knowledge of MDE and can hardly achieve
optimal performance. In this work, we propose a novel adversarial training
method for self-supervised MDE models based on view synthesis without using
ground-truth depth. We improve adversarial robustness against physical-world
attacks using L0-norm-bounded perturbation in training. We compare our method
with supervised learning based and contrastive learning based methods that are
tailored for MDE. Results on two representative MDE networks show that we
achieve better robustness against various adversarial attacks with nearly no
benign performance degradation.


http://arxiv.org/abs/2301.13486
Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond. (47%)
Meyer Scetbon; Elvis Dohmatob
  In this work we study the robustness to adversarial attacks, of
early-stopping strategies on gradient-descent (GD) methods for linear
regression. More precisely, we show that early-stopped GD is optimally robust
(up to an absolute constant) against Euclidean-norm adversarial attacks.
However, we show that this strategy can be arbitrarily sub-optimal in the case
of general Mahalanobis attacks. This observation is compatible with recent
findings in the case of classification~\cite{Vardi2022GradientMP} that show
that GD provably converges to non-robust models. To alleviate this issue, we
propose to apply instead a GD scheme on a transformation of the data adapted to
the attack. This data transformation amounts to apply feature-depending
learning rates and we show that this modified GD is able to handle any
Mahalanobis attack, as well as more general attacks under some conditions.
Unfortunately, choosing such adapted transformations can be hard for general
attacks. To the rescue, we design a simple and tractable estimator whose
adversarial risk is optimal up to within a multiplicative constant of 1.1124 in
the population regime, and works for any norm.


http://arxiv.org/abs/2301.13803
Fairness-aware Vision Transformer via Debiased Self-Attention. (47%)
Yao Qiang; Chengyin Li; Prashant Khanduri; Dongxiao Zhu
  Vision Transformer (ViT) has recently gained significant interest in solving
computer vision (CV) problems due to its capability of extracting informative
features and modeling long-range dependencies through the self-attention
mechanism. To fully realize the advantages of ViT in real-world applications,
recent works have explored the trustworthiness of ViT, including its robustness
and explainability. However, another desiderata, fairness has not yet been
adequately addressed in the literature. We establish that the existing
fairness-aware algorithms (primarily designed for CNNs) do not perform well on
ViT. This necessitates the need for developing our novel framework via Debiased
Self-Attention (DSA). DSA is a fairness-through-blindness approach that
enforces ViT to eliminate spurious features correlated with the sensitive
attributes for bias mitigation. Notably, adversarial examples are leveraged to
locate and mask the spurious features in the input image patches. In addition,
DSA utilizes an attention weights alignment regularizer in the training
objective to encourage learning informative features for target prediction.
Importantly, our DSA framework leads to improved fairness guarantees over prior
works on multiple prediction tasks without compromising target prediction
performance


http://arxiv.org/abs/2301.13838
Image Shortcut Squeezing: Countering Perturbative Availability Poisons with Compression. (12%)
Zhuoran Liu; Zhengyu Zhao; Martha Larson
  Perturbative availability poisoning (PAP) adds small changes to images to
prevent their use for model training. Current research adopts the belief that
practical and effective approaches to countering such poisons do not exist. In
this paper, we argue that it is time to abandon this belief. We present
extensive experiments showing that 12 state-of-the-art PAP methods are
vulnerable to Image Shortcut Squeezing (ISS), which is based on simple
compression. For example, on average, ISS restores the CIFAR-10 model accuracy
to $81.73\%$, surpassing the previous best preprocessing-based countermeasures
by $37.97\%$ absolute. ISS also (slightly) outperforms adversarial training and
has higher generalizability to unseen perturbation norms and also higher
efficiency. Our investigation reveals that the property of PAP perturbations
depends on the type of surrogate model used for poison generation, and it
explains why a specific ISS compression yields the best performance for a
specific type of PAP perturbation. We further test stronger, adaptive
poisoning, and show it falls short of being an ideal defense against ISS.
Overall, our results demonstrate the importance of considering various (simple)
countermeasures to ensure the meaningfulness of analysis carried out during the
development of availability poisons.


http://arxiv.org/abs/2301.13807
Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search. (1%)
Sepehr Sharifi; Donghwan Shin; Lionel C. Briand; Nathan Aschbacher
  In Machine Learning (ML)-enabled autonomous systems (MLASs), it is essential
to identify the hazard boundary of ML Components (MLCs) in the MLAS under
analysis. Given that such boundary captures the conditions in terms of MLC
behavior and system context that can lead to hazards, it can then be used to,
for example, build a safety monitor that can take any predefined fallback
mechanisms at runtime when reaching the hazard boundary. However, determining
such hazard boundary for an ML component is challenging. This is due to the
space combining system contexts (i.e., scenarios) and MLC behaviors (i.e.,
inputs and outputs) being far too large for exhaustive exploration and even to
handle using conventional metaheuristics, such as genetic algorithms.
Additionally, the high computational cost of simulations required to determine
any MLAS safety violations makes the problem even more challenging.
Furthermore, it is unrealistic to consider a region in the problem space
deterministically safe or unsafe due to the uncontrollable parameters in
simulations and the non-linear behaviors of ML models (e.g., deep neural
networks) in the MLAS under analysis. To address the challenges, we propose
MLCSHE (ML Component Safety Hazard Envelope), a novel method based on a
Cooperative Co-Evolutionary Algorithm (CCEA), which aims to tackle a
high-dimensional problem by decomposing it into two lower-dimensional search
subproblems. Moreover, we take a probabilistic view of safe and unsafe regions
and define a novel fitness function to measure the distance from the
probabilistic hazard boundary and thus drive the search effectively. We
evaluate the effectiveness and efficiency of MLCSHE on a complex Autonomous
Vehicle (AV) case study. Our evaluation results show that MLCSHE is
significantly more effective and efficient compared to a standard genetic
algorithm and random search.


http://arxiv.org/abs/2301.13122
Towards Adversarial Realism and Robust Learning for IoT Intrusion Detection and Classification. (99%)
João Vitorino; Isabel Praça; Eva Maia
  The Internet of Things (IoT) faces tremendous security challenges. Machine
learning models can be used to tackle the growing number of cyber-attack
variations targeting IoT systems, but the increasing threat posed by
adversarial attacks restates the need for reliable defense strategies. This
work describes the types of constraints required for an adversarial
cyber-attack example to be realistic and proposes a methodology for a
trustworthy adversarial robustness analysis with a realistic adversarial
evasion attack vector. The proposed methodology was used to evaluate three
supervised algorithms, Random Forest (RF), Extreme Gradient Boosting (XGB), and
Light Gradient Boosting Machine (LGBM), and one unsupervised algorithm,
Isolation Forest (IFOR). Constrained adversarial examples were generated with
the Adaptative Perturbation Pattern Method (A2PM), and evasion attacks were
performed against models created with regular and adversarial training. Even
though RF was the least affected in binary classification, XGB consistently
achieved the highest accuracy in multi-class classification. The obtained
results evidence the inherent susceptibility of tree-based algorithms and
ensembles to adversarial evasion attacks and demonstrates the benefits of
adversarial training and a security by design approach for a more robust IoT
network intrusion detection.


http://arxiv.org/abs/2301.12680
Feature-Space Bayesian Adversarial Learning Improved Malware Detector Robustness. (99%)
Bao Gia Doan; Shuiqiao Yang; Paul Montague; Vel Olivier De; Tamas Abraham; Seyit Camtepe; Salil S. Kanhere; Ehsan Abbasnejad; Damith C. Ranasinghe
  We present a new algorithm to train a robust malware detector. Modern malware
detectors rely on machine learning algorithms. Now, the adversarial objective
is to devise alterations to the malware code to decrease the chance of being
detected whilst preserving the functionality and realism of the malware.
Adversarial learning is effective in improving robustness but generating
functional and realistic adversarial malware samples is non-trivial. Because:
i) in contrast to tasks capable of using gradient-based feedback, adversarial
learning in a domain without a differentiable mapping function from the problem
space (malware code inputs) to the feature space is hard; and ii) it is
difficult to ensure the adversarial malware is realistic and functional. This
presents a challenge for developing scalable adversarial machine learning
algorithms for large datasets at a production or commercial scale to realize
robust malware detectors. We propose an alternative; perform adversarial
learning in the feature space in contrast to the problem space. We prove the
projection of perturbed, yet valid malware, in the problem space into feature
space will always be a subset of adversarials generated in the feature space.
Hence, by generating a robust network against feature-space adversarial
examples, we inherently achieve robustness against problem-space adversarial
examples. We formulate a Bayesian adversarial learning objective that captures
the distribution of models for improved robustness. We prove that our learning
method bounds the difference between the adversarial risk and empirical risk
explaining the improved robustness. We show that adversarially trained BNNs
achieve state-of-the-art robustness. Notably, adversarially trained BNNs are
robust against stronger attacks with larger attack budgets by a margin of up to
15% on a recent production-scale malware dataset of more than 20 million
samples.


http://arxiv.org/abs/2301.12968
Improving Adversarial Transferability with Scheduled Step Size and Dual Example. (99%)
Zeliang Zhang; Peihan Liu; Xiaosen Wang; Chenliang Xu
  Deep neural networks are widely known to be vulnerable to adversarial
examples, especially showing significantly poor performance on adversarial
examples generated under the white-box setting. However, most white-box attack
methods rely heavily on the target model and quickly get stuck in local optima,
resulting in poor adversarial transferability. The momentum-based methods and
their variants are proposed to escape the local optima for better
transferability. In this work, we notice that the transferability of
adversarial examples generated by the iterative fast gradient sign method
(I-FGSM) exhibits a decreasing trend when increasing the number of iterations.
Motivated by this finding, we argue that the information of adversarial
perturbations near the benign sample, especially the direction, benefits more
on the transferability. Thus, we propose a novel strategy, which uses the
Scheduled step size and the Dual example (SD), to fully utilize the adversarial
information near the benign sample. Our proposed strategy can be easily
integrated with existing adversarial attack methods for better adversarial
transferability. Empirical evaluations on the standard ImageNet dataset
demonstrate that our proposed method can significantly enhance the
transferability of existing adversarial attacks.


http://arxiv.org/abs/2302.01757
Certified Robustness of Learning-based Static Malware Detectors. (99%)
Zhuoqun Huang; Neil G. Marchant; Keane Lucas; Lujo Bauer; Olga Ohrimenko; Benjamin I. P. Rubinstein
  Certified defenses are a recent development in adversarial machine learning
(ML), which aim to rigorously guarantee the robustness of ML models to
adversarial perturbations. A large body of work studies certified defenses in
computer vision, where $\ell_p$ norm-bounded evasion attacks are adopted as a
tractable threat model. However, this threat model has known limitations in
vision, and is not applicable to other domains -- e.g., where inputs may be
discrete or subject to complex constraints. Motivated by this gap, we study
certified defenses for malware detection, a domain where attacks against
ML-based systems are a real and current threat. We consider static malware
detection systems that operate on byte-level data. Our certified defense is
based on the approach of randomized smoothing which we adapt by: (1) replacing
the standard Gaussian randomization scheme with a novel deletion randomization
scheme that operates on bytes or chunks of an executable; and (2) deriving a
certificate that measures robustness to evasion attacks in terms of generalized
edit distance. To assess the size of robustness certificates that are
achievable while maintaining high accuracy, we conduct experiments on malware
datasets using a popular convolutional malware detection model, MalConv. We are
able to accurately classify 91% of the inputs while being certifiably robust to
any adversarial perturbations of edit distance 128 bytes or less. By
comparison, an existing certification of up to 128 bytes of substitutions
(without insertions or deletions) achieves an accuracy of 78%. In addition,
given that robustness certificates are conservative, we evaluate practical
robustness to several recently published evasion attacks and, in some cases,
find robustness beyond certified guarantees.


http://arxiv.org/abs/2301.12896
Identifying Adversarially Attackable and Robust Samples. (98%)
Vyas Raina; Mark Gales
  This work proposes a novel perspective on adversarial attacks by introducing
the concept of sample attackability and robustness. Adversarial attacks insert
small, imperceptible perturbations to the input that cause large, undesired
changes to the output of deep learning models. Despite extensive research on
generating adversarial attacks and building defense systems, there has been
limited research on understanding adversarial attacks from an input-data
perspective. We propose a deep-learning-based method for detecting the most
attackable and robust samples in an unseen dataset for an unseen target model.
The proposed method is based on a neural network architecture that takes as
input a sample and outputs a measure of attackability or robustness. The
proposed method is evaluated using a range of different models and different
attack methods, and the results demonstrate its effectiveness in detecting the
samples that are most likely to be affected by adversarial attacks.
Understanding sample attackability can have important implications for future
work in sample-selection tasks. For example in active learning, the acquisition
function can be designed to select the most attackable samples, or in
adversarial training, only the most attackable samples are selected for
augmentation.


http://arxiv.org/abs/2301.12868
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex. (98%)
Terry Yue Zhuo; Zhuang Li; Yujin Huang; Fatemeh Shiri; Weiqing Wang; Gholamreza Haffari; Yuan-Fang Li
  Semantic parsing is a technique aimed at constructing a structured
representation of the meaning of a natural-language question. Recent
advancements in few-shot language models trained on code have demonstrated
superior performance in generating these representations compared to
traditional unimodal language models, which are trained on downstream tasks.
Despite these advancements, existing fine-tuned neural semantic parsers are
susceptible to adversarial attacks on natural-language inputs. While it has
been established that the robustness of smaller semantic parsers can be
enhanced through adversarial training, this approach is not feasible for large
language models in real-world scenarios, as it requires both substantial
computational resources and expensive human annotation on in-domain semantic
parsing data. This paper presents the first empirical study on the adversarial
robustness of a large prompt-based language model of code, \codex. Our results
demonstrate that the state-of-the-art (SOTA) code-language models are
vulnerable to carefully crafted adversarial examples. To address this
challenge, we propose methods for improving robustness without the need for
significant amounts of labeled data or heavy computational resources.


http://arxiv.org/abs/2301.13096
Anchor-Based Adversarially Robust Zero-Shot Learning Driven by Language. (96%)
Xiao Li; Wei Zhang; Yining Liu; Zhanhao Hu; Bo Zhang; Xiaolin Hu
  Deep neural networks are vulnerable to adversarial attacks. We consider
adversarial defense in the case of zero-shot image classification setting,
which has rarely been explored because both adversarial defense and zero-shot
learning are challenging. We propose LAAT, a novel Language-driven,
Anchor-based Adversarial Training strategy, to improve the adversarial
robustness in a zero-shot setting. LAAT uses a text encoder to obtain fixed
anchors (normalized feature embeddings) of each category, then uses these
anchors to perform adversarial training. The text encoder has the property that
semantically similar categories can be mapped to neighboring anchors in the
feature space. By leveraging this property, LAAT can make the image model
adversarially robust on novel categories without any extra examples.
Experimental results show that our method achieves impressive zero-shot
adversarial performance, even surpassing the previous state-of-the-art
adversarially robust one-shot methods in most attacking settings. When models
are trained with LAAT on large datasets like ImageNet-1K, they can have
substantial zero-shot adversarial robustness across several downstream
datasets.


http://arxiv.org/abs/2301.13356
Inference Time Evidences of Adversarial Attacks for Forensic on Transformers. (87%)
Hugo Lemarchant; Liangzi Li; Yiming Qian; Yuta Nakashima; Hajime Nagahara
  Vision Transformers (ViTs) are becoming a very popular paradigm for vision
tasks as they achieve state-of-the-art performance on image classification.
However, although early works implied that this network structure had increased
robustness against adversarial attacks, some works argue ViTs are still
vulnerable. This paper presents our first attempt toward detecting adversarial
attacks during inference time using the network's input and outputs as well as
latent features. We design four quantifications (or derivatives) of input,
output, and latent vectors of ViT-based models that provide a signature of the
inference, which could be beneficial for the attack detection, and empirically
study their behavior over clean samples and adversarial samples. The results
demonstrate that the quantifications from input (images) and output (posterior
probabilities) are promising for distinguishing clean and adversarial samples,
while latent vectors offer less discriminative power, though they give some
insights on how adversarial perturbations work.


http://arxiv.org/abs/2301.13028
On the Efficacy of Metrics to Describe Adversarial Attacks. (82%)
Tommaso Puccetti; Tommaso Zoppi; Andrea Ceccarelli
  Adversarial defenses are naturally evaluated on their ability to tolerate
adversarial attacks. To test defenses, diverse adversarial attacks are crafted,
that are usually described in terms of their evading capability and the L0, L1,
L2, and Linf norms. We question if the evading capability and L-norms are the
most effective information to claim that defenses have been tested against a
representative attack set. To this extent, we select image quality metrics from
the state of the art and search correlations between image perturbation and
detectability. We observe that computing L-norms alone is rarely the preferable
solution. We observe a strong correlation between the identified metrics
computed on an adversarial image and the output of a detector on such an image,
to the extent that they can predict the response of a detector with
approximately 0.94 accuracy. Further, we observe that metrics can classify
attacks based on similar perturbations and similar detectability. This suggests
a possible review of the approach to evaluate detectors, where additional
metrics are included to assure that a representative attack dataset is
selected.


http://arxiv.org/abs/2301.12993
Benchmarking Robustness to Adversarial Image Obfuscations. (74%)
Florian Stimberg; Ayan Chakrabarti; Chun-Ta Lu; Hussein Hazimeh; Otilia Stretcu; Wei Qiao; Yintao Liu; Merve Kaya; Cyrus Rashtchian; Ariel Fuxman; Mehmet Tek; Sven Gowal
  Automated content filtering and moderation is an important tool that allows
online platforms to build striving user communities that facilitate cooperation
and prevent abuse. Unfortunately, resourceful actors try to bypass automated
filters in a bid to post content that violate platform policies and codes of
conduct. To reach this goal, these malicious actors may obfuscate policy
violating images (e.g. overlay harmful images by carefully selected benign
images or visual patterns) to prevent machine learning models from reaching the
correct decision. In this paper, we invite researchers to tackle this specific
issue and present a new image benchmark. This benchmark, based on ImageNet,
simulates the type of obfuscations created by malicious actors. It goes beyond
ImageNet-$\textrm{C}$ and ImageNet-$\bar{\textrm{C}}$ by proposing general,
drastic, adversarial modifications that preserve the original content intent.
It aims to tackle a more common adversarial threat than the one considered by
$\ell_p$-norm bounded adversaries. We evaluate 33 pretrained models on the
benchmark and train models with different augmentations, architectures and
training methods on subsets of the obfuscations to measure generalization. We
hope this benchmark will encourage researchers to test their models and methods
and try to find new approaches that are more robust to these obfuscations.


http://arxiv.org/abs/2301.13188
Extracting Training Data from Diffusion Models. (5%)
Nicholas Carlini; Jamie Hayes; Milad Nasr; Matthew Jagielski; Vikash Sehwag; Florian Tramèr; Borja Balle; Daphne Ippolito; Eric Wallace
  Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have
attracted significant attention due to their ability to generate high-quality
synthetic images. In this work, we show that diffusion models memorize
individual images from their training data and emit them at generation time.
With a generate-and-filter pipeline, we extract over a thousand training
examples from state-of-the-art models, ranging from photographs of individual
people to trademarked company logos. We also train hundreds of diffusion models
in various settings to analyze how different modeling and data decisions affect
privacy. Overall, our results show that diffusion models are much less private
than prior generative models such as GANs, and that mitigating these
vulnerabilities may require new advances in privacy-preserving training.


http://arxiv.org/abs/2301.12831
M3FAS: An Accurate and Robust MultiModal Mobile Face Anti-Spoofing System. (1%)
Chenqi Kong; Kexin Zheng; Yibing Liu; Shiqi Wang; Anderson Rocha; Haoliang Li
  Face presentation attacks (FPA), also known as face spoofing, have brought
increasing concerns to the public through various malicious applications, such
as financial fraud and privacy leakage. Therefore, safeguarding face
recognition systems against FPA is of utmost importance. Although existing
learning-based face anti-spoofing (FAS) models can achieve outstanding
detection performance, they lack generalization capability and suffer
significant performance drops in unforeseen environments. Many methodologies
seek to use auxiliary modality data (e.g., depth and infrared maps) during the
presentation attack detection (PAD) to address this limitation. However, these
methods can be limited since (1) they require specific sensors such as depth
and infrared cameras for data capture, which are rarely available on commodity
mobile devices, and (2) they cannot work properly in practical scenarios when
either modality is missing or of poor quality. In this paper, we devise an
accurate and robust MultiModal Mobile Face Anti-Spoofing system named M3FAS to
overcome the issues above. The innovation of this work mainly lies in the
following aspects: (1) To achieve robust PAD, our system combines visual and
auditory modalities using three pervasively available sensors: camera, speaker,
and microphone; (2) We design a novel two-branch neural network with three
hierarchical feature aggregation modules to perform cross-modal feature fusion;
(3). We propose a multi-head training strategy. The model outputs three
predictions from the vision, acoustic, and fusion heads, enabling a more
flexible PAD. Extensive experiments have demonstrated the accuracy, robustness,
and flexibility of M3FAS under various challenging experimental settings.


http://arxiv.org/abs/2301.12549
Scaling in Depth: Unlocking Robustness Certification on ImageNet. (98%)
Kai Hu; Andy Zou; Zifan Wang; Klas Leino; Matt Fredrikson
  Notwithstanding the promise of Lipschitz-based approaches to
\emph{deterministically} train and certify robust deep networks, the
state-of-the-art results only make successful use of feed-forward Convolutional
Networks (ConvNets) on low-dimensional data, e.g. CIFAR-10. Because ConvNets
often suffer from vanishing gradients when going deep, large-scale datasets
with many classes, e.g., ImageNet, have remained out of practical reach. This
paper investigates ways to scale up certifiably robust training to Residual
Networks (ResNets). First, we introduce the \emph{Linear ResNet} (LiResNet)
architecture, which utilizes a new residual block designed to facilitate
\emph{tighter} Lipschitz bounds compared to a conventional residual block.
Second, we introduce Efficient Margin MAximization (EMMA), a loss function that
stabilizes robust training by simultaneously penalizing worst-case adversarial
examples from \emph{all} classes. Combining LiResNet and EMMA, we achieve new
\emph{state-of-the-art} robust accuracy on CIFAR-10/100 and Tiny-ImageNet under
$\ell_2$-norm-bounded perturbations. Moreover, for the first time, we are able
to scale up deterministic robustness guarantees to ImageNet, bringing hope to
the possibility of applying deterministic certification to real-world
applications.


http://arxiv.org/abs/2301.12487
Mitigating Adversarial Effects of False Data Injection Attacks in Power Grid. (93%)
Farhin Farhad Riya; Shahinul Hoque; Jinyuan Stella Sun; Jiangnan Li; Hairong Qi
  Deep Neural Networks have proven to be highly accurate at a variety of tasks
in recent years. The benefits of Deep Neural Networks have also been embraced
in power grids to detect False Data Injection Attacks (FDIA) while conducting
critical tasks like state estimation. However, the vulnerabilities of DNNs
along with the distinct infrastructure of cyber-physical-system (CPS) can favor
the attackers to bypass the detection mechanism. Moreover, the divergent nature
of CPS engenders limitations to the conventional defense mechanisms for False
Data Injection Attacks. In this paper, we propose a DNN framework with
additional layer which utilizes randomization to mitigate the adversarial
effect by padding the inputs. The primary advantage of our method is when
deployed to a DNN model it has trivial impact on the models performance even
with larger padding sizes. We demonstrate the favorable outcome of the
framework through simulation using the IEEE 14-bus, 30-bus, 118-bus and 300-bus
systems. Furthermore to justify the framework we select attack techniques that
generate subtle adversarial examples that can bypass the detection mechanism
effortlessly.


http://arxiv.org/abs/2301.12576
Uncovering Adversarial Risks of Test-Time Adaptation. (82%)
Tong Wu; Feiran Jia; Xiangyu Qi; Jiachen T. Wang; Vikash Sehwag; Saeed Mahloujifar; Prateek Mittal
  Recently, test-time adaptation (TTA) has been proposed as a promising
solution for addressing distribution shifts. It allows a base model to adapt to
an unforeseen distribution during inference by leveraging the information from
the batch of (unlabeled) test data. However, we uncover a novel security
vulnerability of TTA based on the insight that predictions on benign samples
can be impacted by malicious samples in the same batch. To exploit this
vulnerability, we propose Distribution Invading Attack (DIA), which injects a
small fraction of malicious data into the test batch. DIA causes models using
TTA to misclassify benign and unperturbed test data, providing an entirely new
capability for adversaries that is infeasible in canonical machine learning
pipelines. Through comprehensive evaluations, we demonstrate the high
effectiveness of our attack on multiple benchmarks across six TTA methods. In
response, we investigate two countermeasures to robustify the existing insecure
TTA implementations, following the principle of "security by design". Together,
we hope our findings can make the community aware of the utility-security
tradeoffs in deploying TTA and provide valuable insights for developing robust
TTA approaches.


http://arxiv.org/abs/2301.12554
Improving the Accuracy-Robustness Trade-off of Classifiers via Adaptive Smoothing. (75%)
Yatong Bai; Brendon G. Anderson; Aerin Kim; Somayeh Sojoudi
  While it is shown in the literature that simultaneously accurate and robust
classifiers exist for common datasets, previous methods that improve the
adversarial robustness of classifiers often manifest an accuracy-robustness
trade-off. We build upon recent advancements in data-driven ``locally biased
smoothing'' to develop classifiers that treat benign and adversarial test data
differently. Specifically, we tailor the smoothing operation to the usage of a
robust neural network as the source of robustness. We then extend the smoothing
procedure to the multi-class setting and adapt an adversarial input detector
into a policy network. The policy adaptively adjusts the mixture of the robust
base classifier and a standard network, where the standard network is optimized
for clean accuracy and is not robust in general. We provide theoretical
analyses to motivate the use of the adaptive smoothing procedure, certify the
robustness of the smoothed classifier under realistic assumptions, and justify
the introduction of the policy network. We use various attack methods,
including AutoAttack and adaptive attack, to empirically verify that the
smoothed model noticeably improves the accuracy-robustness trade-off. On the
CIFAR-100 dataset, our method simultaneously achieves an 80.09\% clean accuracy
and a 32.94\% AutoAttacked accuracy. The code that implements adaptive
smoothing is available at https://github.com/Bai-YT/AdaptiveSmoothing.


http://arxiv.org/abs/2301.12595
Adversarial Attacks on Adversarial Bandits. (69%)
Yuzhe Ma; Zhijin Zhou
  We study a security threat to adversarial multi-armed bandits, in which an
attacker perturbs the loss or reward signal to control the behavior of the
victim bandit player. We show that the attacker is able to mislead any
no-regret adversarial bandit algorithm into selecting a suboptimal target arm
in every but sublinear (T-o(T)) number of rounds, while incurring only
sublinear (o(T)) cumulative attack cost. This result implies critical security
concern in real-world bandit-based systems, e.g., in online recommendation, an
attacker might be able to hijack the recommender system and promote a desired
product. Our proposed attack algorithms require knowledge of only the regret
rate, thus are agnostic to the concrete bandit algorithm employed by the victim
player. We also derived a theoretical lower bound on the cumulative attack cost
that any victim-agnostic attack algorithm must incur. The lower bound matches
the upper bound achieved by our attack, which shows that our attack is
asymptotically optimal.


http://arxiv.org/abs/2301.12456
Towards Verifying the Geometric Robustness of Large-scale Neural Networks. (54%)
Fu Wang; Peipei Xu; Wenjie Ruan; Xiaowei Huang
  Deep neural networks (DNNs) are known to be vulnerable to adversarial
geometric transformation. This paper aims to verify the robustness of
large-scale DNNs against the combination of multiple geometric transformations
with a provable guarantee. Given a set of transformations (e.g., rotation,
scaling, etc.), we develop GeoRobust, a black-box robustness analyser built
upon a novel global optimisation strategy, for locating the worst-case
combination of transformations that affect and even alter a network's output.
GeoRobust can provide provable guarantees on finding the worst-case combination
based on recent advances in Lipschitzian theory. Due to its black-box nature,
GeoRobust can be deployed on large-scale DNNs regardless of their
architectures, activation functions, and the number of neurons. In practice,
GeoRobust can locate the worst-case geometric transformation with high
precision for the ResNet50 model on ImageNet in a few seconds on average. We
examined 18 ImageNet classifiers, including the ResNet family and vision
transformers, and found a positive correlation between the geometric robustness
of the networks and the parameter numbers. We also observe that increasing the
depth of DNN is more beneficial than increasing its width in terms of improving
its geometric robustness. Our tool GeoRobust is available at
https://github.com/TrustAI/GeoRobust.


http://arxiv.org/abs/2301.12637
Lateralized Learning for Multi-Class Visual Classification Tasks. (13%)
Abubakar Siddique; Will N. Browne; Gina M. Grimshaw
  The majority of computer vision algorithms fail to find higher-order
(abstract) patterns in an image so are not robust against adversarial attacks,
unlike human lateralized vision. Deep learning considers each input pixel in a
homogeneous manner such that different parts of a ``locality-sensitive hashing
table'' are often not connected, meaning higher-order patterns are not
discovered. Hence these systems are not robust against noisy, irrelevant, and
redundant data, resulting in the wrong prediction being made with high
confidence. Conversely, vertebrate brains afford heterogeneous knowledge
representation through lateralization, enabling modular learning at different
levels of abstraction. This work aims to verify the effectiveness, scalability,
and robustness of a lateralized approach to real-world problems that contain
noisy, irrelevant, and redundant data. The experimental results of multi-class
(200 classes) image classification show that the novel system effectively
learns knowledge representation at multiple levels of abstraction making it
more robust than other state-of-the-art techniques. Crucially, the novel
lateralized system outperformed all the state-of-the-art deep learning-based
systems for the classification of normal and adversarial images by 19.05% -
41.02% and 1.36% - 49.22%, respectively. Findings demonstrate the value of
heterogeneous and lateralized learning for computer vision applications.


http://arxiv.org/abs/2301.12527
Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification. (3%)
Ali Borji
  Test sets are an integral part of evaluating models and gauging progress in
object recognition, and more broadly in computer vision and AI. Existing test
sets for object recognition, however, suffer from shortcomings such as bias
towards the ImageNet characteristics and idiosyncrasies (e.g., ImageNet-V2),
being limited to certain types of stimuli (e.g., indoor scenes in ObjectNet),
and underestimating the model performance (e.g., ImageNet-A). To mitigate these
problems, we introduce a new test set, called D2O, which is sufficiently
different from existing test sets. Images are a mix of generated images as well
as images crawled from the web. They are diverse, unmodified, and
representative of real-world scenarios and cause state-of-the-art models to
misclassify them with high confidence. To emphasize generalization, our dataset
by design does not come paired with a training set. It contains 8,060 images
spread across 36 categories, out of which 29 appear in ImageNet. The best Top-1
accuracy on our dataset is around 60% which is much lower than 91% best Top-1
accuracy on ImageNet. We find that popular vision APIs perform very poorly in
detecting objects over D2O categories such as ``faces'', ``cars'', and
``cats''. Our dataset also comes with a ``miscellaneous'' category, over which
we test the image tagging models. Overall, our investigations demonstrate that
the D2O test set contain a mix of images with varied levels of difficulty and
is predictive of the average-case performance of models. It can challenge
object recognition models for years to come and can spur more research in this
fundamental area.


http://arxiv.org/abs/2301.12643
Adversarial Style Augmentation for Domain Generalization. (2%)
Yabin Zhang; Bin Deng; Ruihuang Li; Kui Jia; Lei Zhang
  It is well-known that the performance of well-trained deep neural networks
may degrade significantly when they are applied to data with even slightly
shifted distributions. Recent studies have shown that introducing certain
perturbation on feature statistics (\eg, mean and standard deviation) during
training can enhance the cross-domain generalization ability. Existing methods
typically conduct such perturbation by utilizing the feature statistics within
a mini-batch, limiting their representation capability. Inspired by the domain
generalization objective, we introduce a novel Adversarial Style Augmentation
(ASA) method, which explores broader style spaces by generating more effective
statistics perturbation via adversarial training. Specifically, we first search
for the most sensitive direction and intensity for statistics perturbation by
maximizing the task loss. By updating the model against the adversarial
statistics perturbation during training, we allow the model to explore the
worst-case domain and hence improve its generalization performance. To
facilitate the application of ASA, we design a simple yet effective module,
namely AdvStyle, which instantiates the ASA method in a plug-and-play manner.
We justify the efficacy of AdvStyle on tasks of cross-domain classification and
instance retrieval. It achieves higher mean accuracy and lower performance
fluctuation. Especially, our method significantly outperforms its competitors
on the PACS dataset under the single source generalization setting, \eg,
boosting the classification accuracy from 61.2\% to 67.1\% with a ResNet50
backbone. Our code will be available at \url{https://github.com/YBZh/AdvStyle}.


http://arxiv.org/abs/2301.12589
Confidence-Aware Calibration and Scoring Functions for Curriculum Learning. (1%)
Shuang Ao; Stefan Rueger; Advaith Siddharthan
  Despite the great success of state-of-the-art deep neural networks, several
studies have reported models to be over-confident in predictions, indicating
miscalibration. Label Smoothing has been proposed as a solution to the
over-confidence problem and works by softening hard targets during training,
typically by distributing part of the probability mass from a `one-hot' label
uniformly to all other labels. However, neither model nor human confidence in a
label are likely to be uniformly distributed in this manner, with some labels
more likely to be confused than others. In this paper we integrate notions of
model confidence and human confidence with label smoothing, respectively
\textit{Model Confidence LS} and \textit{Human Confidence LS}, to achieve
better model calibration and generalization. To enhance model generalization,
we show how our model and human confidence scores can be successfully applied
to curriculum learning, a training strategy inspired by learning of `easier to
harder' tasks. A higher model or human confidence score indicates a more
recognisable and therefore easier sample, and can therefore be used as a
scoring function to rank samples in curriculum learning. We evaluate our
proposed methods with four state-of-the-art architectures for image and text
classification task, using datasets with multi-rater label annotations by
humans. We report that integrating model or human confidence information in
label smoothing and curriculum learning improves both model performance and
model calibration. The code are available at
\url{https://github.com/AoShuang92/Confidence_Calibration_CL}.


http://arxiv.org/abs/2301.12277
Node Injection for Class-specific Network Poisoning. (82%)
Ansh Kumar Sharma; Rahul Kukreja; Mayank Kharbanda; Tanmoy Chakraborty
  Graph Neural Networks (GNNs) are powerful in learning rich network
representations that aid the performance of downstream tasks. However, recent
studies showed that GNNs are vulnerable to adversarial attacks involving node
injection and network perturbation. Among these, node injection attacks are
more practical as they don't require manipulation in the existing network and
can be performed more realistically. In this paper, we propose a novel problem
statement - a class-specific poison attack on graphs in which the attacker aims
to misclassify specific nodes in the target class into a different class using
node injection. Additionally, nodes are injected in such a way that they
camouflage as benign nodes. We propose NICKI, a novel attacking strategy that
utilizes an optimization-based approach to sabotage the performance of
GNN-based node classifiers. NICKI works in two phases - it first learns the
node representation and then generates the features and edges of the injected
nodes. Extensive experiments and ablation studies on four benchmark networks
show that NICKI is consistently better than four baseline attacking strategies
for misclassifying nodes in the target class. We also show that the injected
nodes are properly camouflaged as benign, thus making the poisoned graph
indistinguishable from its clean version w.r.t various topological properties.


http://arxiv.org/abs/2301.12318
Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering. (13%)
Rui Zhu; Di Tang; Siyuan Tang; Guanhong Tao; Shiqing Ma; Xiaofeng Wang; Haixu Tang
  Most existing methods to detect backdoored machine learning (ML) models take
one of the two approaches: trigger inversion (aka. reverse engineer) and weight
analysis (aka. model diagnosis). In particular, the gradient-based trigger
inversion is considered to be among the most effective backdoor detection
techniques, as evidenced by the TrojAI competition, Trojan Detection Challenge
and backdoorBench. However, little has been done to understand why this
technique works so well and, more importantly, whether it raises the bar to the
backdoor attack. In this paper, we report the first attempt to answer this
question by analyzing the change rate of the backdoored model around its
trigger-carrying inputs. Our study shows that existing attacks tend to inject
the backdoor characterized by a low change rate around trigger-carrying inputs,
which are easy to capture by gradient-based trigger inversion. In the meantime,
we found that the low change rate is not necessary for a backdoor attack to
succeed: we design a new attack enhancement called \textit{Gradient Shaping}
(GRASP), which follows the opposite direction of adversarial training to reduce
the change rate of a backdoored model with regard to the trigger, without
undermining its backdoor effect. Also, we provide a theoretic analysis to
explain the effectiveness of this new technique and the fundamental weakness of
gradient-based trigger inversion. Finally, we perform both theoretical and
experimental analysis, showing that the GRASP enhancement does not reduce the
effectiveness of the stealthy attacks against the backdoor detection methods
based on weight analysis, as well as other backdoor mitigation methods without
using detection.


http://arxiv.org/abs/2301.12151
Selecting Models based on the Risk of Damage Caused by Adversarial Attacks. (1%)
Jona Klemenc; Holger Trittenbach
  Regulation, legal liabilities, and societal concerns challenge the adoption
of AI in safety and security-critical applications. One of the key concerns is
that adversaries can cause harm by manipulating model predictions without being
detected. Regulation hence demands an assessment of the risk of damage caused
by adversaries. Yet, there is no method to translate this high-level demand
into actionable metrics that quantify the risk of damage.
  In this article, we propose a method to model and statistically estimate the
probability of damage arising from adversarial attacks. We show that our
proposed estimator is statistically consistent and unbiased. In experiments, we
demonstrate that the estimation results of our method have a clear and
actionable interpretation and outperform conventional metrics. We then show how
operators can use the estimation results to reliably select the model with the
lowest risk.


http://arxiv.org/abs/2301.12046
Semantic Adversarial Attacks on Face Recognition through Significant Attributes. (99%)
Yasmeen M. Khedr; Yifeng Xiong; Kun He
  Face recognition is known to be vulnerable to adversarial face images.
Existing works craft face adversarial images by indiscriminately changing a
single attribute without being aware of the intrinsic attributes of the images.
To this end, we propose a new Semantic Adversarial Attack called SAA-StarGAN
that tampers with the significant facial attributes for each image. We predict
the most significant attributes by applying the cosine similarity or
probability score. The probability score method is based on training a Face
Verification model for an attribute prediction task to obtain a class
probability score for each attribute. The prediction process will help craft
adversarial face images more easily and efficiently, as well as improve the
adversarial transferability. Then, we change the most significant facial
attributes, with either one or more of the facial attributes for impersonation
and dodging attacks in white-box and black-box settings. Experimental results
show that our method could generate diverse and realistic adversarial face
images meanwhile avoid affecting human perception of the face recognition.
SAA-StarGAN achieves an 80.5% attack success rate against black-box models,
outperforming existing methods by 35.5% under the impersonation attack.
Concerning the black-box setting, SAA-StarGAN achieves high attack success
rates on various models. The experiments confirm that predicting the most
important attributes significantly affects the success of adversarial attacks
in both white-box and black-box settings and could enhance the transferability
of the crafted adversarial examples.


http://arxiv.org/abs/2301.11544
Targeted Attacks on Timeseries Forecasting. (99%)
Yuvaraj Govindarajulu; Avinash Amballa; Pavan Kulkarni; Manojkumar Parmar
  Real-world deep learning models developed for Time Series Forecasting are
used in several critical applications ranging from medical devices to the
security domain. Many previous works have shown how deep learning models are
prone to adversarial attacks and studied their vulnerabilities. However, the
vulnerabilities of time series models for forecasting due to adversarial inputs
are not extensively explored. While the attack on a forecasting model might aim
to deteriorate the performance of the model, it is more effective, if the
attack is focused on a specific impact on the model's output. In this paper, we
propose a novel formulation of Directional, Amplitudinal, and Temporal targeted
adversarial attacks on time series forecasting models. These targeted attacks
create a specific impact on the amplitude and direction of the output
prediction. We use the existing adversarial attack techniques from the computer
vision domain and adapt them for time series. Additionally, we propose a
modified version of the Auto Projected Gradient Descent attack for targeted
attacks. We examine the impact of the proposed targeted attacks versus
untargeted attacks. We use KS-Tests to statistically demonstrate the impact of
the attack. Our experimental results show how targeted attacks on time series
models are viable and are more powerful in terms of statistical similarity. It
is, hence difficult to detect through statistical methods. We believe that this
work opens a new paradigm in the time series forecasting domain and represents
an important consideration for developing better defenses.


http://arxiv.org/abs/2301.11546
Adapting Step-size: A Unified Perspective to Analyze and Improve Gradient-based Methods for Adversarial Attacks. (98%)
Wei Tao; Lei Bao; Long Sheng; Gaowei Wu; Qing Tao
  Learning adversarial examples can be formulated as an optimization problem of
maximizing the loss function with some box-constraints. However, for solving
this induced optimization problem, the state-of-the-art gradient-based methods
such as FGSM, I-FGSM and MI-FGSM look different from their original methods
especially in updating the direction, which makes it difficult to understand
them and then leaves some theoretical issues to be addressed in viewpoint of
optimization. In this paper, from the perspective of adapting step-size, we
provide a unified theoretical interpretation of these gradient-based
adversarial learning methods. We show that each of these algorithms is in fact
a specific reformulation of their original gradient methods but using the
step-size rules with only current gradient information. Motivated by such
analysis, we present a broad class of adaptive gradient-based algorithms based
on the regular gradient methods, in which the step-size strategy utilizing
information of the accumulated gradients is integrated. Such adaptive step-size
strategies directly normalize the scale of the gradients rather than use some
empirical operations. The important benefit is that convergence for the
iterative algorithms is guaranteed and then the whole optimization process can
be stabilized. The experiments demonstrate that our AdaI-FGM consistently
outperforms I-FGSM and AdaMI-FGM remains competitive with MI-FGSM for black-box
attacks.


http://arxiv.org/abs/2301.11824
PECAN: A Deterministic Certified Defense Against Backdoor Attacks. (97%)
Yuhao Zhang; Aws Albarghouthi; Loris D'Antoni
  Neural networks are vulnerable to backdoor poisoning attacks, where the
attackers maliciously poison the training set and insert triggers into the test
input to change the prediction of the victim model. Existing defenses for
backdoor attacks either provide no formal guarantees or come with
expensive-to-compute and ineffective probabilistic guarantees. We present
PECAN, an efficient and certified approach for defending against backdoor
attacks. The key insight powering PECAN is to apply off-the-shelf test-time
evasion certification techniques on a set of neural networks trained on
disjoint partitions of the data. We evaluate PECAN on image classification and
malware detection datasets. Our results demonstrate that PECAN can (1)
significantly outperform the state-of-the-art certified backdoor defense, both
in defense strength and efficiency, and (2) on real back-door attacks, PECAN
can reduce attack success rate by order of magnitude when compared to a range
of baselines from the literature.


http://arxiv.org/abs/2301.12001
Vertex-based reachability analysis for verifying ReLU deep neural networks. (93%)
João Zago; Eduardo Camponogara; Eric Antonelo
  Neural networks achieved high performance over different tasks, i.e. image
identification, voice recognition and other applications. Despite their
success, these models are still vulnerable regarding small perturbations, which
can be used to craft the so-called adversarial examples. Different approaches
have been proposed to circumvent their vulnerability, including formal
verification systems, which employ a variety of techniques, including
reachability, optimization and search procedures, to verify that the model
satisfies some property. In this paper we propose three novel reachability
algorithms for verifying deep neural networks with ReLU activations. The first
and third algorithms compute an over-approximation for the reachable set,
whereas the second one computes the exact reachable set. Differently from
previously proposed approaches, our algorithms take as input a V-polytope. Our
experiments on the ACAS Xu problem show that the Exact Polytope Network Mapping
(EPNM) reachability algorithm proposed in this work surpass the
state-of-the-art results from the literature, specially in relation to other
reachability methods.


http://arxiv.org/abs/2301.11912
OccRob: Efficient SMT-Based Occlusion Robustness Verification of Deep Neural Networks. (92%)
Xingwu Guo; Ziwei Zhou; Yueling Zhang; Guy Katz; Min Zhang
  Occlusion is a prevalent and easily realizable semantic perturbation to deep
neural networks (DNNs). It can fool a DNN into misclassifying an input image by
occluding some segments, possibly resulting in severe errors. Therefore, DNNs
planted in safety-critical systems should be verified to be robust against
occlusions prior to deployment. However, most existing robustness verification
approaches for DNNs are focused on non-semantic perturbations and are not
suited to the occlusion case. In this paper, we propose the first efficient,
SMT-based approach for formally verifying the occlusion robustness of DNNs. We
formulate the occlusion robustness verification problem and prove it is
NP-complete. Then, we devise a novel approach for encoding occlusions as a part
of neural networks and introduce two acceleration techniques so that the
extended neural networks can be efficiently verified using off-the-shelf,
SMT-based neural network verification tools. We implement our approach in a
prototype called OccRob and extensively evaluate its performance on benchmark
datasets with various occlusion variants. The experimental results demonstrate
our approach's effectiveness and efficiency in verifying DNNs' robustness
against various occlusions, and its ability to generate counterexamples when
these DNNs are not robust.


http://arxiv.org/abs/2301.11806
PCV: A Point Cloud-Based Network Verifier. (88%)
Arup Kumar Sarker; Farzana Yasmin Ahmad; Matthew B. Dwyer
  3D vision with real-time LiDAR-based point cloud data became a vital part of
autonomous system research, especially perception and prediction modules use
for object classification, segmentation, and detection. Despite their success,
point cloud-based network models are vulnerable to multiple adversarial
attacks, where the certain factor of changes in the validation set causes
significant performance drop in well-trained networks. Most of the existing
verifiers work perfectly on 2D convolution. Due to complex architecture,
dimension of hyper-parameter, and 3D convolution, no verifiers can perform the
basic layer-wise verification. It is difficult to conclude the robustness of a
3D vision model without performing the verification. Because there will be
always corner cases and adversarial input that can compromise the model's
effectiveness.
  In this project, we describe a point cloud-based network verifier that
successfully deals state of the art 3D classifier PointNet verifies the
robustness by generating adversarial inputs. We have used extracted properties
from the trained PointNet and changed certain factors for perturbation input.
We calculate the impact on model accuracy versus property factor and can test
PointNet network's robustness against a small collection of perturbing input
states resulting from adversarial attacks like the suggested hybrid reverse
signed attack. The experimental results reveal that the resilience property of
PointNet is affected by our hybrid reverse signed perturbation strategy


http://arxiv.org/abs/2301.11553
Robust Transformer with Locality Inductive Bias and Feature Normalization. (88%)
Omid Nejati Manzari; Hossein Kashiani; Hojat Asgarian Dehkordi; Shahriar Baradaran Shokouhi
  Vision transformers have been demonstrated to yield state-of-the-art results
on a variety of computer vision tasks using attention-based networks. However,
research works in transformers mostly do not investigate robustness/accuracy
trade-off, and they still struggle to handle adversarial perturbations. In this
paper, we explore the robustness of vision transformers against adversarial
perturbations and try to enhance their robustness/accuracy trade-off in white
box attack settings. To this end, we propose Locality iN Locality (LNL)
transformer model. We prove that the locality introduction to LNL contributes
to the robustness performance since it aggregates local information such as
lines, edges, shapes, and even objects. In addition, to further improve the
robustness performance, we encourage LNL to extract training signal from the
moments (a.k.a., mean and standard deviation) and the normalized features. We
validate the effectiveness and generality of LNL by achieving state-of-the-art
results in terms of accuracy and robustness metrics on German Traffic Sign
Recognition Benchmark (GTSRB) and Canadian Institute for Advanced Research
(CIFAR-10). More specifically, for traffic sign classification, the proposed
LNL yields gains of 1.1% and ~35% in terms of clean and robustness accuracy
compared to the state-of-the-art studies.


http://arxiv.org/abs/2301.12036
Analyzing Robustness of the Deep Reinforcement Learning Algorithm in Ramp Metering Applications Considering False Data Injection Attack and Defense. (87%)
Diyi Liu; Lanmin Liu; Lee D Han
  Decades of practices of ramp metering, by controlling downstream volume and
smoothing the interweaving traffic, have proved that ramp metering can decrease
total travel time, mitigate shockwaves, decrease rear-end collisions, reduce
pollution, etc. Besides traditional methods like ALIENA algorithms, Deep
Reinforcement Learning algorithms have been established recently to build finer
control on ramp metering. However, those Deep Learning models may be venerable
to adversarial attacks. Thus, it is important to investigate the robustness of
those models under False Data Injection adversarial attack. Furthermore,
algorithms capable of detecting anomaly data from clean data are the key to
safeguard Deep Learning algorithm. In this study, an online algorithm that can
distinguish adversarial data from clean data are tested. Results found that in
most cases anomaly data can be distinguished from clean data, although their
difference is too small to be manually distinguished by humans. In practice,
whenever adversarial/hazardous data is detected, the system can fall back to a
fixed control program, and experts should investigate the detectors status or
security protocols afterwards before real damages happen.


http://arxiv.org/abs/2301.11578
Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers. (80%)
Sungmin Cha; Sungjun Cho; Dasol Hwang; Honglak Lee; Taesup Moon; Moontae Lee
  Since the recent advent of regulations for data protection (e.g., the General
Data Protection Regulation), there has been increasing demand in deleting
information learned from sensitive data in pre-trained models without
retraining from scratch. The inherent vulnerability of neural networks towards
adversarial attacks and unfairness also calls for a robust method to remove or
correct information in an instance-wise fashion, while retaining the predictive
performance across remaining data. To this end, we define instance-wise
unlearning, of which the goal is to delete information on a set of instances
from a pre-trained model, by either misclassifying each instance away from its
original prediction or relabeling the instance to a different label. We also
propose two methods that reduce forgetting on the remaining data: 1) utilizing
adversarial examples to overcome forgetting at the representation-level and 2)
leveraging weight importance metrics to pinpoint network parameters guilty of
propagating unwanted information. Both methods only require the pre-trained
model and data instances to forget, allowing painless application to real-life
settings where the entire training set is unavailable. Through extensive
experimentation on various image classification benchmarks, we show that our
approach effectively preserves knowledge of remaining data while unlearning
given instances in both single-task and continual unlearning scenarios.


http://arxiv.org/abs/2301.11783
Certified Invertibility in Neural Networks via Mixed-Integer Programming. (62%)
Tianqi Cui; Thomas Bertalan; George J. Pappas; Manfred Morari; Ioannis G. Kevrekidis; Mahyar Fazlyab
  Neural networks are notoriously vulnerable to adversarial attacks -- small
imperceptible perturbations that can change the network's output drastically.
In the reverse direction, there may exist large, meaningful perturbations that
leave the network's decision unchanged (excessive invariance, nonivertibility).
We study the latter phenomenon in two contexts: (a) discrete-time dynamical
system identification, as well as (b) calibration of the output of one neural
network to the output of another (neural network matching). For ReLU networks
and $L_p$ norms ($p=1,2,\infty$), we formulate these optimization problems as
mixed-integer programs (MIPs) that apply to neural network approximators of
dynamical systems. We also discuss the applicability of our results to
invertibility certification in transformations between neural networks (e.g. at
different levels of pruning).


http://arxiv.org/abs/2301.11457
Attacking Important Pixels for Anchor-free Detectors. (99%)
Yunxu Xie; Shu Hu; Xin Wang; Quanyu Liao; Bin Zhu; Xi Wu; Siwei Lyu
  Deep neural networks have been demonstrated to be vulnerable to adversarial
attacks: subtle perturbation can completely change the prediction result.
Existing adversarial attacks on object detection focus on attacking
anchor-based detectors, which may not work well for anchor-free detectors. In
this paper, we propose the first adversarial attack dedicated to anchor-free
detectors. It is a category-wise attack that attacks important pixels of all
instances of a category simultaneously. Our attack manifests in two forms,
sparse category-wise attack (SCA) and dense category-wise attack (DCA), that
minimize the $L_0$ and $L_\infty$ norm-based perturbations, respectively. For
DCA, we present three variants, DCA-G, DCA-L, and DCA-S, that select a global
region, a local region, and a semantic region, respectively, to attack. Our
experiments on large-scale benchmark datasets including PascalVOC, MS-COCO, and
MS-COCO Keypoints indicate that our proposed methods achieve state-of-the-art
attack performance and transferability on both object detection and human pose
estimation tasks.


http://arxiv.org/abs/2301.11324
Certified Interpretability Robustness for Class Activation Mapping. (92%)
Alex Gu; Tsui-Wei Weng; Pin-Yu Chen; Sijia Liu; Luca Daniel
  Interpreting machine learning models is challenging but crucial for ensuring
the safety of deep networks in autonomous driving systems. Due to the
prevalence of deep learning based perception models in autonomous vehicles,
accurately interpreting their predictions is crucial. While a variety of such
methods have been proposed, most are shown to lack robustness. Yet, little has
been done to provide certificates for interpretability robustness. Taking a
step in this direction, we present CORGI, short for Certifiably prOvable
Robustness Guarantees for Interpretability mapping. CORGI is an algorithm that
takes in an input image and gives a certifiable lower bound for the robustness
of the top k pixels of its CAM interpretability map. We show the effectiveness
of CORGI via a case study on traffic sign data, certifying lower bounds on the
minimum adversarial perturbation not far from (4-5x) state-of-the-art attack
methods.


http://arxiv.org/abs/2301.10964
Interaction-level Membership Inference Attack Against Federated Recommender Systems. (31%)
Wei Yuan; Chaoqun Yang; Quoc Viet Hung Nguyen; Lizhen Cui; Tieke He; Hongzhi Yin
  The marriage of federated learning and recommender system (FedRec) has been
widely used to address the growing data privacy concerns in personalized
recommendation services. In FedRecs, users' attribute information and behavior
data (i.e., user-item interaction data) are kept locally on their personal
devices, therefore, it is considered a fairly secure approach to protect user
privacy. As a result, the privacy issue of FedRecs is rarely explored.
Unfortunately, several recent studies reveal that FedRecs are vulnerable to
user attribute inference attacks, highlighting the privacy concerns of FedRecs.
In this paper, we further investigate the privacy problem of user behavior data
(i.e., user-item interactions) in FedRecs. Specifically, we perform the first
systematic study on interaction-level membership inference attacks on FedRecs.
An interaction-level membership inference attacker is first designed, and then
the classical privacy protection mechanism, Local Differential Privacy (LDP),
is adopted to defend against the membership inference attack. Unfortunately,
the empirical analysis shows that LDP is not effective against such new attacks
unless the recommendation performance is largely compromised. To mitigate the
interaction-level membership attack threats, we design a simple yet effective
defense method to significantly reduce the attacker's inference accuracy
without losing recommendation performance. Extensive experiments are conducted
with two widely used FedRecs (Fed-NCF and Fed-LightGCN) on three real-world
recommendation datasets (MovieLens-100K, Steam-200K, and Amazon Cell Phone),
and the experimental results show the effectiveness of our solutions.


http://arxiv.org/abs/2301.11050
Minerva: A File-Based Ransomware Detector. (13%)
Dorjan Hitaj; Giulio Pagnotta; Gaspari Fabio De; Carli Lorenzo De; Luigi V. Mancini
  Ransomware is a rapidly evolving type of malware designed to encrypt user
files on a device, making them inaccessible in order to exact a ransom.
Ransomware attacks resulted in billions of dollars in damages in recent years
and are expected to cause hundreds of billions more in the next decade. With
current state-of-the-art process-based detectors being heavily susceptible to
evasion attacks, no comprehensive solution to this problem is available today.
This paper presents Minerva, a new approach to ransomware detection. Unlike
current methods focused on identifying ransomware based on process-level
behavioral modeling, Minerva detects ransomware by building behavioral profiles
of files based on all the operations they receive in a time window. Minerva
addresses some of the critical challenges associated with process-based
approaches, specifically their vulnerability to complex evasion attacks. Our
evaluation of Minerva demonstrates its effectiveness in detecting ransomware
attacks, including those that are able to bypass existing defenses. Our results
show that Minerva identifies ransomware activity with an average accuracy of
99.45% and an average recall of 99.66%, with 99.97% of ransomware detected
within 1 second.


http://arxiv.org/abs/2301.10822
RobustPdM: Designing Robust Predictive Maintenance against Adversarial Attacks. (99%)
Ayesha Siddique; Ripan Kumar Kundu; Gautam Raj Mode; Khaza Anuarul Hoque
  The state-of-the-art predictive maintenance (PdM) techniques have shown great
success in reducing maintenance costs and downtime of complicated machines
while increasing overall productivity through extensive utilization of
Internet-of-Things (IoT) and Deep Learning (DL). Unfortunately, IoT sensors and
DL algorithms are both prone to cyber-attacks. For instance, DL algorithms are
known for their susceptibility to adversarial examples. Such adversarial
attacks are vastly under-explored in the PdM domain. This is because the
adversarial attacks in the computer vision domain for classification tasks
cannot be directly applied to the PdM domain for multivariate time series (MTS)
regression tasks. In this work, we propose an end-to-end methodology to design
adversarially robust PdM systems by extensively analyzing the effect of
different types of adversarial attacks and proposing a novel adversarial
defense technique for DL-enabled PdM models. First, we propose novel MTS
Projected Gradient Descent (PGD) and MTS PGD with random restarts (PGD_r)
attacks. Then, we evaluate the impact of MTS PGD and PGD_r along with MTS Fast
Gradient Sign Method (FGSM) and MTS Basic Iterative Method (BIM) on Long
Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural
Network (CNN), and Bi-directional LSTM based PdM system. Our results using
NASA's turbofan engine dataset show that adversarial attacks can cause a severe
defect (up to 11X) in the RUL prediction, outperforming the effectiveness of
the state-of-the-art PdM attacks by 3X. Furthermore, we present a novel
approximate adversarial training method to defend against adversarial attacks.
We observe that approximate adversarial training can significantly improve the
robustness of PdM models (up to 54X) and outperforms the state-of-the-art PdM
defense methods by offering 3X more robustness.


http://arxiv.org/abs/2301.10412
BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testing. (98%)
Jiali Wei; Ming Fan; Wenjing Jiao; Wuxia Jin; Ting Liu
  Deep neural networks (DNNs) and natural language processing (NLP) systems
have developed rapidly and have been widely used in various real-world fields.
However, they have been shown to be vulnerable to backdoor attacks.
Specifically, the adversary injects a backdoor into the model during the
training phase, so that input samples with backdoor triggers are classified as
the target class. Some attacks have achieved high attack success rates on the
pre-trained language models (LMs), but there have yet to be effective defense
methods. In this work, we propose a defense method based on deep model mutation
testing. Our main justification is that backdoor samples are much more robust
than clean samples if we impose random mutations on the LMs and that backdoors
are generalizable. We first confirm the effectiveness of model mutation testing
in detecting backdoor samples and select the most appropriate mutation
operators. We then systematically defend against three extensively studied
backdoor attack levels (i.e., char-level, word-level, and sentence-level) by
detecting backdoor samples. We also make the first attempt to defend against
the latest style-level backdoor attacks. We evaluate our approach on three
benchmark datasets (i.e., IMDB, Yelp, and AG news) and three style transfer
datasets (i.e., SST-2, Hate-speech, and AG news). The extensive experimental
results demonstrate that our approach can detect backdoor samples more
efficiently and accurately than the three state-of-the-art defense approaches.


http://arxiv.org/abs/2301.10454
A Data-Centric Approach for Improving Adversarial Training Through the Lens of Out-of-Distribution Detection. (96%)
Mohammad Azizmalayeri; Arman Zarei; Alireza Isavand; Mohammad Taghi Manzuri; Mohammad Hossein Rohban
  Current machine learning models achieve super-human performance in many
real-world applications. Still, they are susceptible against imperceptible
adversarial perturbations. The most effective solution for this problem is
adversarial training that trains the model with adversarially perturbed samples
instead of original ones. Various methods have been developed over recent years
to improve adversarial training such as data augmentation or modifying training
attacks. In this work, we examine the same problem from a new data-centric
perspective. For this purpose, we first demonstrate that the existing
model-based methods can be equivalent to applying smaller perturbation or
optimization weights to the hard training examples. By using this finding, we
propose detecting and removing these hard samples directly from the training
procedure rather than applying complicated algorithms to mitigate their
effects. For detection, we use maximum softmax probability as an effective
method in out-of-distribution detection since we can consider the hard samples
as the out-of-distribution samples for the whole data distribution. Our results
on SVHN and CIFAR-10 datasets show the effectiveness of this method in
improving the adversarial training without adding too much computational cost.


http://arxiv.org/abs/2301.10766
On the Adversarial Robustness of Camera-based 3D Object Detection. (81%)
Shaoyuan Xie; Zichao Li; Zeyu Wang; Cihang Xie
  In recent years, camera-based 3D object detection has gained widespread
attention for its ability to achieve high performance with low computational
cost. However, the robustness of these methods to adversarial attacks has not
been thoroughly examined. In this study, we conduct the first comprehensive
investigation of the robustness of leading camera-based 3D object detection
methods under various adversarial conditions. Our experiments reveal five
interesting findings: (a) the use of accurate depth estimation effectively
improves robustness; (b) depth-estimation-free approaches do not show superior
robustness; (c) bird's-eye-view-based representations exhibit greater
robustness against localization attacks; (d) incorporating multi-frame benign
inputs can effectively mitigate adversarial attacks; and (e) addressing
long-tail problems can enhance robustness. We hope our work can provide
guidance for the design of future camera-based object detection modules with
improved adversarial robustness.


http://arxiv.org/abs/2301.10576
A Study on FGSM Adversarial Training for Neural Retrieval. (75%)
Simon Lupart; Stéphane Clinchant
  Neural retrieval models have acquired significant effectiveness gains over
the last few years compared to term-based methods. Nevertheless, those models
may be brittle when faced to typos, distribution shifts or vulnerable to
malicious attacks. For instance, several recent papers demonstrated that such
variations severely impacted models performances, and then tried to train more
resilient models. Usual approaches include synonyms replacements or typos
injections -- as data-augmentation -- and the use of more robust tokenizers
(characterBERT, BPE-dropout). To further complement the literature, we
investigate in this paper adversarial training as another possible solution to
this robustness issue. Our comparison includes the two main families of
BERT-based neural retrievers, i.e. dense and sparse, with and without
distillation techniques. We then demonstrate that one of the most simple
adversarial training techniques -- the Fast Gradient Sign Method (FGSM) -- can
improve first stage rankers robustness and effectiveness. In particular, FGSM
increases models performances on both in-domain and out-of-domain
distributions, and also on queries with typos, for multiple neural retrievers.


http://arxiv.org/abs/2301.10908
Distilling Cognitive Backdoor Patterns within an Image. (2%)
Hanxun Huang; Xingjun Ma; Sarah Erfani; James Bailey
  This paper proposes a simple method to distill and detect backdoor patterns
within an image: \emph{Cognitive Distillation} (CD). The idea is to extract the
"minimal essence" from an input image responsible for the model's prediction.
CD optimizes an input mask to extract a small pattern from the input image that
can lead to the same model output (i.e., logits or deep features). The
extracted pattern can help understand the cognitive mechanism of a model on
clean vs. backdoor images and is thus called a \emph{Cognitive Pattern} (CP).
Using CD and the distilled CPs, we uncover an interesting phenomenon of
backdoor attacks: despite the various forms and sizes of trigger patterns used
by different attacks, the CPs of backdoor samples are all surprisingly and
suspiciously small. One thus can leverage the learned mask to detect and remove
backdoor examples from poisoned training datasets. We conduct extensive
experiments to show that CD can robustly detect a wide range of advanced
backdoor attacks. We also show that CD can potentially be applied to help
detect potential biases from face datasets. Code is available at
\url{https://github.com/HanxunH/CognitiveDistillation}.


http://arxiv.org/abs/2301.10608
Connecting metrics for shape-texture knowledge in computer vision. (1%)
Tiago Oliveira; Tiago Marques; Arlindo L. Oliveira
  Modern artificial neural networks, including convolutional neural networks
and vision transformers, have mastered several computer vision tasks, including
object recognition. However, there are many significant differences between the
behavior and robustness of these systems and of the human visual system. Deep
neural networks remain brittle and susceptible to many changes in the image
that do not cause humans to misclassify images. Part of this different behavior
may be explained by the type of features humans and deep neural networks use in
vision tasks. Humans tend to classify objects according to their shape while
deep neural networks seem to rely mostly on texture. Exploring this question is
relevant, since it may lead to better performing neural network architectures
and to a better understanding of the workings of the vision system of primates.
In this work, we advance the state of the art in our understanding of this
phenomenon, by extending previous analyses to a much larger set of deep neural
network architectures. We found that the performance of models in image
classification tasks is highly correlated with their shape bias measured at the
output and penultimate layer. Furthermore, our results showed that the number
of neurons that represent shape and texture are strongly anti-correlated, thus
providing evidence that there is competition between these two types of
features. Finally, we observed that while in general there is a correlation
between performance and shape bias, there are significant variations between
architecture families.


http://arxiv.org/abs/2301.11289
Blockchain-aided Secure Semantic Communication for AI-Generated Content in Metaverse. (13%)
Yijing Lin; Hongyang Du; Dusit Niyato; Jiangtian Nie; Jiayi Zhang; Yanyu Cheng; Zhaohui Yang
  The construction of virtual transportation networks requires massive data to
be transmitted from edge devices to Virtual Service Providers (VSP) to
facilitate circulations between the physical and virtual domains in Metaverse.
Leveraging semantic communication for reducing information redundancy, VSPs can
receive semantic data from edge devices to provide varied services through
advanced techniques, e.g., AI-Generated Content (AIGC), for users to explore
digital worlds. But the use of semantic communication raises a security issue
because attackers could send malicious semantic data with similar semantic
information but different desired content to break Metaverse services and cause
wrong output of AIGC. Therefore, in this paper, we first propose a
blockchain-aided semantic communication framework for AIGC services in virtual
transportation networks to facilitate interactions of the physical and virtual
domains among VSPs and edge devices. We illustrate a training-based targeted
semantic attack scheme to generate adversarial semantic data by various loss
functions. We also design a semantic defense scheme that uses the blockchain
and zero-knowledge proofs to tell the difference between the semantic
similarities of adversarial and authentic semantic data and to check the
authenticity of semantic data transformations. Simulation results show that the
proposed defense method can reduce the semantic similarity of the adversarial
semantic data and the authentic ones by up to 30% compared with the attack
scheme.


http://arxiv.org/abs/2301.09892
Learning Effective Strategies for Moving Target Defense with Switching Costs. (1%)
Vignesh Viswanathan; Megha Bose; Praveen Paruchuri
  Moving Target Defense (MTD) has emerged as a key technique in various
security applications as it takes away the attacker's ability to perform
reconnaissance for exploiting a system's vulnerabilities. However, most of the
existing research in the field assumes unrealistic access to information about
the attacker's motivations and/or actions when developing MTD strategies. Many
of the existing approaches also assume complete knowledge regarding the
vulnerabilities of a system and how each of these vulnerabilities can be
exploited by an attacker. In this work, we aim to create algorithms that
generate effective Moving Target Defense strategies that do not rely on prior
knowledge about the attackers. Our work assumes that the only way the defender
receives information about its own reward is via interaction with the attacker
in a repeated game setting. Depending on the amount of information that can be
obtained from the interactions, we devise two different algorithms using
multi-armed bandit formulation to identify efficient strategies. We then
evaluate our algorithms using data mined from the National Vulnerability
Database to showcase that they match the performance of the state-of-the-art
techniques, despite using a lot less amount of information.


http://arxiv.org/abs/2301.09879
Data Augmentation Alone Can Improve Adversarial Training. (1%)
Lin Li; Michael Spratling
  Adversarial training suffers from the issue of robust overfitting, which
seriously impairs its generalization performance. Data augmentation, which is
effective at preventing overfitting in standard training, has been observed by
many previous works to be ineffective in mitigating overfitting in adversarial
training. This work proves that, contrary to previous findings, data
augmentation alone can significantly boost accuracy and robustness in
adversarial training. We find that the hardness and the diversity of data
augmentation are important factors in combating robust overfitting. In general,
diversity can improve both accuracy and robustness, while hardness can boost
robustness at the cost of accuracy within a certain limit and degrade them both
over that limit. To mitigate robust overfitting, we first propose a new crop
transformation, Cropshift, which has improved diversity compared to the
conventional one (Padcrop). We then propose a new data augmentation scheme,
based on Cropshift, with much improved diversity and well-balanced hardness.
Empirically, our augmentation method achieves the state-of-the-art accuracy and
robustness for data augmentations in adversarial training. Furthermore, when
combined with weight averaging it matches, or even exceeds, the performance of
the best contemporary regularization methods for alleviating robust
overfitting. Code is available at:
https://github.com/TreeLLi/DA-Alone-Improves-AT.


http://arxiv.org/abs/2301.09740
DODEM: DOuble DEfense Mechanism Against Adversarial Attacks Towards Secure Industrial Internet of Things Analytics. (99%)
Onat Gungor; Tajana Rosing; Baris Aksanli
  Industrial Internet of Things (I-IoT) is a collaboration of devices, sensors,
and networking equipment to monitor and collect data from industrial
operations. Machine learning (ML) methods use this data to make high-level
decisions with minimal human intervention. Data-driven predictive maintenance
(PDM) is a crucial ML-based I-IoT application to find an optimal maintenance
schedule for industrial assets. The performance of these ML methods can
seriously be threatened by adversarial attacks where an adversary crafts
perturbed data and sends it to the ML model to deteriorate its prediction
performance. The models should be able to stay robust against these attacks
where robustness is measured by how much perturbation in input data affects
model performance. Hence, there is a need for effective defense mechanisms that
can protect these models against adversarial attacks. In this work, we propose
a double defense mechanism to detect and mitigate adversarial attacks in I-IoT
environments. We first detect if there is an adversarial attack on a given
sample using novelty detection algorithms. Then, based on the outcome of our
algorithm, marking an instance as attack or normal, we select adversarial
retraining or standard training to provide a secondary defense layer. If there
is an attack, adversarial retraining provides a more robust model, while we
apply standard training for regular samples. Since we may not know if an attack
will take place, our adaptive mechanism allows us to consider irregular changes
in data. The results show that our double defense strategy is highly efficient
where we can improve model robustness by up to 64.6% and 52% compared to
standard and adversarial retraining, respectively.


http://arxiv.org/abs/2301.09305
Practical Adversarial Attacks Against AI-Driven Power Allocation in a Distributed MIMO Network. (92%)
Ömer Faruk Tuna; Fehmi Emre Kadan; Leyli Karaçay
  In distributed multiple-input multiple-output (D-MIMO) networks, power
control is crucial to optimize the spectral efficiencies of users and max-min
fairness (MMF) power control is a commonly used strategy as it satisfies
uniform quality-of-service to all users. The optimal solution of MMF power
control requires high complexity operations and hence deep neural network based
artificial intelligence (AI) solutions are proposed to decrease the complexity.
Although quite accurate models can be achieved by using AI, these models have
some intrinsic vulnerabilities against adversarial attacks where carefully
crafted perturbations are applied to the input of the AI model. In this work,
we show that threats against the target AI model which might be originated from
malicious users or radio units can substantially decrease the network
performance by applying a successful adversarial sample, even in the most
constrained circumstances. We also demonstrate that the risk associated with
these kinds of adversarial attacks is higher than the conventional attack
threats. Detailed simulations reveal the effectiveness of adversarial attacks
and the necessity of smart defense techniques.


http://arxiv.org/abs/2301.09508
BayBFed: Bayesian Backdoor Defense for Federated Learning. (78%)
Kavita Kumari; Phillip Rieger; Hossein Fereidooni; Murtuza Jadliwala; Ahmad-Reza Sadeghi
  Federated learning (FL) allows participants to jointly train a machine
learning model without sharing their private data with others. However, FL is
vulnerable to poisoning attacks such as backdoor attacks. Consequently, a
variety of defenses have recently been proposed, which have primarily utilized
intermediary states of the global model (i.e., logits) or distance of the local
models (i.e., L2-norm) from the global model to detect malicious backdoors.
However, as these approaches directly operate on client updates, their
effectiveness depends on factors such as clients' data distribution or the
adversary's attack strategies. In this paper, we introduce a novel and more
generic backdoor defense framework, called BayBFed, which proposes to utilize
probability distributions over client updates to detect malicious updates in
FL: it computes a probabilistic measure over the clients' updates to keep track
of any adjustments made in the updates, and uses a novel detection algorithm
that can leverage this probabilistic measure to efficiently detect and filter
out malicious updates. Thus, it overcomes the shortcomings of previous
approaches that arise due to the direct usage of client updates; as our
probabilistic measure will include all aspects of the local client training
strategies. BayBFed utilizes two Bayesian Non-Parametric extensions: (i) a
Hierarchical Beta-Bernoulli process to draw a probabilistic measure given the
clients' updates, and (ii) an adaptation of the Chinese Restaurant Process
(CRP), referred by us as CRP-Jensen, which leverages this probabilistic measure
to detect and filter out malicious updates. We extensively evaluate our defense
approach on five benchmark datasets: CIFAR10, Reddit, IoT intrusion detection,
MNIST, and FMNIST, and show that it can effectively detect and eliminate
malicious updates in FL without deteriorating the benign performance of the
global model.


http://arxiv.org/abs/2301.09732
Backdoor Attacks in Peer-to-Peer Federated Learning. (31%)
Gokberk Yar; Cristina Nita-Rotaru; Alina Oprea
  We study backdoor attacks in peer-to-peer federated learning systems on
different graph topologies and datasets. We show that only 5% attacker nodes
are sufficient to perform a backdoor attack with 42% attack success without
decreasing the accuracy on clean data by more than 2%. We also demonstrate that
the attack can be amplified by the attacker crashing a small number of nodes.
We evaluate defenses proposed in the context of centralized federated learning
and show they are ineffective in peer-to-peer settings. Finally, we propose a
defense that mitigates the attacks by applying different clipping norms to the
model updates received from peers and local model trained by a node.


http://arxiv.org/abs/2301.09069
Provable Unrestricted Adversarial Training without Compromise with Generalizability. (99%)
Lilin Zhang; Ning Yang; Yanchao Sun; Philip S. Yu
  Adversarial training (AT) is widely considered as the most promising strategy
to defend against adversarial attacks and has drawn increasing interest from
researchers. However, the existing AT methods still suffer from two challenges.
First, they are unable to handle unrestricted adversarial examples (UAEs),
which are built from scratch, as opposed to restricted adversarial examples
(RAEs), which are created by adding perturbations bound by an $l_p$ norm to
observed examples. Second, the existing AT methods often achieve adversarial
robustness at the expense of standard generalizability (i.e., the accuracy on
natural examples) because they make a tradeoff between them. To overcome these
challenges, we propose a unique viewpoint that understands UAEs as
imperceptibly perturbed unobserved examples. Also, we find that the tradeoff
results from the separation of the distributions of adversarial examples and
natural examples. Based on these ideas, we propose a novel AT approach called
Provable Unrestricted Adversarial Training (PUAT), which can provide a target
classifier with comprehensive adversarial robustness against both UAE and RAE,
and simultaneously improve its standard generalizability. Particularly, PUAT
utilizes partially labeled data to achieve effective UAE generation by
accurately capturing the natural data distribution through a novel augmented
triple-GAN. At the same time, PUAT extends the traditional AT by introducing
the supervised loss of the target classifier into the adversarial loss and
achieves the alignment between the UAE distribution, the natural data
distribution, and the distribution learned by the classifier, with the
collaboration of the augmented triple-GAN. Finally, the solid theoretical
analysis and extensive experiments conducted on widely-used benchmarks
demonstrate the superiority of PUAT.


http://arxiv.org/abs/2301.09072
ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learning. (8%)
Shangqing Liu; Bozhi Wu; Xiaofei Xie; Guozhu Meng; Yang Liu
  Large-scale pre-trained models such as CodeBERT, GraphCodeBERT have earned
widespread attention from both academia and industry. Attributed to the
superior ability in code representation, they have been further applied in
multiple downstream tasks such as clone detection, code search and code
translation. However, it is also observed that these state-of-the-art
pre-trained models are susceptible to adversarial attacks. The performance of
these pre-trained models drops significantly with simple perturbations such as
renaming variable names. This weakness may be inherited by their downstream
models and thereby amplified at an unprecedented scale. To this end, we propose
an approach namely ContraBERT that aims to improve the robustness of
pre-trained models via contrastive learning. Specifically, we design nine kinds
of simple and complex data augmentation operators on the programming language
(PL) and natural language (NL) data to construct different variants.
Furthermore, we continue to train the existing pre-trained models by masked
language modeling (MLM) and contrastive pre-training task on the original
samples with their augmented variants to enhance the robustness of the model.
The extensive experiments demonstrate that ContraBERT can effectively improve
the robustness of the existing pre-trained models. Further study also confirms
that these robustness-enhanced models provide improvements as compared to
original models over four popular downstream tasks.


http://arxiv.org/abs/2301.08842
Limitations of Piecewise Linearity for Efficient Robustness Certification. (95%)
Klas Leino
  Certified defenses against small-norm adversarial examples have received
growing attention in recent years; though certified accuracies of
state-of-the-art methods remain far below their non-robust counterparts,
despite the fact that benchmark datasets have been shown to be well-separated
at far larger radii than the literature generally attempts to certify. In this
work, we offer insights that identify potential factors in this performance
gap. Specifically, our analysis reveals that piecewise linearity imposes
fundamental limitations on the tightness of leading certification techniques.
These limitations are felt in practical terms as a greater need for capacity in
models hoped to be certified efficiently. Moreover, this is in addition to the
capacity necessary to learn a robust boundary, studied in prior work. However,
we argue that addressing the limitations of piecewise linearity through scaling
up model capacity may give rise to potential difficulties -- particularly
regarding robust generalization -- therefore, we conclude by suggesting that
developing smooth activation functions may be the way forward for advancing the
performance of certified neural networks.


http://arxiv.org/abs/2301.08751
Towards Understanding How Self-training Tolerates Data Backdoor Poisoning. (16%)
Soumyadeep Pal; Ren Wang; Yuguang Yao; Sijia Liu
  Recent studies on backdoor attacks in model training have shown that
polluting a small portion of training data is sufficient to produce incorrect
manipulated predictions on poisoned test-time data while maintaining high clean
accuracy in downstream tasks. The stealthiness of backdoor attacks has imposed
tremendous defense challenges in today's machine learning paradigm. In this
paper, we explore the potential of self-training via additional unlabeled data
for mitigating backdoor attacks. We begin by making a pilot study to show that
vanilla self-training is not effective in backdoor mitigation. Spurred by that,
we propose to defend the backdoor attacks by leveraging strong but proper data
augmentations in the self-training pseudo-labeling stage. We find that the new
self-training regime help in defending against backdoor attacks to a great
extent. Its effectiveness is demonstrated through experiments for different
backdoor triggers on CIFAR-10 and a combination of CIFAR-10 with an additional
unlabeled 500K TinyImages dataset. Finally, we explore the direction of
combining self-supervised representation learning with self-training for
further improvement in backdoor defense.


http://arxiv.org/abs/2301.08881
Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness. (8%)
Shuaichen Chang; Jun Wang; Mingwen Dong; Lin Pan; Henghui Zhu; Alexander Hanbo Li; Wuwei Lan; Sheng Zhang; Jiarong Jiang; Joseph Lilien; Steve Ash; William Yang Wang; Zhiguo Wang; Vittorio Castelli; Patrick Ng; Bing Xiang
  Neural text-to-SQL models have achieved remarkable performance in translating
natural language questions into SQL queries. However, recent studies reveal
that text-to-SQL models are vulnerable to task-specific perturbations. Previous
curated robustness test sets usually focus on individual phenomena. In this
paper, we propose a comprehensive robustness benchmark based on Spider, a
cross-domain text-to-SQL benchmark, to diagnose the model robustness. We design
17 perturbations on databases, natural language questions, and SQL queries to
measure the robustness from different angles. In order to collect more
diversified natural question perturbations, we utilize large pretrained
language models (PLMs) to simulate human behaviors in creating natural
questions. We conduct a diagnostic study of the state-of-the-art models on the
robustness set. Experimental results reveal that even the most robust model
suffers from a 14.0% performance drop overall and a 50.7% performance drop on
the most challenging perturbation. We also present a breakdown analysis
regarding text-to-SQL model designs and provide insights for improving model
robustness.


http://arxiv.org/abs/2301.08428
Defending SDN against packet injection attacks using deep learning. (2%)
Anh Tuan Phu; Bo Li; Faheem Ullah; Tanvir Ul Huque; Ranesh Naha; Ali Babar; Hung Nguyen
  The (logically) centralised architecture of the software-defined networks
makes them an easy target for packet injection attacks. In these attacks, the
attacker injects malicious packets into the SDN network to affect the services
and performance of the SDN controller and overflow the capacity of the SDN
switches. Such attacks have been shown to ultimately stop the network
functioning in real-time, leading to network breakdowns. There have been
significant works on detecting and defending against similar DoS attacks in
non-SDN networks, but detection and protection techniques for SDN against
packet injection attacks are still in their infancy. Furthermore, many of the
proposed solutions have been shown to be easily by-passed by simple
modifications to the attacking packets or by altering the attacking profile. In
this paper, we develop novel Graph Convolutional Neural Network models and
algorithms for grouping network nodes/users into security classes by learning
from network data. We start with two simple classes - nodes that engage in
suspicious packet injection attacks and nodes that are not. From these classes,
we then partition the network into separate segments with different security
policies using distributed Ryu controllers in an SDN network. We show in
experiments on an emulated SDN that our detection solution outperforms
alternative approaches with above 99\% detection accuracy on various types
(both old and new) of injection attacks. More importantly, our mitigation
solution maintains continuous functions of non-compromised nodes while
isolating compromised/suspicious nodes in real-time. All code and data are
publicly available for reproducibility of our results.


http://arxiv.org/abs/2301.08170
On the Vulnerability of Backdoor Defenses for Federated Learning. (62%)
Pei Fang; Jinghui Chen
  Federated Learning (FL) is a popular distributed machine learning paradigm
that enables jointly training a global model without sharing clients' data.
However, its repetitive server-client communication gives room for backdoor
attacks with aim to mislead the global model into a targeted misprediction when
a specific trigger pattern is presented. In response to such backdoor threats
on federated learning, various defense measures have been proposed. In this
paper, we study whether the current defense mechanisms truly neutralize the
backdoor threats from federated learning in a practical setting by proposing a
new federated backdoor attack method for possible countermeasures. Different
from traditional training (on triggered data) and rescaling (the malicious
client model) based backdoor injection, the proposed backdoor attack framework
(1) directly modifies (a small proportion of) local model weights to inject the
backdoor trigger via sign flips; (2) jointly optimize the trigger pattern with
the client model, thus is more persistent and stealthy for circumventing
existing defenses. In a case study, we examine the strength and weaknesses of
recent federated backdoor defenses from three major categories and provide
suggestions to the practitioners when training federated models in practice.


http://arxiv.org/abs/2301.08401
On the Relationship Between Information-Theoretic Privacy Metrics And Probabilistic Information Privacy. (31%)
Chong Xiao Wang; Wee Peng Tay
  Information-theoretic (IT) measures based on $f$-divergences have recently
gained interest as a measure of privacy leakage as they allow for trading off
privacy against utility using only a single-value characterization. However,
their operational interpretations in the privacy context are unclear. In this
paper, we relate the notion of probabilistic information privacy (IP) to
several IT privacy metrics based on $f$-divergences. We interpret probabilistic
IP under both the detection and estimation frameworks and link it to
differential privacy, thus allowing a precise operational interpretation of
these IT privacy metrics. We show that the $\chi^2$-divergence privacy metric
is stronger than those based on total variation distance and Kullback-Leibler
divergence. Therefore, we further develop a data-driven empirical risk
framework based on the $\chi^2$-divergence privacy metric and realized using
deep neural networks. This framework is agnostic to the adversarial attack
model. Empirical experiments demonstrate the efficacy of our approach.


http://arxiv.org/abs/2301.08092
RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation. (16%)
Utkarsh Nath; Yancheng Wang; Yingzhen Yang
  Deep Neural Networks are vulnerable to adversarial attacks. Neural
Architecture Search (NAS), one of the driving tools of deep neural networks,
demonstrates superior performance in prediction accuracy in various machine
learning applications. However, it is unclear how it performs against
adversarial attacks. Given the presence of a robust teacher, it would be
interesting to investigate if NAS would produce robust neural architecture by
inheriting robustness from the teacher. In this paper, we propose Robust Neural
Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel
NAS algorithm that improves the robustness of NAS by learning from a robust
teacher through cross-layer knowledge distillation. Unlike previous knowledge
distillation methods that encourage close student/teacher output only in the
last layer, RNAS-CL automatically searches for the best teacher layer to
supervise each student layer. Experimental result evidences the effectiveness
of RNAS-CL and shows that RNAS-CL produces small and robust neural
architecture.


http://arxiv.org/abs/2301.08114
Enhancing Deep Learning with Scenario-Based Override Rules: a Case Study. (1%)
Adiel Ashrov; Guy Katz
  Deep neural networks (DNNs) have become a crucial instrument in the software
development toolkit, due to their ability to efficiently solve complex
problems. Nevertheless, DNNs are highly opaque, and can behave in an unexpected
manner when they encounter unfamiliar input. One promising approach for
addressing this challenge is by extending DNN-based systems with hand-crafted
override rules, which override the DNN's output when certain conditions are
met. Here, we advocate crafting such override rules using the well-studied
scenario-based modeling paradigm, which produces rules that are simple,
extensible, and powerful enough to ensure the safety of the DNN, while also
rendering the system more translucent. We report on two extensive case studies,
which demonstrate the feasibility of the approach; and through them, propose an
extension to scenario-based modeling, which facilitates its integration with
DNN components. We regard this work as a step towards creating safer and more
reliable DNN-based systems and models.


http://arxiv.org/abs/2301.06871
Denoising Diffusion Probabilistic Models as a Defense against Adversarial Attacks. (98%)
Lars Lien Ankile; Anna Midgley; Sebastian Weisshaar
  Neural Networks are infamously sensitive to small perturbations in their
inputs, making them vulnerable to adversarial attacks. This project evaluates
the performance of Denoising Diffusion Probabilistic Models (DDPM) as a
purification technique to defend against adversarial attacks. This works by
adding noise to an adversarial example before removing it through the reverse
process of the diffusion model. We evaluate the approach on the PatchCamelyon
data set for histopathologic scans of lymph node sections and find an
improvement of the robust accuracy by up to 88\% of the original model's
accuracy, constituting a considerable improvement over the vanilla model and
our baselines. The project code is located at
https://github.com/ankile/Adversarial-Diffusion.


http://arxiv.org/abs/2301.07487
Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness. (68%)
Ezgi Korkmaz
  Learning from raw high dimensional data via interaction with a given
environment has been effectively achieved through the utilization of deep
neural networks. Yet the observed degradation in policy performance caused by
imperceptible worst-case policy dependent translations along high sensitivity
directions (i.e. adversarial perturbations) raises concerns on the robustness
of deep reinforcement learning policies. In our paper, we show that these high
sensitivity directions do not lie only along particular worst-case directions,
but rather are more abundant in the deep neural policy landscape and can be
found via more natural means in a black-box setting. Furthermore, we show that
vanilla training techniques intriguingly result in learning more robust
policies compared to the policies learnt via the state-of-the-art adversarial
training techniques. We believe our work lays out intriguing properties of the
deep reinforcement learning policy manifold and our results can help to build
robust and generalizable deep reinforcement learning policies.


http://arxiv.org/abs/2301.07284
Label Inference Attack against Split Learning under Regression Setting. (8%)
Shangyu Xie; Xin Yang; Yuanshun Yao; Tianyi Liu; Taiqing Wang; Jiankai Sun
  As a crucial building block in vertical Federated Learning (vFL), Split
Learning (SL) has demonstrated its practice in the two-party model training
collaboration, where one party holds the features of data samples and another
party holds the corresponding labels. Such method is claimed to be private
considering the shared information is only the embedding vectors and gradients
instead of private raw data and labels. However, some recent works have shown
that the private labels could be leaked by the gradients. These existing attack
only works under the classification setting where the private labels are
discrete. In this work, we step further to study the leakage in the scenario of
the regression model, where the private labels are continuous numbers (instead
of discrete labels in classification). This makes previous attacks harder to
infer the continuous labels due to the unbounded output range. To address the
limitation, we propose a novel learning-based attack that integrates gradient
information and extra learning regularization objectives in aspects of model
training properties, which can infer the labels under regression settings
effectively. The comprehensive experiments on various datasets and models have
demonstrated the effectiveness of our proposed attack. We hope our work can
pave the way for future analyses that make the vFL framework more secure.


http://arxiv.org/abs/2301.06393
$\beta$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search. (1%)
Peng Ye; Tong He; Baopu Li; Tao Chen; Lei Bai; Wanli Ouyang
  Neural Architecture Search has attracted increasing attention in recent
years. Among them, differential NAS approaches such as DARTS, have gained
popularity for the search efficiency. However, they still suffer from three
main issues, that are, the weak stability due to the performance collapse, the
poor generalization ability of the searched architectures, and the inferior
robustness to different kinds of proxies. To solve the stability and
generalization problems, a simple-but-effective regularization method, termed
as Beta-Decay, is proposed to regularize the DARTS-based NAS searching process
(i.e., $\beta$-DARTS). Specifically, Beta-Decay regularization can impose
constraints to keep the value and variance of activated architecture parameters
from being too large, thereby ensuring fair competition among architecture
parameters and making the supernet less sensitive to the impact of input on the
operation set. In-depth theoretical analyses on how it works and why it works
are provided. Comprehensive experiments validate that Beta-Decay regularization
can help to stabilize the searching process and makes the searched network more
transferable across different datasets. To address the robustness problem, we
first benchmark different NAS methods under a wide range of proxy data, proxy
channels, proxy layers and proxy epochs, since the robustness of NAS under
different kinds of proxies has not been explored before. We then conclude some
interesting findings and find that $\beta$-DARTS always achieves the best
result among all compared NAS methods under almost all proxies. We further
introduce the novel flooding regularization to the weight optimization of
$\beta$-DARTS (i.e., Bi-level regularization), and experimentally and
theoretically verify its effectiveness for improving the proxy robustness of
differentiable NAS.


http://arxiv.org/abs/2301.06442
Modeling Uncertain Feature Representation for Domain Generalization. (1%)
Xiaotong Li; Zixuan Hu; Jun Liu; Yixiao Ge; Yongxing Dai; Ling-Yu Duan
  Though deep neural networks have achieved impressive success on various
vision tasks, obvious performance degradation still exists when models are
tested in out-of-distribution scenarios. In addressing this limitation, we
ponder that the feature statistics (mean and standard deviation), which carry
the domain characteristics of the training data, can be properly manipulated to
improve the generalization ability of deep learning models. Existing methods
commonly consider feature statistics as deterministic values measured from the
learned features and do not explicitly model the uncertain statistics
discrepancy caused by potential domain shifts during testing. In this paper, we
improve the network generalization ability by modeling domain shifts with
uncertainty (DSU), i.e., characterizing the feature statistics as uncertain
distributions during training. Specifically, we hypothesize that the feature
statistic, after considering the potential uncertainties, follows a
multivariate Gaussian distribution. During inference, we propose an
instance-wise adaptation strategy that can adaptively deal with the
unforeseeable shift and further enhance the generalization ability of the
trained model with negligible additional cost. We also conduct theoretical
analysis on the aspects of generalization error bound and the implicit
regularization effect, showing the efficacy of our method. Extensive
experiments demonstrate that our method consistently improves the network
generalization ability on multiple vision tasks, including image
classification, semantic segmentation, instance retrieval, and pose estimation.
Our methods are simple yet effective and can be readily integrated into
networks without additional trainable parameters or loss constraints. Code will
be released in https://github.com/lixiaotong97/DSU.


http://arxiv.org/abs/2301.06241
BEAGLE: Forensics of Deep Learning Backdoor Attack for Better Defense. (4%)
Siyuan Cheng; Guanhong Tao; Yingqi Liu; Shengwei An; Xiangzhe Xu; Shiwei Feng; Guangyu Shen; Kaiyuan Zhang; Qiuling Xu; Shiqing Ma; Xiangyu Zhang
  Deep Learning backdoor attacks have a threat model similar to traditional
cyber attacks. Attack forensics, a critical counter-measure for traditional
cyber attacks, is hence of importance for defending model backdoor attacks. In
this paper, we propose a novel model backdoor forensics technique. Given a few
attack samples such as inputs with backdoor triggers, which may represent
different types of backdoors, our technique automatically decomposes them to
clean inputs and the corresponding triggers. It then clusters the triggers
based on their properties to allow automatic attack categorization and
summarization. Backdoor scanners can then be automatically synthesized to find
other instances of the same type of backdoor in other models. Our evaluation on
2,532 pre-trained models, 10 popular attacks, and comparison with 9 baselines
show that our technique is highly effective. The decomposed clean inputs and
triggers closely resemble the ground truth. The synthesized scanners
substantially outperform the vanilla versions of existing scanners that can
hardly generalize to different kinds of attacks.


http://arxiv.org/abs/2301.05506
On the feasibility of attacking Thai LPR systems with adversarial examples. (99%)
Chissanupong Jiamsuchon; Jakapan Suaboot; Norrathep Rattanavipanon
  Recent advances in deep neural networks (DNNs) have significantly enhanced
the capabilities of optical character recognition (OCR) technology, enabling
its adoption to a wide range of real-world applications. Despite this success,
DNN-based OCR is shown to be vulnerable to adversarial attacks, in which the
adversary can influence the DNN model's prediction by carefully manipulating
input to the model. Prior work has demonstrated the security impacts of
adversarial attacks on various OCR languages. However, to date, no studies have
been conducted and evaluated on an OCR system tailored specifically for the
Thai language. To bridge this gap, this work presents a feasibility study of
performing adversarial attacks on a specific Thai OCR application -- Thai
License Plate Recognition (LPR). Moreover, we propose a new type of adversarial
attack based on the \emph{semi-targeted} scenario and show that this scenario
is highly realistic in LPR applications. Our experimental results show the
feasibility of our attacks as they can be performed on a commodity computer
desktop with over 90% attack success rate.


http://arxiv.org/abs/2301.05264
Security-Aware Approximate Spiking Neural Networks. (87%)
Syed Tihaam Ahmad; Ayesha Siddique; Khaza Anuarul Hoque
  Deep Neural Networks (DNNs) and Spiking Neural Networks (SNNs) are both known
for their susceptibility to adversarial attacks. Therefore, researchers in the
recent past have extensively studied the robustness and defense of DNNs and
SNNs under adversarial attacks. Compared to accurate SNNs (AccSNN), approximate
SNNs (AxSNNs) are known to be up to 4X more energy-efficient for ultra-low
power applications. Unfortunately, the robustness of AxSNNs under adversarial
attacks is yet unexplored. In this paper, we first extensively analyze the
robustness of AxSNNs with different structural parameters and approximation
levels under two gradient-based and two neuromorphic attacks. Then, we propose
two novel defense methods, i.e., precision scaling and approximate
quantization-aware filtering (AQF), for securing AxSNNs. We evaluated the
effectiveness of these two defense methods using both static and neuromorphic
datasets. Our results demonstrate that AxSNNs are more prone to adversarial
attacks than AccSNNs, but precision scaling and AQF significantly improve the
robustness of AxSNNs. For instance, a PGD attack on AxSNN results in a 72\%
accuracy loss compared to AccSNN without any attack, whereas the same attack on
the precision-scaled AxSNN leads to only a 17\% accuracy loss in the static
MNIST dataset (4X robustness improvement). Similarly, a Sparse Attack on AxSNN
leads to a 77\% accuracy loss when compared to AccSNN without any attack,
whereas the same attack on an AxSNN with AQF leads to only a 2\% accuracy loss
in the neuromorphic DVS128 Gesture dataset (38X robustness improvement).


http://arxiv.org/abs/2301.05250
Jamming Attacks on Decentralized Federated Learning in General Multi-Hop Wireless Networks. (3%)
Yi Shi; Yalin E. Sagduyu; Tugba Erpek
  Decentralized federated learning (DFL) is an effective approach to train a
deep learning model at multiple nodes over a multi-hop network, without the
need of a server having direct connections to all nodes. In general, as long as
nodes are connected potentially via multiple hops, the DFL process will
eventually allow each node to experience the effects of models from all other
nodes via either direct connections or multi-hop paths, and thus is able to
train a high-fidelity model at each node. We consider an effective attack that
uses jammers to prevent the model exchanges between nodes. There are two attack
scenarios. First, the adversary can attack any link under a certain budget.
Once attacked, two end nodes of a link cannot exchange their models. Secondly,
some jammers with limited jamming ranges are deployed in the network and a
jammer can only jam nodes within its jamming range. Once a directional link is
attacked, the receiver node cannot receive the model from the transmitter node.
We design algorithms to select links to be attacked for both scenarios. For the
second scenario, we also design algorithms to deploy jammers at optimal
locations so that they can attack critical nodes and achieve the highest impact
on the DFL process. We evaluate these algorithms by using wireless signal
classification over a large network area as the use case and identify how these
attack mechanisms exploits various learning, connectivity, and sensing aspects.
We show that the DFL performance can be significantly reduced by jamming
attacks launched in a wireless network and characterize the attack surface as a
vulnerability study before the safe deployment of DFL over wireless networks.


http://arxiv.org/abs/2301.04785
Phase-shifted Adversarial Training. (82%)
Yeachan Kim; Seongyeon Kim; Ihyeok Seo; Bonggun Shin
  Adversarial training has been considered an imperative component for safely
deploying neural network-based applications to the real world. To achieve
stronger robustness, existing methods primarily focus on how to generate strong
attacks by increasing the number of update steps, regularizing the models with
the smoothed loss function, and injecting the randomness into the attack.
Instead, we analyze the behavior of adversarial training through the lens of
response frequency. We empirically discover that adversarial training causes
neural networks to have low convergence to high-frequency information,
resulting in highly oscillated predictions near each data. To learn
high-frequency contents efficiently and effectively, we first prove that a
universal phenomenon of frequency principle, i.e., \textit{lower frequencies
are learned first}, still holds in adversarial training. Based on that, we
propose phase-shifted adversarial training (PhaseAT) in which the model learns
high-frequency components by shifting these frequencies to the low-frequency
range where the fast convergence occurs. For evaluations, we conduct the
experiments on CIFAR-10 and ImageNet with the adaptive attack carefully
designed for reliable evaluation. Comprehensive results show that PhaseAT
significantly improves the convergence for high-frequency information. This
results in improved adversarial robustness by enabling the model to have
smoothed predictions near each data.


http://arxiv.org/abs/2301.04554
Universal Detection of Backdoor Attacks via Density-based Clustering and Centroids Analysis. (68%)
Wei Guo; Benedetta Tondi; Mauro Barni
  In this paper, we propose a Universal Defence based on Clustering and
Centroids Analysis (CCA-UD) against backdoor attacks. The goal of the proposed
defence is to reveal whether a Deep Neural Network model is subject to a
backdoor attack by inspecting the training dataset. CCA-UD first clusters the
samples of the training set by means of density-based clustering. Then, it
applies a novel strategy to detect the presence of poisoned clusters. The
proposed strategy is based on a general misclassification behaviour obtained
when the features of a representative example of the analysed cluster are added
to benign samples. The capability of inducing a misclassification error is a
general characteristic of poisoned samples, hence the proposed defence is
attack-agnostic. This mask a significant difference with respect to existing
defences, that, either can defend against only some types of backdoor attacks,
e.g., when the attacker corrupts the label of the poisoned samples, or are
effective only when some conditions on the poisoning ratios adopted by the
attacker or the kind of triggering pattern used by the attacker are satisfied.
Experiments carried out on several classification tasks, considering different
types of backdoor attacks and triggering patterns, including both local and
global triggers, reveal that the proposed method is very effective to defend
against backdoor attacks in all the cases, always outperforming the state of
the art techniques.


http://arxiv.org/abs/2301.04093
On the Robustness of AlphaFold: A COVID-19 Case Study. (73%)
Ismail Alkhouri; Sumit Jha; Andre Beckus; George Atia; Alvaro Velasquez; Rickard Ewetz; Arvind Ramanathan; Susmit Jha
  Protein folding neural networks (PFNNs) such as AlphaFold predict remarkably
accurate structures of proteins compared to other approaches. However, the
robustness of such networks has heretofore not been explored. This is
particularly relevant given the broad social implications of such technologies
and the fact that biologically small perturbations in the protein sequence do
not generally lead to drastic changes in the protein structure. In this paper,
we demonstrate that AlphaFold does not exhibit such robustness despite its high
accuracy. This raises the challenge of detecting and quantifying the extent to
which these predicted protein structures can be trusted. To measure the
robustness of the predicted structures, we utilize (i) the root-mean-square
deviation (RMSD) and (ii) the Global Distance Test (GDT) similarity measure
between the predicted structure of the original sequence and the structure of
its adversarially perturbed version. We prove that the problem of minimally
perturbing protein sequences to fool protein folding neural networks is
NP-complete. Based on the well-established BLOSUM62 sequence alignment scoring
matrix, we generate adversarial protein sequences and show that the RMSD
between the predicted protein structure and the structure of the original
sequence are very large when the adversarial changes are bounded by (i) 20
units in the BLOSUM62 distance, and (ii) five residues (out of hundreds or
thousands of residues) in the given protein sequence. In our experimental
evaluation, we consider 111 COVID-19 proteins in the Universal Protein resource
(UniProt), a central resource for protein data managed by the European
Bioinformatics Institute, Swiss Institute of Bioinformatics, and the US Protein
Information Resource. These result in an overall GDT similarity test score
average of around 34%, demonstrating a substantial drop in the performance of
AlphaFold.


http://arxiv.org/abs/2301.03826
CDA: Contrastive-adversarial Domain Adaptation. (38%)
Nishant Yadav; Mahbubul Alam; Ahmed Farahat; Dipanjan Ghosh; Chetan Gupta; Auroop R. Ganguly
  Recent advances in domain adaptation reveal that adversarial learning on deep
neural networks can learn domain invariant features to reduce the shift between
source and target domains. While such adversarial approaches achieve
domain-level alignment, they ignore the class (label) shift. When
class-conditional data distributions are significantly different between the
source and target domain, it can generate ambiguous features near class
boundaries that are more likely to be misclassified. In this work, we propose a
two-stage model for domain adaptation called \textbf{C}ontrastive-adversarial
\textbf{D}omain \textbf{A}daptation \textbf{(CDA)}. While the adversarial
component facilitates domain-level alignment, two-stage contrastive learning
exploits class information to achieve higher intra-class compactness across
domains resulting in well-separated decision boundaries. Furthermore, the
proposed contrastive framework is designed as a plug-and-play module that can
be easily embedded with existing adversarial methods for domain adaptation. We
conduct experiments on two widely used benchmark datasets for domain
adaptation, namely, \textit{Office-31} and \textit{Digits-5}, and demonstrate
that CDA achieves state-of-the-art results on both datasets.


http://arxiv.org/abs/2301.04230
User-Centered Security in Natural Language Processing. (12%)
Chris Emmery
  This dissertation proposes a framework of user-centered security in Natural
Language Processing (NLP), and demonstrates how it can improve the
accessibility of related research. Accordingly, it focuses on two security
domains within NLP with great public interest. First, that of author profiling,
which can be employed to compromise online privacy through invasive inferences.
Without access and detailed insight into these models' predictions, there is no
reasonable heuristic by which Internet users might defend themselves from such
inferences. Secondly, that of cyberbullying detection, which by default
presupposes a centralized implementation; i.e., content moderation across
social platforms. As access to appropriate data is restricted, and the nature
of the task rapidly evolves (both through lexical variation, and cultural
shifts), the effectiveness of its classifiers is greatly diminished and thereby
often misrepresented.
  Under the proposed framework, we predominantly investigate the use of
adversarial attacks on language; i.e., changing a given input (generating
adversarial samples) such that a given model does not function as intended.
These attacks form a common thread between our user-centered security problems;
they are highly relevant for privacy-preserving obfuscation methods against
author profiling, and adversarial samples might also prove useful to assess the
influence of lexical variation and augmentation on cyberbullying detection.


http://arxiv.org/abs/2301.04218
Diffusion Models For Stronger Face Morphing Attacks. (3%)
Zander Blasingame; Chen Liu
  Face morphing attacks seek to deceive a Face Recognition (FR) system by
presenting a morphed image consisting of the biometric qualities from two
different identities with the aim of triggering a false acceptance with one of
the two identities, thereby presenting a significant threat to biometric
systems. The success of a morphing attack is dependent on the ability of the
morphed image to represent the biometric characteristics of both identities
that were used to create the image. We present a novel morphing attack that
uses a Diffusion-based architecture to improve the visual fidelity of the image
and improve the ability of the morphing attack to represent characteristics
from both identities. We demonstrate the high fidelity of the proposed attack
by evaluating its visual fidelity via the Frechet Inception Distance. Extensive
experiments are conducted to measure the vulnerability of FR systems to the
proposed attack. The proposed attack is compared to two state-of-the-art
GAN-based morphing attacks along with two Landmark-based attacks. The ability
of a morphing attack detector to detect the proposed attack is measured and
compared against the other attacks. Additionally, a novel metric to measure the
relative strength between morphing attacks is introduced and evaluated.


http://arxiv.org/abs/2301.03760
Over-The-Air Adversarial Attacks on Deep Learning Wi-Fi Fingerprinting. (99%)
Fei Xiao; Yong Huang; Yingying Zuo; Wei Kuang; Wei Wang
  Empowered by deep neural networks (DNNs), Wi-Fi fingerprinting has recently
achieved astonishing localization performance to facilitate many
security-critical applications in wireless networks, but it is inevitably
exposed to adversarial attacks, where subtle perturbations can mislead DNNs to
wrong predictions. Such vulnerability provides new security breaches to
malicious devices for hampering wireless network security, such as
malfunctioning geofencing or asset management. The prior adversarial attack on
localization DNNs uses additive perturbations on channel state information
(CSI) measurements, which is impractical in Wi-Fi transmissions. To transcend
this limitation, this paper presents FooLoc, which fools Wi-Fi CSI
fingerprinting DNNs over the realistic wireless channel between the attacker
and the victim access point (AP). We observe that though uplink CSIs are
unknown to the attacker, the accessible downlink CSIs could be their reasonable
substitutes at the same spot. We thoroughly investigate the multiplicative and
repetitive properties of over-the-air perturbations and devise an efficient
optimization problem to generate imperceptible yet robust adversarial
perturbations. We implement FooLoc using commercial Wi-Fi APs and Wireless
Open-Access Research Platform (WARP) v3 boards in offline and online
experiments, respectively. The experimental results show that FooLoc achieves
overall attack success rates of about 70% in targeted attacks and of above 90%
in untargeted attacks with small perturbation-to-signal ratios of about -18dB.


http://arxiv.org/abs/2301.03703
On the Susceptibility and Robustness of Time Series Models through Adversarial Attack and Defense. (98%)
Asadullah Hill Galib; Bidhan Bashyal
  Under adversarial attacks, time series regression and classification are
vulnerable. Adversarial defense, on the other hand, can make the models more
resilient. It is important to evaluate how vulnerable different time series
models are to attacks and how well they recover using defense. The sensitivity
to various attacks and the robustness using the defense of several time series
models are investigated in this study. Experiments are run on seven-time series
models with three adversarial attacks and one adversarial defense. According to
the findings, all models, particularly GRU and RNN, appear to be vulnerable.
LSTM and GRU also have better defense recovery. FGSM exceeds the competitors in
terms of attacks. PGD attacks are more difficult to recover from than other
sorts of attacks.


http://arxiv.org/abs/2301.04017
Is Federated Learning a Practical PET Yet? (4%)
Franziska Boenisch; Adam Dziedzic; Roei Schuster; Ali Shahin Shamsabadi; Ilia Shumailov; Nicolas Papernot
  Federated learning (FL) is a framework for users to jointly train a machine
learning model. FL is promoted as a privacy-enhancing technology (PET) that
provides data minimization: data never "leaves" personal devices and users
share only model updates with a server (e.g., a company) coordinating the
distributed training. We assess the realistic (i.e., worst-case) privacy
guarantees that are provided to users who are unable to trust the server. To
this end, we propose an attack against FL protected with distributed
differential privacy (DDP) and secure aggregation (SA). The attack method is
based on the introduction of Sybil devices that deviate from the protocol to
expose individual users' data for reconstruction by the server. The underlying
root cause for the vulnerability to our attack is the power imbalance. The
server orchestrates the whole protocol and users are given little guarantees
about the selection of other users participating in the protocol. Moving
forward, we discuss requirements for an FL protocol to guarantee DDP without
asking users to trust the server. We conclude that such systems are not yet
practical.


http://arxiv.org/abs/2301.03724
SoK: Hardware Defenses Against Speculative Execution Attacks. (1%)
Guangyuan Hu; Zecheng He; Ruby Lee
  Speculative execution attacks leverage the speculative and out-of-order
execution features in modern computer processors to access secret data or
execute code that should not be executed. Secret information can then be leaked
through a covert channel. While software patches can be installed for
mitigation on existing hardware, these solutions can incur big performance
overhead. Hardware mitigation is being studied extensively by the computer
architecture community. It has the benefit of preserving software compatibility
and the potential for much smaller performance overhead than software
solutions.
  This paper presents a systematization of the hardware defenses against
speculative execution attacks that have been proposed. We show that speculative
execution attacks consist of 6 critical attack steps. We propose defense
strategies, each of which prevents a critical attack step from happening, thus
preventing the attack from succeeding. We then summarize 20 hardware defenses
and overhead-reducing features that have been proposed. We show that each
defense proposed can be classified under one of our defense strategies, which
also explains why it can thwart the attack from succeeding. We discuss the
scope of the defenses, their performance overhead, and the security-performance
trade-offs that can be made.


http://arxiv.org/abs/2301.03110
RobArch: Designing Robust Architectures against Adversarial Attacks. (76%)
ShengYun Peng; Weilin Xu; Cory Cornelius; Kevin Li; Rahul Duggal; Duen Horng Chau; Jason Martin
  Adversarial Training is the most effective approach for improving the
robustness of Deep Neural Networks (DNNs). However, compared to the large body
of research in optimizing the adversarial training process, there are few
investigations into how architecture components affect robustness, and they
rarely constrain model capacity. Thus, it is unclear where robustness precisely
comes from. In this work, we present the first large-scale systematic study on
the robustness of DNN architecture components under fixed parameter budgets.
Through our investigation, we distill 18 actionable robust network design
guidelines that empower model developers to gain deep insights. We demonstrate
these guidelines' effectiveness by introducing the novel Robust Architecture
(RobArch) model that instantiates the guidelines to build a family of
top-performing models across parameter capacities against strong adversarial
attacks. RobArch achieves the new state-of-the-art AutoAttack accuracy on the
RobustBench ImageNet leaderboard. The code is available at
$\href{https://github.com/ShengYun-Peng/RobArch}{\text{this url}}$.


http://arxiv.org/abs/2301.02905
REaaS: Enabling Adversarially Robust Downstream Classifiers via Robust Encoder as a Service. (99%)
Wenjie Qu; Jinyuan Jia; Neil Zhenqiang Gong
  Encoder as a service is an emerging cloud service. Specifically, a service
provider first pre-trains an encoder (i.e., a general-purpose feature
extractor) via either supervised learning or self-supervised learning and then
deploys it as a cloud service API. A client queries the cloud service API to
obtain feature vectors for its training/testing inputs when training/testing
its classifier (called downstream classifier). A downstream classifier is
vulnerable to adversarial examples, which are testing inputs with carefully
crafted perturbation that the downstream classifier misclassifies. Therefore,
in safety and security critical applications, a client aims to build a robust
downstream classifier and certify its robustness guarantees against adversarial
examples.
  What APIs should the cloud service provide, such that a client can use any
certification method to certify the robustness of its downstream classifier
against adversarial examples while minimizing the number of queries to the
APIs? How can a service provider pre-train an encoder such that clients can
build more certifiably robust downstream classifiers? We aim to answer the two
questions in this work. For the first question, we show that the cloud service
only needs to provide two APIs, which we carefully design, to enable a client
to certify the robustness of its downstream classifier with a minimal number of
queries to the APIs. For the second question, we show that an encoder
pre-trained using a spectral-norm regularization term enables clients to build
more robust downstream classifiers.


http://arxiv.org/abs/2301.04472
Adversarial training with informed data selection. (99%)
Marcele O. K. Mendonça; Javier Maroto; Pascal Frossard; Paulo S. R. Diniz
  With the increasing amount of available data and advances in computing
capabilities, deep neural networks (DNNs) have been successfully employed to
solve challenging tasks in various areas, including healthcare, climate, and
finance. Nevertheless, state-of-the-art DNNs are susceptible to
quasi-imperceptible perturbed versions of the original images -- adversarial
examples. These perturbations of the network input can lead to disastrous
implications in critical areas where wrong decisions can directly affect human
lives. Adversarial training is the most efficient solution to defend the
network against these malicious attacks. However, adversarial trained networks
generally come with lower clean accuracy and higher computational complexity.
This work proposes a data selection (DS) strategy to be applied in the
mini-batch training. Based on the cross-entropy loss, the most relevant samples
in the batch are selected to update the model parameters in the
backpropagation. The simulation results show that a good compromise can be
obtained regarding robustness and standard accuracy, whereas the computational
complexity of the backpropagation pass is reduced.


http://arxiv.org/abs/2301.02412
Adversarial Attacks on Neural Models of Code via Code Difference Reduction. (99%)
Zhao Tian; Junjie Chen; Zhi Jin
  Deep learning has been widely used to solve various code-based tasks by
building deep code models based on a large number of code snippets. However,
deep code models are still vulnerable to adversarial attacks. As source code is
discrete and has to strictly stick to the grammar and semantics constraints,
the adversarial attack techniques in other domains are not applicable.
Moreover, the attack techniques specific to deep code models suffer from the
effectiveness issue due to the enormous attack space. In this work, we propose
a novel adversarial attack technique (i.e., CODA). Its key idea is to use the
code differences between the target input and reference inputs (that have small
code differences but different prediction results with the target one) to guide
the generation of adversarial examples. It considers both structure differences
and identifier differences to preserve the original semantics. Hence, the
attack space can be largely reduced as the one constituted by the two kinds of
code differences, and thus the attack process can be largely improved by
designing corresponding equivalent structure transformations and identifier
renaming transformations. Our experiments on 10 deep code models (i.e., two pre
trained models with five code-based tasks) demonstrate the effectiveness and
efficiency of CODA, the naturalness of its generated examples, and its
capability of defending against attacks after adversarial fine-tuning. For
example, CODA improves the state-of-the-art techniques (i.e., CARROT and ALERT)
by 79.25% and 72.20% on average in terms of the attack success rate,
respectively.


http://arxiv.org/abs/2301.02496
Stealthy Backdoor Attack for Code Models. (98%)
Zhou Yang; Bowen Xu; Jie M. Zhang; Hong Jin Kang; Jieke Shi; Junda He; David Lo
  Code models, such as CodeBERT and CodeT5, offer general-purpose
representations of code and play a vital role in supporting downstream
automated software engineering tasks. Most recently, code models were revealed
to be vulnerable to backdoor attacks. A code model that is backdoor-attacked
can behave normally on clean examples but will produce pre-defined malicious
outputs on examples injected with triggers that activate the backdoors.
Existing backdoor attacks on code models use unstealthy and easy-to-detect
triggers. This paper aims to investigate the vulnerability of code models with
stealthy backdoor attacks. To this end, we propose AFRAIDOOR (Adversarial
Feature as Adaptive Backdoor). AFRAIDOOR achieves stealthiness by leveraging
adversarial perturbations to inject adaptive triggers into different inputs. We
evaluate AFRAIDOOR on three widely adopted code models (CodeBERT, PLBART and
CodeT5) and two downstream tasks (code summarization and method name
prediction). We find that around 85% of adaptive triggers in AFRAIDOOR bypass
the detection in the defense process. By contrast, only less than 12% of the
triggers from previous work bypass the defense. When the defense method is not
applied, both AFRAIDOOR and baselines have almost perfect attack success rates.
However, once a defense is applied, the success rates of baselines decrease
dramatically to 10.47% and 12.06%, while the success rate of AFRAIDOOR are
77.05% and 92.98% on the two tasks. Our finding exposes security weaknesses in
code models under stealthy backdoor attacks and shows that the state-of-the-art
defense method cannot provide sufficient protection. We call for more research
efforts in understanding security threats to code models and developing more
effective countermeasures.


http://arxiv.org/abs/2301.02615
Silent Killer: Optimizing Backdoor Trigger Yields a Stealthy and Powerful Data Poisoning Attack. (98%)
Tzvi Lederer; Gallil Maimon; Lior Rokach
  We propose a stealthy and powerful backdoor attack on neural networks based
on data poisoning (DP). In contrast to previous attacks, both the poison and
the trigger in our method are stealthy. We are able to change the model's
classification of samples from a source class to a target class chosen by the
attacker. We do so by using a small number of poisoned training samples with
nearly imperceptible perturbations, without changing their labels. At inference
time, we use a stealthy perturbation added to the attacked samples as a
trigger. This perturbation is crafted as a universal adversarial perturbation
(UAP), and the poison is crafted using gradient alignment coupled to this
trigger. Our method is highly efficient in crafting time compared to previous
methods and requires only a trained surrogate model without additional
retraining. Our attack achieves state-of-the-art results in terms of attack
success rate while maintaining high accuracy on clean samples.


http://arxiv.org/abs/2301.02288
gRoMA: a Tool for Measuring Deep Neural Networks Global Robustness. (96%)
Natan Levy; Raz Yerushalmi; Guy Katz
  Deep neural networks (DNNs) are a state-of-the-art technology, capable of
outstanding performance in many key tasks. However, it is challenging to
integrate DNNs into safety-critical systems, such as those in the aerospace or
automotive domains, due to the risk of adversarial inputs: slightly perturbed
inputs that can cause the DNN to make grievous mistakes. Adversarial inputs
have been shown to plague even modern DNNs; and so the risks they pose must be
measured and mitigated to allow the safe deployment of DNNs in safety-critical
systems. Here, we present a novel and scalable tool called gRoMA, which uses a
statistical approach for formally measuring the global categorial robustness of
a DNN - i.e., the probability of randomly encountering an adversarial input for
a specific output category. Our tool operates on pre-trained, black-box
classification DNNs. It randomly generates input samples that belong to an
output category of interest, measures the DNN's susceptibility to adversarial
inputs around these inputs, and then aggregates the results to infer the
overall global robustness of the DNN up to some small bounded error. For
evaluation purposes, we used gRoMA to measure the global robustness of the
widespread Densenet DNN model over the CIFAR10 dataset and our results exposed
significant gaps in the robustness of the different output categories. This
experiment demonstrates the scalability of the new approach and showcases its
potential for allowing DNNs to be deployed within critical systems of interest.


http://arxiv.org/abs/2301.02039
Randomized Message-Interception Smoothing: Gray-box Certificates for Graph Neural Networks. (61%)
Yan Scholten; Jan Schuchardt; Simon Geisler; Aleksandar Bojchevski; Stephan Günnemann
  Randomized smoothing is one of the most promising frameworks for certifying
the adversarial robustness of machine learning models, including Graph Neural
Networks (GNNs). Yet, existing randomized smoothing certificates for GNNs are
overly pessimistic since they treat the model as a black box, ignoring the
underlying architecture. To remedy this, we propose novel gray-box certificates
that exploit the message-passing principle of GNNs: We randomly intercept
messages and carefully analyze the probability that messages from adversarially
controlled nodes reach their target nodes. Compared to existing certificates,
we certify robustness to much stronger adversaries that control entire nodes in
the graph and can arbitrarily manipulate node features. Our certificates
provide stronger guarantees for attacks at larger distances, as messages from
farther-away nodes are more likely to get intercepted. We demonstrate the
effectiveness of our method on various models and datasets. Since our gray-box
certificates consider the underlying graph structure, we can significantly
improve certifiable robustness by applying graph sparsification.


http://arxiv.org/abs/2301.02344
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models. (1%)
Hojjat Aghakhani; Wei Dai; Andre Manoel; Xavier Fernandes; Anant Kharkar; Christopher Kruegel; Giovanni Vigna; David Evans; Ben Zorn; Robert Sim
  With tools like GitHub Copilot, automatic code suggestion is no longer a
dream in software engineering. These tools, based on large language models, are
typically trained on massive corpora of code mined from unvetted public
sources. As a result, these models are susceptible to data poisoning attacks
where an adversary manipulates the model's training or fine-tuning phases by
injecting malicious data. Poisoning attacks could be designed to influence the
model's suggestions at run time for chosen contexts, such as inducing the model
into suggesting insecure code payloads. To achieve this, prior poisoning
attacks explicitly inject the insecure code payload into the training data,
making the poisoning data detectable by static analysis tools that can remove
such malicious data from the training set. In this work, we demonstrate two
novel data poisoning attacks, COVERT and TROJANPUZZLE, that can bypass static
analysis by planting malicious poisoning data in out-of-context regions such as
docstrings. Our most novel attack, TROJANPUZZLE, goes one step further in
generating less suspicious poisoning data by never including certain
(suspicious) parts of the payload in the poisoned data, while still inducing a
model that suggests the entire payload when completing code (i.e., outside
docstrings). This makes TROJANPUZZLE robust against signature-based
dataset-cleansing methods that identify and filter out suspicious sequences
from the training data. Our evaluation against two model sizes demonstrates
that both COVERT and TROJANPUZZLE have significant implications for how
practitioners should select code used to train or tune code-suggestion models.


http://arxiv.org/abs/2301.01832
Availability Adversarial Attack and Countermeasures for Deep Learning-based Load Forecasting. (98%)
Wangkun Xu; Fei Teng
  The forecast of electrical loads is essential for the planning and operation
of the power system. Recently, advances in deep learning have enabled more
accurate forecasts. However, deep neural networks are prone to adversarial
attacks. Although most of the literature focuses on integrity-based attacks,
this paper proposes availability-based adversarial attacks, which can be more
easily implemented by attackers. For each forecast instance, the availability
attack position is optimally solved by mixed-integer reformulation of the
artificial neural network. To tackle this attack, an adversarial training
algorithm is proposed. In simulation, a realistic load forecasting dataset is
considered and the attack performance is compared to the integrity-based
attack. Meanwhile, the adversarial training algorithm is shown to significantly
improve robustness against availability attacks. All codes are available at
https://github.com/xuwkk/AAA_Load_Forecast.


http://arxiv.org/abs/2301.01495
Beckman Defense. (84%)
A. V. Subramanyam
  Optimal transport (OT) based distributional robust optimisation (DRO) has
received some traction in the recent past. However, it is at a nascent stage
but has a sound potential in robustifying the deep learning models.
Interestingly, OT barycenters demonstrate a good robustness against adversarial
attacks. Owing to the computationally expensive nature of OT barycenters, they
have not been investigated under DRO framework. In this work, we propose a new
barycenter, namely Beckman barycenter, which can be computed efficiently and
used for training the network to defend against adversarial attacks in
conjunction with adversarial training. We propose a novel formulation of
Beckman barycenter and analytically obtain the barycenter using the marginals
of the input image. We show that the Beckman barycenter can be used to train
adversarially trained networks to improve the robustness. Our training is
extremely efficient as it requires only a single epoch of training. Elaborate
experiments on CIFAR-10, CIFAR-100 and Tiny ImageNet demonstrate that training
an adversarially robust network with Beckman barycenter can significantly
increase the performance. Under auto attack, we get a a maximum boost of 10\%
in CIFAR-10, 8.34\% in CIFAR-100 and 11.51\% in Tiny ImageNet. Our code is
available at
https://github.com/Visual-Conception-Group/test-barycentric-defense.


http://arxiv.org/abs/2301.01731
GUAP: Graph Universal Attack Through Adversarial Patching. (81%)
Xiao Zang; Jie Chen; Bo Yuan
  Graph neural networks (GNNs) are a class of effective deep learning models
for node classification tasks; yet their predictive capability may be severely
compromised under adversarially designed unnoticeable perturbations to the
graph structure and/or node data. Most of the current work on graph adversarial
attacks aims at lowering the overall prediction accuracy, but we argue that the
resulting abnormal model performance may catch attention easily and invite
quick counterattack. Moreover, attacks through modification of existing graph
data may be hard to conduct if good security protocols are implemented. In this
work, we consider an easier attack harder to be noticed, through adversarially
patching the graph with new nodes and edges. The attack is universal: it
targets a single node each time and flips its connection to the same set of
patch nodes. The attack is unnoticeable: it does not modify the predictions of
nodes other than the target. We develop an algorithm, named GUAP, that achieves
high attack success rate but meanwhile preserves the prediction accuracy. GUAP
is fast to train by employing a sampling strategy. We demonstrate that a 5%
sampling in each epoch yields 20x speedup in training, with only a slight
degradation in attack performance. Additionally, we show that the adversarial
patch trained with the graph convolutional network transfers well to other
GNNs, such as the graph attention network.


http://arxiv.org/abs/2301.01885
Enhancement attacks in biomedical machine learning. (1%)
Matthew Rosenblatt; Javid Dadashkarimi; Dustin Scheinost
  The prevalence of machine learning in biomedical research is rapidly growing,
yet the trustworthiness of such research is often overlooked. While some
previous works have investigated the ability of adversarial attacks to degrade
model performance in medical imaging, the ability to falsely improve
performance via recently-developed "enhancement attacks" may be a greater
threat to biomedical machine learning. In the spirit of developing attacks to
better understand trustworthiness, we developed three techniques to drastically
enhance prediction performance of classifiers with minimal changes to features,
including the enhancement of 1) within-dataset predictions, 2) a particular
method over another, and 3) cross-dataset generalization. Our within-dataset
enhancement framework falsely improved classifiers' accuracy from 50% to almost
100% while maintaining high feature similarities between original and enhanced
data (Pearson's r's>0.99). Similarly, the method-specific enhancement framework
was effective in falsely improving the performance of one method over another.
For example, a simple neural network outperformed LR by 50% on our enhanced
dataset, although no performance differences were present in the original
dataset. Crucially, the original and enhanced data were still similar (r=0.95).
Finally, we demonstrated that enhancement is not specific to within-dataset
predictions but can also be adapted to enhance the generalization accuracy of
one dataset to another by up to 38%. Overall, our results suggest that more
robust data sharing and provenance tracking pipelines are necessary to maintain
data integrity in biomedical machine learning research.


http://arxiv.org/abs/2301.01343
Explainability and Robustness of Deep Visual Classification Models. (92%)
Jindong Gu
  In the computer vision community, Convolutional Neural Networks (CNNs), first
proposed in the 1980's, have become the standard visual classification model.
Recently, as alternatives to CNNs, Capsule Networks (CapsNets) and Vision
Transformers (ViTs) have been proposed. CapsNets, which were inspired by the
information processing of the human brain, are considered to have more
inductive bias than CNNs, whereas ViTs are considered to have less inductive
bias than CNNs. All three classification models have received great attention
since they can serve as backbones for various downstream tasks. However, these
models are far from being perfect. As pointed out by the community, there are
two weaknesses in standard Deep Neural Networks (DNNs). One of the limitations
of DNNs is the lack of explainability. Even though they can achieve or surpass
human expert performance in the image classification task, the DNN-based
decisions are difficult to understand. In many real-world applications,
however, individual decisions need to be explained. The other limitation of
DNNs is adversarial vulnerability. Concretely, the small and imperceptible
perturbations of inputs can mislead DNNs. The vulnerability of deep neural
networks poses challenges to current visual classification models. The
potential threats thereof can lead to unacceptable consequences. Besides,
studying model adversarial vulnerability can lead to a better understanding of
the underlying models. Our research aims to address the two limitations of
DNNs. Specifically, we focus on deep visual classification models, especially
the core building parts of each classification model, e.g. dynamic routing in
CapsNets and self-attention module in ViTs.


http://arxiv.org/abs/2301.00986
Look, Listen, and Attack: Backdoor Attacks Against Video Action Recognition. (83%)
Hasan Abed Al Kader Hammoud; Shuming Liu; Mohammed Alkhrashi; Fahad AlBalawi; Bernard Ghanem
  Deep neural networks (DNNs) are vulnerable to a class of attacks called
"backdoor attacks", which create an association between a backdoor trigger and
a target label the attacker is interested in exploiting. A backdoored DNN
performs well on clean test images, yet persistently predicts an
attacker-defined label for any sample in the presence of the backdoor trigger.
Although backdoor attacks have been extensively studied in the image domain,
there are very few works that explore such attacks in the video domain, and
they tend to conclude that image backdoor attacks are less effective in the
video domain. In this work, we revisit the traditional backdoor threat model
and incorporate additional video-related aspects to that model. We show that
poisoned-label image backdoor attacks could be extended temporally in two ways,
statically and dynamically, leading to highly effective attacks in the video
domain. In addition, we explore natural video backdoors to highlight the
seriousness of this vulnerability in the video domain. And, for the first time,
we study multi-modal (audiovisual) backdoor attacks against video action
recognition models, where we show that attacking a single modality is enough
for achieving a high attack success rate.


http://arxiv.org/abs/2301.01197
Backdoor Attacks Against Dataset Distillation. (50%)
Yugeng Liu; Zheng Li; Michael Backes; Yun Shen; Yang Zhang
  Dataset distillation has emerged as a prominent technique to improve data
efficiency when training machine learning models. It encapsulates the knowledge
from a large dataset into a smaller synthetic dataset. A model trained on this
smaller distilled dataset can attain comparable performance to a model trained
on the original training dataset. However, the existing dataset distillation
techniques mainly aim at achieving the best trade-off between resource usage
efficiency and model utility. The security risks stemming from them have not
been explored. This study performs the first backdoor attack against the models
trained on the data distilled by dataset distillation models in the image
domain. Concretely, we inject triggers into the synthetic data during the
distillation procedure rather than during the model training stage, where all
previous attacks are performed. We propose two types of backdoor attacks,
namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw
data at the initial distillation phase, while DOORPING iteratively updates the
triggers during the entire distillation procedure. We conduct extensive
evaluations on multiple datasets, architectures, and dataset distillation
techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack
success rate (ASR) scores in some cases, while DOORPING reaches higher ASR
scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive
ablation study to analyze the factors that may affect the attack performance.
Finally, we evaluate multiple defense mechanisms against our backdoor attacks
and show that our attacks can practically circumvent these defense mechanisms.


http://arxiv.org/abs/2301.01044
Analysis of Label-Flip Poisoning Attack on Machine Learning Based Malware Detector. (33%)
Kshitiz Aryal; Maanak Gupta; Mahmoud Abdelsalam
  With the increase in machine learning (ML) applications in different domains,
incentives for deceiving these models have reached more than ever. As data is
the core backbone of ML algorithms, attackers shifted their interest toward
polluting the training data. Data credibility is at even higher risk with the
rise of state-of-art research topics like open design principles, federated
learning, and crowd-sourcing. Since the machine learning model depends on
different stakeholders for obtaining data, there are no reliable automated
mechanisms to verify the veracity of data from each source.
  Malware detection is arduous due to its malicious nature with the addition of
metamorphic and polymorphic ability in the evolving samples. ML has proven to
solve the zero-day malware detection problem, which is unresolved by
traditional signature-based approaches. The poisoning of malware training data
can allow the malware files to go undetected by the ML-based malware detectors,
helping the attackers to fulfill their malicious goals. A feasibility analysis
of the data poisoning threat in the malware detection domain is still lacking.
Our work will focus on two major sections: training ML-based malware detectors
and poisoning the training data using the label-poisoning approach. We will
analyze the robustness of different machine learning models against data
poisoning with varying volumes of poisoning data.


http://arxiv.org/abs/2301.00896
Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos. (92%)
Wei Xingxing; Wang Songping; Yan Huanqian
  Adversarial robustness assessment for video recognition models has raised
concerns owing to their wide applications on safety-critical tasks. Compared
with images, videos have much high dimension, which brings huge computational
costs when generating adversarial videos. This is especially serious for the
query-based black-box attacks where gradient estimation for the threat models
is usually utilized, and high dimensions will lead to a large number of
queries. To mitigate this issue, we propose to simultaneously eliminate the
temporal and spatial redundancy within the video to achieve an effective and
efficient gradient estimation on the reduced searching space, and thus query
number could decrease. To implement this idea, we design the novel Adversarial
spatial-temporal Focus (AstFocus) attack on videos, which performs attacks on
the simultaneously focused key frames and key regions from the inter-frames and
intra-frames in the video. AstFocus attack is based on the cooperative
Multi-Agent Reinforcement Learning (MARL) framework. One agent is responsible
for selecting key frames, and another agent is responsible for selecting key
regions. These two agents are jointly trained by the common rewards received
from the black-box threat models to perform a cooperative prediction. By
continuously querying, the reduced searching space composed of key frames and
key regions is becoming precise, and the whole query number becomes less than
that on the original video. Extensive experiments on four mainstream video
recognition models and three widely used action recognition datasets
demonstrate that the proposed AstFocus attack outperforms the SOTA methods,
which is prevenient in fooling rate, query number, time, and perturbation
magnitude at the same.


http://arxiv.org/abs/2301.00364
Generalizable Black-Box Adversarial Attack with Meta Learning. (99%)
Fei Yin; Yong Zhang; Baoyuan Wu; Yan Feng; Jingyi Zhang; Yanbo Fan; Yujiu Yang
  In the scenario of black-box adversarial attack, the target model's
parameters are unknown, and the attacker aims to find a successful adversarial
perturbation based on query feedback under a query budget. Due to the limited
feedback information, existing query-based black-box attack methods often
require many queries for attacking each benign example. To reduce query cost,
we propose to utilize the feedback information across historical attacks,
dubbed example-level adversarial transferability. Specifically, by treating the
attack on each benign example as one task, we develop a meta-learning framework
by training a meta-generator to produce perturbations conditioned on benign
examples. When attacking a new benign example, the meta generator can be
quickly fine-tuned based on the feedback information of the new task as well as
a few historical attacks to produce effective perturbations. Moreover, since
the meta-train procedure consumes many queries to learn a generalizable
generator, we utilize model-level adversarial transferability to train the
meta-generator on a white-box surrogate model, then transfer it to help the
attack against the target model. The proposed framework with the two types of
adversarial transferability can be naturally combined with any off-the-shelf
query-based attack methods to boost their performance, which is verified by
extensive experiments.


http://arxiv.org/abs/2301.01223
ExploreADV: Towards exploratory attack for Neural Networks. (99%)
Tianzuo Luo; Yuyi Zhong; Siaucheng Khoo
  Although deep learning has made remarkable progress in processing various
types of data such as images, text and speech, they are known to be susceptible
to adversarial perturbations: perturbations specifically designed and added to
the input to make the target model produce erroneous output. Most of the
existing studies on generating adversarial perturbations attempt to perturb the
entire input indiscriminately. In this paper, we propose ExploreADV, a general
and flexible adversarial attack system that is capable of modeling regional and
imperceptible attacks, allowing users to explore various kinds of adversarial
examples as needed. We adapt and combine two existing boundary attack methods,
DeepFool and Brendel\&Bethge Attack, and propose a mask-constrained adversarial
attack system, which generates minimal adversarial perturbations under the
pixel-level constraints, namely ``mask-constraints''. We study different ways
of generating such mask-constraints considering the variance and importance of
the input features, and show that our adversarial attack system offers users
good flexibility to focus on sub-regions of inputs, explore imperceptible
perturbations and understand the vulnerability of pixels/regions to adversarial
attacks. We demonstrate our system to be effective based on extensive
experiments and user study.


http://arxiv.org/abs/2301.00435
Trojaning semi-supervised learning model via poisoning wild images on the web. (47%)
Le Feng; Zhenxing Qian; Sheng Li; Xinpeng Zhang
  Wild images on the web are vulnerable to backdoor (also called trojan)
poisoning, causing machine learning models learned on these images to be
injected with backdoors. Most previous attacks assumed that the wild images are
labeled. In reality, however, most images on the web are unlabeled.
Specifically, we study the effects of unlabeled backdoor images under
semi-supervised learning (SSL) on widely studied deep neural networks. To be
realistic, we assume that the adversary is zero-knowledge and that the
semi-supervised learning model is trained from scratch. Firstly, we find the
fact that backdoor poisoning always fails when poisoned unlabeled images come
from different classes, which is different from poisoning the labeled images.
The reason is that the SSL algorithms always strive to correct them during
training. Therefore, for unlabeled images, we implement backdoor poisoning on
images from the target class. Then, we propose a gradient matching strategy to
craft poisoned images such that their gradients match the gradients of target
images on the SSL model, which can fit poisoned images to the target class and
realize backdoor injection. To the best of our knowledge, this may be the first
approach to backdoor poisoning on unlabeled images of trained-from-scratch SSL
models. Experiments show that our poisoning achieves state-of-the-art attack
success rates on most SSL algorithms while bypassing modern backdoor defenses.


http://arxiv.org/abs/2301.01218
Tracing the Origin of Adversarial Attack for Forensic Investigation and Deterrence. (99%)
Han Fang; Jiyi Zhang; Yupeng Qiu; Ke Xu; Chengfang Fang; Ee-Chien Chang
  Deep neural networks are vulnerable to adversarial attacks. In this paper, we
take the role of investigators who want to trace the attack and identify the
source, that is, the particular model which the adversarial examples are
generated from. Techniques derived would aid forensic investigation of attack
incidents and serve as deterrence to potential attacks. We consider the
buyers-seller setting where a machine learning model is to be distributed to
various buyers and each buyer receives a slightly different copy with same
functionality. A malicious buyer generates adversarial examples from a
particular copy $\mathcal{M}_i$ and uses them to attack other copies. From
these adversarial examples, the investigator wants to identify the source
$\mathcal{M}_i$. To address this problem, we propose a two-stage
separate-and-trace framework. The model separation stage generates multiple
copies of a model for a same classification task. This process injects unique
characteristics into each copy so that adversarial examples generated have
distinct and traceable features. We give a parallel structure which embeds a
``tracer'' in each copy, and a noise-sensitive training loss to achieve this
goal. The tracing stage takes in adversarial examples and a few candidate
models, and identifies the likely source. Based on the unique features induced
by the noise-sensitive loss function, we could effectively trace the potential
adversarial copy by considering the output logits from each tracer. Empirical
results show that it is possible to trace the origin of the adversarial example
and the mechanism can be applied to a wide range of architectures and datasets.


http://arxiv.org/abs/2212.14875
Guidance Through Surrogate: Towards a Generic Diagnostic Attack. (99%)
Muzammal Naseer; Salman Khan; Fatih Porikli; Fahad Shahbaz Khan
  Adversarial training is an effective approach to make deep neural networks
robust against adversarial attacks. Recently, different adversarial training
defenses are proposed that not only maintain a high clean accuracy but also
show significant robustness against popular and well studied adversarial
attacks such as PGD. High adversarial robustness can also arise if an attack
fails to find adversarial gradient directions, a phenomenon known as `gradient
masking'. In this work, we analyse the effect of label smoothing on adversarial
training as one of the potential causes of gradient masking. We then develop a
guided mechanism to avoid local minima during attack optimization, leading to a
novel attack dubbed Guided Projected Gradient Attack (G-PGA). Our attack
approach is based on a `match and deceive' loss that finds optimal adversarial
directions through guidance from a surrogate model. Our modified attack does
not require random restarts, large number of attack iterations or search for an
optimal step-size. Furthermore, our proposed G-PGA is generic, thus it can be
combined with an ensemble attack strategy as we demonstrate for the case of
Auto-Attack, leading to efficiency and convergence speed improvements. More
than an effective attack, G-PGA can be used as a diagnostic tool to reveal
elusive robustness due to gradient masking in adversarial defenses.


http://arxiv.org/abs/2212.14597
Defense Against Adversarial Attacks on Audio DeepFake Detection. (86%)
Piotr Kawa; Marcin Plata; Piotr Syga
  Audio DeepFakes are artificially generated utterances created using deep
learning methods with the main aim to fool the listeners, most of such audio is
highly convincing. Their quality is sufficient to pose a serious threat in
terms of security and privacy, such as the reliability of news or defamation.
To prevent the threats, multiple neural networks-based methods to detect
generated speech have been proposed. In this work, we cover the topic of
adversarial attacks, which decrease the performance of detectors by adding
superficial (difficult to spot by a human) changes to input data. Our
contribution contains evaluating the robustness of 3 detection architectures
against adversarial attacks in two scenarios (white-box and using
transferability mechanism) and enhancing it later by the use of adversarial
training performed by our novel adaptive training method.


http://arxiv.org/abs/2212.14677
Adversarial attacks and defenses on ML- and hardware-based IoT device fingerprinting and identification. (82%)
Pedro Miguel Sánchez Sánchez; Alberto Huertas Celdrán; Gérôme Bovet; Gregorio Martínez Pérez
  In the last years, the number of IoT devices deployed has suffered an
undoubted explosion, reaching the scale of billions. However, some new
cybersecurity issues have appeared together with this development. Some of
these issues are the deployment of unauthorized devices, malicious code
modification, malware deployment, or vulnerability exploitation. This fact has
motivated the requirement for new device identification mechanisms based on
behavior monitoring. Besides, these solutions have recently leveraged Machine
and Deep Learning techniques due to the advances in this field and the increase
in processing capabilities. In contrast, attackers do not stay stalled and have
developed adversarial attacks focused on context modification and ML/DL
evaluation evasion applied to IoT device identification solutions. This work
explores the performance of hardware behavior-based individual device
identification, how it is affected by possible context- and ML/DL-focused
attacks, and how its resilience can be improved using defense techniques. In
this sense, it proposes an LSTM-CNN architecture based on hardware performance
behavior for individual device identification. Then, previous techniques have
been compared with the proposed architecture using a hardware performance
dataset collected from 45 Raspberry Pi devices running identical software. The
LSTM-CNN improves previous solutions achieving a +0.96 average F1-Score and 0.8
minimum TPR for all devices. Afterward, context- and ML/DL-focused adversarial
attacks were applied against the previous model to test its robustness. A
temperature-based context attack was not able to disrupt the identification.
However, some ML/DL state-of-the-art evasion attacks were successful. Finally,
adversarial training and model distillation defense techniques are selected to
improve the model resilience to evasion attacks, without degrading its
performance.


http://arxiv.org/abs/2301.01217
Unlearnable Clusters: Towards Label-agnostic Unlearnable Examples. (13%)
Jiaming Zhang; Xingjun Ma; Qi Yi; Jitao Sang; Yugang Jiang; Yaowei Wang; Changsheng Xu
  There is a growing interest in developing unlearnable examples (UEs) against
visual privacy leaks on the Internet. UEs are training samples added with
invisible but unlearnable noise, which have been found can prevent unauthorized
training of machine learning models. UEs typically are generated via a bilevel
optimization framework with a surrogate model to remove (minimize) errors from
the original samples, and then applied to protect the data against unknown
target models. However, existing UE generation methods all rely on an ideal
assumption called label-consistency, where the hackers and protectors are
assumed to hold the same label for a given sample. In this work, we propose and
promote a more practical label-agnostic setting, where the hackers may exploit
the protected data quite differently from the protectors. E.g., a m-class
unlearnable dataset held by the protector may be exploited by the hacker as a
n-class dataset. Existing UE generation methods are rendered ineffective in
this challenging setting. To tackle this challenge, we present a novel
technique called Unlearnable Clusters (UCs) to generate label-agnostic
unlearnable examples with cluster-wise perturbations. Furthermore, we propose
to leverage VisionandLanguage Pre-trained Models (VLPMs) like CLIP as the
surrogate model to improve the transferability of the crafted UCs to diverse
domains. We empirically verify the effectiveness of our proposed approach under
a variety of settings with different datasets, target models, and even
commercial platforms Microsoft Azure and Baidu PaddlePaddle.


http://arxiv.org/abs/2301.00108
Targeted k-node Collapse Problem: Towards Understanding the Robustness of Local k-core Structure. (1%)
Yuqian Lv; Bo Zhou; Jinhuan Wang; Qi Xuan
  The concept of k-core, which indicates the largest induced subgraph where
each node has k or more neighbors, plays a significant role in measuring the
cohesiveness and the engagement of a network, and it is exploited in diverse
applications, e.g., network analysis, anomaly detection, community detection,
etc. Recent works have demonstrated the vulnerability of k-core under malicious
perturbations which focuses on removing the minimal number of edges to make a
whole k-core structure collapse. However, to the best of our knowledge, there
is no existing research concentrating on how many edges should be removed at
least to make an arbitrary node in k-core collapse. Therefore, in this paper,
we make the first attempt to study the Targeted k-node Collapse Problem (TNCP)
with four novel contributions. Firstly, we offer the general definition of TNCP
problem with the proof of its NP-hardness. Secondly, in order to address the
TNCP problem, we propose a heuristic algorithm named TNC and its improved
version named ATNC for implementations on large-scale networks. After that, the
experiments on 16 real-world networks across various domains verify the
superiority of our proposed algorithms over 4 baseline methods along with
detailed comparisons and analyses. Finally, the significance of TNCP problem
for precisely evaluating the resilience of k-core structures in networks is
validated.


http://arxiv.org/abs/2212.14315
"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice. (68%)
Giovanni Apruzzese; Hyrum S. Anderson; Savino Dambra; David Freeman; Fabio Pierazzi; Kevin A. Roundy
  Recent years have seen a proliferation of research on adversarial machine
learning. Numerous papers demonstrate powerful algorithmic attacks against a
wide variety of machine learning (ML) models, and numerous other papers propose
defenses that can withstand most attacks. However, abundant real-world evidence
suggests that actual attackers use simple tactics to subvert ML-driven systems,
and as a result security practitioners have not prioritized adversarial ML
defenses.
  Motivated by the apparent gap between researchers and practitioners, this
position paper aims to bridge the two domains. We first present three
real-world case studies from which we can glean practical insights unknown or
neglected in research. Next we analyze all adversarial ML papers recently
published in top security conferences, highlighting positive trends and blind
spots. Finally, we state positions on precise and cost-driven threat modeling,
collaboration between industry and academia, and reproducible research. We
believe that our positions, if adopted, will increase the real-world impact of
future endeavours in adversarial ML, bringing both researchers and
practitioners closer to their shared goal of improving the security of ML
systems.


http://arxiv.org/abs/2212.14268
Detection of out-of-distribution samples using binary neuron activation patterns. (9%)
Bartlomiej Olber; Krystian Radlak; Adam Popowicz; Michal Szczepankiewicz; Krystian Chachula
  Deep neural networks (DNN) have outstanding performance in various
applications. Despite numerous efforts of the research community,
out-of-distribution (OOD) samples remain significant limitation of DNN
classifiers. The ability to identify previously unseen inputs as novel is
crucial in safety-critical applications such as self-driving cars, unmanned
aerial vehicles and robots. Existing approaches to detect OOD samples treat a
DNN as a black box and assess the confidence score of the output predictions.
Unfortunately, this method frequently fails, because DNN are not trained to
reduce their confidence for OOD inputs. In this work, we introduce a novel
method for OOD detection. Our method is motivated by theoretical analysis of
neuron activation patterns (NAP) in ReLU based architectures. The proposed
method does not introduce high computational workload due to the binary
representation of the activation patterns extracted from convolutional layers.
The extensive empirical evaluation proves its high performance on various DNN
architectures and seven image datasets. ion.


http://arxiv.org/abs/2212.13707
Thermal Heating in ReRAM Crossbar Arrays: Challenges and Solutions. (99%)
Kamilya Smagulova; Mohammed E. Fouda; Ahmed Eltawil
  Increasing popularity of deep-learning-powered applications raises the issue
of vulnerability of neural networks to adversarial attacks. In other words,
hardly perceptible changes in input data lead to the output error in neural
network hindering their utilization in applications that involve decisions with
security risks. A number of previous works have already thoroughly evaluated
the most commonly used configuration - Convolutional Neural Networks (CNNs)
against different types of adversarial attacks. Moreover, recent works
demonstrated transferability of the some adversarial examples across different
neural network models. This paper studied robustness of the new emerging models
such as SpinalNet-based neural networks and Compact Convolutional Transformers
(CCT) on image classification problem of CIFAR-10 dataset. Each architecture
was tested against four White-box attacks and three Black-box attacks. Unlike
VGG and SpinalNet models, attention-based CCT configuration demonstrated large
span between strong robustness and vulnerability to adversarial examples.
Eventually, the study of transferability between VGG, VGG-inspired SpinalNet
and pretrained CCT 7/3x1 models was conducted. It was shown that despite high
effectiveness of the attack on the certain individual model, this does not
guarantee the transferability to other models.


http://arxiv.org/abs/2212.14115
Certifying Safety in Reinforcement Learning under Adversarial Perturbation Attacks. (98%)
Junlin Wu; Hussein Sibai; Yevgeniy Vorobeychik
  Function approximation has enabled remarkable advances in applying
reinforcement learning (RL) techniques in environments with high-dimensional
inputs, such as images, in an end-to-end fashion, mapping such inputs directly
to low-level control. Nevertheless, these have proved vulnerable to small
adversarial input perturbations. A number of approaches for improving or
certifying robustness of end-to-end RL to adversarial perturbations have
emerged as a result, focusing on cumulative reward. However, what is often at
stake in adversarial scenarios is the violation of fundamental properties, such
as safety, rather than the overall reward that combines safety with efficiency.
Moreover, properties such as safety can only be defined with respect to true
state, rather than the high-dimensional raw inputs to end-to-end policies. To
disentangle nominal efficiency and adversarial safety, we situate RL in
deterministic partially-observable Markov decision processes (POMDPs) with the
goal of maximizing cumulative reward subject to safety constraints. We then
propose a partially-supervised reinforcement learning (PSRL) framework that
takes advantage of an additional assumption that the true state of the POMDP is
known at training time. We present the first approach for certifying safety of
PSRL policies under adversarial input perturbations, and two adversarial
training approaches that make direct use of PSRL. Our experiments demonstrate
both the efficacy of the proposed approach for certifying safety in adversarial
environments, and the value of the PSRL framework coupled with adversarial
training in improving certified safety while preserving high nominal reward and
high-quality predictions of true state.


http://arxiv.org/abs/2212.13700
Publishing Efficient On-device Models Increases Adversarial Vulnerability. (95%)
Sanghyun Hong; Nicholas Carlini; Alexey Kurakin
  Recent increases in the computational demands of deep neural networks (DNNs)
have sparked interest in efficient deep learning mechanisms, e.g., quantization
or pruning. These mechanisms enable the construction of a small, efficient
version of commercial-scale models with comparable accuracy, accelerating their
deployment to resource-constrained devices.
  In this paper, we study the security considerations of publishing on-device
variants of large-scale models. We first show that an adversary can exploit
on-device models to make attacking the large models easier. In evaluations
across 19 DNNs, by exploiting the published on-device models as a transfer
prior, the adversarial vulnerability of the original commercial-scale models
increases by up to 100x. We then show that the vulnerability increases as the
similarity between a full-scale and its efficient model increase. Based on the
insights, we propose a defense, $similarity$-$unpairing$, that fine-tunes
on-device models with the objective of reducing the similarity. We evaluated
our defense on all the 19 DNNs and found that it reduces the transferability up
to 90% and the number of queries required by a factor of 10-100x. Our results
suggest that further research is needed on the security (or even privacy)
threats caused by publishing those efficient siblings.


http://arxiv.org/abs/2212.14049
Differentiable Search of Accurate and Robust Architectures. (92%)
Yuwei Ou; Xiangning Xie; Shangce Gao; Yanan Sun; Kay Chen Tan; Jiancheng Lv
  Deep neural networks (DNNs) are found to be vulnerable to adversarial
attacks, and various methods have been proposed for the defense. Among these
methods, adversarial training has been drawing increasing attention because of
its simplicity and effectiveness. However, the performance of the adversarial
training is greatly limited by the architectures of target DNNs, which often
makes the resulting DNNs with poor accuracy and unsatisfactory robustness. To
address this problem, we propose DSARA to automatically search for the neural
architectures that are accurate and robust after adversarial training. In
particular, we design a novel cell-based search space specially for adversarial
training, which improves the accuracy and the robustness upper bound of the
searched architectures by carefully designing the placement of the cells and
the proportional relationship of the filter numbers. Then we propose a
two-stage search strategy to search for both accurate and robust neural
architectures. At the first stage, the architecture parameters are optimized to
minimize the adversarial loss, which makes full use of the effectiveness of the
adversarial training in enhancing the robustness. At the second stage, the
architecture parameters are optimized to minimize both the natural loss and the
adversarial loss utilizing the proposed multi-objective adversarial training
method, so that the searched neural architectures are both accurate and robust.
We evaluate the proposed algorithm under natural data and various adversarial
attacks, which reveals the superiority of the proposed method in terms of both
accurate and robust architectures. We also conclude that accurate and robust
neural architectures tend to deploy very different structures near the input
and the output, which has great practical significance on both hand-crafting
and automatically designing of accurate and robust neural architectures.


http://arxiv.org/abs/2212.14106
Robust Ranking Explanations. (74%)
Chao Chen; Chenghua Guo; Guixiang Ma; Xi Zhang; Sihong Xie
  Gradient-based explanation is the cornerstone of explainable deep networks,
but it has been shown to be vulnerable to adversarial attacks. However,
existing works measure the explanation robustness based on $\ell_p$-norm, which
can be counter-intuitive to humans, who only pay attention to the top few
salient features. We propose explanation ranking thickness as a more suitable
explanation robustness metric. We then present a new practical adversarial
attacking goal for manipulating explanation rankings. To mitigate the
ranking-based attacks while maintaining computational feasibility, we derive
surrogate bounds of the thickness that involve expensive sampling and
integration. We use a multi-objective approach to analyze the convergence of a
gradient-based attack to confirm that the explanation robustness can be
measured by the thickness metric. We conduct experiments on various network
architectures and diverse datasets to prove the superiority of the proposed
methods, while the widely accepted Hessian-based curvature smoothing approaches
are not as robust as our method.


http://arxiv.org/abs/2212.13929
Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset. (1%)
Suba S; Nita Parekh; Ramesh Loganathan; Vikram Pudi; Chinnababu Sunkavalli
  Computer tomography (CT) have been routinely used for the diagnosis of lung
diseases and recently, during the pandemic, for detecting the infectivity and
severity of COVID-19 disease. One of the major concerns in using ma-chine
learning (ML) approaches for automatic processing of CT scan images in clinical
setting is that these methods are trained on limited and biased sub-sets of
publicly available COVID-19 data. This has raised concerns regarding the
generalizability of these models on external datasets, not seen by the model
during training. To address some of these issues, in this work CT scan images
from confirmed COVID-19 data obtained from one of the largest public
repositories, COVIDx CT 2A were used for training and internal vali-dation of
machine learning models. For the external validation we generated
Indian-COVID-19 CT dataset, an open-source repository containing 3D CT volumes
and 12096 chest CT images from 288 COVID-19 patients from In-dia. Comparative
performance evaluation of four state-of-the-art machine learning models, viz.,
a lightweight convolutional neural network (CNN), and three other CNN based
deep learning (DL) models such as VGG-16, ResNet-50 and Inception-v3 in
classifying CT images into three classes, viz., normal, non-covid pneumonia,
and COVID-19 is carried out on these two datasets. Our analysis showed that the
performance of all the models is comparable on the hold-out COVIDx CT 2A test
set with 90% - 99% accuracies (96% for CNN), while on the external
Indian-COVID-19 CT dataset a drop in the performance is observed for all the
models (8% - 19%). The traditional ma-chine learning model, CNN performed the
best on the external dataset (accu-racy 88%) in comparison to the deep learning
models, indicating that a light-weight CNN is better generalizable on unseen
data. The data and code are made available at https://github.com/aleesuss/c19.


http://arxiv.org/abs/2212.13607
EDoG: Adversarial Edge Detection For Graph Neural Networks. (98%)
Xiaojun Xu; Yue Yu; Hanzhang Wang; Alok Lal; Carl A. Gunter; Bo Li
  Graph Neural Networks (GNNs) have been widely applied to different tasks such
as bioinformatics, drug design, and social networks. However, recent studies
have shown that GNNs are vulnerable to adversarial attacks which aim to mislead
the node or subgraph classification prediction by adding subtle perturbations.
Detecting these attacks is challenging due to the small magnitude of
perturbation and the discrete nature of graph data. In this paper, we propose a
general adversarial edge detection pipeline EDoG without requiring knowledge of
the attack strategies based on graph generation. Specifically, we propose a
novel graph generation approach combined with link prediction to detect
suspicious adversarial edges. To effectively train the graph generative model,
we sample several sub-graphs from the given graph data. We show that since the
number of adversarial edges is usually low in practice, with low probability
the sampled sub-graphs will contain adversarial edges based on the union bound.
In addition, considering the strong attacks which perturb a large number of
edges, we propose a set of novel features to perform outlier detection as the
preprocessing for our detection. Extensive experimental results on three
real-world graph datasets including a private transaction rule dataset from a
major company and two types of synthetic graphs with controlled properties show
that EDoG can achieve above 0.8 AUC against four state-of-the-art unseen attack
strategies without requiring any knowledge about the attack type; and around
0.85 with knowledge of the attack type. EDoG significantly outperforms
traditional malicious edge detection baselines. We also show that an adaptive
attack with full knowledge of our detection pipeline is difficult to bypass it.


http://arxiv.org/abs/2212.13667
Learning When to Use Adaptive Adversarial Image Perturbations against Autonomous Vehicles. (86%)
Hyung-Jin Yoon; Hamidreza Jafarnejadsani; Petros Voulgaris
  The deep neural network (DNN) models for object detection using camera images
are widely adopted in autonomous vehicles. However, DNN models are shown to be
susceptible to adversarial image perturbations. In the existing methods of
generating the adversarial image perturbations, optimizations take each
incoming image frame as the decision variable to generate an image
perturbation. Therefore, given a new image, the typically
computationally-expensive optimization needs to start over as there is no
learning between the independent optimizations. Very few approaches have been
developed for attacking online image streams while considering the underlying
physical dynamics of autonomous vehicles, their mission, and the environment.
We propose a multi-level stochastic optimization framework that monitors an
attacker's capability of generating the adversarial perturbations. Based on
this capability level, a binary decision attack/not attack is introduced to
enhance the effectiveness of the attacker. We evaluate our proposed multi-level
image attack framework using simulations for vision-guided autonomous vehicles
and actual tests with a small indoor drone in an office environment. The
results show our method's capability to generate the image attack in real-time
while monitoring when the attacker is proficient given state estimates.


http://arxiv.org/abs/2302.03523
Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs. (62%)
Souvik Kundu; Sairam Sundaresan; Sharath Nittur Sridhar; Shunlin Lu; Han Tang; Peter A. Beerel
  Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA)
performance on both clean and adversarially-perturbed images rely on either
activation or weight conditioned convolution operations. However, such
conditional learning costs additional multiply-accumulate (MAC) or addition
operations, increasing inference memory and compute costs. To that end, we
present a sparse mixture once for all adversarial training (SMART), that allows
a model to train once and then in-situ trade-off between accuracy and
robustness, that too at a reduced compute and parameter overhead. In
particular, SMART develops two expert paths, for clean and adversarial images,
respectively, that are then conditionally trained via respective dedicated sets
of binary sparsity masks. Extensive evaluations on multiple image
classification datasets across different models show SMART to have up to 2.72x
fewer non-zero parameters costing proportional reduction in compute overhead,
while yielding SOTA accuracy-robustness trade-off. Additionally, we present
insightful observations in designing sparse masks to successfully condition on
both clean and perturbed images.


http://arxiv.org/abs/2212.13675
XMAM:X-raying Models with A Matrix to Reveal Backdoor Attacks for Federated Learning. (56%)
Jianyi Zhang; Fangjiao Zhang; Qichao Jin; Zhiqiang Wang; Xiaodong Lin; Xiali Hei
  Federated Learning (FL) has received increasing attention due to its privacy
protection capability. However, the base algorithm FedAvg is vulnerable when it
suffers from so-called backdoor attacks. Former researchers proposed several
robust aggregation methods. Unfortunately, many of these aggregation methods
are unable to defend against backdoor attacks. What's more, the attackers
recently have proposed some hiding methods that further improve backdoor
attacks' stealthiness, making all the existing robust aggregation methods fail.
  To tackle the threat of backdoor attacks, we propose a new aggregation
method, X-raying Models with A Matrix (XMAM), to reveal the malicious local
model updates submitted by the backdoor attackers. Since we observe that the
output of the Softmax layer exhibits distinguishable patterns between malicious
and benign updates, we focus on the Softmax layer's output in which the
backdoor attackers are difficult to hide their malicious behavior.
Specifically, like X-ray examinations, we investigate the local model updates
by using a matrix as an input to get their Softmax layer's outputs. Then, we
preclude updates whose outputs are abnormal by clustering. Without any training
dataset in the server, the extensive evaluations show that our XMAM can
effectively distinguish malicious local model updates from benign ones. For
instance, when other methods fail to defend against the backdoor attacks at no
more than 20% malicious clients, our method can tolerate 45% malicious clients
in the black-box mode and about 30% in Projected Gradient Descent (PGD) mode.
Besides, under adaptive attacks, the results demonstrate that XMAM can still
complete the global model training task even when there are 40% malicious
clients. Finally, we analyze our method's screening complexity, and the results
show that XMAM is about 10-10000 times faster than the existing methods.


http://arxiv.org/abs/2212.12995
Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks. (99%)
Xingxing Wei; Ying Guo; Jie Yu; Bo Zhang
  Adversarial patch is an important form of real-world adversarial attack that
brings serious risks to the robustness of deep neural networks. Previous
methods generate adversarial patches by either optimizing their perturbation
values while fixing the pasting position or manipulating the position while
fixing the patch's content. This reveals that the positions and perturbations
are both important to the adversarial attack. For that, in this paper, we
propose a novel method to simultaneously optimize the position and perturbation
for an adversarial patch, and thus obtain a high attack success rate in the
black-box setting. Technically, we regard the patch's position, the
pre-designed hyper-parameters to determine the patch's perturbations as the
variables, and utilize the reinforcement learning framework to simultaneously
solve for the optimal solution based on the rewards obtained from the target
model with a small number of queries. Extensive experiments are conducted on
the Face Recognition (FR) task, and results on four representative FR models
show that our method can significantly improve the attack success rate and
query efficiency. Besides, experiments on the commercial FR service and
physical environments confirm its practical application value. We also extend
our method to the traffic sign recognition task to verify its generalization
ability.


http://arxiv.org/abs/2212.12732
Frequency Regularization for Improving Adversarial Robustness. (99%)
Binxiao Huang; Chaofan Tao; Rui Lin; Ngai Wong
  Deep neural networks are incredibly vulnerable to crafted,
human-imperceptible adversarial perturbations. Although adversarial training
(AT) has proven to be an effective defense approach, we find that the
AT-trained models heavily rely on the input low-frequency content for judgment,
accounting for the low standard accuracy. To close the large gap between the
standard and robust accuracies during AT, we investigate the frequency
difference between clean and adversarial inputs, and propose a frequency
regularization (FR) to align the output difference in the spectral domain.
Besides, we find Stochastic Weight Averaging (SWA), by smoothing the kernels
over epochs, further improves the robustness. Among various defense schemes,
our method achieves the strongest robustness against attacks by PGD-20, C\&W
and Autoattack, on a WideResNet trained on CIFAR-10 without any extra data.


http://arxiv.org/abs/2212.12641
Out-of-Distribution Detection with Reconstruction Error and Typicality-based Penalty. (61%)
Genki Osada; Takahashi Tsubasa; Budrul Ahsan; Takashi Nishide
  The task of out-of-distribution (OOD) detection is vital to realize safe and
reliable operation for real-world applications. After the failure of
likelihood-based detection in high dimensions had been shown, approaches based
on the \emph{typical set} have been attracting attention; however, they still
have not achieved satisfactory performance. Beginning by presenting the failure
case of the typicality-based approach, we propose a new reconstruction
error-based approach that employs normalizing flow (NF). We further introduce a
typicality-based penalty, and by incorporating it into the reconstruction error
in NF, we propose a new OOD detection method, penalized reconstruction error
(PRE). Because the PRE detects test inputs that lie off the in-distribution
manifold, it effectively detects adversarial examples as well as OOD examples.
We show the effectiveness of our method through the evaluation using natural
image datasets, CIFAR-10, TinyImageNet, and ILSVRC2012.


http://arxiv.org/abs/2212.12380
Towards Scalable Physically Consistent Neural Networks: an Application to Data-driven Multi-zone Thermal Building Models. (1%)
Natale Loris Di; Bratislav Svetozarevic; Philipp Heer; Colin Neil Jones
  With more and more data being collected, data-driven modeling methods have
been gaining in popularity in recent years. While physically sound, classical
gray-box models are often cumbersome to identify and scale, and their accuracy
might be hindered by their limited expressiveness. On the other hand, classical
black-box methods, typically relying on Neural Networks (NNs) nowadays, often
achieve impressive performance, even at scale, by deriving statistical patterns
from data. However, they remain completely oblivious to the underlying physical
laws, which may lead to potentially catastrophic failures if decisions for
real-world physical systems are based on them. Physically Consistent Neural
Networks (PCNNs) were recently developed to address these aforementioned
issues, ensuring physical consistency while still leveraging NNs to attain
state-of-the-art accuracy.
  In this work, we scale PCNNs to model building temperature dynamics and
propose a thorough comparison with classical gray-box and black-box methods.
More precisely, we design three distinct PCNN extensions, thereby exemplifying
the modularity and flexibility of the architecture, and formally prove their
physical consistency. In the presented case study, PCNNs are shown to achieve
state-of-the-art accuracy, even outperforming classical NN-based models despite
their constrained structure. Our investigations furthermore provide a clear
illustration of NNs achieving seemingly good performance while remaining
completely physics-agnostic, which can be misleading in practice. While this
performance comes at the cost of computational complexity, PCNNs on the other
hand show accuracy improvements of 17-35% compared to all other physically
consistent methods, paving the way for scalable physically consistent models
with state-of-the-art performance.


http://arxiv.org/abs/2212.11778
Adversarial Machine Learning and Defense Game for NextG Signal Classification with Deep Learning. (98%)
Yalin E. Sagduyu
  This paper presents a game-theoretic framework to study the interactions of
attack and defense for deep learning-based NextG signal classification. NextG
systems such as the one envisioned for a massive number of IoT devices can
employ deep neural networks (DNNs) for various tasks such as user equipment
identification, physical layer authentication, and detection of incumbent users
(such as in the Citizens Broadband Radio Service (CBRS) band). By training
another DNN as the surrogate model, an adversary can launch an inference
(exploratory) attack to learn the behavior of the victim model, predict
successful operation modes (e.g., channel access), and jam them. A defense
mechanism can increase the adversary's uncertainty by introducing controlled
errors in the victim model's decisions (i.e., poisoning the adversary's
training data). This defense is effective against an attack but reduces the
performance when there is no attack. The interactions between the defender and
the adversary are formulated as a non-cooperative game, where the defender
selects the probability of defending or the defense level itself (i.e., the
ratio of falsified decisions) and the adversary selects the probability of
attacking. The defender's objective is to maximize its reward (e.g., throughput
or transmission success ratio), whereas the adversary's objective is to
minimize this reward and its attack cost. The Nash equilibrium strategies are
determined as operation modes such that no player can unilaterally improve its
utility given the other's strategy is fixed. A fictitious play is formulated
for each player to play the game repeatedly in response to the empirical
frequency of the opponent's actions. The performance in Nash equilibrium is
compared to the fixed attack and defense cases, and the resilience of NextG
signal classification against attacks is quantified.


http://arxiv.org/abs/2212.11760
Aliasing is a Driver of Adversarial Attacks. (80%)
Adrián Rodríguez-Muñoz; Antonio Torralba
  Aliasing is a highly important concept in signal processing, as careful
consideration of resolution changes is essential in ensuring transmission and
processing quality of audio, image, and video. Despite this, up until recently
aliasing has received very little consideration in Deep Learning, with all
common architectures carelessly sub-sampling without considering aliasing
effects. In this work, we investigate the hypothesis that the existence of
adversarial perturbations is due in part to aliasing in neural networks. Our
ultimate goal is to increase robustness against adversarial attacks using
explainable, non-trained, structural changes only, derived from aliasing first
principles. Our contributions are the following. First, we establish a
sufficient condition for no aliasing for general image transformations. Next,
we study sources of aliasing in common neural network layers, and derive simple
modifications from first principles to eliminate or reduce it. Lastly, our
experimental results show a solid link between anti-aliasing and adversarial
attacks. Simply reducing aliasing already results in more robust classifiers,
and combining anti-aliasing with robust training out-performs solo robust
training on $L_2$ attacks with none or minimal losses in performance on
$L_{\infty}$ attacks.


http://arxiv.org/abs/2212.11810
GAN-based Domain Inference Attack. (2%)
Yuechun Gu; Keke Chen
  Model-based attacks can infer training data information from deep neural
network models. These attacks heavily depend on the attacker's knowledge of the
application domain, e.g., using it to determine the auxiliary data for
model-inversion attacks. However, attackers may not know what the model is used
for in practice. We propose a generative adversarial network (GAN) based method
to explore likely or similar domains of a target model -- the model domain
inference (MDI) attack. For a given target (classification) model, we assume
that the attacker knows nothing but the input and output formats and can use
the model to derive the prediction for any input in the desired form. Our basic
idea is to use the target model to affect a GAN training process for a
candidate domain's dataset that is easy to obtain. We find that the target
model may distract the training procedure less if the domain is more similar to
the target domain. We then measure the distraction level with the distance
between GAN-generated datasets, which can be used to rank candidate domains for
the target model. Our experiments show that the auxiliary dataset from an MDI
top-ranked domain can effectively boost the result of model-inversion attacks.


http://arxiv.org/abs/2212.11614
Hybrid Quantum-Classical Generative Adversarial Network for High Resolution Image Generation. (1%)
Shu Lok Tsang; Maxwell T. West; Sarah M. Erfani; Muhammad Usman
  Quantum machine learning (QML) has received increasing attention due to its
potential to outperform classical machine learning methods in various problems.
A subclass of QML methods is quantum generative adversarial networks (QGANs)
which have been studied as a quantum counterpart of classical GANs widely used
in image manipulation and generation tasks. The existing work on QGANs is still
limited to small-scale proof-of-concept examples based on images with
significant down-scaling. Here we integrate classical and quantum techniques to
propose a new hybrid quantum-classical GAN framework. We demonstrate its
superior learning capabilities by generating $28 \times 28$ pixels grey-scale
images without dimensionality reduction or classical pre/post-processing on
multiple classes of the standard MNIST and Fashion MNIST datasets, which
achieves comparable results to classical frameworks with 3 orders of magnitude
less trainable generator parameters. To gain further insight into the working
of our hybrid approach, we systematically explore the impact of its parameter
space by varying the number of qubits, the size of image patches, the number of
layers in the generator, the shape of the patches and the choice of prior
distribution. Our results show that increasing the quantum generator size
generally improves the learning capability of the network. The developed
framework provides a foundation for future design of QGANs with optimal
parameter set tailored for complex image generation tasks.


http://arxiv.org/abs/2212.11005
Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspective. (80%)
Shihua Huang; Zhichao Lu; Kalyanmoy Deb; Vishnu Naresh Boddeti
  Efforts to improve the adversarial robustness of convolutional neural
networks have primarily focused on developing more effective adversarial
training methods. In contrast, little attention was devoted to analyzing the
role of architectural elements (such as topology, depth, and width) on
adversarial robustness. This paper seeks to bridge this gap and present a
holistic study on the impact of architectural design on adversarial robustness.
We focus on residual networks and consider architecture design at the block
level, i.e., topology, kernel size, activation, and normalization, as well as
at the network scaling level, i.e., depth and width of each block in the
network. In both cases, we first derive insights through systematic ablative
experiments. Then we design a robust residual block, dubbed RobustResBlock, and
a compound scaling rule, dubbed RobustScaling, to distribute depth and width at
the desired FLOP count. Finally, we combine RobustResBlock and RobustScaling
and present a portfolio of adversarially robust residual networks,
RobustResNets, spanning a broad spectrum of model capacities. Experimental
validation across multiple datasets and adversarial attacks demonstrate that
RobustResNets consistently outperform both the standard WRNs and other existing
robust architectures, achieving state-of-the-art AutoAttack robust accuracy of
61.1% without additional data and 63.7% with 500K external data while being
$2\times$ more compact in terms of parameters. Code is available at \url{
https://github.com/zhichao-lu/robust-residual-network}


http://arxiv.org/abs/2212.11205
Vulnerabilities of Deep Learning-Driven Semantic Communications to Backdoor (Trojan) Attacks. (67%)
Yalin E. Sagduyu; Tugba Erpek; Sennur Ulukus; Aylin Yener
  This paper highlights vulnerabilities of deep learning-driven semantic
communications to backdoor (Trojan) attacks. Semantic communications aims to
convey a desired meaning while transferring information from a transmitter to
its receiver. An encoder-decoder pair that is represented by two deep neural
networks (DNNs) as part of an autoencoder is trained to reconstruct signals
such as images at the receiver by transmitting latent features of small size
over a limited number of channel uses. In the meantime, another DNN of a
semantic task classifier at the receiver is jointly trained with the
autoencoder to check the meaning conveyed to the receiver. The complex decision
space of the DNNs makes semantic communications susceptible to adversarial
manipulations. In a backdoor (Trojan) attack, the adversary adds triggers to a
small portion of training samples and changes the label to a target label. When
the transfer of images is considered, the triggers can be added to the images
or equivalently to the corresponding transmitted or received signals. In test
time, the adversary activates these triggers by providing poisoned samples as
input to the encoder (or decoder) of semantic communications. The backdoor
attack can effectively change the semantic information transferred for the
poisoned input samples to a target meaning. As the performance of semantic
communications improves with the signal-to-noise ratio and the number of
channel uses, the success of the backdoor attack increases as well. Also,
increasing the Trojan ratio in training data makes the attack more successful.
In the meantime, the effect of this attack on the unpoisoned input samples
remains limited. Overall, this paper shows that the backdoor attack poses a
serious threat to semantic communications and presents novel design guidelines
to preserve the meaning of transferred information in the presence of backdoor
attacks.


http://arxiv.org/abs/2212.11209
A Theoretical Study of The Effects of Adversarial Attacks on Sparse Regression. (13%)
Deepak Maurya; Jean Honorio
  This paper analyzes $\ell_1$ regularized linear regression under the
challenging scenario of having only adversarially corrupted data for training.
We use the primal-dual witness paradigm to provide provable performance
guarantees for the support of the estimated regression parameter vector to
match the actual parameter. Our theoretical analysis shows the
counter-intuitive result that an adversary can influence sample complexity by
corrupting the irrelevant features, i.e., those corresponding to zero
coefficients of the regression parameter vector, which, consequently, do not
affect the dependent variable. As any adversarially robust algorithm has its
limitations, our theoretical analysis identifies the regimes under which the
learning algorithm and adversary can dominate over each other. It helps us to
analyze these fundamental limits and address critical scientific questions of
which parameters (like mutual incoherence, the maximum and minimum eigenvalue
of the covariance matrix, and the budget of adversarial perturbation) play a
role in the high or low probability of success of the LASSO algorithm. Also,
the derived sample complexity is logarithmic with respect to the size of the
regression parameter vector, and our theoretical claims are validated by
empirical analysis on synthetic and real-world datasets.


http://arxiv.org/abs/2212.10230
A Comprehensive Study and Comparison of the Robustness of 3D Object Detectors Against Adversarial Attacks. (98%)
Yifan Zhang; Junhui Hou; Yixuan Yuan
  Deep learning-based 3D object detectors have made significant progress in
recent years and have been deployed in a wide range of applications. It is
crucial to understand the robustness of detectors against adversarial attacks
when employing detectors in security-critical applications. In this paper, we
make the first attempt to conduct a thorough evaluation and analysis of the
robustness of 3D detectors under adversarial attacks. Specifically, we first
extend three kinds of adversarial attacks to the 3D object detection task to
benchmark the robustness of state-of-the-art 3D object detectors against
attacks on KITTI and Waymo datasets, subsequently followed by the analysis of
the relationship between robustness and properties of detectors. Then, we
explore the transferability of cross-model, cross-task, and cross-data attacks.
We finally conduct comprehensive experiments of defense for 3D detectors,
demonstrating that simple transformations like flipping are of little help in
improving robustness when the strategy of transformation imposed on input point
cloud data is exposed to attackers. Our findings will facilitate investigations
in understanding and defending the adversarial attacks against 3D object
detectors to advance this field.


http://arxiv.org/abs/2212.10006
Multi-head Uncertainty Inference for Adversarial Attack Detection. (98%)
Yuqi Yang; Songyun Yang; Jiyang Xie. Zhongwei Si; Kai Guo; Ke Zhang; Kongming Liang
  Deep neural networks (DNNs) are sensitive and susceptible to tiny
perturbation by adversarial attacks which causes erroneous predictions. Various
methods, including adversarial defense and uncertainty inference (UI), have
been developed in recent years to overcome the adversarial attacks. In this
paper, we propose a multi-head uncertainty inference (MH-UI) framework for
detecting adversarial attack examples. We adopt a multi-head architecture with
multiple prediction heads (i.e., classifiers) to obtain predictions from
different depths in the DNNs and introduce shallow information for the UI.
Using independent heads at different depths, the normalized predictions are
assumed to follow the same Dirichlet distribution, and we estimate distribution
parameter of it by moment matching. Cognitive uncertainty brought by the
adversarial attacks will be reflected and amplified on the distribution.
Experimental results show that the proposed MH-UI framework can outperform all
the referred UI methods in the adversarial attack detection task with different
settings.


http://arxiv.org/abs/2212.10258
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing. (98%)
Yahan Yang; Soham Dan; Dan Roth; Insup Lee
  Recently it has been shown that state-of-the-art NLP models are vulnerable to
adversarial attacks, where the predictions of a model can be drastically
altered by slight modifications to the input (such as synonym substitutions).
While several defense techniques have been proposed, and adapted, to the
discrete nature of text adversarial attacks, the benefits of general-purpose
regularization methods such as label smoothing for language models, have not
been studied. In this paper, we study the adversarial robustness provided by
various label smoothing strategies in foundational models for diverse NLP tasks
in both in-domain and out-of-domain settings. Our experiments show that label
smoothing significantly improves adversarial robustness in pre-trained models
like BERT, against various popular attacks. We also analyze the relationship
between prediction confidence and robustness, showing that label smoothing
reduces over-confident errors on adversarial examples.


http://arxiv.org/abs/2212.10438
Is Semantic Communications Secure? A Tale of Multi-Domain Adversarial Attacks. (96%)
Yalin E. Sagduyu; Tugba Erpek; Sennur Ulukus; Aylin Yener
  Semantic communications seeks to transfer information from a source while
conveying a desired meaning to its destination. We model the
transmitter-receiver functionalities as an autoencoder followed by a task
classifier that evaluates the meaning of the information conveyed to the
receiver. The autoencoder consists of an encoder at the transmitter to jointly
model source coding, channel coding, and modulation, and a decoder at the
receiver to jointly model demodulation, channel decoding and source decoding.
By augmenting the reconstruction loss with a semantic loss, the two deep neural
networks (DNNs) of this encoder-decoder pair are interactively trained with the
DNN of the semantic task classifier. This approach effectively captures the
latent feature space and reliably transfers compressed feature vectors with a
small number of channel uses while keeping the semantic loss low. We identify
the multi-domain security vulnerabilities of using the DNNs for semantic
communications. Based on adversarial machine learning, we introduce test-time
(targeted and non-targeted) adversarial attacks on the DNNs by manipulating
their inputs at different stages of semantic communications. As a computer
vision attack, small perturbations are injected to the images at the input of
the transmitter's encoder. As a wireless attack, small perturbations signals
are transmitted to interfere with the input of the receiver's decoder. By
launching these stealth attacks individually or more effectively in a combined
form as a multi-domain attack, we show that it is possible to change the
semantics of the transferred information even when the reconstruction loss
remains low. These multi-domain adversarial attacks pose as a serious threat to
the semantics of information transfer (with larger impact than conventional
jamming) and raise the need of defense methods for the safe adoption of
semantic communications.


http://arxiv.org/abs/2212.10556
Unleashing the Power of Visual Prompting At the Pixel Level. (92%)
Junyang Wu; Xianhang Li; Chen Wei; Huiyu Wang; Alan Yuille; Yuyin Zhou; Cihang Xie
  This paper presents a simple and effective visual prompting method for
adapting pre-trained models to downstream recognition tasks. Our method
includes two key designs. First, rather than directly adding together the
prompt and the image, we treat the prompt as an extra and independent learnable
component. We show that the strategy of reconciling the prompt and the image
matters, and find that warping the prompt around a properly shrinked image
empirically works the best. Second, we re-introduce two "old tricks" commonly
used in building transferable adversarial examples, i.e., input diversity and
gradient normalization, into visual prompting. These techniques improve
optimization and enable the prompt to generalize better. We provide extensive
experimental results to demonstrate the effectiveness of our method. Using a
CLIP model, our prompting method sets a new record of 82.8% average accuracy
across 12 popular classification datasets, substantially surpassing the prior
art by +5.6%. It is worth noting that this prompting performance already
outperforms linear probing by +2.1% and can even match fully fine-tuning in
certain datasets. In addition, our prompting method shows competitive
performance across different data scales and against distribution shifts. The
code is publicly available at https://github.com/UCSC-VLAA/EVP.


http://arxiv.org/abs/2212.10318
Learned Systems Security. (78%)
Roei Schuster; Jin Peng Zhou; Paul Grubbs; Thorsten Eisenhofer; Nicolas Papernot
  A learned system uses machine learning (ML) internally to improve
performance. We can expect such systems to be vulnerable to some adversarial-ML
attacks. Often, the learned component is shared between mutually-distrusting
users or processes, much like microarchitectural resources such as caches,
potentially giving rise to highly-realistic attacker models. However, compared
to attacks on other ML-based systems, attackers face a level of indirection as
they cannot interact directly with the learned model. Additionally, the
difference between the attack surface of learned and non-learned versions of
the same system is often subtle. These factors obfuscate the de-facto risks
that the incorporation of ML carries. We analyze the root causes of
potentially-increased attack surface in learned systems and develop a framework
for identifying vulnerabilities that stem from the use of ML. We apply our
framework to a broad set of learned systems under active development. To
empirically validate the many vulnerabilities surfaced by our framework, we
choose 3 of them and implement and evaluate exploits against prominent
learned-system instances. We show that the use of ML caused leakage of past
queries in a database, enabled a poisoning attack that causes exponential
memory blowup in an index structure and crashes it in seconds, and enabled
index users to snoop on each others' key distributions by timing queries over
their own keys. We find that adversarial ML is a universal threat against
learned systems, point to open research gaps in our understanding of
learned-systems security, and conclude by discussing mitigations, while noting
that data leakage is inherent in systems whose learned component is shared
between multiple parties.


http://arxiv.org/abs/2212.10717
Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks. (22%)
Jimmy Z. Di; Jack Douglas; Jayadev Acharya; Gautam Kamath; Ayush Sekhari
  We introduce camouflaged data poisoning attacks, a new attack vector that
arises in the context of machine unlearning and other settings when model
retraining may be induced. An adversary first adds a few carefully crafted
points to the training dataset such that the impact on the model's predictions
is minimal. The adversary subsequently triggers a request to remove a subset of
the introduced points at which point the attack is unleashed and the model's
predictions are negatively affected. In particular, we consider clean-label
targeted attacks (in which the goal is to cause the model to misclassify a
specific test point) on datasets including CIFAR-10, Imagenette, and Imagewoof.
This attack is realized by constructing camouflage datapoints that mask the
effect of a poisoned dataset.


http://arxiv.org/abs/2212.10264
ReCode: Robustness Evaluation of Code Generation Models. (10%)
Shiqi Wang; Zheng Li; Haifeng Qian; Chenghao Yang; Zijian Wang; Mingyue Shang; Varun Kumar; Samson Tan; Baishakhi Ray; Parminder Bhatia; Ramesh Nallapati; Murali Krishna Ramanathan; Dan Roth; Bing Xiang
  Code generation models have achieved impressive performance. However, they
tend to be brittle as slight edits to a prompt could lead to very different
generations; these robustness properties, critical for user experience when
deployed in real-life applications, are not well understood. Most existing
works on robustness in text or code tasks have focused on classification, while
robustness in generation tasks is an uncharted area and to date there is no
comprehensive benchmark for robustness in code generation. In this paper, we
propose ReCode, a comprehensive robustness evaluation benchmark for code
generation models. We customize over 30 transformations specifically for code
on docstrings, function and variable names, code syntax, and code format. They
are carefully designed to be natural in real-life coding practice, preserve the
original semantic meaning, and thus provide multifaceted assessments of a
model's robustness performance. With human annotators, we verified that over
90% of the perturbed prompts do not alter the semantic meaning of the original
prompt. In addition, we define robustness metrics for code generation models
considering the worst-case behavior under each type of perturbation, taking
advantage of the fact that executing the generated code can serve as objective
evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well
as function completion tasks derived from them. Interesting observations
include: better robustness for CodeGen over InCoder and GPT-J; models are most
sensitive to syntax perturbations; more challenging robustness evaluation on
MBPP over HumanEval.


http://arxiv.org/abs/2212.10221
SoK: Analysis of Root Causes and Defense Strategies for Attacks on Microarchitectural Optimizations. (5%)
Nadja Ramhöj Holtryd; Madhavan Manivannan; Per Stenström
  Microarchitectural optimizations are expected to play a crucial role in
ensuring performance scalability in future technology nodes. However, recent
attacks have demonstrated that microarchitectural optimizations, which were
assumed to be secure, can be exploited. Moreover, new attacks surface at a
rapid pace limiting the scope of existing defenses. These developments prompt
the need to review microarchitectural optimizations with an emphasis on
security, understand the attack landscape and the potential defense strategies.
  We analyze timing-based side-channel attacks targeting a diverse set of
microarchitectural optimizations. We provide a framework for analysing
non-transient and transient attacks, which highlights the similarities. We
identify the four root causes of timing-based side-channel attacks:
determinism, sharing, access violation and information flow, through our
systematic analysis. Our key insight is that a subset (or all) of the root
causes are exploited by attacks and eliminating any of the exploited root
causes, in any attack step, is enough to provide protection. Leveraging our
framework, we systematize existing defenses and show that they target these
root causes in the different attack steps.


http://arxiv.org/abs/2212.10002
Defending Against Poisoning Attacks in Open-Domain Question Answering. (2%)
Orion Weller; Aleem Khan; Nathaniel Weir; Dawn Lawrie; Durme Benjamin Van
  Recent work in open-domain question answering (ODQA) has shown that
adversarial poisoning of the input contexts can cause large drops in accuracy
for production systems. However, little to no work has proposed methods to
defend against these attacks. To do so, we introduce a new method that uses
query augmentation to search for a diverse set of retrieved passages that could
answer the original question. We integrate these new passages into the model
through the design of a novel confidence method, comparing the predicted answer
to its appearance in the retrieved contexts (what we call Confidence from
Answer Redundancy, e.g. CAR). Together these methods allow for a simple but
effective way to defend against poisoning attacks and provide gains of 5-20%
exact match across varying levels of data poisoning.


http://arxiv.org/abs/2212.10534
DISCO: Distilling Phrasal Counterfactuals with Large Language Models. (1%)
Zeming Chen; Qiyue Gao; Kyle Richardson; Antoine Bosselut; Ashish Sabharwal
  Recent methods demonstrate that data augmentation using counterfactual
knowledge can teach models the causal structure of a task, leading to robust
and generalizable models. However, such counterfactual data often has a limited
scale and diversity if crowdsourced and is computationally expensive to extend
to new perturbation types if generated using supervised methods. To address
this, we introduce a new framework called DISCO for automatically generating
high-quality counterfactual data at scale. DISCO engineers prompts to generate
phrasal perturbations with a large general language model. Then, a
task-specific teacher model filters the generation to distill high-quality
counterfactual data. We show that learning with this counterfactual data yields
a comparatively small student model that is 6% (absolute) more robust and
generalizes 5% better across distributions than baselines on various
challenging evaluations. This model is also 15% more sensitive in
differentiating original and counterfactual examples, on three evaluation sets
written by human workers and via human-AI collaboration.


http://arxiv.org/abs/2212.09254
TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimization. (99%)
Bairu Hou; Jinghan Jia; Yihua Zhang; Guanhua Zhang; Yang Zhang; Sijia Liu; Shiyu Chang
  Robustness evaluation against adversarial examples has become increasingly
important to unveil the trustworthiness of the prevailing deep models in
natural language processing (NLP). However, in contrast to the computer vision
domain where the first-order projected gradient descent (PGD) is used as the
benchmark approach to generate adversarial examples for robustness evaluation,
there lacks a principled first-order gradient-based robustness evaluation
framework in NLP. The emerging optimization challenges lie in 1) the discrete
nature of textual inputs together with the strong coupling between the
perturbation location and the actual content, and 2) the additional constraint
that the perturbed text should be fluent and achieve a low perplexity under a
language model. These challenges make the development of PGD-like NLP attacks
difficult. To bridge the gap, we propose TextGrad, a new attack generator using
gradient-driven optimization, supporting high-accuracy and high-quality
assessment of adversarial robustness in NLP. Specifically, we address the
aforementioned challenges in a unified optimization framework. And we develop
an effective convex relaxation method to co-optimize the continuously-relaxed
site selection and perturbation variables and leverage an effective sampling
method to establish an accurate mapping from the continuous optimization
variables to the discrete textual perturbations. Moreover, as a first-order
attack generation method, TextGrad can be baked into adversarial training to
further improve the robustness of NLP models. Extensive experiments are
provided to demonstrate the effectiveness of TextGrad not only in attack
generation for robustness evaluation but also in adversarial defense.


http://arxiv.org/abs/2212.09994
Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation. (75%)
Xinyu Pi; Bing Wang; Yan Gao; Jiaqi Guo; Zhoujun Li; Jian-Guang Lou
  The robustness of Text-to-SQL parsers against adversarial perturbations plays
a crucial role in delivering highly reliable applications. Previous studies
along this line primarily focused on perturbations in the natural language
question side, neglecting the variability of tables. Motivated by this, we
propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm to
measure the robustness of Text-to-SQL models. Following this proposition, we
curate ADVETA, the first robustness evaluation benchmark featuring natural and
realistic ATPs. All tested state-of-the-art models experience dramatic
performance drops on ADVETA, revealing models' vulnerability in real-world
practices. To defend against ATP, we build a systematic adversarial training
example generation framework tailored for better contextualization of tabular
data. Experiments show that our approach not only brings the best robustness
improvement against table-side perturbations but also substantially empowers
models against NL-side perturbations. We release our benchmark and code at:
https://github.com/microsoft/ContextualSP.


http://arxiv.org/abs/2212.09360
AI Security for Geoscience and Remote Sensing: Challenges and Future Trends. (50%)
Yonghao Xu; Tao Bai; Weikang Yu; Shizhen Chang; Peter M. Atkinson; Pedram Ghamisi
  Recent advances in artificial intelligence (AI) have significantly
intensified research in the geoscience and remote sensing (RS) field. AI
algorithms, especially deep learning-based ones, have been developed and
applied widely to RS data analysis. The successful application of AI covers
almost all aspects of Earth observation (EO) missions, from low-level vision
tasks like super-resolution, denoising, and inpainting, to high-level vision
tasks like scene classification, object detection, and semantic segmentation.
While AI techniques enable researchers to observe and understand the Earth more
accurately, the vulnerability and uncertainty of AI models deserve further
attention, considering that many geoscience and RS tasks are highly
safety-critical. This paper reviews the current development of AI security in
the geoscience and RS field, covering the following five important aspects:
adversarial attack, backdoor attack, federated learning, uncertainty, and
explainability. Moreover, the potential opportunities and trends are discussed
to provide insights for future research. To the best of the authors' knowledge,
this paper is the first attempt to provide a systematic review of AI
security-related research in the geoscience and RS community. Available code
and datasets are also listed in the paper to move this vibrant field of
research forward.


http://arxiv.org/abs/2212.09668
Task-Oriented Communications for NextG: End-to-End Deep Learning and AI Security Aspects. (26%)
Yalin E. Sagduyu; Sennur Ulukus; Aylin Yener
  Communications systems to date are primarily designed with the goal of
reliable (error-free) transfer of digital sequences (bits). Next generation
(NextG) communication systems are beginning to explore shifting this design
paradigm of reliably decoding bits to reliably executing a given task.
Task-oriented communications system design is likely to find impactful
applications, for example, considering the relative importance of messages. In
this paper, a wireless signal classification is considered as the task to be
performed in the NextG Radio Access Network (RAN) for signal intelligence and
spectrum awareness applications such as user equipment (UE) identification and
authentication, and incumbent signal detection for spectrum co-existence. For
that purpose, edge devices collect wireless signals and communicate with the
NextG base station (gNodeB) that needs to know the signal class. Edge devices
may not have sufficient processing power and may not be trusted to perform the
signal classification task, whereas the transfer of the captured signals from
the edge devices to the gNodeB may not be efficient or even feasible subject to
stringent delay, rate, and energy restrictions. We present a task-oriented
communications approach, where all the transmitter, receiver and classifier
functionalities are jointly trained as two deep neural networks (DNNs), one for
the edge device and another for the gNodeB. We show that this approach achieves
better accuracy with smaller DNNs compared to the baselines that treat
communications and signal classification as two separate tasks. Finally, we
discuss how adversarial machine learning poses a major security threat for the
use of DNNs for task-oriented communications. We demonstrate the major
performance loss under backdoor (Trojan) attacks and adversarial (evasion)
attacks that target the training and test processes of task-oriented
communications.


http://arxiv.org/abs/2212.09979
Flareon: Stealthy any2any Backdoor Injection via Poisoned Augmentation. (2%)
Tianrui Qin; Xianghuan He; Xitong Gao; Yiren Zhao; Kejiang Ye; Cheng-Zhong Xu
  Open software supply chain attacks, once successful, can exact heavy costs in
mission-critical applications. As open-source ecosystems for deep learning
flourish and become increasingly universal, they present attackers previously
unexplored avenues to code-inject malicious backdoors in deep neural network
models. This paper proposes Flareon, a small, stealthy, seemingly harmless code
modification that specifically targets the data augmentation pipeline with
motion-based triggers. Flareon neither alters ground-truth labels, nor modifies
the training loss objective, nor does it assume prior knowledge of the victim
model architecture, training data, and training hyperparameters. Yet, it has a
surprisingly large ramification on training -- models trained under Flareon
learn powerful target-conditional (or "any2any") backdoors. The resulting
models can exhibit high attack success rates for any target choices and better
clean accuracies than backdoor attacks that not only seize greater control, but
also assume more restrictive attack capabilities. We also demonstrate the
effectiveness of Flareon against recent defenses. Flareon is fully open-source
and available online to the deep learning community:
https://github.com/lafeat/flareon.


http://arxiv.org/abs/2212.09458
Exploring Optimal Substructure for Out-of-distribution Generalization via Feature-targeted Model Pruning. (1%)
Yingchun Wang; Jingcai Guo; Song Guo; Weizhan Zhang; Jie Zhang
  Recent studies show that even highly biased dense networks contain an
unbiased substructure that can achieve better out-of-distribution (OOD)
generalization than the original model. Existing works usually search the
invariant subnetwork using modular risk minimization (MRM) with out-domain
data. Such a paradigm may bring about two potential weaknesses: 1) Unfairness,
due to the insufficient observation of out-domain data during training; and 2)
Sub-optimal OOD generalization, due to the feature-untargeted model pruning on
the whole data distribution. In this paper, we propose a novel Spurious
Feature-targeted model Pruning framework, dubbed SFP, to automatically explore
invariant substructures without referring to the above weaknesses.
Specifically, SFP identifies in-distribution (ID) features during training
using our theoretically verified task loss, upon which, SFP can perform ID
targeted-model pruning that removes branches with strong dependencies on ID
features. Notably, by attenuating the projections of spurious features into
model space, SFP can push the model learning toward invariant features and pull
that out of environmental features, devising optimal OOD generalization.
Moreover, we also conduct detailed theoretical analysis to provide the
rationality guarantee and a proof framework for OOD structures via model
sparsity, and for the first time, reveal how a highly biased data distribution
affects the model's OOD generalization. Extensive experiments on various OOD
datasets show that SFP can significantly outperform both structure-based and
non-structure OOD generalization SOTAs, with accuracy improvement up to 4.72%
and 23.35%, respectively.


http://arxiv.org/abs/2212.09155
Estimating the Adversarial Robustness of Attributions in Text with Transformers. (99%)
Adam Ivankay; Mattia Rigotti; Ivan Girardi; Chiara Marchiori; Pascal Frossard
  Explanations are crucial parts of deep neural network (DNN) classifiers. In
high stakes applications, faithful and robust explanations are important to
understand and gain trust in DNN classifiers. However, recent work has shown
that state-of-the-art attribution methods in text classifiers are susceptible
to imperceptible adversarial perturbations that alter explanations
significantly while maintaining the correct prediction outcome. If undetected,
this can critically mislead the users of DNNs. Thus, it is crucial to
understand the influence of such adversarial perturbations on the networks'
explanations and their perceptibility. In this work, we establish a novel
definition of attribution robustness (AR) in text classification, based on
Lipschitz continuity. Crucially, it reflects both attribution change induced by
adversarial input alterations and perceptibility of such alterations. Moreover,
we introduce a wide set of text similarity measures to effectively capture
locality between two text samples and imperceptibility of adversarial
perturbations in text. We then propose our novel TransformerExplanationAttack
(TEA), a strong adversary that provides a tight estimation for attribution
robustness in text classification. TEA uses state-of-the-art language models to
extract word substitutions that result in fluent, contextual adversarial
samples. Finally, with experiments on several text classification
architectures, we show that TEA consistently outperforms current
state-of-the-art AR estimators, yielding perturbations that alter explanations
to a greater extent while being more fluent and less perceptible.


http://arxiv.org/abs/2212.09035
Minimizing Maximum Model Discrepancy for Transferable Black-box Targeted Attacks. (99%)
Anqi Zhao; Tong Chu; Yahao Liu; Wen Li; Jingjing Li; Lixin Duan
  In this work, we study the black-box targeted attack problem from the model
discrepancy perspective. On the theoretical side, we present a generalization
error bound for black-box targeted attacks, which gives a rigorous theoretical
analysis for guaranteeing the success of the attack. We reveal that the attack
error on a target model mainly depends on empirical attack error on the
substitute model and the maximum model discrepancy among substitute models. On
the algorithmic side, we derive a new algorithm for black-box targeted attacks
based on our theoretical analysis, in which we additionally minimize the
maximum model discrepancy(M3D) of the substitute models when training the
generator to generate adversarial examples. In this way, our model is capable
of crafting highly transferable adversarial examples that are robust to the
model variation, thus improving the success rate for attacking the black-box
model. We conduct extensive experiments on the ImageNet dataset with different
classification models, and our proposed approach outperforms existing
state-of-the-art methods by a significant margin. Our codes will be released.


http://arxiv.org/abs/2301.06083
Discrete Point-wise Attack Is Not Enough: Generalized Manifold Adversarial Attack for Face Recognition. (99%)
Qian Li; Yuxiao Hu; Ye Liu; Dongxiao Zhang; Xin Jin; Yuntian Chen
  Classical adversarial attacks for Face Recognition (FR) models typically
generate discrete examples for target identity with a single state image.
However, such paradigm of point-wise attack exhibits poor generalization
against numerous unknown states of identity and can be easily defended. In this
paper, by rethinking the inherent relationship between the face of target
identity and its variants, we introduce a new pipeline of Generalized Manifold
Adversarial Attack (GMAA) to achieve a better attack performance by expanding
the attack range. Specifically, this expansion lies on two aspects - GMAA not
only expands the target to be attacked from one to many to encourage a good
generalization ability for the generated adversarial examples, but it also
expands the latter from discrete points to manifold by leveraging the domain
knowledge that face expression change can be continuous, which enhances the
attack effect as a data augmentation mechanism did. Moreover, we further design
a dual supervision with local and global constraints as a minor contribution to
improve the visual quality of the generated adversarial examples. We
demonstrate the effectiveness of our method based on extensive experiments, and
reveal that GMAA promises a semantic continuous adversarial space with a higher
generalization ability and visual quality


http://arxiv.org/abs/2212.09067
Fine-Tuning Is All You Need to Mitigate Backdoor Attacks. (4%)
Zeyang Sha; Xinlei He; Pascal Berrang; Mathias Humbert; Yang Zhang
  Backdoor attacks represent one of the major threats to machine learning
models. Various efforts have been made to mitigate backdoors. However, existing
defenses have become increasingly complex and often require high computational
resources or may also jeopardize models' utility. In this work, we show that
fine-tuning, one of the most common and easy-to-adopt machine learning training
operations, can effectively remove backdoors from machine learning models while
maintaining high model utility. Extensive experiments over three machine
learning paradigms show that fine-tuning and our newly proposed
super-fine-tuning achieve strong defense performance. Furthermore, we coin a
new term, namely backdoor sequela, to measure the changes in model
vulnerabilities to other attacks before and after the backdoor has been
removed. Empirical evaluation shows that, compared to other defense methods,
super-fine-tuning leaves limited backdoor sequela. We hope our results can help
machine learning model owners better protect their models from backdoor
threats. Also, it calls for the design of more advanced attacks in order to
comprehensively assess machine learning models' backdoor vulnerabilities.


http://arxiv.org/abs/2212.09000
Confidence-aware Training of Smoothed Classifiers for Certified Robustness. (86%)
Jongheon Jeong; Seojin Kim; Jinwoo Shin
  Any classifier can be "smoothed out" under Gaussian noise to build a new
classifier that is provably robust to $\ell_2$-adversarial perturbations, viz.,
by averaging its predictions over the noise via randomized smoothing. Under the
smoothed classifiers, the fundamental trade-off between accuracy and
(adversarial) robustness has been well evidenced in the literature: i.e.,
increasing the robustness of a classifier for an input can be at the expense of
decreased accuracy for some other inputs. In this paper, we propose a simple
training method leveraging this trade-off to obtain robust smoothed
classifiers, in particular, through a sample-wise control of robustness over
the training samples. We make this control feasible by using "accuracy under
Gaussian noise" as an easy-to-compute proxy of adversarial robustness for an
input. Specifically, we differentiate the training objective depending on this
proxy to filter out samples that are unlikely to benefit from the worst-case
(adversarial) objective. Our experiments show that the proposed method, despite
its simplicity, consistently exhibits improved certified robustness upon
state-of-the-art training methods. Somewhat surprisingly, we find these
improvements persist even for other notions of robustness, e.g., to various
types of common corruptions.


http://arxiv.org/abs/2212.09006
A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness. (2%)
Tiantian Feng; Rajat Hebbar; Nicholas Mehlman; Xuan Shi; Aditya Kommineni; and Shrikanth Narayanan
  Speech-centric machine learning systems have revolutionized many leading
domains ranging from transportation and healthcare to education and defense,
profoundly changing how people live, work, and interact with each other.
However, recent studies have demonstrated that many speech-centric ML systems
may need to be considered more trustworthy for broader deployment.
Specifically, concerns over privacy breaches, discriminating performance, and
vulnerability to adversarial attacks have all been discovered in ML research
fields. In order to address the above challenges and risks, a significant
number of efforts have been made to ensure these ML systems are trustworthy,
especially private, safe, and fair. In this paper, we conduct the first
comprehensive survey on speech-centric trustworthy ML topics related to
privacy, safety, and fairness. In addition to serving as a summary report for
the research community, we point out several promising future research
directions to inspire the researchers who wish to explore further in this area.


http://arxiv.org/abs/2212.08853
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation. (1%)
Hongyi Yuan; Zheng Yuan; Chuanqi Tan; Fei Huang; Songfang Huang
  Language models with the Transformers structure have shown great performance
in natural language processing. However, there still poses problems when
fine-tuning pre-trained language models on downstream tasks, such as
over-fitting or representation collapse. In this work, we propose HyPe, a
simple yet effective fine-tuning technique to alleviate such problems by
perturbing hidden representations of Transformers layers. Unlike previous works
that only add noise to inputs or parameters, we argue that the hidden
representations of Transformers layers convey more diverse and meaningful
language information. Therefore, making the Transformers layers more robust to
hidden representation perturbations can further benefit the fine-tuning of PLMs
en bloc. We conduct extensive experiments and analyses on GLUE and other
natural language inference datasets. Results demonstrate that HyPe outperforms
vanilla fine-tuning and enhances generalization of hidden representations from
different layers. In addition, HyPe acquires negligible computational
overheads, and is better than and compatible with previous state-of-the-art
fine-tuning techniques.


http://arxiv.org/abs/2212.08341
Adversarial Example Defense via Perturbation Grading Strategy. (99%)
Shaowei Zhu; Wanli Lyu; Bin Li; Zhaoxia Yin; Bin Luo
  Deep Neural Networks have been widely used in many fields. However, studies
have shown that DNNs are easily attacked by adversarial examples, which have
tiny perturbations and greatly mislead the correct judgment of DNNs.
Furthermore, even if malicious attackers cannot obtain all the underlying model
parameters, they can use adversarial examples to attack various DNN-based task
systems. Researchers have proposed various defense methods to protect DNNs,
such as reducing the aggressiveness of adversarial examples by preprocessing or
improving the robustness of the model by adding modules. However, some defense
methods are only effective for small-scale examples or small perturbations but
have limited defense effects for adversarial examples with large perturbations.
This paper assigns different defense strategies to adversarial perturbations of
different strengths by grading the perturbations on the input examples.
Experimental results show that the proposed method effectively improves defense
performance. In addition, the proposed method does not modify any task model,
which can be used as a preprocessing module, which significantly reduces the
deployment cost in practical applications.


http://arxiv.org/abs/2212.08568
Biomedical image analysis competitions: The state of current participation practice. (4%)
Matthias Eisenmann; Annika Reinke; Vivienn Weru; Minu Dietlinde Tizabi; Fabian Isensee; Tim J. Adler; Patrick Godau; Veronika Cheplygina; Michal Kozubek; Sharib Ali; Anubha Gupta; Jan Kybic; Alison Noble; Solórzano Carlos Ortiz de; Samiksha Pachade; Caroline Petitjean; Daniel Sage; Donglai Wei; Elizabeth Wilden; Deepak Alapatt; Vincent Andrearczyk; Ujjwal Baid; Spyridon Bakas; Niranjan Balu; Sophia Bano; Vivek Singh Bawa; Jorge Bernal; Sebastian Bodenstedt; Alessandro Casella; Jinwook Choi; Olivier Commowick; Marie Daum; Adrien Depeursinge; Reuben Dorent; Jan Egger; Hannah Eichhorn; Sandy Engelhardt; Melanie Ganz; Gabriel Girard; Lasse Hansen; Mattias Heinrich; Nicholas Heller; Alessa Hering; Arnaud Huaulmé; Hyunjeong Kim; Bennett Landman; Hongwei Bran Li; Jianning Li; Jun Ma; Anne Martel; Carlos Martín-Isla; Bjoern Menze; Chinedu Innocent Nwoye; Valentin Oreiller; Nicolas Padoy; Sarthak Pati; Kelly Payette; Carole Sudre; Wijnen Kimberlin van; Armine Vardazaryan; Tom Vercauteren; Martin Wagner; Chuanbo Wang; Moi Hoon Yap; Zeyun Yu; Chun Yuan; Maximilian Zenk; Aneeq Zia; David Zimmerer; Rina Bao; Chanyeol Choi; Andrew Cohen; Oleh Dzyubachyk; Adrian Galdran; Tianyuan Gan; Tianqi Guo; Pradyumna Gupta; Mahmood Haithami; Edward Ho; Ikbeom Jang; Zhili Li; Zhengbo Luo; Filip Lux; Sokratis Makrogiannis; Dominik Müller; Young-tack Oh; Subeen Pang; Constantin Pape; Gorkem Polat; Charlotte Rosalie Reed; Kanghyun Ryu; Tim Scherr; Vajira Thambawita; Haoyu Wang; Xinliang Wang; Kele Xu; Hung Yeh; Doyeob Yeo; Yixuan Yuan; Yan Zeng; Xin Zhao; Julian Abbing; Jannes Adam; Nagesh Adluru; Niklas Agethen; Salman Ahmed; Yasmina Al Khalil; Mireia Alenyà; Esa Alhoniemi; Chengyang An; Talha Anwar; Tewodros Weldebirhan Arega; Netanell Avisdris; Dogu Baran Aydogan; Yingbin Bai; Maria Baldeon Calisto; Berke Doga Basaran; Marcel Beetz; Cheng Bian; Hao Bian; Kevin Blansit; Louise Bloch; Robert Bohnsack; Sara Bosticardo; Jack Breen; Mikael Brudfors; Raphael Brüngel; Mariano Cabezas; Alberto Cacciola; Zhiwei Chen; Yucong Chen; Daniel Tianming Chen; Minjeong Cho; Min-Kook Choi; Chuantao Xie Chuantao Xie; Dana Cobzas; Julien Cohen-Adad; Jorge Corral Acero; Sujit Kumar Das; Oliveira Marcela de; Hanqiu Deng; Guiming Dong; Lars Doorenbos; Cory Efird; Di Fan; Mehdi Fatan Serj; Alexandre Fenneteau; Lucas Fidon; Patryk Filipiak; René Finzel; Nuno R. Freitas; Christoph M. Friedrich; Mitchell Fulton; Finn Gaida; Francesco Galati; Christoforos Galazis; Chang Hee Gan; Zheyao Gao; Shengbo Gao; Matej Gazda; Beerend Gerats; Neil Getty; Adam Gibicar; Ryan Gifford; Sajan Gohil; Maria Grammatikopoulou; Daniel Grzech; Orhun Güley; Timo Günnemann; Chunxu Guo; Sylvain Guy; Heonjin Ha; Luyi Han; Il Song Han; Ali Hatamizadeh; Tian He; Jimin Heo; Sebastian Hitziger; SeulGi Hong; SeungBum Hong; Rian Huang; Ziyan Huang; Markus Huellebrand; Stephan Huschauer; Mustaffa Hussain; Tomoo Inubushi; Ece Isik Polat; Mojtaba Jafaritadi; SeongHun Jeong; Bailiang Jian; Yuanhong Jiang; Zhifan Jiang; Yueming Jin; Smriti Joshi; Abdolrahim Kadkhodamohammadi; Reda Abdellah Kamraoui; Inha Kang; Junghwa Kang; Davood Karimi; April Khademi; Muhammad Irfan Khan; Suleiman A. Khan; Rishab Khantwal; Kwang-Ju Kim; Timothy Kline; Satoshi Kondo; Elina Kontio; Adrian Krenzer; Artem Kroviakov; Hugo Kuijf; Satyadwyoom Kumar; Rosa Francesco La; Abhi Lad; Doohee Lee; Minho Lee; Chiara Lena; Hao Li; Ling Li; Xingyu Li; Fuyuan Liao; KuanLun Liao; Arlindo Limede Oliveira; Chaonan Lin; Shan Lin; Akis Linardos; Marius George Linguraru; Han Liu; Tao Liu; Di Liu; Yanling Liu; João Lourenço-Silva; Jingpei Lu; Jiangshan Lu; Imanol Luengo; Christina B. Lund; Huan Minh Luu; Yi Lv; Yi Lv; Uzay Macar; Leon Maechler; Sina Mansour L.; Kenji Marshall; Moona Mazher; Richard McKinley; Alfonso Medela; Felix Meissen; Mingyuan Meng; Dylan Miller; Seyed Hossein Mirjahanmardi; Arnab Mishra; Samir Mitha; Hassan Mohy-ud-Din; Tony Chi Wing Mok; Gowtham Krishnan Murugesan; Enamundram Naga Karthik; Sahil Nalawade; Jakub Nalepa; Mohamed Naser; Ramin Nateghi; Hammad Naveed; Quang-Minh Nguyen; Cuong Nguyen Quoc; Brennan Nichyporuk; Bruno Oliveira; David Owen; Jimut Bahan Pal; Junwen Pan; Wentao Pan; Winnie Pang; Bogyu Park; Vivek Pawar; Kamlesh Pawar; Michael Peven; Lena Philipp; Tomasz Pieciak; Szymon Plotka; Marcel Plutat; Fattaneh Pourakpour; Domen Preložnik; Kumaradevan Punithakumar; Abdul Qayyum; Sandro Queirós; Arman Rahmim; Salar Razavi; Jintao Ren; Mina Rezaei; Jonathan Adam Rico; ZunHyan Rieu; Markus Rink; Johannes Roth; Yusely Ruiz-Gonzalez; Numan Saeed; Anindo Saha; Mostafa Salem; Ricardo Sanchez-Matilla; Kurt Schilling; Wei Shao; Zhiqiang Shen; Ruize Shi; Pengcheng Shi; Daniel Sobotka; Théodore Soulier; Bella Specktor Fadida; Danail Stoyanov; Timothy Sum Hon Mun; Xiaowu Sun; Rong Tao; Franz Thaler; Antoine Théberge; Felix Thielke; Helena Torres; Kareem A. Wahid; Jiacheng Wang; YiFei Wang; Wei Wang; Xiong Wang; Jianhui Wen; Ning Wen; Marek Wodzinski; Ye Wu; Fangfang Xia; Tianqi Xiang; Chen Xiaofei; Lizhan Xu; Tingting Xue; Yuxuan Yang; Lin Yang; Kai Yao; Huifeng Yao; Amirsaeed Yazdani; Michael Yip; Hwanseung Yoo; Fereshteh Yousefirizi; Shunkai Yu; Lei Yu; Jonathan Zamora; Ramy Ashraf Zeineldin; Dewen Zeng; Jianpeng Zhang; Bokai Zhang; Jiapeng Zhang; Fan Zhang; Huahong Zhang; Zhongchen Zhao; Zixuan Zhao; Jiachen Zhao; Can Zhao; Qingshuo Zheng; Yuheng Zhi; Ziqi Zhou; Baosheng Zou; Klaus Maier-Hein; Paul F. Jäger; Annette Kopp-Schneider; Lena Maier-Hein
  The number of international benchmarking competitions is steadily increasing
in various fields of machine learning (ML) research and practice. So far,
however, little is known about the common practice as well as bottlenecks faced
by the community in tackling the research questions posed. To shed light on the
status quo of algorithm development in the specific field of biomedical imaging
analysis, we designed an international survey that was issued to all
participants of challenges conducted in conjunction with the IEEE ISBI 2021 and
MICCAI 2021 conferences (80 competitions in total). The survey covered
participants' expertise and working environments, their chosen strategies, as
well as algorithm characteristics. A median of 72% challenge participants took
part in the survey. According to our results, knowledge exchange was the
primary incentive (70%) for participation, while the reception of prize money
played only a minor role (16%). While a median of 80 working hours was spent on
method development, a large portion of participants stated that they did not
have enough time for method development (32%). 25% perceived the infrastructure
to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of
these, 84% were based on standard architectures. 43% of the respondents
reported that the data samples (e.g., images) were too large to be processed at
once. This was most commonly addressed by patch-based training (69%),
downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks.
K-fold cross-validation on the training set was performed by only 37% of the
participants and only 50% of the participants performed ensembling based on
multiple identical models (61%) or heterogeneous models (39%). 48% of the
respondents applied postprocessing steps.


http://arxiv.org/abs/2212.08649
Better May Not Be Fairer: Can Data Augmentation Mitigate Subgroup Degradation? (1%)
Ming-Chang Chiu; Pin-Yu Chen; Xuezhe Ma
  It is no secret that deep learning models exhibit undesirable behaviors such
as learning spurious correlations instead of learning correct relationships
between input/output pairs. Prior works on robustness study datasets that mix
low-level features to quantify how spurious correlations affect predictions
instead of considering natural semantic factors due to limitations in accessing
realistic datasets for comprehensive evaluation. To bridge this gap, in this
paper we first investigate how natural background colors play a role as
spurious features in image classification tasks by manually splitting the test
sets of CIFAR10 and CIFAR100 into subgroups based on the background color of
each image. We name our datasets CIFAR10-B and CIFAR100-B. We find that while
standard CNNs achieve human-level accuracy, the subgroup performances are not
consistent, and the phenomenon remains even after data augmentation (DA). To
alleviate this issue, we propose FlowAug, a semantic DA method that leverages
the decoupled semantic representations captured by a pre-trained generative
flow. Experimental results show that FlowAug achieves more consistent results
across subgroups than other types of DA methods on CIFAR10 and CIFAR100.
Additionally, it shows better generalization performance. Furthermore, we
propose a generic metric for studying model robustness to spurious
correlations, where we take a macro average on the weighted standard deviations
across different classes. Per our metric, FlowAug demonstrates less reliance on
spurious correlations. Although this metric is proposed to study our curated
datasets, it applies to all datasets that have subgroups or subclasses. Lastly,
aside from less dependence on spurious correlations and better generalization
on in-distribution test sets, we also show superior out-of-distribution results
on CIFAR10.1 and competitive performances on CIFAR10-C and CIFAR100-C.


http://arxiv.org/abs/2212.08650
On Human Visual Contrast Sensitivity and Machine Vision Robustness: A Comparative Study. (1%)
Ming-Chang Chiu; Yingfei Wang; Derrick Eui Gyu Kim; Pin-Yu Chen; Xuezhe Ma
  It is well established in neuroscience that color vision plays an essential
part in the human visual perception system. Meanwhile, many novel designs for
computer vision inspired by human vision have achieved success in a wide range
of tasks and applications. Nonetheless, how color differences affect machine
vision has not been well explored. Our work tries to bridge this gap between
the human color vision aspect of visual recognition and that of the machine. To
achieve this, we curate two datasets: CIFAR10-F and CIFAR100-F, which are based
on the foreground colors of the popular CIFAR datasets. Together with CIFAR10-B
and CIFAR100-B, the existing counterpart datasets with information on the
background colors of CIFAR test sets, we assign each image based on its color
contrast level per its foreground and background color labels and use this as a
proxy to study how color contrast affects machine vision. We first conduct a
proof-of-concept study, showing the effect of color difference and validate our
datasets. Furthermore, on a broader level, an important characteristic of human
vision is its robustness against ambient changes; therefore, drawing
inspirations from ophthalmology and the robustness literature, we analogize
contrast sensitivity from the human visual aspect to machine vision and
complement the current robustness study using corrupted images with our
CIFAR-CoCo datasets. In summary, motivated by neuroscience and equipped with
the datasets we curate, we devise a new framework in two dimensions to perform
extensive analyses on the effect of color contrast and corrupted images: (1)
model architecture, (2) model size, to measure the perception ability of
machine vision beyond total accuracy. We also explore how task complexity and
data augmentation play a role in this setup. Our results call attention to new
evaluation approaches for human-like machine perception.


http://arxiv.org/abs/2212.07992
Alternating Objectives Generates Stronger PGD-Based Adversarial Attacks. (98%)
Nikolaos Antoniou; Efthymios Georgiou; Alexandros Potamianos
  Designing powerful adversarial attacks is of paramount importance for the
evaluation of $\ell_p$-bounded adversarial defenses. Projected Gradient Descent
(PGD) is one of the most effective and conceptually simple algorithms to
generate such adversaries. The search space of PGD is dictated by the steepest
ascent directions of an objective. Despite the plethora of objective function
choices, there is no universally superior option and robustness overestimation
may arise from ill-suited objective selection. Driven by this observation, we
postulate that the combination of different objectives through a simple loss
alternating scheme renders PGD more robust towards design choices. We
experimentally verify this assertion on a synthetic-data example and by
evaluating our proposed method across 25 different $\ell_{\infty}$-robust
models and 3 datasets. The performance improvement is consistent, when compared
to the single loss counterparts. In the CIFAR-10 dataset, our strongest
adversarial attack outperforms all of the white-box components of AutoAttack
(AA) ensemble, as well as the most powerful attacks existing on the literature,
achieving state-of-the-art results in the computational budget of our study
($T=100$, no restarts).


http://arxiv.org/abs/2212.08130
On Evaluating Adversarial Robustness of Chest X-ray Classification: Pitfalls and Best Practices. (84%)
Salah Ghamizi; Maxime Cordy; Michail Papadakis; Yves Le Traon
  Vulnerability to adversarial attacks is a well-known weakness of Deep Neural
Networks. While most of the studies focus on natural images with standardized
benchmarks like ImageNet and CIFAR, little research has considered real world
applications, in particular in the medical domain. Our research shows that,
contrary to previous claims, robustness of chest x-ray classification is much
harder to evaluate and leads to very different assessments based on the
dataset, the architecture and robustness metric. We argue that previous studies
did not take into account the peculiarity of medical diagnosis, like the
co-occurrence of diseases, the disagreement of labellers (domain experts), the
threat model of the attacks and the risk implications for each successful
attack.
  In this paper, we discuss the methodological foundations, review the pitfalls
and best practices, and suggest new methodological considerations for
evaluating the robustness of chest xray classification models. Our evaluation
on 3 datasets, 7 models, and 18 diseases is the largest evaluation of
robustness of chest x-ray classification models.


http://arxiv.org/abs/2212.08044
Are Multimodal Models Robust to Image and Text Perturbations? (5%)
Jielin Qiu; Yi Zhu; Xingjian Shi; Florian Wenzel; Zhiqiang Tang; Ding Zhao; Bo Li; Mu Li
  Multimodal image-text models have shown remarkable performance in the past
few years. However, evaluating their robustness against distribution shifts is
crucial before adopting them in real-world applications. In this paper, we
investigate the robustness of 9 popular open-sourced image-text models under
common perturbations on five tasks (image-text retrieval, visual reasoning,
visual entailment, image captioning, and text-to-image generation). In
particular, we propose several new multimodal robustness benchmarks by applying
17 image perturbation and 16 text perturbation techniques on top of existing
datasets. We observe that multimodal models are not robust to image and text
perturbations, especially to image perturbations. Among the tested perturbation
methods, character-level perturbations constitute the most severe distribution
shift for text, and zoom blur is the most severe shift for image data. We also
introduce two new robustness metrics (MMI and MOR) for proper evaluations of
multimodal models. We hope our extensive study sheds light on new directions
for the development of robust multimodal models.


http://arxiv.org/abs/2212.10628
Holistic risk assessment of inference attacks in machine learning. (4%)
Yang Yang
  As machine learning expanding application, there are more and more
unignorable privacy and safety issues. Especially inference attacks against
Machine Learning models allow adversaries to infer sensitive information about
the target model, such as training data, model parameters, etc. Inference
attacks can lead to serious consequences, including violating individuals
privacy, compromising the intellectual property of the owner of the machine
learning model. As far as concerned, researchers have studied and analyzed in
depth several types of inference attacks, albeit in isolation, but there is
still a lack of a holistic rick assessment of inference attacks against machine
learning models, such as their application in different scenarios, the common
factors affecting the performance of these attacks and the relationship among
the attacks. As a result, this paper performs a holistic risk assessment of
different inference attacks against Machine Learning models. This paper focuses
on three kinds of representative attacks: membership inference attack,
attribute inference attack and model stealing attack. And a threat model
taxonomy is established. A total of 12 target models using three model
architectures, including AlexNet, ResNet18 and Simple CNN, are trained on four
datasets, namely CelebA, UTKFace, STL10 and FMNIST.


http://arxiv.org/abs/2212.12307
Defending against cybersecurity threats to the payments and banking system. (2%)
Williams Haruna; Toyin Ajiboro Aremu; Yetunde Ajao Modupe
  Cyber security threats to the payment and banking system have become a
worldwide menace. The phenomenon has forced financial institutions to take
risks as part of their business model. Hence, deliberate investment in
sophisticated technologies and security measures has become imperative to
safeguard against heavy financial losses and information breaches that may
occur due to cyber-attacks. The proliferation of cyber crimes is a huge concern
for various stakeholders in the banking sector. Usually, cyber-attacks are
carried out via software systems running on a computing system in cyberspace.
As such, to prevent risks of cyber-attacks on software systems, entities
operating within cyberspace must be identified and the threats to the
application security isolated after analyzing the vulnerabilities and
developing defense mechanisms. This paper will examine various approaches that
identify assets in cyberspace, classify the cyber threats, provide security
defenses and map security measures to control types and functionalities. Thus,
adopting the right application to the security threats and defenses will aid IT
professionals and users alike in making decisions for developing a strong
defense-in-depth mechanism.


http://arxiv.org/abs/2301.03595
White-box Inference Attacks against Centralized Machine Learning and Federated Learning. (1%)
Jingyi Ge
  With the development of information science and technology, various
industries have generated massive amounts of data, and machine learning is
widely used in the analysis of big data. However, if the privacy of machine
learning applications' customers cannot be guaranteed, it will cause security
threats and losses to users' personal privacy information and service
providers. Therefore, the issue of privacy protection of machine learning has
received wide attention. For centralized machine learning models, we evaluate
the impact of different neural network layers, gradient, gradient norm, and
fine-tuned models on member inference attack performance with prior knowledge;
For the federated learning model, we discuss the location of the attacker in
the target model and its attack mode. The results show that the centralized
machine learning model shows more serious member information leakage in all
aspects, and the accuracy of the attacker in the central parameter server is
significantly higher than the local Inference attacks as participants.


http://arxiv.org/abs/2212.07495
SAIF: Sparse Adversarial and Interpretable Attack Framework. (98%)
Tooba Imtiaz; Morgan Kohler; Jared Miller; Zifeng Wang; Mario Sznaier; Octavia Camps; Jennifer Dy
  Adversarial attacks hamper the decision-making ability of neural networks by
perturbing the input signal. The addition of calculated small distortion to
images, for instance, can deceive a well-trained image classification network.
In this work, we propose a novel attack technique called Sparse Adversarial and
Interpretable Attack Framework (SAIF). Specifically, we design imperceptible
attacks that contain low-magnitude perturbations at a small number of pixels
and leverage these sparse attacks to reveal the vulnerability of classifiers.
We use the Frank-Wolfe (conditional gradient) algorithm to simultaneously
optimize the attack perturbations for bounded magnitude and sparsity with
$O(1/\sqrt{T})$ convergence. Empirical results show that SAIF computes highly
imperceptible and interpretable adversarial examples, and outperforms
state-of-the-art sparse attack methods on the ImageNet dataset.


http://arxiv.org/abs/2212.07591
Dissecting Distribution Inference. (88%)
Anshuman Suri; Yifu Lu; Yanjin Chen; David Evans
  A distribution inference attack aims to infer statistical properties of data
used to train machine learning models. These attacks are sometimes surprisingly
potent, but the factors that impact distribution inference risk are not well
understood and demonstrated attacks often rely on strong and unrealistic
assumptions such as full knowledge of training environments even in supposedly
black-box threat scenarios. To improve understanding of distribution inference
risks, we develop a new black-box attack that even outperforms the best known
white-box attack in most settings. Using this new attack, we evaluate
distribution inference risk while relaxing a variety of assumptions about the
adversary's knowledge under black-box access, like known model architectures
and label-only access. Finally, we evaluate the effectiveness of previously
proposed defenses and introduce new defenses. We find that although noise-based
defenses appear to be ineffective, a simple re-sampling defense can be highly
effective. Code is available at
https://github.com/iamgroot42/dissecting_distribution_inference


http://arxiv.org/abs/2212.07283
Generative Robust Classification. (11%)
Xuwang Yin
  Training adversarially robust discriminative (i.e., softmax) classifier has
been the dominant approach to robust classification. Building on recent work on
adversarial training (AT)-based generative models, we investigate using AT to
learn unnormalized class-conditional density models and then performing
generative robust classification. Our result shows that, under the condition of
similar model capacities, the generative robust classifier achieves comparable
performance to a baseline softmax robust classifier when the test data is clean
or when the test perturbation is of limited size, and much better performance
when the test perturbation size exceeds the training perturbation size. The
generative classifier is also able to generate samples or counterfactuals that
more closely resemble the training data, suggesting that the generative
classifier can better capture the class-conditional distributions. In contrast
to standard discriminative adversarial training where advanced data
augmentation techniques are only effective when combined with weight averaging,
we find it straightforward to apply advanced data augmentation to achieve
better robustness in our approach. Our result suggests that the generative
classifier is a competitive alternative to robust classification, especially
for problems with limited number of classes.


http://arxiv.org/abs/2212.14109
Synthesis of Adversarial DDOS Attacks Using Tabular Generative Adversarial Networks. (8%)
Abdelmageed Ahmed Hassan; Mohamed Sayed Hussein; Ahmed Shehata AboMoustafa; Sarah Hossam Elmowafy
  Network Intrusion Detection Systems (NIDS) are tools or software that are
widely used to maintain the computer networks and information systems keeping
them secure and preventing malicious traffics from penetrating into them, as
they flag when somebody is trying to break into the system. Best effort has
been set up on these systems, and the results achieved so far are quite
satisfying, however, new types of attacks stand out as the technology of
attacks keep evolving, one of these attacks are the attacks based on Generative
Adversarial Networks (GAN) that can evade machine learning IDS leaving them
vulnerable. This project investigates the impact of the Adversarial Attacks
synthesized using real DDoS attacks generated using GANs on the IDS. The
objective is to discover how will these systems react towards synthesized
attacks. marking the vulnerability and weakness points of these systems so we
could fix them.


http://arxiv.org/abs/2212.07558
DOC-NAD: A Hybrid Deep One-class Classifier for Network Anomaly Detection. (1%)
Mohanad Sarhan; Gayan Kulatilleke; Wai Weng Lo; Siamak Layeghy; Marius Portmann
  Machine Learning (ML) approaches have been used to enhance the detection
capabilities of Network Intrusion Detection Systems (NIDSs). Recent work has
achieved near-perfect performance by following binary- and multi-class network
anomaly detection tasks. Such systems depend on the availability of both
(benign and malicious) network data classes during the training phase. However,
attack data samples are often challenging to collect in most organisations due
to security controls preventing the penetration of known malicious traffic to
their networks. Therefore, this paper proposes a Deep One-Class (DOC)
classifier for network intrusion detection by only training on benign network
data samples. The novel one-class classification architecture consists of a
histogram-based deep feed-forward classifier to extract useful network data
features and use efficient outlier detection. The DOC classifier has been
extensively evaluated using two benchmark NIDS datasets. The results
demonstrate its superiority over current state-of-the-art one-class classifiers
in terms of detection and false positive rates.


http://arxiv.org/abs/2212.06776
Unfolding Local Growth Rate Estimates for (Almost) Perfect Adversarial Detection. (99%)
Peter Lorenz; Margret Keuper; Janis Keuper
  Convolutional neural networks (CNN) define the state-of-the-art solution on
many perceptual tasks. However, current CNN approaches largely remain
vulnerable against adversarial perturbations of the input that have been
crafted specifically to fool the system while being quasi-imperceptible to the
human eye. In recent years, various approaches have been proposed to defend
CNNs against such attacks, for example by model hardening or by adding explicit
defence mechanisms. Thereby, a small "detector" is included in the network and
trained on the binary classification task of distinguishing genuine data from
data containing adversarial perturbations. In this work, we propose a simple
and light-weight detector, which leverages recent findings on the relation
between networks' local intrinsic dimensionality (LID) and adversarial attacks.
Based on a re-interpretation of the LID measure and several simple adaptations,
we surpass the state-of-the-art on adversarial detection by a significant
margin and reach almost perfect results in terms of F1-score for several
networks and datasets. Sources available at:
https://github.com/adverML/multiLID


http://arxiv.org/abs/2212.06431
Object-fabrication Targeted Attack for Object Detection. (99%)
Xuchong Zhang; Changfeng Sun; Haoliang Han; Hang Wang; Hongbin Sun; Nanning Zheng
  Recent researches show that the deep learning based object detection is
vulnerable to adversarial examples. Generally, the adversarial attack for
object detection contains targeted attack and untargeted attack. According to
our detailed investigations, the research on the former is relatively fewer
than the latter and all the existing methods for the targeted attack follow the
same mode, i.e., the object-mislabeling mode that misleads detectors to
mislabel the detected object as a specific wrong label. However, this mode has
limited attack success rate, universal and generalization performances. In this
paper, we propose a new object-fabrication targeted attack mode which can
mislead detectors to `fabricate' extra false objects with specific target
labels. Furthermore, we design a dual attention based targeted feature space
attack method to implement the proposed targeted attack mode. The attack
performances of the proposed mode and method are evaluated on MS COCO and
BDD100K datasets using FasterRCNN and YOLOv5. Evaluation results demonstrate
that, the proposed object-fabrication targeted attack mode and the
corresponding targeted feature space attack method show significant
improvements in terms of image-specific attack, universal performance and
generalization capability, compared with the previous targeted attack for
object detection. Code will be made available.


http://arxiv.org/abs/2212.06822
Adversarial Attacks and Defences for Skin Cancer Classification. (99%)
Vinay Jogani; Joy Purohit; Ishaan Shivhare; Samina Attari; Shraddha Surtkar
  There has been a concurrent significant improvement in the medical images
used to facilitate diagnosis and the performance of machine learning techniques
to perform tasks such as classification, detection, and segmentation in recent
years. As a result, a rapid increase in the usage of such systems can be
observed in the healthcare industry, for instance in the form of medical image
classification systems, where these models have achieved diagnostic parity with
human physicians. One such application where this can be observed is in
computer vision tasks such as the classification of skin lesions in
dermatoscopic images. However, as stakeholders in the healthcare industry, such
as insurance companies, continue to invest extensively in machine learning
infrastructure, it becomes increasingly important to understand the
vulnerabilities in such systems. Due to the highly critical nature of the tasks
being carried out by these machine learning models, it is necessary to analyze
techniques that could be used to take advantage of these vulnerabilities and
methods to defend against them. This paper explores common adversarial attack
techniques. The Fast Sign Gradient Method and Projected Descent Gradient are
used against a Convolutional Neural Network trained to classify dermatoscopic
images of skin lesions. Following that, it also discusses one of the most
popular adversarial defense techniques, adversarial training. The performance
of the model that has been trained on adversarial examples is then tested
against the previously mentioned attacks, and recommendations to improve neural
networks robustness are thus provided based on the results of the experiment.


http://arxiv.org/abs/2212.06836
Towards Efficient and Domain-Agnostic Evasion Attack with High-dimensional Categorical Inputs. (80%)
Hongyan Bao; Yufei Han; Yujun Zhou; Xin Gao; Xiangliang Zhang
  Our work targets at searching feasible adversarial perturbation to attack a
classifier with high-dimensional categorical inputs in a domain-agnostic
setting. This is intrinsically an NP-hard knapsack problem where the
exploration space becomes explosively larger as the feature dimension
increases. Without the help of domain knowledge, solving this problem via
heuristic method, such as Branch-and-Bound, suffers from exponential
complexity, yet can bring arbitrarily bad attack results. We address the
challenge via the lens of multi-armed bandit based combinatorial search. Our
proposed method, namely FEAT, treats modifying each categorical feature as
pulling an arm in multi-armed bandit programming. Our objective is to achieve
highly efficient and effective attack using an Orthogonal Matching Pursuit
(OMP)-enhanced Upper Confidence Bound (UCB) exploration strategy. Our
theoretical analysis bounding the regret gap of FEAT guarantees its practical
attack performance. In empirical analysis, we compare FEAT with other
state-of-the-art domain-agnostic attack methods over various real-world
categorical data sets of different applications. Substantial experimental
observations confirm the expected efficiency and attack effectiveness of FEAT
applied in different application scenarios. Our work further hints the
applicability of FEAT for assessing the adversarial vulnerability of
classification systems with high-dimensional categorical inputs.


http://arxiv.org/abs/2212.07016
Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. (73%)
Chengzhi Mao; Scott Geng; Junfeng Yang; Xin Wang; Carl Vondrick
  Pretrained large-scale vision-language models like CLIP have exhibited strong
generalization over unseen tasks. Yet imperceptible adversarial perturbations
can significantly reduce CLIP's performance on new tasks. In this work, we
identify and explore the problem of \emph{adapting large-scale models for
zero-shot adversarial robustness}. We first identify two key factors during
model adaption -- training losses and adaptation methods -- that affect the
model's zero-shot adversarial robustness. We then propose a text-guided
contrastive adversarial training loss, which aligns the text embeddings and the
adversarial visual features with contrastive learning on a small set of
training data. We apply this training loss to two adaption methods, model
finetuning and visual prompt tuning. We find that visual prompt tuning is more
effective in the absence of texts, while finetuning wins in the existence of
text guidance. Overall, our approach significantly improves the zero-shot
adversarial robustness over CLIP, seeing an average improvement of over 31
points over ImageNet and 15 zero-shot datasets. We hope this work can shed
light on understanding the zero-shot adversarial robustness of large-scale
models.


http://arxiv.org/abs/2212.06493
Pixel is All You Need: Adversarial Trajectory-Ensemble Active Learning for Salient Object Detection. (56%)
Zhenyu Wu; Lin Wang; Wei Wang; Qing Xia; Chenglizhao Chen; Aimin Hao; Shuo Li
  Although weakly-supervised techniques can reduce the labeling effort, it is
unclear whether a saliency model trained with weakly-supervised data (e.g.,
point annotation) can achieve the equivalent performance of its
fully-supervised version. This paper attempts to answer this unexplored
question by proving a hypothesis: there is a point-labeled dataset where
saliency models trained on it can achieve equivalent performance when trained
on the densely annotated dataset. To prove this conjecture, we proposed a novel
yet effective adversarial trajectory-ensemble active learning (ATAL). Our
contributions are three-fold: 1) Our proposed adversarial attack triggering
uncertainty can conquer the overconfidence of existing active learning methods
and accurately locate these uncertain pixels. {2)} Our proposed
trajectory-ensemble uncertainty estimation method maintains the advantages of
the ensemble networks while significantly reducing the computational cost. {3)}
Our proposed relationship-aware diversity sampling algorithm can conquer
oversampling while boosting performance. Experimental results show that our
ATAL can find such a point-labeled dataset, where a saliency model trained on
it obtained $97\%$ -- $99\%$ performance of its fully-supervised version with
only ten annotated points per image.


http://arxiv.org/abs/2212.13989
AdvCat: Domain-Agnostic Robustness Assessment for Cybersecurity-Critical Applications with Categorical Inputs. (56%)
Helene Orsini; Hongyan Bao; Yujun Zhou; Xiangrui Xu; Yufei Han; Longyang Yi; Wei Wang; Xin Gao; Xiangliang Zhang
  Machine Learning-as-a-Service systems (MLaaS) have been largely developed for
cybersecurity-critical applications, such as detecting network intrusions and
fake news campaigns. Despite effectiveness, their robustness against
adversarial attacks is one of the key trust concerns for MLaaS deployment. We
are thus motivated to assess the adversarial robustness of the Machine Learning
models residing at the core of these security-critical applications with
categorical inputs. Previous research efforts on accessing model robustness
against manipulation of categorical inputs are specific to use cases and
heavily depend on domain knowledge, or require white-box access to the target
ML model. Such limitations prevent the robustness assessment from being as a
domain-agnostic service provided to various real-world applications. We propose
a provably optimal yet computationally highly efficient adversarial robustness
assessment protocol for a wide band of ML-driven cybersecurity-critical
applications. We demonstrate the use of the domain-agnostic robustness
assessment method with substantial experimental study on fake news detection
and intrusion detection problems.


http://arxiv.org/abs/2212.06428
Privacy-preserving Security Inference Towards Cloud-Edge Collaborative Using Differential Privacy. (1%)
Yulong Wang; Xingshu Chen; Qixu Wang
  Cloud-edge collaborative inference approach splits deep neural networks
(DNNs) into two parts that run collaboratively on resource-constrained edge
devices and cloud servers, aiming at minimizing inference latency and
protecting data privacy. However, even if the raw input data from edge devices
is not directly exposed to the cloud, state-of-the-art attacks targeting
collaborative inference are still able to reconstruct the raw private data from
the intermediate outputs of the exposed local models, introducing serious
privacy risks. In this paper, a secure privacy inference framework for
cloud-edge collaboration is proposed, termed CIS, which supports adaptively
partitioning the network according to the dynamically changing network
bandwidth and fully releases the computational power of edge devices. To
mitigate the influence introduced by private perturbation, CIS provides a way
to achieve differential privacy protection by adding refined noise to the
intermediate layer feature maps offloaded to the cloud. Meanwhile, with a given
total privacy budget, the budget is reasonably allocated by the size of the
feature graph rank generated by different convolution filters, which makes the
inference in the cloud robust to the perturbed data, thus effectively trade-off
the conflicting problem between privacy and availability. Finally, we construct
a real cloud-edge collaborative inference computing scenario to verify the
effectiveness of inference latency and model partitioning on
resource-constrained edge devices. Furthermore, the state-of-the-art cloud-edge
collaborative reconstruction attack is used to evaluate the practical
availability of the end-to-end privacy protection mechanism provided by CIS.


http://arxiv.org/abs/2212.06643
Boosting Semi-Supervised Learning with Contrastive Complementary Labeling. (1%)
Qinyi Deng; Yong Guo; Zhibang Yang; Haolin Pan; Jian Chen
  Semi-supervised learning (SSL) has achieved great success in leveraging a
large amount of unlabeled data to learn a promising classifier. A popular
approach is pseudo-labeling that generates pseudo labels only for those
unlabeled data with high-confidence predictions. As for the low-confidence
ones, existing methods often simply discard them because these unreliable
pseudo labels may mislead the model. Nevertheless, we highlight that these data
with low-confidence pseudo labels can be still beneficial to the training
process. Specifically, although the class with the highest probability in the
prediction is unreliable, we can assume that this sample is very unlikely to
belong to the classes with the lowest probabilities. In this way, these data
can be also very informative if we can effectively exploit these complementary
labels, i.e., the classes that a sample does not belong to. Inspired by this,
we propose a novel Contrastive Complementary Labeling (CCL) method that
constructs a large number of reliable negative pairs based on the complementary
labels and adopts contrastive learning to make use of all the unlabeled data.
Extensive experiments demonstrate that CCL significantly improves the
performance on top of existing methods. More critically, our CCL is
particularly effective under the label-scarce settings. For example, we yield
an improvement of 2.43% over FixMatch on CIFAR-10 only with 40 labeled data.


http://arxiv.org/abs/2212.05917
SRoUDA: Meta Self-training for Robust Unsupervised Domain Adaptation. (98%)
Wanqing Zhu; Jia-Li Yin; Bo-Hao Chen; Ximeng Liu
  As acquiring manual labels on data could be costly, unsupervised domain
adaptation (UDA), which transfers knowledge learned from a rich-label dataset
to the unlabeled target dataset, is gaining increasing popularity. While
extensive studies have been devoted to improving the model accuracy on target
domain, an important issue of model robustness is neglected. To make things
worse, conventional adversarial training (AT) methods for improving model
robustness are inapplicable under UDA scenario since they train models on
adversarial examples that are generated by supervised loss function. In this
paper, we present a new meta self-training pipeline, named SRoUDA, for
improving adversarial robustness of UDA models. Based on self-training
paradigm, SRoUDA starts with pre-training a source model by applying UDA
baseline on source labeled data and taraget unlabeled data with a developed
random masked augmentation (RMA), and then alternates between adversarial
target model training on pseudo-labeled target data and finetuning source model
by a meta step. While self-training allows the direct incorporation of AT in
UDA, the meta step in SRoUDA further helps in mitigating error propagation from
noisy pseudo labels. Extensive experiments on various benchmark datasets
demonstrate the state-of-the-art performance of SRoUDA where it achieves
significant model robustness improvement without harming clean accuracy. Code
is available at https://github.com/Vision.


http://arxiv.org/abs/2212.07815
Adversarially Robust Video Perception by Seeing Motion. (98%)
Lingyu Zhang; Chengzhi Mao; Junfeng Yang; Carl Vondrick
  Despite their excellent performance, state-of-the-art computer vision models
often fail when they encounter adversarial examples. Video perception models
tend to be more fragile under attacks, because the adversary has more places to
manipulate in high-dimensional data. In this paper, we find one reason for
video models' vulnerability is that they fail to perceive the correct motion
under adversarial perturbations. Inspired by the extensive evidence that motion
is a key factor for the human visual system, we propose to correct what the
model sees by restoring the perceived motion information. Since motion
information is an intrinsic structure of the video data, recovering motion
signals can be done at inference time without any human annotation, which
allows the model to adapt to unforeseen, worst-case inputs. Visualizations and
empirical experiments on UCF-101 and HMDB-51 datasets show that restoring
motion information in deep vision models improves adversarial robustness. Even
under adaptive attacks where the adversary knows our defense, our algorithm is
still effective. Our work provides new insight into robust video perception
algorithms by using intrinsic structures from the data. Our webpage is
available at https://motion4robust.cs.columbia.edu.


http://arxiv.org/abs/2212.06123
A Survey on Reinforcement Learning Security with Application to Autonomous Driving. (96%)
Ambra Demontis; Maura Pintor; Luca Demetrio; Kathrin Grosse; Hsiao-Ying Lin; Chengfang Fang; Battista Biggio; Fabio Roli
  Reinforcement learning allows machines to learn from their own experience.
Nowadays, it is used in safety-critical applications, such as autonomous
driving, despite being vulnerable to attacks carefully crafted to either
prevent that the reinforcement learning algorithm learns an effective and
reliable policy, or to induce the trained agent to make a wrong decision. The
literature about the security of reinforcement learning is rapidly growing, and
some surveys have been proposed to shed light on this field. However, their
categorizations are insufficient for choosing an appropriate defense given the
kind of system at hand. In our survey, we do not only overcome this limitation
by considering a different perspective, but we also discuss the applicability
of state-of-the-art attacks and defenses when reinforcement learning algorithms
are used in the context of autonomous driving.


http://arxiv.org/abs/2212.05709
HOTCOLD Block: Fooling Thermal Infrared Detectors with a Novel Wearable Design. (96%)
Hui Wei; Zhixiang Wang; Xuemei Jia; Yinqiang Zheng; Hao Tang; Shin'ichi Satoh; Zheng Wang
  Adversarial attacks on thermal infrared imaging expose the risk of related
applications. Estimating the security of these systems is essential for safely
deploying them in the real world. In many cases, realizing the attacks in the
physical space requires elaborate special perturbations. These solutions are
often \emph{impractical} and \emph{attention-grabbing}. To address the need for
a physically practical and stealthy adversarial attack, we introduce
\textsc{HotCold} Block, a novel physical attack for infrared detectors that
hide persons utilizing the wearable Warming Paste and Cooling Paste. By
attaching these readily available temperature-controlled materials to the body,
\textsc{HotCold} Block evades human eyes efficiently. Moreover, unlike existing
methods that build adversarial patches with complex texture and structure
features, \textsc{HotCold} Block utilizes an SSP-oriented adversarial
optimization algorithm that enables attacks with pure color blocks and explores
the influence of size, shape, and position on attack performance. Extensive
experimental results in both digital and physical environments demonstrate the
performance of our proposed \textsc{HotCold} Block. \emph{Code is available:
\textcolor{magenta}{https://github.com/weihui1308/HOTCOLDBlock}}.


http://arxiv.org/abs/2212.06079
Robust Perception through Equivariance. (96%)
Chengzhi Mao; Lingyu Zhang; Abhishek Joshi; Junfeng Yang; Hao Wang; Carl Vondrick
  Deep networks for computer vision are not reliable when they encounter
adversarial examples. In this paper, we introduce a framework that uses the
dense intrinsic constraints in natural images to robustify inference. By
introducing constraints at inference time, we can shift the burden of
robustness from training to the inference algorithm, thereby allowing the model
to adjust dynamically to each individual image's unique and potentially novel
characteristics at inference time. Among different constraints, we find that
equivariance-based constraints are most effective, because they allow dense
constraints in the feature space without overly constraining the representation
at a fine-grained level. Our theoretical results validate the importance of
having such dense constraints at inference time. Our empirical experiments show
that restoring feature equivariance at inference time defends against
worst-case adversarial perturbations. The method obtains improved adversarial
robustness on four datasets (ImageNet, Cityscapes, PASCAL VOC, and MS-COCO) on
image recognition, semantic segmentation, and instance segmentation tasks.
Project page is available at equi4robust.cs.columbia.edu.


http://arxiv.org/abs/2212.06295
Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety. (75%)
Joshua Albrecht; Ellie Kitanidis; Abraham J. Fetterman
  Large language models (LLMs) have exploded in popularity in the past few
years and have achieved undeniably impressive results on benchmarks as varied
as question answering and text summarization. We provide a simple new prompting
strategy that leads to yet another supposedly "super-human" result, this time
outperforming humans at common sense ethical reasoning (as measured by accuracy
on a subset of the ETHICS dataset). Unfortunately, we find that relying on
average performance to judge capabilities can be highly misleading. LLM errors
differ systematically from human errors in ways that make it easy to craft
adversarial examples, or even perturb existing examples to flip the output
label. We also observe signs of inverse scaling with model size on some
examples, and show that prompting models to "explain their reasoning" often
leads to alarming justifications of unethical actions. Our results highlight
how human-like performance does not necessarily imply human-like understanding
or reasoning.


http://arxiv.org/abs/2212.06325
AFLGuard: Byzantine-robust Asynchronous Federated Learning. (15%)
Minghong Fang; Jia Liu; Neil Zhenqiang Gong; Elizabeth S. Bentley
  Federated learning (FL) is an emerging machine learning paradigm, in which
clients jointly learn a model with the help of a cloud server. A fundamental
challenge of FL is that the clients are often heterogeneous, e.g., they have
different computing powers, and thus the clients may send model updates to the
server with substantially different delays. Asynchronous FL aims to address
this challenge by enabling the server to update the model once any client's
model update reaches it without waiting for other clients' model updates.
However, like synchronous FL, asynchronous FL is also vulnerable to poisoning
attacks, in which malicious clients manipulate the model via poisoning their
local data and/or model updates sent to the server. Byzantine-robust FL aims to
defend against poisoning attacks. In particular, Byzantine-robust FL can learn
an accurate model even if some clients are malicious and have Byzantine
behaviors. However, most existing studies on Byzantine-robust FL focused on
synchronous FL, leaving asynchronous FL largely unexplored. In this work, we
bridge this gap by proposing AFLGuard, a Byzantine-robust asynchronous FL
method. We show that, both theoretically and empirically, AFLGuard is robust
against various existing and adaptive poisoning attacks (both untargeted and
targeted). Moreover, AFLGuard outperforms existing Byzantine-robust
asynchronous FL methods.


http://arxiv.org/abs/2212.05827
Carpet-bombing patch: attacking a deep network without usual requirements. (2%)
Pol Labarbarie; Adrien Chan-Hon-Tong; Stéphane Herbin; Milad Leyli-Abadi
  Although deep networks have shown vulnerability to evasion attacks, such
attacks have usually unrealistic requirements. Recent literature discussed the
possibility to remove or not some of these requirements. This paper contributes
to this literature by introducing a carpet-bombing patch attack which has
almost no requirement. Targeting the feature representations, this patch attack
does not require knowing the network task. This attack decreases accuracy on
Imagenet, mAP on Pascal Voc, and IoU on Cityscapes without being aware that the
underlying tasks involved classification, detection or semantic segmentation,
respectively. Beyond the potential safety issues raised by this attack, the
impact of the carpet-bombing attack highlights some interesting property of
deep network layer dynamic.


http://arxiv.org/abs/2212.05630
DISCO: Adversarial Defense with Local Implicit Functions. (99%)
Chih-Hui Ho; Nuno Vasconcelos
  The problem of adversarial defenses for image classification, where the goal
is to robustify a classifier against adversarial examples, is considered.
Inspired by the hypothesis that these examples lie beyond the natural image
manifold, a novel aDversarIal defenSe with local impliCit functiOns (DISCO) is
proposed to remove adversarial perturbations by localized manifold projections.
DISCO consumes an adversarial image and a query pixel location and outputs a
clean RGB value at the location. It is implemented with an encoder and a local
implicit module, where the former produces per-pixel deep features and the
latter uses the features in the neighborhood of query pixel for predicting the
clean RGB value. Extensive experiments demonstrate that both DISCO and its
cascade version outperform prior defenses, regardless of whether the defense is
known to the attacker. DISCO is also shown to be data and parameter efficient
and to mount defenses that transfers across datasets, classifiers and attacks.


http://arxiv.org/abs/2212.05680
REAP: A Large-Scale Realistic Adversarial Patch Benchmark. (98%)
Nabeel Hingun; Chawin Sitawarin; Jerry Li; David Wagner
  Machine learning models are known to be susceptible to adversarial
perturbation. One famous attack is the adversarial patch, a sticker with a
particularly crafted pattern that makes the model incorrectly predict the
object it is placed on. This attack presents a critical threat to
cyber-physical systems that rely on cameras such as autonomous cars. Despite
the significance of the problem, conducting research in this setting has been
difficult; evaluating attacks and defenses in the real world is exceptionally
costly while synthetic data are unrealistic. In this work, we propose the REAP
(REalistic Adversarial Patch) benchmark, a digital benchmark that allows the
user to evaluate patch attacks on real images, and under real-world conditions.
Built on top of the Mapillary Vistas dataset, our benchmark contains over
14,000 traffic signs. Each sign is augmented with a pair of geometric and
lighting transformations, which can be used to apply a digitally generated
patch realistically onto the sign. Using our benchmark, we perform the first
large-scale assessments of adversarial patch attacks under realistic
conditions. Our experiments suggest that adversarial patch attacks may present
a smaller threat than previously believed and that the success rate of an
attack on simpler digital simulations is not predictive of its actual
effectiveness in practice. We release our benchmark publicly at
https://github.com/wagner-group/reap-benchmark.


http://arxiv.org/abs/2212.05387
General Adversarial Defense Against Black-box Attacks via Pixel Level and Feature Level Distribution Alignments. (99%)
Xiaogang Xu; Hengshuang Zhao; Philip Torr; Jiaya Jia
  Deep Neural Networks (DNNs) are vulnerable to the black-box adversarial
attack that is highly transferable. This threat comes from the distribution gap
between adversarial and clean samples in feature space of the target DNNs. In
this paper, we use Deep Generative Networks (DGNs) with a novel training
mechanism to eliminate the distribution gap. The trained DGNs align the
distribution of adversarial samples with clean ones for the target DNNs by
translating pixel values. Different from previous work, we propose a more
effective pixel level training constraint to make this achievable, thus
enhancing robustness on adversarial samples. Further, a class-aware
feature-level constraint is formulated for integrated distribution alignment.
Our approach is general and applicable to multiple tasks, including image
classification, semantic segmentation, and object detection. We conduct
extensive experiments on different datasets. Our strategy demonstrates its
unique effectiveness and generality against black-box attacks.


http://arxiv.org/abs/2212.05399
Untargeted Attack against Federated Recommendation Systems via Poisonous Item Embeddings and the Defense. (93%)
Yang Yu; Qi Liu; Likang Wu; Runlong Yu; Sanshi Lei Yu; Zaixi Zhang
  Federated recommendation (FedRec) can train personalized recommenders without
collecting user data, but the decentralized nature makes it susceptible to
poisoning attacks. Most previous studies focus on the targeted attack to
promote certain items, while the untargeted attack that aims to degrade the
overall performance of the FedRec system remains less explored. In fact,
untargeted attacks can disrupt the user experience and bring severe financial
loss to the service provider. However, existing untargeted attack methods are
either inapplicable or ineffective against FedRec systems. In this paper, we
delve into the untargeted attack and its defense for FedRec systems. (i) We
propose ClusterAttack, a novel untargeted attack method. It uploads poisonous
gradients that converge the item embeddings into several dense clusters, which
make the recommender generate similar scores for these items in the same
cluster and perturb the ranking order. (ii) We propose a uniformity-based
defense mechanism (UNION) to protect FedRec systems from such attacks. We
design a contrastive learning task that regularizes the item embeddings toward
a uniform distribution. Then the server filters out these malicious gradients
by estimating the uniformity of updated item embeddings. Experiments on two
public datasets show that ClusterAttack can effectively degrade the performance
of FedRec systems while circumventing many defense methods, and UNION can
improve the resistance of the system against various untargeted attacks,
including our ClusterAttack.


http://arxiv.org/abs/2212.05337
Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checking. (93%)
Dennis Gross; Thiago D. Simao; Nils Jansen; Guillermo A. Perez
  Deep Reinforcement Learning (RL) agents are susceptible to adversarial noise
in their observations that can mislead their policies and decrease their
performance. However, an adversary may be interested not only in decreasing the
reward, but also in modifying specific temporal logic properties of the policy.
This paper presents a metric that measures the exact impact of adversarial
attacks against such properties. We use this metric to craft optimal
adversarial attacks. Furthermore, we introduce a model checking method that
allows us to verify the robustness of RL policies against adversarial attacks.
Our empirical analysis confirms (1) the quality of our metric to craft
adversarial attacks against temporal logic properties, and (2) that we are able
to concisely assess a system's robustness against attacks.


http://arxiv.org/abs/2212.05380
Mitigating Adversarial Gray-Box Attacks Against Phishing Detectors. (54%)
Giovanni Apruzzese; V. S. Subrahmanian
  Although machine learning based algorithms have been extensively used for
detecting phishing websites, there has been relatively little work on how
adversaries may attack such "phishing detectors" (PDs for short). In this
paper, we propose a set of Gray-Box attacks on PDs that an adversary may use
which vary depending on the knowledge that he has about the PD. We show that
these attacks severely degrade the effectiveness of several existing PDs. We
then propose the concept of operation chains that iteratively map an original
set of features to a new set of features and develop the "Protective Operation
Chain" (POC for short) algorithm. POC leverages the combination of random
feature selection and feature mappings in order to increase the attacker's
uncertainty about the target PD. Using 3 existing publicly available datasets
plus a fourth that we have created and will release upon the publication of
this paper, we show that POC is more robust to these attacks than past
competing work, while preserving predictive performance when no adversarial
attacks are present. Moreover, POC is robust to attacks on 13 different
classifiers, not just one. These results are shown to be statistically
significant at the p < 0.001 level.


http://arxiv.org/abs/2212.05400
How to Backdoor Diffusion Models? (10%)
Sheng-Yen Chou; Pin-Yu Chen; Tsung-Yi Ho
  Diffusion models are state-of-the-art deep learning empowered generative
models that are trained based on the principle of learning forward and reverse
diffusion processes via progressive noise-addition and denoising. To gain a
better understanding of the limitations and potential risks, this paper
presents the first study on the robustness of diffusion models against backdoor
attacks. Specifically, we propose BadDiffusion, a novel attack framework that
engineers compromised diffusion processes during model training for backdoor
implantation. At the inference stage, the backdoored diffusion model will
behave just like an untampered generator for regular data inputs, while falsely
generating some targeted outcome designed by the bad actor upon receiving the
implanted trigger signal. Such a critical risk can be dreadful for downstream
tasks and applications built upon the problematic model. Our extensive
experiments on various backdoor attack settings show that BadDiffusion can
consistently lead to compromised diffusion models with high utility and target
specificity. Even worse, BadDiffusion can be made cost-effective by simply
finetuning a clean pre-trained diffusion model to implant backdoors. We also
explore some possible countermeasures for risk mitigation. Our results call
attention to potential risks and possible misuse of diffusion models.


http://arxiv.org/abs/2212.05327
Identifying the Source of Vulnerability in Explanation Discrepancy: A Case Study in Neural Text Classification. (1%)
Ruixuan Tang; Hanjie Chen; Yangfeng Ji
  Some recent works observed the instability of post-hoc explanations when
input side perturbations are applied to the model. This raises the interest and
concern in the stability of post-hoc explanations. However, the remaining
question is: is the instability caused by the neural network model or the
post-hoc explanation method? This work explores the potential source that leads
to unstable post-hoc explanations. To separate the influence from the model, we
propose a simple output probability perturbation method. Compared to prior
input side perturbation methods, the output probability perturbation method can
circumvent the neural model's potential effect on the explanations and allow
the analysis on the explanation method. We evaluate the proposed method with
three widely-used post-hoc explanation methods (LIME (Ribeiro et al., 2016),
Kernel Shapley (Lundberg and Lee, 2017a), and Sample Shapley (Strumbelj and
Kononenko, 2010)). The results demonstrate that the post-hoc methods are
stable, barely producing discrepant explanations under output probability
perturbations. The observation suggests that neural network models may be the
primary source of fragile explanations.


http://arxiv.org/abs/2212.04985
Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization. (98%)
Lin Li; Michael Spratling
  Adversarial training is widely used to improve the robustness of deep neural
networks to adversarial attack. However, adversarial training is prone to
overfitting, and the cause is far from clear. This work sheds light on the
mechanisms underlying overfitting through analyzing the loss landscape w.r.t.
the input. We find that robust overfitting results from standard training,
specifically the minimization of the clean loss, and can be mitigated by
regularization of the loss gradients. Moreover, we find that robust overfitting
turns severer during adversarial training partially because the gradient
regularization effect of adversarial training becomes weaker due to the
increase in the loss landscapes curvature. To improve robust generalization, we
propose a new regularizer to smooth the loss landscape by penalizing the
weighted logits variation along the adversarial direction. Our method
significantly mitigates robust overfitting and achieves the highest robustness
and efficiency compared to similar previous methods. Code is available at
https://github.com/TreeLLi/Combating-RO-AdvLC.


http://arxiv.org/abs/2212.04875
Expeditious Saliency-guided Mix-up through Random Gradient Thresholding. (2%)
Minh-Long Luu; Zeyi Huang; Eric P. Xing; Yong Jae Lee; Haohan Wang
  Mix-up training approaches have proven to be effective in improving the
generalization ability of Deep Neural Networks. Over the years, the research
community expands mix-up methods into two directions, with extensive efforts to
improve saliency-guided procedures but minimal focus on the arbitrary path,
leaving the randomization domain unexplored. In this paper, inspired by the
superior qualities of each direction over one another, we introduce a novel
method that lies at the junction of the two routes. By combining the best
elements of randomness and saliency utilization, our method balances speed,
simplicity, and accuracy. We name our method R-Mix following the concept of
"Random Mix-up". We demonstrate its effectiveness in generalization, weakly
supervised object localization, calibration, and robustness to adversarial
attacks. Finally, in order to address the question of whether there exists a
better decision protocol, we train a Reinforcement Learning agent that decides
the mix-up policies based on the classifier's performance, reducing dependency
on human-designed objectives and hyperparameter tuning. Extensive experiments
further show that the agent is capable of performing at the cutting-edge level,
laying the foundation for a fully automatic mix-up. Our code is released at
[https://github.com/minhlong94/Random-Mixup].


http://arxiv.org/abs/2212.05015
Robustness Implies Privacy in Statistical Estimation. (1%)
Samuel B. Hopkins; Gautam Kamath; Mahbod Majid; Shyam Narayanan
  We study the relationship between adversarial robustness and differential
privacy in high-dimensional algorithmic statistics. We give the first black-box
reduction from privacy to robustness which can produce private estimators with
optimal tradeoffs among sample complexity, accuracy, and privacy for a wide
range of fundamental high-dimensional parameter estimation problems, including
mean and covariance estimation. We show that this reduction can be implemented
in polynomial time in some important special cases. In particular, using
nearly-optimal polynomial-time robust estimators for the mean and covariance of
high-dimensional Gaussians which are based on the Sum-of-Squares method, we
design the first polynomial-time private estimators for these problems with
nearly-optimal samples-accuracy-privacy tradeoffs. Our algorithms are also
robust to a constant fraction of adversarially-corrupted samples.


http://arxiv.org/abs/2212.04687
Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Models. (1%)
Rui Zhu; Di Tang; Siyuan Tang; XiaoFeng Wang; Haixu Tang
  In this paper, we present a simple yet surprisingly effective technique to
induce "selective amnesia" on a backdoored model. Our approach, called SEAM,
has been inspired by the problem of catastrophic forgetting (CF), a long
standing issue in continual learning. Our idea is to retrain a given DNN model
on randomly labeled clean data, to induce a CF on the model, leading to a
sudden forget on both primary and backdoor tasks; then we recover the primary
task by retraining the randomized model on correctly labeled clean data. We
analyzed SEAM by modeling the unlearning process as continual learning and
further approximating a DNN using Neural Tangent Kernel for measuring CF. Our
analysis shows that our random-labeling approach actually maximizes the CF on
an unknown backdoor in the absence of triggered inputs, and also preserves some
feature extraction in the network to enable a fast revival of the primary task.
We further evaluated SEAM on both image processing and Natural Language
Processing tasks, under both data contamination and training manipulation
attacks, over thousands of models either trained on popular image datasets or
provided by the TrojAI competition. Our experiments show that SEAM vastly
outperforms the state-of-the-art unlearning techniques, achieving a high
Fidelity (measuring the gap between the accuracy of the primary task and that
of the backdoor) within a few minutes (about 30 times faster than training a
model from scratch using the MNIST dataset), with only a small amount of clean
data (0.1% of training data for TrojAI models).


http://arxiv.org/abs/2212.11138
QVIP: An ILP-based Formal Verification Approach for Quantized Neural Networks. (1%)
Yedi Zhang; Zhe Zhao; Fu Song; Min Zhang; Taolue Chen; Jun Sun
  Deep learning has become a promising programming paradigm in software
development, owing to its surprising performance in solving many challenging
tasks. Deep neural networks (DNNs) are increasingly being deployed in practice,
but are limited on resource-constrained devices owing to their demand for
computational power. Quantization has emerged as a promising technique to
reduce the size of DNNs with comparable accuracy as their floating-point
numbered counterparts. The resulting quantized neural networks (QNNs) can be
implemented energy-efficiently. Similar to their floating-point numbered
counterparts, quality assurance techniques for QNNs, such as testing and formal
verification, are essential but are currently less explored. In this work, we
propose a novel and efficient formal verification approach for QNNs. In
particular, we are the first to propose an encoding that reduces the
verification problem of QNNs into the solving of integer linear constraints,
which can be solved using off-the-shelf solvers. Our encoding is both sound and
complete. We demonstrate the application of our approach on local robustness
verification and maximum robustness radius computation. We implement our
approach in a prototype tool QVIP and conduct a thorough evaluation.
Experimental results on QNNs with different quantization bits confirm the
effectiveness and efficiency of our approach, e.g., two orders of magnitude
faster and able to solve more verification tasks in the same time limit than
the state-of-the-art methods.


http://arxiv.org/abs/2212.04138
Targeted Adversarial Attacks against Neural Network Trajectory Predictors. (99%)
Kaiyuan Tan; Jun Wang; Yiannis Kantaros
  Trajectory prediction is an integral component of modern autonomous systems
as it allows for envisioning future intentions of nearby moving agents. Due to
the lack of other agents' dynamics and control policies, deep neural network
(DNN) models are often employed for trajectory forecasting tasks. Although
there exists an extensive literature on improving the accuracy of these models,
there is a very limited number of works studying their robustness against
adversarially crafted input trajectories. To bridge this gap, in this paper, we
propose a targeted adversarial attack against DNN models for trajectory
forecasting tasks. We call the proposed attack TA4TP for Targeted adversarial
Attack for Trajectory Prediction. Our approach generates adversarial input
trajectories that are capable of fooling DNN models into predicting
user-specified target/desired trajectories. Our attack relies on solving a
nonlinear constrained optimization problem where the objective function
captures the deviation of the predicted trajectory from a target one while the
constraints model physical requirements that the adversarial input should
satisfy. The latter ensures that the inputs look natural and they are safe to
execute (e.g., they are close to nominal inputs and away from obstacles). We
demonstrate the effectiveness of TA4TP on two state-of-the-art DNN models and
two datasets. To the best of our knowledge, we propose the first targeted
adversarial attack against DNN models used for trajectory forecasting.


http://arxiv.org/abs/2212.04454
XRand: Differentially Private Defense against Explanation-Guided Attacks. (68%)
Truc Nguyen; Phung Lai; NhatHai Phan; My T. Thai
  Recent development in the field of explainable artificial intelligence (XAI)
has helped improve trust in Machine-Learning-as-a-Service (MLaaS) systems, in
which an explanation is provided together with the model prediction in response
to each query. However, XAI also opens a door for adversaries to gain insights
into the black-box models in MLaaS, thereby making the models more vulnerable
to several attacks. For example, feature-based explanations (e.g., SHAP) could
expose the top important features that a black-box model focuses on. Such
disclosure has been exploited to craft effective backdoor triggers against
malware classifiers. To address this trade-off, we introduce a new concept of
achieving local differential privacy (LDP) in the explanations, and from that
we establish a defense, called XRand, against such attacks. We show that our
mechanism restricts the information that the adversary can learn about the top
important features, while maintaining the faithfulness of the explanations.


http://arxiv.org/abs/2212.04656
Robust Graph Representation Learning via Predictive Coding. (22%)
Billy Byiringiro; Tommaso Salvatori; Thomas Lukasiewicz
  Predictive coding is a message-passing framework initially developed to model
information processing in the brain, and now also topic of research in machine
learning due to some interesting properties. One of such properties is the
natural ability of generative models to learn robust representations thanks to
their peculiar credit assignment rule, that allows neural activities to
converge to a solution before updating the synaptic weights. Graph neural
networks are also message-passing models, which have recently shown outstanding
results in diverse types of tasks in machine learning, providing
interdisciplinary state-of-the-art performance on structured data. However,
they are vulnerable to imperceptible adversarial attacks, and unfit for
out-of-distribution generalization. In this work, we address this by building
models that have the same structure of popular graph neural network
architectures, but rely on the message-passing rule of predictive coding.
Through an extensive set of experiments, we show that the proposed models are
(i) comparable to standard ones in terms of performance in both inductive and
transductive tasks, (ii) better calibrated, and (iii) robust against multiple
kinds of adversarial attacks.


http://arxiv.org/abs/2212.03334
Pre-trained Encoders in Self-Supervised Learning Improve Secure and Privacy-preserving Supervised Learning. (96%)
Hongbin Liu; Wenjie Qu; Jinyuan Jia; Neil Zhenqiang Gong
  Classifiers in supervised learning have various security and privacy issues,
e.g., 1) data poisoning attacks, backdoor attacks, and adversarial examples on
the security side as well as 2) inference attacks and the right to be forgotten
for the training data on the privacy side. Various secure and
privacy-preserving supervised learning algorithms with formal guarantees have
been proposed to address these issues. However, they suffer from various
limitations such as accuracy loss, small certified security guarantees, and/or
inefficiency. Self-supervised learning is an emerging technique to pre-train
encoders using unlabeled data. Given a pre-trained encoder as a feature
extractor, supervised learning can train a simple yet accurate classifier using
a small amount of labeled training data. In this work, we perform the first
systematic, principled measurement study to understand whether and when a
pre-trained encoder can address the limitations of secure or privacy-preserving
supervised learning algorithms. Our key findings are that a pre-trained encoder
substantially improves 1) both accuracy under no attacks and certified security
guarantees against data poisoning and backdoor attacks of state-of-the-art
secure learning algorithms (i.e., bagging and KNN), 2) certified security
guarantees of randomized smoothing against adversarial examples without
sacrificing its accuracy under no attacks, 3) accuracy of differentially
private classifiers, and 4) accuracy and/or efficiency of exact machine
unlearning.


http://arxiv.org/abs/2212.02531
Enhancing Quantum Adversarial Robustness by Randomized Encodings. (99%)
Weiyuan Gong; Dong Yuan; Weikang Li; Dong-Ling Deng
  The interplay between quantum physics and machine learning gives rise to the
emergent frontier of quantum machine learning, where advanced quantum learning
models may outperform their classical counterparts in solving certain
challenging problems. However, quantum learning systems are vulnerable to
adversarial attacks: adding tiny carefully-crafted perturbations on legitimate
input samples can cause misclassifications. To address this issue, we propose a
general scheme to protect quantum learning systems from adversarial attacks by
randomly encoding the legitimate data samples through unitary or quantum error
correction encoders. In particular, we rigorously prove that both global and
local random unitary encoders lead to exponentially vanishing gradients (i.e.
barren plateaus) for any variational quantum circuits that aim to add
adversarial perturbations, independent of the input data and the inner
structures of adversarial circuits and quantum classifiers. In addition, we
prove a rigorous bound on the vulnerability of quantum classifiers under local
unitary adversarial attacks. We show that random black-box quantum error
correction encoders can protect quantum classifiers against local adversarial
noises and their robustness increases as we concatenate error correction codes.
To quantify the robustness enhancement, we adapt quantum differential privacy
as a measure of the prediction stability for quantum classifiers. Our results
establish versatile defense strategies for quantum classifiers against
adversarial perturbations, which provide valuable guidance to enhance the
reliability and security for both near-term and future quantum learning
technologies.


http://arxiv.org/abs/2212.03069
Multiple Perturbation Attack: Attack Pixelwise Under Different $\ell_p$-norms For Better Adversarial Performance. (99%)
Ngoc N. Tran; Anh Tuan Bui; Dinh Phung; Trung Le
  Adversarial machine learning has been both a major concern and a hot topic
recently, especially with the ubiquitous use of deep neural networks in the
current landscape. Adversarial attacks and defenses are usually likened to a
cat-and-mouse game in which defenders and attackers evolve over the time. On
one hand, the goal is to develop strong and robust deep networks that are
resistant to malicious actors. On the other hand, in order to achieve that, we
need to devise even stronger adversarial attacks to challenge these defense
models. Most of existing attacks employs a single $\ell_p$ distance (commonly,
$p\in\{1,2,\infty\}$) to define the concept of closeness and performs steepest
gradient ascent w.r.t. this $p$-norm to update all pixels in an adversarial
example in the same way. These $\ell_p$ attacks each has its own pros and cons;
and there is no single attack that can successfully break through defense
models that are robust against multiple $\ell_p$ norms simultaneously.
Motivated by these observations, we come up with a natural approach: combining
various $\ell_p$ gradient projections on a pixel level to achieve a joint
adversarial perturbation. Specifically, we learn how to perturb each pixel to
maximize the attack performance, while maintaining the overall visual
imperceptibility of adversarial examples. Finally, through various experiments
with standardized benchmarks, we show that our method outperforms most current
strong attacks across state-of-the-art defense mechanisms, while retaining its
ability to remain clean visually.


http://arxiv.org/abs/2212.02127
FaceQAN: Face Image Quality Assessment Through Adversarial Noise Exploration. (92%)
Žiga Babnik; Peter Peer; Vitomir Štruc
  Recent state-of-the-art face recognition (FR) approaches have achieved
impressive performance, yet unconstrained face recognition still represents an
open problem. Face image quality assessment (FIQA) approaches aim to estimate
the quality of the input samples that can help provide information on the
confidence of the recognition decision and eventually lead to improved results
in challenging scenarios. While much progress has been made in face image
quality assessment in recent years, computing reliable quality scores for
diverse facial images and FR models remains challenging. In this paper, we
propose a novel approach to face image quality assessment, called FaceQAN, that
is based on adversarial examples and relies on the analysis of adversarial
noise which can be calculated with any FR model learned by using some form of
gradient descent. As such, the proposed approach is the first to link image
quality to adversarial attacks. Comprehensive (cross-model as well as
model-specific) experiments are conducted with four benchmark datasets, i.e.,
LFW, CFP-FP, XQLFW and IJB-C, four FR models, i.e., CosFace, ArcFace,
CurricularFace and ElasticFace, and in comparison to seven state-of-the-art
FIQA methods to demonstrate the performance of FaceQAN. Experimental results
show that FaceQAN achieves competitive results, while exhibiting several
desirable characteristics.


http://arxiv.org/abs/2212.02042
Refiner: Data Refining against Gradient Leakage Attacks in Federated Learning. (47%)
Mingyuan Fan; Cen Chen; Chengyu Wang; Wenmeng Zhou; Jun Huang; Ximeng Liu; Wenzhong Guo
  Federated Learning (FL) is pervasive in privacy-focused IoT environments
since it enables avoiding privacy leakage by training models with gradients
instead of data. Recent works show the uploaded gradients can be employed to
reconstruct data, i.e., gradient leakage attacks, and several defenses are
designed to alleviate the risk by tweaking the gradients. However, these
defenses exhibit weak resilience against threatening attacks, as the
effectiveness builds upon the unrealistic assumptions that deep neural networks
are simplified as linear models. In this paper, without such unrealistic
assumptions, we present a novel defense, called Refiner, instead of perturbing
gradients, which refines ground-truth data to craft robust data that yields
sufficient utility but with the least amount of privacy information, and then
the gradients of robust data are uploaded. To craft robust data, Refiner
promotes the gradients of critical parameters associated with robust data to
close ground-truth ones while leaving the gradients of trivial parameters to
safeguard privacy. Moreover, to exploit the gradients of trivial parameters,
Refiner utilizes a well-designed evaluation network to steer robust data far
away from ground-truth data, thereby alleviating privacy leakage risk.
Extensive experiments across multiple benchmark datasets demonstrate the
superior defense effectiveness of Refiner at defending against state-of-the-art
threats.


http://arxiv.org/abs/2212.02457
Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria. (8%)
Tengyuan Liang
  Covariate distribution shifts and adversarial perturbations present
robustness challenges to the conventional statistical learning framework:
seemingly small unconceivable shifts in the test covariate distribution can
significantly affect the performance of the statistical model learned based on
the training distribution. The model performance typically deteriorates when
extrapolation happens: namely, covariates shift to a region where the training
distribution is scarce, and naturally, the learned model has little
information. For robustness and regularization considerations, adversarial
perturbation techniques are proposed as a remedy; however, more needs to be
studied about what extrapolation region adversarial covariate shift will focus
on, given a learned model. This paper precisely characterizes the extrapolation
region, examining both regression and classification in an infinite-dimensional
setting. We study the implications of adversarial covariate shifts to
subsequent learning of the equilibrium -- the Bayes optimal model -- in a
sequential game framework. We exploit the dynamics of the adversarial learning
game and reveal the curious effects of the covariate shift to equilibrium
learning and experimental design. In particular, we establish two directional
convergence results that exhibit distinctive phenomena: (1) a blessing in
regression, the adversarial covariate shifts in an exponential rate to an
optimal experimental design for rapid subsequent learning, (2) a curse in
classification, the adversarial covariate shifts in a subquadratic rate fast to
the hardest experimental design trapping subsequent learning.


http://arxiv.org/abs/2212.02705
What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning? (3%)
Songyang Han; Sanbao Su; Sihong He; Shuo Han; Haizhao Yang; Fei Miao
  Various types of Multi-Agent Reinforcement Learning (MARL) methods have been
developed, assuming that agents' policies are based on true states. Recent
works have improved the robustness of MARL under uncertainties from the reward,
transition probability, or other partners' policies. However, in real-world
multi-agent systems, state estimations may be perturbed by sensor measurement
noise or even adversaries. Agents' policies trained with only true state
information will deviate from optimal solutions when facing adversarial state
perturbations during execution. MARL under adversarial state perturbations has
limited study. Hence, in this work, we propose a State-Adversarial Markov Game
(SAMG) and make the first attempt to study the fundamental properties of MARL
under state uncertainties. We prove that the optimal agent policy and the
robust Nash equilibrium do not always exist for an SAMG. Instead, we define the
solution concept, robust agent policy, of the proposed SAMG under adversarial
state perturbations, where agents want to maximize the worst-case expected
state value. We then design a gradient descent ascent-based robust MARL
algorithm to learn the robust policies for the MARL agents. Our experiments
show that adversarial state perturbations decrease agents' rewards for several
baselines from the existing literature, while our algorithm outperforms
baselines with state perturbations and significantly improves the robustness of
the MARL policies under state uncertainties.


http://arxiv.org/abs/2212.02648
Spuriosity Rankings: Sorting Data for Spurious Correlation Robustness. (1%)
Mazda Moayeri; Wenxiao Wang; Sahil Singla; Soheil Feizi
  We present a framework for ranking images within their class based on the
strength of spurious cues present. By measuring the gap in accuracy on the
highest and lowest ranked images (we call this spurious gap), we assess
spurious feature reliance for $89$ diverse ImageNet models, finding that even
the best models underperform in images with weak spurious presence. However,
the effect of spurious cues varies far more dramatically across classes,
emphasizing the crucial, often overlooked, class-dependence of the spurious
correlation problem. While most spurious features we observe are clarifying
(i.e. improving test-time accuracy when present, as is typically expected), we
surprisingly find many cases of confusing spurious features, where models
perform better when they are absent. We then close the spurious gap by training
new classification heads on lowly ranked (i.e. without common spurious cues)
images, resulting in improved effective robustness to distribution shifts
(ObjectNet, ImageNet-R, ImageNet-Sketch). We also propose a second metric to
assess feature reliability, finding that spurious features are generally less
reliable than non-spurious (core) ones, though again, spurious features can be
more reliable for certain classes. To enable our analysis, we annotated $5,000$
feature-class dependencies over {\it all} of ImageNet as core or spurious using
minimal human supervision. Finally, we show the feature discovery and
spuriosity ranking framework can be extended to other datasets like CelebA and
WaterBirds in a lightweight fashion with only linear layer training, leading to
discovering a previously unknown racial bias in the Celeb-A hair
classification.


http://arxiv.org/abs/2212.02663
Efficient Malware Analysis Using Metric Embeddings. (1%)
Ethan M. Rudd; David Krisiloff; Scott Coull; Daniel Olszewski; Edward Raff; James Holt
  In this paper, we explore the use of metric learning to embed Windows PE
files in a low-dimensional vector space for downstream use in a variety of
applications, including malware detection, family classification, and malware
attribute tagging. Specifically, we enrich labeling on malicious and benign PE
files using computationally expensive, disassembly-based malicious
capabilities. Using these capabilities, we derive several different types of
metric embeddings utilizing an embedding neural network trained via contrastive
loss, Spearman rank correlation, and combinations thereof. We then examine
performance on a variety of transfer tasks performed on the EMBER and SOREL
datasets, demonstrating that for several tasks, low-dimensional,
computationally efficient metric embeddings maintain performance with little
decay, which offers the potential to quickly retrain for a variety of transfer
tasks at significantly reduced storage overhead. We conclude with an
examination of practical considerations for the use of our proposed embedding
approach, such as robustness to adversarial evasion and introduction of
task-specific auxiliary objectives to improve performance on mission critical
tasks.


http://arxiv.org/abs/2212.02003
Bayesian Learning with Information Gain Provably Bounds Risk for a Robust Adversarial Defense. (98%)
Bao Gia Doan; Ehsan Abbasnejad; Javen Qinfeng Shi; Damith C. Ranasinghe
  We present a new algorithm to learn a deep neural network model robust
against adversarial attacks. Previous algorithms demonstrate an adversarially
trained Bayesian Neural Network (BNN) provides improved robustness. We
recognize the adversarial learning approach for approximating the multi-modal
posterior distribution of a Bayesian model can lead to mode collapse;
consequently, the model's achievements in robustness and performance are
sub-optimal. Instead, we first propose preventing mode collapse to better
approximate the multi-modal posterior distribution. Second, based on the
intuition that a robust model should ignore perturbations and only consider the
informative content of the input, we conceptualize and formulate an information
gain objective to measure and force the information learned from both benign
and adversarial training instances to be similar. Importantly. we prove and
demonstrate that minimizing the information gain objective allows the
adversarial risk to approach the conventional empirical risk. We believe our
efforts provide a step toward a basis for a principled method of adversarially
training BNNs. Our model demonstrate significantly improved robustness--up to
20%--compared with adversarial training and Adv-BNN under PGD attacks with
0.035 distortion on both CIFAR-10 and STL-10 datasets.


http://arxiv.org/abs/2212.01806
Recognizing Object by Components with Human Prior Knowledge Enhances Adversarial Robustness of Deep Neural Networks. (88%)
Xiao Li; Ziqi Wang; Bo Zhang; Fuchun Sun; Xiaolin Hu
  Adversarial attacks can easily fool object recognition systems based on deep
neural networks (DNNs). Although many defense methods have been proposed in
recent years, most of them can still be adaptively evaded. One reason for the
weak adversarial robustness may be that DNNs are only supervised by category
labels and do not have part-based inductive bias like the recognition process
of humans. Inspired by a well-known theory in cognitive psychology --
recognition-by-components, we propose a novel object recognition model ROCK
(Recognizing Object by Components with human prior Knowledge). It first
segments parts of objects from images, then scores part segmentation results
with predefined human prior knowledge, and finally outputs prediction based on
the scores. The first stage of ROCK corresponds to the process of decomposing
objects into parts in human vision. The second stage corresponds to the
decision process of the human brain. ROCK shows better robustness than
classical recognition models across various attack settings. These results
encourage researchers to rethink the rationality of currently widely-used
DNN-based object recognition models and explore the potential of part-based
models, once important but recently ignored, for improving robustness.


http://arxiv.org/abs/2212.01957
CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness. (82%)
Huy Phan; Miao Yin; Yang Sui; Bo Yuan; Saman Zonouz
  Model compression and model defense for deep neural networks (DNNs) have been
extensively and individually studied. Considering the co-importance of model
compactness and robustness in practical applications, several prior works have
explored to improve the adversarial robustness of the sparse neural networks.
However, the structured sparse models obtained by the exiting works suffer
severe performance degradation for both benign and robust accuracy, thereby
causing a challenging dilemma between robustness and structuredness of the
compact DNNs. To address this problem, in this paper, we propose CSTAR, an
efficient solution that can simultaneously impose the low-rankness-based
Compactness, high STructuredness and high Adversarial Robustness on the target
DNN models. By formulating the low-rankness and robustness requirement within
the same framework and globally determining the ranks, the compressed DNNs can
simultaneously achieve high compression performance and strong adversarial
robustness. Evaluations for various DNN models on different datasets
demonstrate the effectiveness of CSTAR. Compared with the state-of-the-art
robust structured pruning methods, CSTAR shows consistently better performance.
For instance, when compressing ResNet-18 on CIFAR-10, CSTAR can achieve up to
20.07% and 11.91% improvement for benign accuracy and robust accuracy,
respectively. For compressing ResNet-18 with 16x compression ratio on Imagenet,
CSTAR can obtain 8.58% benign accuracy gain and 4.27% robust accuracy gain
compared to the existing robust structured pruning method.


http://arxiv.org/abs/2212.01976
FedCC: Robust Federated Learning against Model Poisoning Attacks. (45%)
Hyejun Jeong; Hamin Son; Seohu Lee; Jayun Hyun; Tai-Myoung Chung
  Federated Learning has emerged to cope with raising concerns about privacy
breaches in using Machine or Deep Learning models. This new paradigm allows the
leverage of deep learning models in a distributed manner, enhancing privacy
preservation. However, the server's blindness to local datasets introduces its
vulnerability to model poisoning attacks and data heterogeneity, tampering with
the global model performance. Numerous works have proposed robust aggregation
algorithms and defensive mechanisms, but the approaches are orthogonal to
individual attacks or issues. FedCC, the proposed method, provides robust
aggregation by comparing the Centered Kernel Alignment of Penultimate Layers
Representations. The experiment results on FedCC demonstrate that it mitigates
untargeted and targeted model poisoning or backdoor attacks while also being
effective in non-Independently and Identically Distributed data environments.
By applying FedCC against untargeted attacks, global model accuracy is
recovered the most. Against targeted backdoor attacks, FedCC nullified attack
confidence while preserving the test accuracy. Most of the experiment results
outstand the baseline methods.


http://arxiv.org/abs/2212.01767
ConfounderGAN: Protecting Image Data Privacy with Causal Confounder. (8%)
Qi Tian; Kun Kuang; Kelu Jiang; Furui Liu; Zhihua Wang; Fei Wu
  The success of deep learning is partly attributed to the availability of
massive data downloaded freely from the Internet. However, it also means that
users' private data may be collected by commercial organizations without
consent and used to train their models. Therefore, it's important and necessary
to develop a method or tool to prevent unauthorized data exploitation. In this
paper, we propose ConfounderGAN, a generative adversarial network (GAN) that
can make personal image data unlearnable to protect the data privacy of its
owners. Specifically, the noise produced by the generator for each image has
the confounder property. It can build spurious correlations between images and
labels, so that the model cannot learn the correct mapping from images to
labels in this noise-added dataset. Meanwhile, the discriminator is used to
ensure that the generated noise is small and imperceptible, thereby remaining
the normal utility of the encrypted image for humans. The experiments are
conducted in six image classification datasets, consisting of three natural
object datasets and three medical datasets. The results demonstrate that our
method not only outperforms state-of-the-art methods in standard settings, but
can also be applied to fast encryption scenarios. Moreover, we show a series of
transferability and stability experiments to further illustrate the
effectiveness and superiority of our method.


http://arxiv.org/abs/2212.01688
LDL: A Defense for Label-Based Membership Inference Attacks. (83%)
Arezoo Rajabi; Dinuka Sahabandu; Luyao Niu; Bhaskar Ramasubramanian; Radha Poovendran
  The data used to train deep neural network (DNN) models in applications such
as healthcare and finance typically contain sensitive information. A DNN model
may suffer from overfitting. Overfitted models have been shown to be
susceptible to query-based attacks such as membership inference attacks (MIAs).
MIAs aim to determine whether a sample belongs to the dataset used to train a
classifier (members) or not (nonmembers). Recently, a new class of label based
MIAs (LAB MIAs) was proposed, where an adversary was only required to have
knowledge of predicted labels of samples. Developing a defense against an
adversary carrying out a LAB MIA on DNN models that cannot be retrained remains
an open problem. We present LDL, a light weight defense against LAB MIAs. LDL
works by constructing a high-dimensional sphere around queried samples such
that the model decision is unchanged for (noisy) variants of the sample within
the sphere. This sphere of label-invariance creates ambiguity and prevents a
querying adversary from correctly determining whether a sample is a member or a
nonmember. We analytically characterize the success rate of an adversary
carrying out a LAB MIA when LDL is deployed, and show that the formulation is
consistent with experimental observations. We evaluate LDL on seven datasets --
CIFAR-10, CIFAR-100, GTSRB, Face, Purchase, Location, and Texas -- with varying
sizes of training data. All of these datasets have been used by SOTA LAB MIAs.
Our experiments demonstrate that LDL reduces the success rate of an adversary
carrying out a LAB MIA in each case. We empirically compare LDL with defenses
against LAB MIAs that require retraining of DNN models, and show that LDL
performs favorably despite not needing to retrain the DNNs.


http://arxiv.org/abs/2212.01716
Security Analysis of SplitFed Learning. (8%)
Momin Ahmad Khan; Virat Shejwalkar; Amir Houmansadr; Fatima Muhammad Anwar
  Split Learning (SL) and Federated Learning (FL) are two prominent distributed
collaborative learning techniques that maintain data privacy by allowing
clients to never share their private data with other clients and servers, and
fined extensive IoT applications in smart healthcare, smart cities, and smart
industry. Prior work has extensively explored the security vulnerabilities of
FL in the form of poisoning attacks. To mitigate the effect of these attacks,
several defenses have also been proposed. Recently, a hybrid of both learning
techniques has emerged (commonly known as SplitFed) that capitalizes on their
advantages (fast training) and eliminates their intrinsic disadvantages
(centralized model updates). In this paper, we perform the first ever empirical
analysis of SplitFed's robustness to strong model poisoning attacks. We observe
that the model updates in SplitFed have significantly smaller dimensionality as
compared to FL that is known to have the curse of dimensionality. We show that
large models that have higher dimensionality are more susceptible to privacy
and security attacks, whereas the clients in SplitFed do not have the complete
model and have lower dimensionality, making them more robust to existing model
poisoning attacks. Our results show that the accuracy reduction due to the
model poisoning attack is 5x lower for SplitFed compared to FL.


http://arxiv.org/abs/2212.01082
Membership Inference Attacks Against Semantic Segmentation Models. (45%)
Tomas Chobola; Dmitrii Usynin; Georgios Kaissis
  Membership inference attacks aim to infer whether a data record has been used
to train a target model by observing its predictions. In sensitive domains such
as healthcare, this can constitute a severe privacy violation. In this work we
attempt to address the existing knowledge gap by conducting an exhaustive study
of membership inference attacks and defences in the domain of semantic image
segmentation. Our findings indicate that for certain threat models, these
learning settings can be considerably more vulnerable than the previously
considered classification settings. We additionally investigate a threat model
where a dishonest adversary can perform model poisoning to aid their inference
and evaluate the effects that these adaptations have on the success of
membership inference attacks. We quantitatively evaluate the attacks on a
number of popular model architectures across a variety of semantic segmentation
tasks, demonstrating that membership inference attacks in this domain can
achieve a high success rate and defending against them may result in
unfavourable privacy-utility trade-offs or increased computational costs.


http://arxiv.org/abs/2212.01346
Guaranteed Conformance of Neurosymbolic Models to Natural Constraints. (1%)
Kaustubh Sridhar; Souradeep Dutta; James Weimer; Insup Lee
  Deep neural networks have emerged as the workhorse for a large section of
robotics and control applications, especially as models for dynamical systems.
Such data-driven models are in turn used for designing and verifying autonomous
systems. This is particularly useful in modeling medical systems where data can
be leveraged to individualize treatment. In safety-critical applications, it is
important that the data-driven model is conformant to established knowledge
from the natural sciences. Such knowledge is often available or can often be
distilled into a (possibly black-box) model $M$. For instance, the unicycle
model for an F1 racing car. In this light, we consider the following problem -
given a model $M$ and state transition dataset, we wish to best approximate the
system model while being bounded distance away from $M$. We propose a method to
guarantee this conformance. Our first step is to distill the dataset into few
representative samples called memories, using the idea of a growing neural gas.
Next, using these memories we partition the state space into disjoint subsets
and compute bounds that should be respected by the neural network, when the
input is drawn from a particular subset. This serves as a symbolic wrapper for
guaranteed conformance. We argue theoretically that this only leads to bounded
increase in approximation error; which can be controlled by increasing the
number of memories. We experimentally show that on three case studies (Car
Model, Drones, and Artificial Pancreas), our constrained neurosymbolic models
conform to specified $M$ models (each encoding various constraints) with
order-of-magnitude improvements compared to the augmented Lagrangian and
vanilla training methods.


http://arxiv.org/abs/2212.00612
Purifier: Defending Data Inference Attacks via Transforming Confidence Scores. (89%)
Ziqi Yang; Lijin Wang; Da Yang; Jie Wan; Ziming Zhao; Ee-Chien Chang; Fan Zhang; Kui Ren
  Neural networks are susceptible to data inference attacks such as the
membership inference attack, the adversarial model inversion attack and the
attribute inference attack, where the attacker could infer useful information
such as the membership, the reconstruction or the sensitive attributes of a
data sample from the confidence scores predicted by the target classifier. In
this paper, we propose a method, namely PURIFIER, to defend against membership
inference attacks. It transforms the confidence score vectors predicted by the
target classifier and makes purified confidence scores indistinguishable in
individual shape, statistical distribution and prediction label between members
and non-members. The experimental results show that PURIFIER helps defend
membership inference attacks with high effectiveness and efficiency,
outperforming previous defense methods, and also incurs negligible utility
loss. Besides, our further experiments show that PURIFIER is also effective in
defending adversarial model inversion attacks and attribute inference attacks.
For example, the inversion error is raised about 4+ times on the Facescrub530
classifier, and the attribute inference accuracy drops significantly when
PURIFIER is deployed in our experiment.


http://arxiv.org/abs/2212.00884
Pareto Regret Analyses in Multi-objective Multi-armed Bandit. (31%)
Mengfan Xu; Diego Klabjan
  We study Pareto optimality in multi-objective multi-armed bandit by providing
a formulation of adversarial multi-objective multi-armed bandit and properly
defining its Pareto regrets that can be generalized to stochastic settings as
well. The regrets do not rely on any scalarization functions and reflect Pareto
optimality compared to scalarized regrets. We also present new algorithms
assuming both with and without prior information of the multi-objective
multi-armed bandit setting. The algorithms are shown optimal in adversarial
settings and nearly optimal in stochastic settings simultaneously by our
established upper bounds and lower bounds on Pareto regrets. Moreover, the
lower bound analyses show that the new regrets are consistent with the existing
Pareto regret for stochastic settings and extend an adversarial attack
mechanism from bandit to the multi-objective one.


http://arxiv.org/abs/2212.00325
All You Need Is Hashing: Defending Against Data Reconstruction Attack in Vertical Federated Learning. (2%)
Pengyu Qiu; Xuhong Zhang; Shouling Ji; Yuwen Pu; Ting Wang
  Vertical federated learning is a trending solution for multi-party
collaboration in training machine learning models. Industrial frameworks adopt
secure multi-party computation methods such as homomorphic encryption to
guarantee data security and privacy. However, a line of work has revealed that
there are still leakage risks in VFL. The leakage is caused by the correlation
between the intermediate representations and the raw data. Due to the powerful
approximation ability of deep neural networks, an adversary can capture the
correlation precisely and reconstruct the data. To deal with the threat of the
data reconstruction attack, we propose a hashing-based VFL framework, called
\textit{HashVFL}, to cut off the reversibility directly. The one-way nature of
hashing allows our framework to block all attempts to recover data from hash
codes. However, integrating hashing also brings some challenges, e.g., the loss
of information. This paper proposes and addresses three challenges to
integrating hashing: learnability, bit balance, and consistency. Experimental
results demonstrate \textit{HashVFL}'s efficiency in keeping the main task's
performance and defending against data reconstruction attacks. Furthermore, we
also analyze its potential value in detecting abnormal inputs. In addition, we
conduct extensive experiments to prove \textit{HashVFL}'s generalization in
various settings. In summary, \textit{HashVFL} provides a new perspective on
protecting multi-party's data security and privacy in VFL. We hope our study
can attract more researchers to expand the application domains of
\textit{HashVFL}.


http://arxiv.org/abs/2212.00311
Generalizing and Improving Jacobian and Hessian Regularization. (1%)
Chenwei Cui; Zehao Yan; Guangshen Liu; Liangfu Lu
  Jacobian and Hessian regularization aim to reduce the magnitude of the first
and second-order partial derivatives with respect to neural network inputs, and
they are predominantly used to ensure the adversarial robustness of image
classifiers. In this work, we generalize previous efforts by extending the
target matrix from zero to any matrix that admits efficient matrix-vector
products. The proposed paradigm allows us to construct novel regularization
terms that enforce symmetry or diagonality on square Jacobian and Hessian
matrices. On the other hand, the major challenge for Jacobian and Hessian
regularization has been high computational complexity. We introduce
Lanczos-based spectral norm minimization to tackle this difficulty. This
technique uses a parallelized implementation of the Lanczos algorithm and is
capable of effective and stable regularization of large Jacobian and Hessian
matrices. Theoretical justifications and empirical evidence are provided for
the proposed paradigm and technique. We carry out exploratory experiments to
validate the effectiveness of our novel regularization terms. We also conduct
comparative experiments to evaluate Lanczos-based spectral norm minimization
against prior methods. Results show that the proposed methodologies are
advantageous for a wide range of tasks.


http://arxiv.org/abs/2212.00952
On the Limit of Explaining Black-box Temporal Graph Neural Networks. (1%)
Minh N. Vu; My T. Thai
  Temporal Graph Neural Network (TGNN) has been receiving a lot of attention
recently due to its capability in modeling time-evolving graph-related tasks.
Similar to Graph Neural Networks, it is also non-trivial to interpret
predictions made by a TGNN due to its black-box nature. A major approach
tackling this problems in GNNs is by analyzing the model' responses on some
perturbations of the model's inputs, called perturbation-based explanation
methods. While these methods are convenient and flexible since they do not need
internal access to the model, does this lack of internal access prevent them
from revealing some important information of the predictions? Motivated by that
question, this work studies the limit of some classes of perturbation-based
explanation methods. Particularly, by constructing some specific instances of
TGNNs, we show (i) node-perturbation cannot reliably identify the paths
carrying out the prediction, (ii) edge-perturbation is not reliable in
determining all nodes contributing to the prediction and (iii) perturbing both
nodes and edges does not reliably help us identify the graph's components
carrying out the temporal aggregation in TGNNs.


http://arxiv.org/abs/2212.00951
SimpleMind adds thinking to deep neural networks. (1%)
Youngwon Choi; M. Wasil Wahi-Anwar; Matthew S. Brown
  Deep neural networks (DNNs) detect patterns in data and have shown
versatility and strong performance in many computer vision applications.
However, DNNs alone are susceptible to obvious mistakes that violate simple,
common sense concepts and are limited in their ability to use explicit
knowledge to guide their search and decision making. While overall DNN
performance metrics may be good, these obvious errors, coupled with a lack of
explainability, have prevented widespread adoption for crucial tasks such as
medical image analysis. The purpose of this paper is to introduce SimpleMind,
an open-source software framework for Cognitive AI focused on medical image
understanding. It allows creation of a knowledge base that describes expected
characteristics and relationships between image objects in an intuitive
human-readable form. The SimpleMind framework brings thinking to DNNs by: (1)
providing methods for reasoning with the knowledge base about image content,
such as spatial inferencing and conditional reasoning to check DNN outputs; (2)
applying process knowledge, in the form of general-purpose software agents,
that are chained together to accomplish image preprocessing, DNN prediction,
and result post-processing, and (3) performing automatic co-optimization of all
knowledge base parameters to adapt agents to specific problems. SimpleMind
enables reasoning on multiple detected objects to ensure consistency, providing
cross checking between DNN outputs. This machine reasoning improves the
reliability and trustworthiness of DNNs through an interpretable model and
explainable decisions. Example applications are provided that demonstrate how
SimpleMind supports and improves deep neural networks by embedding them within
a Cognitive AI framework.


http://arxiv.org/abs/2211.17071
Towards Interpreting Vulnerability of Multi-Instance Learning via Customized and Universal Adversarial Perturbations. (97%)
Yu-Xuan Zhang; Hua Meng; Xue-Mei Cao; Zhengchun Zhou; Mei Yang; Avik Ranjan Adhikary
  Multiple-Instance Learning (MIL) is a recent machine learning paradigm which
is immensely useful in various real-life applications, like image analysis,
video anomaly detection, text classification, etc. It is well known that most
of the existing machine learning classifiers are highly vulnerable to
adversarial perturbations. Since MIL is a weakly supervised learning, where
information is available for a set of instances, called bag and not for every
instances, adversarial perturbations can be fatal. In this paper, we have
proposed two adversarial perturbation methods to analyze the effect of
adversarial perturbations to interpret the vulnerability of MIL methods. Out of
the two algorithms, one can be customized for every bag, and the other is a
universal one, which can affect all bags in a given data set and thus has some
generalizability. Through simulations, we have also shown the effectiveness of
the proposed algorithms to fool the state-of-the-art (SOTA) MIL methods.
Finally, we have discussed through experiments, about taking care of these kind
of adversarial perturbations through a simple strategy. Source codes are
available at https://github.com/InkiInki/MI-UAP.


http://arxiv.org/abs/2212.03095
Interpretation of Neural Networks is Susceptible to Universal Adversarial Perturbations. (84%)
Haniyeh Ehsani Oskouie; Farzan Farnia
  Interpreting neural network classifiers using gradient-based saliency maps
has been extensively studied in the deep learning literature. While the
existing algorithms manage to achieve satisfactory performance in application
to standard image recognition datasets, recent works demonstrate the
vulnerability of widely-used gradient-based interpretation schemes to
norm-bounded perturbations adversarially designed for every individual input
sample. However, such adversarial perturbations are commonly designed using the
knowledge of an input sample, and hence perform sub-optimally in application to
an unknown or constantly changing data point. In this paper, we show the
existence of a Universal Perturbation for Interpretation (UPI) for standard
image datasets, which can alter a gradient-based feature map of neural networks
over a significant fraction of test samples. To design such a UPI, we propose a
gradient-based optimization method as well as a principal component analysis
(PCA)-based approach to compute a UPI which can effectively alter a neural
network's gradient-based interpretation on different samples. We support the
proposed UPI approaches by presenting several numerical results of their
successful applications to standard image datasets.


http://arxiv.org/abs/2211.16808
Efficient Adversarial Input Generation via Neural Net Patching. (73%)
Tooba Khan; Kumar Madhukar; Subodh Vishnu Sharma
  The adversarial input generation problem has become central in establishing
the robustness and trustworthiness of deep neural nets, especially when they
are used in safety-critical application domains such as autonomous vehicles and
precision medicine. This is also practically challenging for multiple
reasons-scalability is a common issue owing to large-sized networks, and the
generated adversarial inputs often lack important qualities such as naturalness
and output-impartiality. We relate this problem to the task of patching neural
nets, i.e. applying small changes in some of the network$'$s weights so that
the modified net satisfies a given property. Intuitively, a patch can be used
to produce an adversarial input because the effect of changing the weights can
also be brought about by changing the inputs instead. This work presents a
novel technique to patch neural networks and an innovative approach of using it
to produce perturbations of inputs which are adversarial for the original net.
We note that the proposed solution is significantly more effective than the
prior state-of-the-art techniques.


http://arxiv.org/abs/2211.16806
Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detection. (69%)
Kun Xiang; Xing Zhang; Jinwen She; Jinpeng Liu; Haohan Wang; Shiqi Deng; Shancheng Jiang
  As the COVID-19 pandemic puts pressure on healthcare systems worldwide, the
computed tomography image based AI diagnostic system has become a sustainable
solution for early diagnosis. However, the model-wise vulnerability under
adversarial perturbation hinders its deployment in practical situation. The
existing adversarial training strategies are difficult to generalized into
medical imaging field challenged by complex medical texture features. To
overcome this challenge, we propose a Contour Attention Preserving (CAP) method
based on lung cavity edge extraction. The contour prior features are injected
to attention layer via a parameter regularization and we optimize the robust
empirical risk with hybrid distance metric. We then introduce a new
cross-nation CT scan dataset to evaluate the generalization capability of the
adversarial robustness under distribution shift. Experimental results indicate
that the proposed method achieves state-of-the-art performance in multiple
adversarial defense and generalization tasks. The code and dataset are
available at https://github.com/Quinn777/CAP.


http://arxiv.org/abs/2211.16908
Improved Smoothed Analysis of 2-Opt for the Euclidean TSP. (8%)
Bodo Manthey; Rhijn Jesse van
  The 2-opt heuristic is a simple local search heuristic for the Travelling
Salesperson Problem (TSP). Although it usually performs well in practice, its
worst-case running time is poor. Attempts to reconcile this difference have
used smoothed analysis, in which adversarial instances are perturbed
probabilistically.
  We are interested in the classical model of smoothed analysis for the
Euclidean TSP, in which the perturbations are Gaussian. This model was
previously used by Manthey \& Veenstra, who obtained smoothed complexity bounds
polynomial in $n$, the dimension $d$, and the perturbation strength
$\sigma^{-1}$. However, their analysis only works for $d \geq 4$. The only
previous analysis for $d \leq 3$ was performed by Englert, R\"oglin \&
V\"ocking, who used a different perturbation model which can be translated to
Gaussian perturbations. Their model yields bounds polynomial in $n$ and
$\sigma^{-d}$, and super-exponential in $d$.
  As no direct analysis existed for Gaussian perturbations that yields
polynomial bounds for all $d$, we perform this missing analysis. Along the way,
we improve all existing smoothed complexity bounds for Euclidean 2-opt.


http://arxiv.org/abs/2211.16247
Ada3Diff: Defending against 3D Adversarial Point Clouds via Adaptive Diffusion. (99%)
Kui Zhang; Hang Zhou; Jie Zhang; Qidong Huang; Weiming Zhang; Nenghai Yu
  Deep 3D point cloud models are sensitive to adversarial attacks, which poses
threats to safety-critical applications such as autonomous driving. Robust
training and defend-by-denoise are typical strategies for defending adversarial
perturbations, including adversarial training and statistical filtering,
respectively. However, they either induce massive computational overhead or
rely heavily upon specified noise priors, limiting generalized robustness
against attacks of all kinds. This paper introduces a new defense mechanism
based on denoising diffusion models that can adaptively remove diverse noises
with a tailored intensity estimator. Specifically, we first estimate
adversarial distortions by calculating the distance of the points to their
neighborhood best-fit plane. Depending on the distortion degree, we choose
specific diffusion time steps for the input point cloud and perform the forward
diffusion to disrupt potential adversarial shifts. Then we conduct the reverse
denoising process to restore the disrupted point cloud back to a clean
distribution. This approach enables effective defense against adaptive attacks
with varying noise budgets, achieving accentuated robustness of existing 3D
deep recognition models.


http://arxiv.org/abs/2211.16080
Understanding and Enhancing Robustness of Concept-based Models. (99%)
Sanchit Sinha; Mengdi Huai; Jianhui Sun; Aidong Zhang
  Rising usage of deep neural networks to perform decision making in critical
applications like medical diagnosis and financial analysis have raised concerns
regarding their reliability and trustworthiness. As automated systems become
more mainstream, it is important their decisions be transparent, reliable and
understandable by humans for better trust and confidence. To this effect,
concept-based models such as Concept Bottleneck Models (CBMs) and
Self-Explaining Neural Networks (SENN) have been proposed which constrain the
latent space of a model to represent high level concepts easily understood by
domain experts in the field. Although concept-based models promise a good
approach to both increasing explainability and reliability, it is yet to be
shown if they demonstrate robustness and output consistent concepts under
systematic perturbations to their inputs. To better understand performance of
concept-based models on curated malicious samples, in this paper, we aim to
study their robustness to adversarial perturbations, which are also known as
the imperceptible changes to the input data that are crafted by an attacker to
fool a well-learned concept-based model. Specifically, we first propose and
analyze different malicious attacks to evaluate the security vulnerability of
concept based models. Subsequently, we propose a potential general adversarial
training-based defense mechanism to increase robustness of these systems to the
proposed malicious attacks. Extensive experiments on one synthetic and two
real-world datasets demonstrate the effectiveness of the proposed attacks and
the defense approach.


http://arxiv.org/abs/2211.16253
Advancing Deep Metric Learning Through Multiple Batch Norms And Multi-Targeted Adversarial Examples. (88%)
Inderjeet Singh; Kazuya Kakizaki; Toshinori Araki
  Deep Metric Learning (DML) is a prominent field in machine learning with
extensive practical applications that concentrate on learning visual
similarities. It is known that inputs such as Adversarial Examples (AXs), which
follow a distribution different from that of clean data, result in false
predictions from DML systems. This paper proposes MDProp, a framework to
simultaneously improve the performance of DML models on clean data and inputs
following multiple distributions. MDProp utilizes multi-distribution data
through an AX generation process while leveraging disentangled learning through
multiple batch normalization layers during the training of a DML model. MDProp
is the first to generate feature space multi-targeted AXs to perform targeted
regularization on the training model's denser embedding space regions,
resulting in improved embedding space densities contributing to the improved
generalization in the trained models. From a comprehensive experimental
analysis, we show that MDProp results in up to 2.95% increased clean data
Recall@1 scores and up to 2.12 times increased robustness against different
input distributions compared to the conventional methods.


http://arxiv.org/abs/2211.16093
Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering. (83%)
Kazutoshi Shinoda; Saku Sugawara; Akiko Aizawa
  Question answering (QA) models are shown to be insensitive to large
perturbations to inputs; that is, they make correct and confident predictions
even when given largely perturbed inputs from which humans can not correctly
derive answers. In addition, QA models fail to generalize to other domains and
adversarial test sets, while humans maintain high accuracy. Based on these
observations, we assume that QA models do not use intended features necessary
for human reading but rely on spurious features, causing the lack of
generalization ability. Therefore, we attempt to answer the question: If the
overconfident predictions of QA models for various types of perturbations are
penalized, will the out-of-distribution (OOD) generalization be improved? To
prevent models from making confident predictions on perturbed inputs, we first
follow existing studies and maximize the entropy of the output probability for
perturbed inputs. However, we find that QA models trained to be sensitive to a
certain perturbation type are often insensitive to unseen types of
perturbations. Thus, we simultaneously maximize the entropy for the four
perturbation types (i.e., word- and sentence-level shuffling and deletion) to
further close the gap between models and humans. Contrary to our expectations,
although models become sensitive to the four types of perturbations, we find
that the OOD generalization is not improved. Moreover, the OOD generalization
is sometimes degraded after entropy maximization. Making unconfident
predictions on largely perturbed inputs per se may be beneficial to gaining
human trust. However, our negative results suggest that researchers should pay
attention to the side effect of entropy maximization.


http://arxiv.org/abs/2211.16187
Quantization-aware Interval Bound Propagation for Training Certifiably Robust Quantized Neural Networks. (73%)
Mathias Lechner; Đorđe Žikelić; Krishnendu Chatterjee; Thomas A. Henzinger; Daniela Rus
  We study the problem of training and certifying adversarially robust
quantized neural networks (QNNs). Quantization is a technique for making neural
networks more efficient by running them using low-bit integer arithmetic and is
therefore commonly adopted in industry. Recent work has shown that
floating-point neural networks that have been verified to be robust can become
vulnerable to adversarial attacks after quantization, and certification of the
quantized representation is necessary to guarantee robustness. In this work, we
present quantization-aware interval bound propagation (QA-IBP), a novel method
for training robust QNNs. Inspired by advances in robust learning of
non-quantized networks, our training algorithm computes the gradient of an
abstract representation of the actual network. Unlike existing approaches, our
method can handle the discrete semantics of QNNs. Based on QA-IBP, we also
develop a complete verification procedure for verifying the adversarial
robustness of QNNs, which is guaranteed to terminate and produce a correct
answer. Compared to existing approaches, the key advantage of our verification
procedure is that it runs entirely on GPU or other accelerator devices. We
demonstrate experimentally that our approach significantly outperforms existing
methods and establish the new state-of-the-art for training and certifying the
robustness of QNNs.


http://arxiv.org/abs/2211.16040
AdvMask: A Sparse Adversarial Attack Based Data Augmentation Method for Image Classification. (54%)
Suorong Yang; Jinqiao Li; Jian Zhao; Furao Shen
  Data augmentation is a widely used technique for enhancing the generalization
ability of convolutional neural networks (CNNs) in image classification tasks.
Occlusion is a critical factor that affects on the generalization ability of
image classification models. In order to generate new samples, existing data
augmentation methods based on information deletion simulate occluded samples by
randomly removing some areas in the images. However, those methods cannot
delete areas of the images according to their structural features of the
images. To solve those problems, we propose a novel data augmentation method,
AdvMask, for image classification tasks. Instead of randomly removing areas in
the images, AdvMask obtains the key points that have the greatest influence on
the classification results via an end-to-end sparse adversarial attack module.
Therefore, we can find the most sensitive points of the classification results
without considering the diversity of various image appearance and shapes of the
object of interest. In addition, a data augmentation module is employed to
generate structured masks based on the key points, thus forcing the CNN
classification models to seek other relevant content when the most
discriminative content is hidden. AdvMask can effectively improve the
performance of classification models in the testing process. The experimental
results on various datasets and CNN models verify that the proposed method
outperforms other previous data augmentation methods in image classification
tasks.


http://arxiv.org/abs/2211.16316
A3T: Accuracy Aware Adversarial Training. (10%)
Enes Altinisik; Safa Messaoud; Husrev Taha Sencar; Sanjay Chawla
  Adversarial training has been empirically shown to be more prone to
overfitting than standard training. The exact underlying reasons still need to
be fully understood. In this paper, we identify one cause of overfitting
related to current practices of generating adversarial samples from
misclassified samples. To address this, we propose an alternative approach that
leverages the misclassified samples to mitigate the overfitting problem. We
show that our approach achieves better generalization while having comparable
robustness to state-of-the-art adversarial training methods on a wide range of
computer vision, natural language processing, and tabular tasks.


http://arxiv.org/abs/2211.16228
Building Resilience to Out-of-Distribution Visual Data via Input Optimization and Model Finetuning. (1%)
Christopher J. Holder; Majid Khonji; Jorge Dias; Muhammad Shafique
  A major challenge in machine learning is resilience to out-of-distribution
data, that is data that exists outside of the distribution of a model's
training data. Training is often performed using limited, carefully curated
datasets and so when a model is deployed there is often a significant
distribution shift as edge cases and anomalies not included in the training
data are encountered. To address this, we propose the Input Optimisation
Network, an image preprocessing model that learns to optimise input data for a
specific target vision model. In this work we investigate several
out-of-distribution scenarios in the context of semantic segmentation for
autonomous vehicles, comparing an Input Optimisation based solution to existing
approaches of finetuning the target model with augmented training data and an
adversarially trained preprocessing model. We demonstrate that our approach can
enable performance on such data comparable to that of a finetuned model, and
subsequently that a combined approach, whereby an input optimization network is
optimised to target a finetuned model, delivers superior performance to either
method in isolation. Finally, we propose a joint optimisation approach, in
which input optimization network and target model are trained simultaneously,
which we demonstrate achieves significant further performance gains,
particularly in challenging edge-case scenarios. We also demonstrate that our
architecture can be reduced to a relatively compact size without a significant
performance impact, potentially facilitating real time embedded applications.


http://arxiv.org/abs/2212.00727
Adversarial Artifact Detection in EEG-Based Brain-Computer Interfaces. (99%)
Xiaoqing Chen; Dongrui Wu
  Machine learning has achieved great success in electroencephalogram (EEG)
based brain-computer interfaces (BCIs). Most existing BCI research focused on
improving its accuracy, but few had considered its security. Recent studies,
however, have shown that EEG-based BCIs are vulnerable to adversarial attacks,
where small perturbations added to the input can cause misclassification.
Detection of adversarial examples is crucial to both the understanding of this
phenomenon and the defense. This paper, for the first time, explores
adversarial detection in EEG-based BCIs. Experiments on two EEG datasets using
three convolutional neural networks were performed to verify the performances
of multiple detection approaches. We showed that both white-box and black-box
attacks can be detected, and the former are easier to detect.


http://arxiv.org/abs/2211.15926
Interpretations Cannot Be Trusted: Stealthy and Effective Adversarial Perturbations against Interpretable Deep Learning. (95%)
Eldor Abdukhamidov; Mohammed Abuhamad; Simon S. Woo; Eric Chan-Tin; Tamer Abuhmed
  Deep learning methods have gained increased attention in various applications
due to their outstanding performance. For exploring how this high performance
relates to the proper use of data artifacts and the accurate problem
formulation of a given task, interpretation models have become a crucial
component in developing deep learning-based systems. Interpretation models
enable the understanding of the inner workings of deep learning models and
offer a sense of security in detecting the misuse of artifacts in the input
data. Similar to prediction models, interpretation models are also susceptible
to adversarial inputs. This work introduces two attacks, AdvEdge and
AdvEdge$^{+}$, that deceive both the target deep learning model and the coupled
interpretation model. We assess the effectiveness of proposed attacks against
two deep learning model architectures coupled with four interpretation models
that represent different categories of interpretation models. Our experiments
include the attack implementation using various attack frameworks. We also
explore the potential countermeasures against such attacks. Our analysis shows
the effectiveness of our attacks in terms of deceiving the deep learning models
and their interpreters, and highlights insights to improve and circumvent the
attacks.


http://arxiv.org/abs/2211.15875
Training Time Adversarial Attack Aiming the Vulnerability of Continual Learning. (83%)
Gyojin Han; Jaehyun Choi; Hyeong Gwon Hong; Junmo Kim
  Generally, regularization-based continual learning models limit access to the
previous task data to imitate the real-world setting which has memory and
privacy issues. However, this introduces a problem in these models by not being
able to track the performance on each task. In other words, current continual
learning methods are vulnerable to attacks done on the previous task. We
demonstrate the vulnerability of regularization-based continual learning
methods by presenting simple task-specific training time adversarial attack
that can be used in the learning process of a new task. Training data generated
by the proposed attack causes performance degradation on a specific task
targeted by the attacker. Experiment results justify the vulnerability proposed
in this paper and demonstrate the importance of developing continual learning
models that are robust to adversarial attack.


http://arxiv.org/abs/2211.15900
Towards More Robust Interpretation via Local Gradient Alignment. (76%)
Sunghwan Joo; Seokhyeon Jeong; Juyeon Heo; Adrian Weller; Taesup Moon
  Neural network interpretation methods, particularly feature attribution
methods, are known to be fragile with respect to adversarial input
perturbations. To address this, several methods for enhancing the local
smoothness of the gradient while training have been proposed for attaining
\textit{robust} feature attributions. However, the lack of considering the
normalization of the attributions, which is essential in their visualizations,
has been an obstacle to understanding and improving the robustness of feature
attribution methods. In this paper, we provide new insights by taking such
normalization into account. First, we show that for every non-negative
homogeneous neural network, a naive $\ell_2$-robust criterion for gradients is
\textit{not} normalization invariant, which means that two functions with the
same normalized gradient can have different values. Second, we formulate a
normalization invariant cosine distance-based criterion and derive its upper
bound, which gives insight for why simply minimizing the Hessian norm at the
input, as has been done in previous work, is not sufficient for attaining
robust feature attribution. Finally, we propose to combine both $\ell_2$ and
cosine distance-based criteria as regularization terms to leverage the
advantages of both in aligning the local gradient. As a result, we
experimentally show that models trained with our method produce much more
robust interpretations on CIFAR-10 and ImageNet-100 without significantly
hurting the accuracy, compared to the recent baselines. To the best of our
knowledge, this is the first work to verify the robustness of interpretation on
a larger-scale dataset beyond CIFAR-10, thanks to the computational efficiency
of our method.


http://arxiv.org/abs/2211.15762
Understanding the Impact of Adversarial Robustness on Accuracy Disparity. (15%)
Yuzheng Hu; Fan Wu; Hongyang Zhang; Han Zhao
  While it has long been empirically observed that adversarial robustness may
be at odds with standard accuracy and may have further disparate impacts on
different classes, it remains an open question to what extent such observations
hold and how the class imbalance plays a role within. In this paper, we attempt
to understand this question of accuracy disparity by taking a closer look at
linear classifiers under a Gaussian mixture model. We decompose the impact of
adversarial robustness into two parts: an inherent effect that will degrade the
standard accuracy on all classes, and the other caused by the class imbalance
ratio, which will increase the accuracy disparity compared to standard
training. Furthermore, we also extend our model to the general family of stable
distributions. We demonstrate that while the constraint of adversarial
robustness consistently degrades the standard accuracy in the balanced class
setting, the class imbalance ratio plays a fundamentally different role in
accuracy disparity compared to the Gaussian case, due to the heavy tail of the
stable distribution. We additionally perform experiments on both synthetic and
real-world datasets. The empirical results not only corroborate our theoretical
findings, but also suggest that the implications may extend to nonlinear models
over real-world datasets.


http://arxiv.org/abs/2211.15844
How Important are Good Method Names in Neural Code Generation? A Model Robustness Perspective. (13%)
Guang Yang; Yu Zhou; Wenhua Yang; Tao Yue; Xiang Chen; Taolue Chen
  Pre-trained code generation models (PCGMs) have been widely applied in neural
code generation which can generate executable code from functional descriptions
in natural languages, possibly together with signatures. Despite substantial
performance improvement of PCGMs, the role of method names in neural code
generation has not been thoroughly investigated. In this paper, we study and
demonstrate the potential of benefiting from method names to enhance the
performance of PCGMs, from a model robustness perspective. Specifically, we
propose a novel approach, named RADAR (neuRAl coDe generAtor Robustifier).
RADAR consists of two components: RADAR-Attack and RADAR-Defense. The former
attacks a PCGM by generating adversarial method names as part of the input,
which are semantic and visual similar to the original input, but may trick the
PCGM to generate completely unrelated code snippets. As a countermeasure to
such attacks, RADAR-Defense synthesizes a new method name from the functional
description and supplies it to the PCGM. Evaluation results show that
RADAR-Attack can, e.g., reduce the CodeBLEU of generated code by 19.72% to
38.74% in three state-of-the-art PCGMs (i.e., CodeGPT, PLBART, and CodeT5).
Moreover, RADAR-Defense is able to reinstate the performance of PCGMs with
synthesized method names. These results highlight the importance of good method
names in neural code generation and implicate the benefits of studying model
robustness in software engineering.


http://arxiv.org/abs/2211.15180
Rethinking the Number of Shots in Robust Model-Agnostic Meta-Learning. (8%)
Xiaoyue Duan; Guoliang Kang; Runqi Wang; Shumin Han; Song Xue; Tian Wang; Baochang Zhang
  Robust Model-Agnostic Meta-Learning (MAML) is usually adopted to train a
meta-model which may fast adapt to novel classes with only a few exemplars and
meanwhile remain robust to adversarial attacks. The conventional solution for
robust MAML is to introduce robustness-promoting regularization during
meta-training stage. With such a regularization, previous robust MAML methods
simply follow the typical MAML practice that the number of training shots
should match with the number of test shots to achieve an optimal adaptation
performance. However, although the robustness can be largely improved, previous
methods sacrifice clean accuracy a lot. In this paper, we observe that
introducing robustness-promoting regularization into MAML reduces the intrinsic
dimension of clean sample features, which results in a lower capacity of clean
representations. This may explain why the clean accuracy of previous robust
MAML methods drops severely. Based on this observation, we propose a simple
strategy, i.e., increasing the number of training shots, to mitigate the loss
of intrinsic dimension caused by robustness-promoting regularization. Though
simple, our method remarkably improves the clean accuracy of MAML without much
loss of robustness, producing a robust yet accurate model. Extensive
experiments demonstrate that our method outperforms prior arts in achieving a
better trade-off between accuracy and robustness. Besides, we observe that our
method is less sensitive to the number of fine-tuning steps during
meta-training, which allows for a reduced number of fine-tuning steps to
improve training efficiency.


http://arxiv.org/abs/2211.15556
Attack on Unfair ToS Clause Detection: A Case Study using Universal Adversarial Triggers. (8%)
Shanshan Xu; Irina Broda; Rashid Haddad; Marco Negrini; Matthias Grabmair
  Recent work has demonstrated that natural language processing techniques can
support consumer protection by automatically detecting unfair clauses in the
Terms of Service (ToS) Agreement. This work demonstrates that transformer-based
ToS analysis systems are vulnerable to adversarial attacks. We conduct
experiments attacking an unfair-clause detector with universal adversarial
triggers. Experiments show that a minor perturbation of the text can
considerably reduce the detection performance. Moreover, to measure the
detectability of the triggers, we conduct a detailed human evaluation study by
collecting both answer accuracy and response time from the participants. The
results show that the naturalness of the triggers remains key to tricking
readers.


http://arxiv.org/abs/2211.15223
Gamma-convergence of a nonlocal perimeter arising in adversarial machine learning. (3%)
Leon Bungert; Kerrek Stinson
  In this paper we prove Gamma-convergence of a nonlocal perimeter of Minkowski
type to a local anisotropic perimeter. The nonlocal model describes the
regularizing effect of adversarial training in binary classifications. The
energy essentially depends on the interaction between two distributions
modelling likelihoods for the associated classes. We overcome typical strict
regularity assumptions for the distributions by only assuming that they have
bounded $BV$ densities. In the natural topology coming from compactness, we
prove Gamma-convergence to a weighted perimeter with weight determined by an
anisotropic function of the two densities. Despite being local, this sharp
interface limit reflects classification stability with respect to adversarial
perturbations. We further apply our results to deduce Gamma-convergence of the
associated total variations, to study the asymptotics of adversarial training,
and to prove Gamma-convergence of graph discretizations for the nonlocal
perimeter.


http://arxiv.org/abs/2211.15718
CoNAL: Anticipating Outliers with Large Language Models. (1%)
Albert Xu; Xiang Ren; Robin Jia
  In many task settings, text classification models are likely to encounter
examples from novel classes on which they cannot predict correctly. Selective
prediction, in which models abstain on low-confidence examples, provides a
possible solution, but existing models are often overly confident on OOD
examples. To remedy this overconfidence, we introduce Contrastive
Novelty-Augmented Learning (CoNAL), a two-step method that generates OOD
examples representative of novel classes, then trains to decrease confidence on
them. First, we generate OOD examples by prompting a large language model
twice: we prompt it to enumerate relevant novel labels, then generate examples
from each novel class matching the task format. Second, we train our classifier
with a novel contrastive objective that encourages lower confidence on
generated OOD examples than training examples. When trained with CoNAL,
classifiers improve in their ability to detect and abstain on OOD examples over
prior methods by an average of 2.3% AUAC and 5.5% AUROC across 4 NLP datasets,
with no cost to in-distribution accuracy.


http://arxiv.org/abs/2211.15030
Imperceptible Adversarial Attack via Invertible Neural Networks. (99%)
Zihan Chen; Ziyue Wang; Junjie Huang; Wentao Zhao; Xiao Liu; Dejian Guan
  Adding perturbations via utilizing auxiliary gradient information or
discarding existing details of the benign images are two common approaches for
generating adversarial examples. Though visual imperceptibility is the desired
property of adversarial examples, conventional adversarial attacks still
generate traceable adversarial perturbations. In this paper, we introduce a
novel Adversarial Attack via Invertible Neural Networks (AdvINN) method to
produce robust and imperceptible adversarial examples. Specifically, AdvINN
fully takes advantage of the information preservation property of Invertible
Neural Networks and thereby generates adversarial examples by simultaneously
adding class-specific semantic information of the target class and dropping
discriminant information of the original class. Extensive experiments on
CIFAR-10, CIFAR-100, and ImageNet-1K demonstrate that the proposed AdvINN
method can produce less imperceptible adversarial images than the
state-of-the-art methods and AdvINN yields more robust adversarial examples
with high confidence compared to other adversarial attacks.


http://arxiv.org/abs/2211.14860
Foiling Explanations in Deep Neural Networks. (98%)
Snir Vitrack Tamam; Raz Lapid; Moshe Sipper
  Deep neural networks (DNNs) have greatly impacted numerous fields over the
past decade. Yet despite exhibiting superb performance over many problems,
their black-box nature still poses a significant challenge with respect to
explainability. Indeed, explainable artificial intelligence (XAI) is crucial in
several fields, wherein the answer alone -- sans a reasoning of how said answer
was derived -- is of little value. This paper uncovers a troubling property of
explanation methods for image-based DNNs: by making small visual changes to the
input image -- hardly influencing the network's output -- we demonstrate how
explanations may be arbitrarily manipulated through the use of evolution
strategies. Our novel algorithm, AttaXAI, a model-agnostic, adversarial attack
on XAI algorithms, only requires access to the output logits of a classifier
and to the explanation map; these weak assumptions render our approach highly
useful where real-world models and data are concerned. We compare our method's
performance on two benchmark datasets -- CIFAR100 and ImageNet -- using four
different pretrained deep-learning models: VGG16-CIFAR100, VGG16-ImageNet,
MobileNet-CIFAR100, and Inception-v3-ImageNet. We find that the XAI methods can
be manipulated without the use of gradients or other model internals. Our novel
algorithm is successfully able to manipulate an image in a manner imperceptible
to the human eye, such that the XAI method outputs a specific explanation map.
To our knowledge, this is the first such method in a black-box setting, and we
believe it has significant value where explainability is desired, required, or
legally mandatory.


http://arxiv.org/abs/2211.14769
Navigation as the Attacker Wishes? Towards Building Byzantine-Robust Embodied Agents under Federated Learning. (84%)
Yunchao Zhang; Zonglin Di; Kaiwen Zhou; Cihang Xie; Xin Wang
  Federated embodied agent learning protects the data privacy of individual
visual environments by keeping data locally at each client (the individual
environment) during training. However, since the local data is inaccessible to
the server under federated learning, attackers may easily poison the training
data of the local client to build a backdoor in the agent without notice.
Deploying such an agent raises the risk of potential harm to humans, as the
attackers may easily navigate and control the agent as they wish via the
backdoor. Towards Byzantine-robust federated embodied agent learning, in this
paper, we study the attack and defense for the task of vision-and-language
navigation (VLN), where the agent is required to follow natural language
instructions to navigate indoor environments. First, we introduce a simple but
effective attack strategy, Navigation as Wish (NAW), in which the malicious
client manipulates local trajectory data to implant a backdoor into the global
model. Results on two VLN datasets (R2R and RxR) show that NAW can easily
navigate the deployed VLN agent regardless of the language instruction, without
affecting its performance on normal test sets. Then, we propose a new
Prompt-Based Aggregation (PBA) to defend against the NAW attack in federated
VLN, which provides the server with a ''prompt'' of the vision-and-language
alignment variance between the benign and malicious clients so that they can be
distinguished during training. We validate the effectiveness of the PBA method
on protecting the global model from the NAW attack, which outperforms other
state-of-the-art defense methods by a large margin in the defense metrics on
R2R and RxR.


http://arxiv.org/abs/2211.14794
Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs. (50%)
Guangrun Wang; Philip H. S. Torr
  Classifiers and generators have long been separated. We break down this
separation and showcase that conventional neural network classifiers can
generate high-quality images of a large number of categories, being comparable
to the state-of-the-art generative models (e.g., DDPMs and GANs). We achieve
this by computing the partial derivative of the classification loss function
with respect to the input to optimize the input to produce an image. Since it
is widely known that directly optimizing the inputs is similar to targeted
adversarial attacks incapable of generating human-meaningful images, we propose
a mask-based stochastic reconstruction module to make the gradients
semantic-aware to synthesize plausible images. We further propose a
progressive-resolution technique to guarantee fidelity, which produces
photorealistic images. Furthermore, we introduce a distance metric loss and a
non-trivial distribution loss to ensure classification neural networks can
synthesize diverse and high-fidelity images. Using traditional neural network
classifiers, we can generate good-quality images of 256$\times$256 resolution
on ImageNet. Intriguingly, our method is also applicable to text-to-image
generation by regarding image-text foundation models as generalized
classifiers.
  Proving that classifiers have learned the data distribution and are ready for
image generation has far-reaching implications, for classifiers are much easier
to train than generative models like DDPMs and GANs. We don't even need to
train classification models because tons of public ones are available for
download. Also, this holds great potential for the interpretability and
robustness of classifiers.


http://arxiv.org/abs/2211.14952
Federated Learning Attacks and Defenses: A Survey. (47%)
Yao Chen; Yijie Gui; Hong Lin; Wensheng Gan; Yongdong Wu
  In terms of artificial intelligence, there are several security and privacy
deficiencies in the traditional centralized training methods of machine
learning models by a server. To address this limitation, federated learning
(FL) has been proposed and is known for breaking down ``data silos" and
protecting the privacy of users. However, FL has not yet gained popularity in
the industry, mainly due to its security, privacy, and high cost of
communication. For the purpose of advancing the research in this field,
building a robust FL system, and realizing the wide application of FL, this
paper sorts out the possible attacks and corresponding defenses of the current
FL system systematically. Firstly, this paper briefly introduces the basic
workflow of FL and related knowledge of attacks and defenses. It reviews a
great deal of research about privacy theft and malicious attacks that have been
studied in recent years. Most importantly, in view of the current three
classification criteria, namely the three stages of machine learning, the three
different roles in federated learning, and the CIA (Confidentiality, Integrity,
and Availability) guidelines on privacy protection, we divide attack approaches
into two categories according to the training stage and the prediction stage in
machine learning. Furthermore, we also identify the CIA property violated for
each attack method and potential attack role. Various defense mechanisms are
then analyzed separately from the level of privacy and security. Finally, we
summarize the possible challenges in the application of FL from the aspect of
attacks and defenses and discuss the future development direction of FL
systems. In this way, the designed FL system has the ability to resist
different attacks and is more secure and stable.


http://arxiv.org/abs/2211.14966
Adversarial Rademacher Complexity of Deep Neural Networks. (47%)
Jiancong Xiao; Yanbo Fan; Ruoyu Sun; Zhi-Quan Luo
  Deep neural networks are vulnerable to adversarial attacks. Ideally, a robust
model shall perform well on both the perturbed training data and the unseen
perturbed test data. It is found empirically that fitting perturbed training
data is not hard, but generalizing to perturbed test data is quite difficult.
To better understand adversarial generalization, it is of great interest to
study the adversarial Rademacher complexity (ARC) of deep neural networks.
However, how to bound ARC in multi-layers cases is largely unclear due to the
difficulty of analyzing adversarial loss in the definition of ARC. There have
been two types of attempts of ARC. One is to provide the upper bound of ARC in
linear and one-hidden layer cases. However, these approaches seem hard to
extend to multi-layer cases. Another is to modify the adversarial loss and
provide upper bounds of Rademacher complexity on such surrogate loss in
multi-layer cases. However, such variants of Rademacher complexity are not
guaranteed to be bounds for meaningful robust generalization gaps (RGG). In
this paper, we provide a solution to this unsolved problem. Specifically, we
provide the first bound of adversarial Rademacher complexity of deep neural
networks. Our approach is based on covering numbers. We provide a method to
handle the robustify function classes of DNNs such that we can calculate the
covering numbers. Finally, we provide experiments to study the empirical
implication of our bounds and provide an analysis of poor adversarial
generalization.


http://arxiv.org/abs/2211.14669
Game Theoretic Mixed Experts for Combinational Adversarial Machine Learning. (99%)
Ethan Rathbun; Kaleel Mahmood; Sohaib Ahmad; Caiwen Ding; Dijk Marten van
  Recent advances in adversarial machine learning have shown that defenses
considered to be robust are actually susceptible to adversarial attacks which
are specifically tailored to target their weaknesses. These defenses include
Barrage of Random Transforms (BaRT), Friendly Adversarial Training (FAT), Trash
is Treasure (TiT) and ensemble models made up of Vision Transformers (ViTs),
Big Transfer models and Spiking Neural Networks (SNNs). A natural question
arises: how can one best leverage a combination of adversarial defenses to
thwart such attacks? In this paper, we provide a game-theoretic framework for
ensemble adversarial attacks and defenses which answers this question. In
addition to our framework we produce the first adversarial defense
transferability study to further motivate a need for combinational defenses
utilizing a diverse set of defense architectures. Our framework is called Game
theoretic Mixed Experts (GaME) and is designed to find the Mixed-Nash strategy
for a defender when facing an attacker employing compositional adversarial
attacks. We show that this framework creates an ensemble of defenses with
greater robustness than multiple state-of-the-art, single-model defenses in
addition to combinational defenses with uniform probability distributions.
Overall, our framework and analyses advance the field of adversarial machine
learning by yielding new insights into compositional attack and defense
formulations.


http://arxiv.org/abs/2211.14088
Boundary Adversarial Examples Against Adversarial Overfitting. (99%)
Muhammad Zaid Hameed; Beat Buesser
  Standard adversarial training approaches suffer from robust overfitting where
the robust accuracy decreases when models are adversarially trained for too
long. The origin of this problem is still unclear and conflicting explanations
have been reported, i.e., memorization effects induced by large loss data or
because of small loss data and growing differences in loss distribution of
training samples as the adversarial training progresses. Consequently, several
mitigation approaches including early stopping, temporal ensembling and weight
perturbations on small loss data have been proposed to mitigate the effect of
robust overfitting. However, a side effect of these strategies is a larger
reduction in clean accuracy compared to standard adversarial training. In this
paper, we investigate if these mitigation approaches are complimentary to each
other in improving adversarial training performance. We further propose the use
of helper adversarial examples that can be obtained with minimal cost in the
adversarial example generation, and show how they increase the clean accuracy
in the existing approaches without compromising the robust accuracy.


http://arxiv.org/abs/2211.14424
Supervised Contrastive Prototype Learning: Augmentation Free Robust Neural Network. (98%)
Iordanis Fostiropoulos; Laurent Itti
  Transformations in the input space of Deep Neural Networks (DNN) lead to
unintended changes in the feature space. Almost perceptually identical inputs,
such as adversarial examples, can have significantly distant feature
representations. On the contrary, Out-of-Distribution (OOD) samples can have
highly similar feature representations to training set samples. Our theoretical
analysis for DNNs trained with a categorical classification head suggests that
the inflexible logit space restricted by the classification problem size is one
of the root causes for the lack of $\textit{robustness}$. Our second
observation is that DNNs over-fit to the training augmentation technique and do
not learn $\textit{nuance invariant}$ representations. Inspired by the recent
success of prototypical and contrastive learning frameworks for both improving
robustness and learning nuance invariant representations, we propose a training
framework, $\textbf{Supervised Contrastive Prototype Learning}$ (SCPL). We use
N-pair contrastive loss with prototypes of the same and opposite classes and
replace a categorical classification head with a $\textbf{Prototype
Classification Head}$ (PCH). Our approach is $\textit{sample efficient}$, does
not require $\textit{sample mining}$, can be implemented on any existing DNN
without modification to their architecture, and combined with other training
augmentation techniques. We empirically evaluate the $\textbf{clean}$
robustness of our method on out-of-distribution and adversarial samples. Our
framework outperforms other state-of-the-art contrastive and prototype learning
approaches in $\textit{robustness}$.


http://arxiv.org/abs/2211.14065
Beyond Smoothing: Unsupervised Graph Representation Learning with Edge Heterophily Discriminating. (3%)
Yixin Liu; Yizhen Zheng; Daokun Zhang; Vincent CS Lee; Shirui Pan
  Unsupervised graph representation learning (UGRL) has drawn increasing
research attention and achieved promising results in several graph analytic
tasks. Relying on the homophily assumption, existing UGRL methods tend to
smooth the learned node representations along all edges, ignoring the existence
of heterophilic edges that connect nodes with distinct attributes. As a result,
current methods are hard to generalize to heterophilic graphs where dissimilar
nodes are widely connected, and also vulnerable to adversarial attacks. To
address this issue, we propose a novel unsupervised Graph Representation
learning method with Edge hEterophily discriminaTing (GREET) which learns
representations by discriminating and leveraging homophilic edges and
heterophilic edges. To distinguish two types of edges, we build an edge
discriminator that infers edge homophily/heterophily from feature and structure
information. We train the edge discriminator in an unsupervised way through
minimizing the crafted pivot-anchored ranking loss, with randomly sampled node
pairs acting as pivots. Node representations are learned through contrasting
the dual-channel encodings obtained from the discriminated homophilic and
heterophilic edges. With an effective interplaying scheme, edge discriminating
and representation learning can mutually boost each other during the training
phase. We conducted extensive experiments on 14 benchmark datasets and multiple
learning scenarios to demonstrate the superiority of GREET.


http://arxiv.org/abs/2211.13991
TrustGAN: Training safe and trustworthy deep learning models through generative adversarial networks. (1%)
Hélion du Mas des Bourboux
  Deep learning models have been developed for a variety of tasks and are
deployed every day to work in real conditions. Some of these tasks are critical
and models need to be trusted and safe, e.g. military communications or cancer
diagnosis. These models are given real data, simulated data or combination of
both and are trained to be highly predictive on them. However, gathering enough
real data or simulating them to be representative of all the real conditions
is: costly, sometimes impossible due to confidentiality and most of the time
impossible. Indeed, real conditions are constantly changing and sometimes are
intractable. A solution is to deploy machine learning models that are able to
give predictions when they are confident enough otherwise raise a flag or
abstain. One issue is that standard models easily fail at detecting
out-of-distribution samples where their predictions are unreliable.
  We present here TrustGAN, a generative adversarial network pipeline targeting
trustness. It is a deep learning pipeline which improves a target model
estimation of the confidence without impacting its predictive power. The
pipeline can accept any given deep learning model which outputs a prediction
and a confidence on this prediction. Moreover, the pipeline does not need to
modify this target model. It can thus be easily deployed in a MLOps (Machine
Learning Operations) setting.
  The pipeline is applied here to a target classification model trained on
MNIST data to recognise numbers based on images. We compare such a model when
trained in the standard way and with TrustGAN. We show that on
out-of-distribution samples, here FashionMNIST and CIFAR10, the estimated
confidence is largely reduced. We observe similar conclusions for a
classification model trained on 1D radio signals from AugMod, tested on
RML2016.04C. We also publicly release the code.


http://arxiv.org/abs/2211.13775
SAGA: Spectral Adversarial Geometric Attack on 3D Meshes. (98%)
Tomer Stolik; Itai Lang; Shai Avidan
  A triangular mesh is one of the most popular 3D data representations. As
such, the deployment of deep neural networks for mesh processing is widely
spread and is increasingly attracting more attention. However, neural networks
are prone to adversarial attacks, where carefully crafted inputs impair the
model's functionality. The need to explore these vulnerabilities is a
fundamental factor in the future development of 3D-based applications.
Recently, mesh attacks were studied on the semantic level, where classifiers
are misled to produce wrong predictions. Nevertheless, mesh surfaces possess
complex geometric attributes beyond their semantic meaning, and their analysis
often includes the need to encode and reconstruct the geometry of the shape.
  We propose a novel framework for a geometric adversarial attack on a 3D mesh
autoencoder. In this setting, an adversarial input mesh deceives the
autoencoder by forcing it to reconstruct a different geometric shape at its
output. The malicious input is produced by perturbing a clean shape in the
spectral domain. Our method leverages the spectral decomposition of the mesh
along with additional mesh-related properties to obtain visually credible
results that consider the delicacy of surface distortions. Our code is publicly
available at https://github.com/StolikTomer/SAGA.


http://arxiv.org/abs/2211.13474
Explainable and Safe Reinforcement Learning for Autonomous Air Mobility. (92%)
Lei Wang; Hongyu Yang; Yi Lin; Suwan Yin; Yuankai Wu
  Increasing traffic demands, higher levels of automation, and communication
enhancements provide novel design opportunities for future air traffic
controllers (ATCs). This article presents a novel deep reinforcement learning
(DRL) controller to aid conflict resolution for autonomous free flight.
Although DRL has achieved important advancements in this field, the existing
works pay little attention to the explainability and safety issues related to
DRL controllers, particularly the safety under adversarial attacks. To address
those two issues, we design a fully explainable DRL framework wherein we: 1)
decompose the coupled Q value learning model into a safety-awareness and
efficiency (reach the target) one; and 2) use information from surrounding
intruders as inputs, eliminating the needs of central controllers. In our
simulated experiments, we show that by decoupling the safety-awareness and
efficiency, we can exceed performance on free flight control tasks while
dramatically improving explainability on practical. In addition, the safety Q
learning module provides rich information about the safety situation of
environments. To study the safety under adversarial attacks, we additionally
propose an adversarial attack strategy that can impose both safety-oriented and
efficiency-oriented attacks. The adversarial aims to minimize safety/efficiency
by only attacking the agent at a few time steps. In the experiments, our attack
strategy increases as many collisions as the uniform attack (i.e., attacking at
every time step) by only attacking the agent four times less often, which
provide insights into the capabilities and restrictions of the DRL in future
ATC designs. The source code is publicly available at
https://github.com/WLeiiiii/Gym-ATC-Attack-Project.


http://arxiv.org/abs/2211.13535
Tracking Dataset IP Use in Deep Neural Networks. (76%)
Seonhye Park; Alsharif Abuadbba; Shuo Wang; Kristen Moore; Yansong Gao; Hyoungshick Kim; Surya Nepal
  Training highly performant deep neural networks (DNNs) typically requires the
collection of a massive dataset and the use of powerful computing resources.
Therefore, unauthorized redistribution of private pre-trained DNNs may cause
severe economic loss for model owners. For protecting the ownership of DNN
models, DNN watermarking schemes have been proposed by embedding secret
information in a DNN model and verifying its presence for model ownership.
However, existing DNN watermarking schemes compromise the model utility and are
vulnerable to watermark removal attacks because a model is modified with a
watermark. Alternatively, a new approach dubbed DEEPJUDGE was introduced to
measure the similarity between a suspect model and a victim model without
modifying the victim model. However, DEEPJUDGE would only be designed to detect
the case where a suspect model's architecture is the same as a victim model's.
In this work, we propose a novel DNN fingerprinting technique dubbed DEEPTASTER
to prevent a new attack scenario in which a victim's data is stolen to build a
suspect model. DEEPTASTER can effectively detect such data theft attacks even
when a suspect model's architecture differs from a victim model's. To achieve
this goal, DEEPTASTER generates a few adversarial images with perturbations,
transforms them into the Fourier frequency domain, and uses the transformed
images to identify the dataset used in a suspect model. The intuition is that
those adversarial images can be used to capture the characteristics of DNNs
built on a specific dataset. We evaluated the detection accuracy of DEEPTASTER
on three datasets with three model architectures under various attack
scenarios, including transfer learning, pruning, fine-tuning, and data
augmentation. Overall, DEEPTASTER achieves a balanced accuracy of 94.95%, which
is significantly better than 61.11% achieved by DEEPJUDGE in the same settings.


http://arxiv.org/abs/2211.13644
Seeds Don't Lie: An Adaptive Watermarking Framework for Computer Vision Models. (8%)
Jacob Shams; Ben Nassi; Ikuya Morikawa; Toshiya Shimizu; Asaf Shabtai; Yuval Elovici
  In recent years, various watermarking methods were suggested to detect
computer vision models obtained illegitimately from their owners, however they
fail to demonstrate satisfactory robustness against model extraction attacks.
In this paper, we present an adaptive framework to watermark a protected model,
leveraging the unique behavior present in the model due to a unique random seed
initialized during the model training. This watermark is used to detect
extracted models, which have the same unique behavior, indicating an
unauthorized usage of the protected model's intellectual property (IP). First,
we show how an initial seed for random number generation as part of model
training produces distinct characteristics in the model's decision boundaries,
which are inherited by extracted models and present in their decision
boundaries, but aren't present in non-extracted models trained on the same
data-set with a different seed. Based on our findings, we suggest the Robust
Adaptive Watermarking (RAW) Framework, which utilizes the unique behavior
present in the protected and extracted models to generate a watermark key-set
and verification model. We show that the framework is robust to (1) unseen
model extraction attacks, and (2) extracted models which undergo a blurring
method (e.g., weight pruning). We evaluate the framework's robustness against a
naive attacker (unaware that the model is watermarked), and an informed
attacker (who employs blurring strategies to remove watermarked behavior from
an extracted model), and achieve outstanding (i.e., >0.9) AUC values. Finally,
we show that the framework is robust to model extraction attacks with different
structure and/or architecture than the protected model.


http://arxiv.org/abs/2211.13772
Generative Joint Source-Channel Coding for Semantic Image Transmission. (1%)
Ecenaz Erdemir; Tze-Yang Tung; Pier Luigi Dragotti; Deniz Gunduz
  Recent works have shown that joint source-channel coding (JSCC) schemes using
deep neural networks (DNNs), called DeepJSCC, provide promising results in
wireless image transmission. However, these methods mostly focus on the
distortion of the reconstructed signals with respect to the input image, rather
than their perception by humans. However, focusing on traditional distortion
metrics alone does not necessarily result in high perceptual quality,
especially in extreme physical conditions, such as very low bandwidth
compression ratio (BCR) and low signal-to-noise ratio (SNR) regimes. In this
work, we propose two novel JSCC schemes that leverage the perceptual quality of
deep generative models (DGMs) for wireless image transmission, namely
InverseJSCC and GenerativeJSCC. While the former is an inverse problem approach
to DeepJSCC, the latter is an end-to-end optimized JSCC scheme. In both, we
optimize a weighted sum of mean squared error (MSE) and learned perceptual
image patch similarity (LPIPS) losses, which capture more semantic similarities
than other distortion metrics. InverseJSCC performs denoising on the distorted
reconstructions of a DeepJSCC model by solving an inverse optimization problem
using style-based generative adversarial network (StyleGAN). Our simulation
results show that InverseJSCC significantly improves the state-of-the-art
(SotA) DeepJSCC in terms of perceptual quality in edge cases. In
GenerativeJSCC, we carry out end-to-end training of an encoder and a
StyleGAN-based decoder, and show that GenerativeJSCC significantly outperforms
DeepJSCC both in terms of distortion and perceptual quality.


http://arxiv.org/abs/2211.13737
CycleGANWM: A CycleGAN watermarking method for ownership verification. (1%)
Dongdong Lin; Benedetta Tondi; Bin Li; Mauro Barni
  Due to the proliferation and widespread use of deep neural networks (DNN),
their Intellectual Property Rights (IPR) protection has become increasingly
important. This paper presents a novel model watermarking method for an
unsupervised image-to-image translation (I2IT) networks, named CycleGAN, which
leverage the image translation visual quality and watermark embedding. In this
method, a watermark decoder is trained initially. Then the decoder is frozen
and used to extract the watermark bits when training the CycleGAN watermarking
model. The CycleGAN watermarking (CycleGANWM) is trained with specific loss
functions and optimized to get a good performance on both I2IT task and
watermark embedding. For watermark verification, this work uses statistical
significance test to identify the ownership of the model from the extract
watermark bits. We evaluate the robustness of the model against image
post-processing and improve it by fine-tuning the model with adding data
augmentation on the output images before extracting the watermark bits. We also
carry out surrogate model attack under black-box access of the model. The
experimental results prove that the proposed method is effective and robust to
some image post-processing, and it is able to resist surrogate model attack.


http://arxiv.org/abs/2211.13171
Query Efficient Cross-Dataset Transferable Black-Box Attack on Action Recognition. (99%)
Rohit Gupta; Naveed Akhtar; Gaurav Kumar Nayak; Ajmal Mian; Mubarak Shah
  Black-box adversarial attacks present a realistic threat to action
recognition systems. Existing black-box attacks follow either a query-based
approach where an attack is optimized by querying the target model, or a
transfer-based approach where attacks are generated using a substitute model.
While these methods can achieve decent fooling rates, the former tends to be
highly query-inefficient while the latter assumes extensive knowledge of the
black-box model's training data. In this paper, we propose a new attack on
action recognition that addresses these shortcomings by generating
perturbations to disrupt the features learned by a pre-trained substitute model
to reduce the number of queries. By using a nearly disjoint dataset to train
the substitute model, our method removes the requirement that the substitute
model be trained using the same dataset as the target model, and leverages
queries to the target model to retain the fooling rate benefits provided by
query-based methods. This ultimately results in attacks which are more
transferable than conventional black-box attacks. Through extensive
experiments, we demonstrate highly query-efficient black-box attacks with the
proposed framework. Our method achieves 8% and 12% higher deception rates
compared to state-of-the-art query-based and transfer-based attacks,
respectively.


http://arxiv.org/abs/2211.12990
Adversarial Attacks are a Surprisingly Strong Baseline for Poisoning Few-Shot Meta-Learners. (99%)
Elre T. Oldewage; John Bronskill; Richard E. Turner
  This paper examines the robustness of deployed few-shot meta-learning systems
when they are fed an imperceptibly perturbed few-shot dataset. We attack
amortized meta-learners, which allows us to craft colluding sets of inputs that
are tailored to fool the system's learning algorithm when used as training
data. Jointly crafted adversarial inputs might be expected to synergistically
manipulate a classifier, allowing for very strong data-poisoning attacks that
would be hard to detect. We show that in a white box setting, these attacks are
very successful and can cause the target model's predictions to become worse
than chance. However, in opposition to the well-known transferability of
adversarial examples in general, the colluding sets do not transfer well to
different classifiers. We explore two hypotheses to explain this: 'overfitting'
by the attack, and mismatch between the model on which the attack is generated
and that to which the attack is transferred. Regardless of the mitigation
strategies suggested by these hypotheses, the colluding inputs transfer no
better than adversarial inputs that are generated independently in the usual
way.


http://arxiv.org/abs/2211.12713
Reliable Robustness Evaluation via Automatically Constructed Attack Ensembles. (76%)
Shengcai Liu; Fu Peng; Ke Tang
  Attack Ensemble (AE), which combines multiple attacks together, provides a
reliable way to evaluate adversarial robustness. In practice, AEs are often
constructed and tuned by human experts, which however tends to be sub-optimal
and time-consuming. In this work, we present AutoAE, a conceptually simple
approach for automatically constructing AEs. In brief, AutoAE repeatedly adds
the attack and its iteration steps to the ensemble that maximizes ensemble
improvement per additional iteration consumed. We show theoretically that
AutoAE yields AEs provably within a constant factor of the optimal for a given
defense. We then use AutoAE to construct two AEs for $l_{\infty}$ and $l_2$
attacks, and apply them without any tuning or adaptation to 45 top adversarial
defenses on the RobustBench leaderboard. In all except one cases we achieve
equal or better (often the latter) robustness evaluation than existing AEs, and
notably, in 29 cases we achieve better robustness evaluation than the best
known one. Such performance of AutoAE shows itself as a reliable evaluation
protocol for adversarial robustness, which further indicates the huge potential
of automatic AE construction. Code is available at
\url{https://github.com/LeegerPENG/AutoAE}.


http://arxiv.org/abs/2211.13305
Dual Graphs of Polyhedral Decompositions for the Detection of Adversarial Attacks. (62%)
Huma Jamil; Yajing Liu; Christina Cole; Nathaniel Blanchard; Emily J. King; Michael Kirby; Christopher Peterson
  Previous work has shown that a neural network with the rectified linear unit
(ReLU) activation function leads to a convex polyhedral decomposition of the
input space. These decompositions can be represented by a dual graph with
vertices corresponding to polyhedra and edges corresponding to polyhedra
sharing a facet, which is a subgraph of a Hamming graph. This paper illustrates
how one can utilize the dual graph to detect and analyze adversarial attacks in
the context of digital images. When an image passes through a network
containing ReLU nodes, the firing or non-firing at a node can be encoded as a
bit ($1$ for ReLU activation, $0$ for ReLU non-activation). The sequence of all
bit activations identifies the image with a bit vector, which identifies it
with a polyhedron in the decomposition and, in turn, identifies it with a
vertex in the dual graph. We identify ReLU bits that are discriminators between
non-adversarial and adversarial images and examine how well collections of
these discriminators can ensemble vote to build an adversarial image detector.
Specifically, we examine the similarities and differences of ReLU bit vectors
for adversarial images, and their non-adversarial counterparts, using a
pre-trained ResNet-50 architecture. While this paper focuses on adversarial
digital images, ResNet-50 architecture, and the ReLU activation function, our
methods extend to other network architectures, activation functions, and types
of datasets.


http://arxiv.org/abs/2211.12864
Privacy-Enhancing Optical Embeddings for Lensless Classification. (11%)
Eric Bezzam; Martin Vetterli; Matthieu Simeoni
  Lensless imaging can provide visual privacy due to the highly multiplexed
characteristic of its measurements. However, this alone is a weak form of
security, as various adversarial attacks can be designed to invert the
one-to-many scene mapping of such cameras. In this work, we enhance the privacy
provided by lensless imaging by (1) downsampling at the sensor and (2) using a
programmable mask with variable patterns as our optical encoder. We build a
prototype from a low-cost LCD and Raspberry Pi components, for a total cost of
around 100 USD. This very low price point allows our system to be deployed and
leveraged in a broad range of applications. In our experiments, we first
demonstrate the viability and reconfigurability of our system by applying it to
various classification tasks: MNIST, CelebA (face attributes), and CIFAR10. By
jointly optimizing the mask pattern and a digital classifier in an end-to-end
fashion, low-dimensional, privacy-enhancing embeddings are learned directly at
the sensor. Secondly, we show how the proposed system, through variable mask
patterns, can thwart adversaries that attempt to invert the system (1) via
plaintext attacks or (2) in the event of camera parameters leaks. We
demonstrate the defense of our system to both risks, with 55% and 26% drops in
image quality metrics for attacks based on model-based convex optimization and
generative neural networks respectively. We open-source a wave propagation and
camera simulator needed for end-to-end optimization, the training software, and
a library for interfacing with the camera.


http://arxiv.org/abs/2211.13345
Principled Data-Driven Decision Support for Cyber-Forensic Investigations. (1%)
Soodeh Atefi; Sakshyam Panda; Manos Panaousis; Aron Laszka
  In the wake of a cybersecurity incident, it is crucial to promptly discover
how the threat actors breached security in order to assess the impact of the
incident and to develop and deploy countermeasures that can protect against
further attacks. To this end, defenders can launch a cyber-forensic
investigation, which discovers the techniques that the threat actors used in
the incident. A fundamental challenge in such an investigation is prioritizing
the investigation of particular techniques since the investigation of each
technique requires time and effort, but forensic analysts cannot know which
ones were actually used before investigating them. To ensure prompt discovery,
it is imperative to provide decision support that can help forensic analysts
with this prioritization. A recent study demonstrated that data-driven decision
support, based on a dataset of prior incidents, can provide state-of-the-art
prioritization. However, this data-driven approach, called DISCLOSE, is based
on a heuristic that utilizes only a subset of the available information and
does not approximate optimal decisions. To improve upon this heuristic, we
introduce a principled approach for data-driven decision support for
cyber-forensic investigations. We formulate the decision-support problem using
a Markov decision process, whose states represent the states of a forensic
investigation. To solve the decision problem, we propose a Monte Carlo tree
search based method, which relies on a k-NN regression over prior incidents to
estimate state-transition probabilities. We evaluate our proposed approach on
multiple versions of the MITRE ATT&CK dataset, which is a knowledge base of
adversarial techniques and tactics based on real-world cyber incidents, and
demonstrate that our approach outperforms DISCLOSE in terms of techniques
discovered per effort spent.


http://arxiv.org/abs/2211.13416
Data Provenance Inference in Machine Learning. (1%)
Mingxue Xu; Xiang-Yang Li
  Unintended memorization of various information granularity has garnered
academic attention in recent years, e.g. membership inference and property
inference. How to inversely use this privacy leakage to facilitate real-world
applications is a growing direction; the current efforts include dataset
ownership inference and user auditing. Standing on the data lifecycle and ML
model production, we propose an inference process named Data Provenance
Inference, which is to infer the generation, collection or processing property
of the ML training data, to assist ML developers in locating the training data
gaps without maintaining strenuous metadata. We formularly define the data
provenance and the data provenance inference task in ML training. Then we
propose a novel inference strategy combining embedded-space multiple instance
classification and shadow learning. Comprehensive evaluations cover language,
visual and structured data in black-box and white-box settings, with diverse
kinds of data provenance (i.e. business, county, movie, user). Our best
inference accuracy achieves 98.96% in the white-box text model when "author" is
the data provenance. The experimental results indicate that, in general, the
inference performance positively correlated with the amount of reference data
for inference, the depth and also the amount of the parameter of the accessed
layer. Furthermore, we give a post-hoc statistical analysis of the data
provenance definition to explain when our proposed method works well.


http://arxiv.org/abs/2211.12681
Benchmarking Adversarially Robust Quantum Machine Learning at Scale. (99%)
Maxwell T. West; Sarah M. Erfani; Christopher Leckie; Martin Sevior; Lloyd C. L. Hollenberg; Muhammad Usman
  Machine learning (ML) methods such as artificial neural networks are rapidly
becoming ubiquitous in modern science, technology and industry. Despite their
accuracy and sophistication, neural networks can be easily fooled by carefully
designed malicious inputs known as adversarial attacks. While such
vulnerabilities remain a serious challenge for classical neural networks, the
extent of their existence is not fully understood in the quantum ML setting. In
this work, we benchmark the robustness of quantum ML networks, such as quantum
variational classifiers (QVC), at scale by performing rigorous training for
both simple and complex image datasets and through a variety of high-end
adversarial attacks. Our results show that QVCs offer a notably enhanced
robustness against classical adversarial attacks by learning features which are
not detected by the classical neural networks, indicating a possible quantum
advantage for ML tasks. Contrarily, and remarkably, the converse is not true,
with attacks on quantum networks also capable of deceiving classical neural
networks. By combining quantum and classical network outcomes, we propose a
novel adversarial attack detection technology. Traditionally quantum advantage
in ML systems has been sought through increased accuracy or algorithmic
speed-up, but our work has revealed the potential for a new kind of quantum
advantage through superior robustness of ML models, whose practical realisation
will address serious security concerns and reliability issues of ML algorithms
employed in a myriad of applications including autonomous vehicles,
cybersecurity, and surveillance robotic systems.


http://arxiv.org/abs/2211.12294
PointCA: Evaluating the Robustness of 3D Point Cloud Completion Models Against Adversarial Examples. (99%)
Shengshan Hu; Junwei Zhang; Wei Liu; Junhui Hou; Minghui Li; Leo Yu Zhang; Hai Jin; Lichao Sun
  Point cloud completion, as the upstream procedure of 3D recognition and
segmentation, has become an essential part of many tasks such as navigation and
scene understanding. While various point cloud completion models have
demonstrated their powerful capabilities, their robustness against adversarial
attacks, which have been proven to be fatally malicious towards deep neural
networks, remains unknown. In addition, existing attack approaches towards
point cloud classifiers cannot be applied to the completion models due to
different output forms and attack purposes. In order to evaluate the robustness
of the completion models, we propose PointCA, the first adversarial attack
against 3D point cloud completion models. PointCA can generate adversarial
point clouds that maintain high similarity with the original ones, while being
completed as another object with totally different semantic information.
Specifically, we minimize the representation discrepancy between the
adversarial example and the target point set to jointly explore the adversarial
point clouds in the geometry space and the feature space. Furthermore, to
launch a stealthier attack, we innovatively employ the neighbourhood density
information to tailor the perturbation constraint, leading to geometry-aware
and distribution-adaptive modifications for each point. Extensive experiments
against different premier point cloud completion networks show that PointCA can
cause a performance degradation from 77.9% to 16.7%, with the structure chamfer
distance kept below 0.01. We conclude that existing completion models are
severely vulnerable to adversarial examples, and state-of-the-art defenses for
point cloud classification will be partially invalid when applied to incomplete
and uneven point cloud data.


http://arxiv.org/abs/2211.12314
Attacking Image Splicing Detection and Localization Algorithms Using Synthetic Traces. (98%)
Shengbang Fang; Matthew C Stamm
  Recent advances in deep learning have enabled forensics researchers to
develop a new class of image splicing detection and localization algorithms.
These algorithms identify spliced content by detecting localized
inconsistencies in forensic traces using Siamese neural networks, either
explicitly during analysis or implicitly during training. At the same time,
deep learning has enabled new forms of anti-forensic attacks, such as
adversarial examples and generative adversarial network (GAN) based attacks.
Thus far, however, no anti-forensic attack has been demonstrated against image
splicing detection and localization algorithms. In this paper, we propose a new
GAN-based anti-forensic attack that is able to fool state-of-the-art splicing
detection and localization algorithms such as EXIF-Net, Noiseprint, and
Forensic Similarity Graphs. This attack operates by adversarially training an
anti-forensic generator against a set of Siamese neural networks so that it is
able to create synthetic forensic traces. Under analysis, these synthetic
traces appear authentic and are self-consistent throughout an image. Through a
series of experiments, we demonstrate that our attack is capable of fooling
forensic splicing detection and localization algorithms without introducing
visually detectable artifacts into an attacked image. Additionally, we
demonstrate that our attack outperforms existing alternative attack approaches.
%


http://arxiv.org/abs/2211.12624
Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization. (70%)
Zifan Wang; Nan Ding; Tomer Levinboim; Xi Chen; Radu Soricut
  Recent research in robust optimization has shown an overfitting-like
phenomenon in which models trained against adversarial attacks exhibit higher
robustness on the training set compared to the test set. Although previous work
provided theoretical explanations for this phenomenon using a robust
PAC-Bayesian bound over the adversarial test error, related algorithmic
derivations are at best only loosely connected to this bound, which implies
that there is still a gap between their empirical success and our understanding
of adversarial robustness theory. To close this gap, in this paper we consider
a different form of the robust PAC-Bayesian bound and directly minimize it with
respect to the model posterior. The derivation of the optimal solution connects
PAC-Bayesian learning to the geometry of the robust loss surface through a
Trace of Hessian (TrH) regularizer that measures the surface flatness. In
practice, we restrict the TrH regularizer to the top layer only, which results
in an analytical solution to the bound whose computational cost does not depend
on the network depth. Finally, we evaluate our TrH regularization approach over
CIFAR-10/100 and ImageNet using Vision Transformers (ViT) and compare against
baseline adversarial robustness algorithms. Experimental results show that TrH
regularization leads to improved ViT robustness that either matches or
surpasses previous state-of-the-art approaches while at the same time requires
less memory and computational cost.


http://arxiv.org/abs/2211.12044
Backdoor Cleansing with Unlabeled Data. (70%)
Lu Pang; Tao Sun; Haibin Ling; Chao Chen
  Due to the increasing computational demand of Deep Neural Networks (DNNs),
companies and organizations have begun to outsource the training process.
However, the externally trained DNNs can potentially be backdoor attacked. It
is crucial to defend against such attacks, i.e., to postprocess a suspicious
model so that its backdoor behavior is mitigated while its normal prediction
power on clean inputs remain uncompromised. To remove the abnormal backdoor
behavior, existing methods mostly rely on additional labeled clean samples.
However, such requirement may be unrealistic as the training data are often
unavailable to end users. In this paper, we investigate the possibility of
circumventing such barrier. We propose a novel defense method that does not
require training labels. Through a carefully designed layer-wise weight
re-initialization and knowledge distillation, our method can effectively
cleanse backdoor behaviors of a suspicious network {with negligible compromise
in} its normal behavior. In experiments, we show that our method, trained
without labels, is on-par with state-of-the-art defense methods trained using
labels. We also observe promising defense results even on out-of-distribution
data. This makes our method very practical.


http://arxiv.org/abs/2211.12087
SoK: Inference Attacks and Defenses in Human-Centered Wireless Sensing. (69%)
Wei Sun; Tingjun Chen; Neil Gong
  Human-centered wireless sensing aims to understand the fine-grained
environment and activities of a human using the diverse wireless signals around
her. The wireless sensing community has demonstrated the superiority of such
techniques in many applications such as smart homes, human-computer
interactions, and smart cities. Like many other technologies, wireless sensing
is also a double-edged sword. While the sensed information about a human can be
used for many good purposes such as enhancing life quality, an adversary can
also abuse it to steal private information about the human (e.g., location,
living habits, and behavioral biometric characteristics). However, the
literature lacks a systematic understanding of the privacy vulnerabilities of
wireless sensing and the defenses against them.
  In this work, we aim to bridge this gap. First, we propose a framework to
systematize wireless sensing-based inference attacks. Our framework consists of
three key steps: deploying a sniffing device, sniffing wireless signals, and
inferring private information. Our framework can be used to guide the design of
new inference attacks since different attacks can instantiate these three steps
differently. Second, we propose a defense-in-depth framework to systematize
defenses against such inference attacks. The prevention component of our
framework aims to prevent inference attacks via obfuscating the wireless
signals around a human, while the detection component aims to detect and
respond to attacks. Third, based on our attack and defense frameworks, we
identify gaps in the existing literature and discuss future research
directions.


http://arxiv.org/abs/2211.12005
Self-Ensemble Protection: Training Checkpoints Are Good Data Protectors. (99%)
Sizhe Chen; Geng Yuan; Xinwen Cheng; Yifan Gong; Minghai Qin; Yanzhi Wang; Xiaolin Huang
  As data become increasingly vital for deep learning, a company would be very
cautious about releasing data, because the competitors could use the released
data to train high-performance models, thereby posing a tremendous threat to
the company's commercial competence. To prevent training good models on the
data, imperceptible perturbations could be added to it. Since such
perturbations aim at hurting the entire training process, they should reflect
the vulnerability of DNN training, rather than that of a single model. Based on
this new idea, we seek adversarial examples that are always unrecognized (never
correctly classified) in training. In this paper, we uncover them by modeling
checkpoints' gradients, forming the proposed self-ensemble protection (SEP),
which is very effective because (1) learning on examples ignored during normal
training tends to yield DNNs ignoring normal examples; (2) checkpoints'
cross-model gradients are close to orthogonal, meaning that they are as diverse
as DNNs with different architectures in conventional ensemble. That is, our
amazing performance of ensemble only requires the computation of training one
model. By extensive experiments with 9 baselines on 3 datasets and 5
architectures, SEP is verified to be a new state-of-the-art, e.g., our small
$\ell_\infty=2/255$ perturbations reduce the accuracy of a CIFAR-10 ResNet18
from 94.56\% to 14.68\%, compared to 41.35\% by the best-known method.Code is
available at https://github.com/Sizhe-Chen/SEP.


http://arxiv.org/abs/2211.11236
Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization. (99%)
Jiafeng Wang; Zhaoyu Chen; Kaixun Jiang; Dingkang Yang; Lingyi Hong; Yan Wang; Wenqiang Zhang
  Deep neural networks are vulnerable to adversarial examples, which attach
human invisible perturbations to benign inputs. Simultaneously, adversarial
examples exhibit transferability under different models, which makes practical
black-box attacks feasible. However, existing methods are still incapable of
achieving desired transfer attack performance. In this work, from the
perspective of gradient optimization and consistency, we analyze and discover
the gradient elimination phenomenon as well as the local momentum optimum
dilemma. To tackle these issues, we propose Global Momentum Initialization (GI)
to suppress gradient elimination and help search for the global optimum.
Specifically, we perform gradient pre-convergence before the attack and carry
out a global search during the pre-convergence stage. Our method can be easily
combined with almost all existing transfer methods, and we improve the success
rate of transfer attacks significantly by an average of 6.4% under various
advanced defense mechanisms compared to state-of-the-art methods. Eventually,
we achieve an attack success rate of 95.4%, fully illustrating the insecurity
of existing defense mechanisms.


http://arxiv.org/abs/2211.11312
Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack. (99%)
Yunfeng Diao; He Wang; Tianjia Shao; Yong-Liang Yang; Kun Zhou; David Hogg
  Human Activity Recognition (HAR) has been employed in a wide range of
applications, e.g. self-driving cars, where safety and lives are at stake.
Recently, the robustness of existing skeleton-based HAR methods has been
questioned due to their vulnerability to adversarial attacks, which causes
concerns considering the scale of the implication. However, the proposed
attacks require the full-knowledge of the attacked classifier, which is overly
restrictive. In this paper, we show such threats indeed exist, even when the
attacker only has access to the input/output of the model. To this end, we
propose the very first black-box adversarial attack approach in skeleton-based
HAR called BASAR. BASAR explores the interplay between the classification
boundary and the natural motion manifold. To our best knowledge, this is the
first time data manifold is introduced in adversarial attacks on time series.
Via BASAR, we find on-manifold adversarial samples are extremely deceitful and
rather common in skeletal motions, in contrast to the common belief that
adversarial samples only exist off-manifold. Through exhaustive evaluation, we
show that BASAR can deliver successful attacks across classifiers, datasets,
and attack modes. By attack, BASAR helps identify the potential causes of the
model vulnerability and provides insights on possible improvements. Finally, to
mitigate the newly identified threat, we propose a new adversarial training
approach by leveraging the sophisticated distributions of on/off-manifold
adversarial samples, called mixed manifold-based adversarial training (MMAT).
MMAT can successfully help defend against adversarial attacks without
compromising classification accuracy.


http://arxiv.org/abs/2211.11880
Addressing Mistake Severity in Neural Networks with Semantic Knowledge. (92%)
Natalie Abreu; Nathan Vaska; Victoria Helus
  Robustness in deep neural networks and machine learning algorithms in general
is an open research challenge. In particular, it is difficult to ensure
algorithmic performance is maintained on out-of-distribution inputs or
anomalous instances that cannot be anticipated at training time. Embodied
agents will be deployed in these conditions, and are likely to make incorrect
predictions. An agent will be viewed as untrustworthy unless it can maintain
its performance in dynamic environments. Most robust training techniques aim to
improve model accuracy on perturbed inputs; as an alternate form of robustness,
we aim to reduce the severity of mistakes made by neural networks in
challenging conditions. We leverage current adversarial training methods to
generate targeted adversarial attacks during the training process in order to
increase the semantic similarity between a model's predictions and true labels
of misclassified instances. Results demonstrate that our approach performs
better with respect to mistake severity compared to standard and adversarially
trained models. We also find an intriguing role that non-robust features play
with regards to semantic similarity.


http://arxiv.org/abs/2211.11489
Efficient Generalization Improvement Guided by Random Weight Perturbation. (68%)
Tao Li; Weihao Yan; Zehao Lei; Yingwen Wu; Kun Fang; Ming Yang; Xiaolin Huang
  To fully uncover the great potential of deep neural networks (DNNs), various
learning algorithms have been developed to improve the model's generalization
ability. Recently, sharpness-aware minimization (SAM) establishes a generic
scheme for generalization improvements by minimizing the sharpness measure
within a small neighborhood and achieves state-of-the-art performance. However,
SAM requires two consecutive gradient evaluations for solving the min-max
problem and inevitably doubles the training time. In this paper, we resort to
filter-wise random weight perturbations (RWP) to decouple the nested gradients
in SAM. Different from the small adversarial perturbations in SAM, RWP is
softer and allows a much larger magnitude of perturbations. Specifically, we
jointly optimize the loss function with random perturbations and the original
loss function: the former guides the network towards a wider flat region while
the latter helps recover the necessary local information. These two loss terms
are complementary to each other and mutually independent. Hence, the
corresponding gradients can be efficiently computed in parallel, enabling
nearly the same training speed as regular training. As a result, we achieve
very competitive performance on CIFAR and remarkably better performance on
ImageNet (e.g. $\mathbf{ +1.1\%}$) compared with SAM, but always require half
of the training time. The code is released at https://github.com/nblt/RWP.


http://arxiv.org/abs/2211.11711
CLAWSAT: Towards Both Robust and Accurate Code Models. (56%)
Jinghan Jia; Shashank Srikant; Tamara Mitrovska; Chuang Gan; Shiyu Chang; Sijia Liu; Una-May O'Reilly
  We integrate contrastive learning (CL) with adversarial learning to
co-optimize the robustness and accuracy of code models. Different from existing
works, we show that code obfuscation, a standard code transformation operation,
provides novel means to generate complementary `views' of a code that enable us
to achieve both robust and accurate code models. To the best of our knowledge,
this is the first systematic study to explore and exploit the robustness and
accuracy benefits of (multi-view) code obfuscations in code models.
Specifically, we first adopt adversarial codes as robustness-promoting views in
CL at the self-supervised pre-training phase. This yields improved robustness
and transferability for downstream tasks. Next, at the supervised fine-tuning
stage, we show that adversarial training with a proper temporally-staggered
schedule of adversarial code generation can further improve robustness and
accuracy of the pre-trained code model. Built on the above two modules, we
develop CLAWSAT, a novel self-supervised learning (SSL) framework for code by
integrating $\underline{\textrm{CL}}$ with $\underline{\textrm{a}}$dversarial
vie$\underline{\textrm{w}}$s (CLAW) with $\underline{\textrm{s}}$taggered
$\underline{\textrm{a}}$dversarial $\underline{\textrm{t}}$raining (SAT). On
evaluating three downstream tasks across Python and Java, we show that CLAWSAT
consistently yields the best robustness and accuracy ($\textit{e.g.}$ 11$\%$ in
robustness and 6$\%$ in accuracy on the code summarization task in Python). We
additionally demonstrate the effectiveness of adversarial learning in CLAW by
analyzing the characteristics of the loss landscape and interpretability of the
pre-trained models.


http://arxiv.org/abs/2211.11835
Fairness Increases Adversarial Vulnerability. (54%)
Cuong Tran; Keyu Zhu; Ferdinando Fioretto; Henternyck Pascal Van
  The remarkable performance of deep learning models and their applications in
consequential domains (e.g., facial recognition) introduces important
challenges at the intersection of equity and security. Fairness and robustness
are two desired notions often required in learning models. Fairness ensures
that models do not disproportionately harm (or benefit) some groups over
others, while robustness measures the models' resilience against small input
perturbations.
  This paper shows the existence of a dichotomy between fairness and
robustness, and analyzes when achieving fairness decreases the model robustness
to adversarial samples. The reported analysis sheds light on the factors
causing such contrasting behavior, suggesting that distance to the decision
boundary across groups as a key explainer for this behavior. Extensive
experiments on non-linear models and different architectures validate the
theoretical findings in multiple vision domains. Finally, the paper proposes a
simple, yet effective, solution to construct models achieving good tradeoffs
between fairness and robustness.


http://arxiv.org/abs/2211.14440
Don't Watch Me: A Spatio-Temporal Trojan Attack on Deep-Reinforcement-Learning-Augment Autonomous Driving. (10%)
Yinbo Yu; Jiajia Liu
  Deep reinforcement learning (DRL) is one of the most popular algorithms to
realize an autonomous driving (AD) system. The key success factor of DRL is
that it embraces the perception capability of deep neural networks which,
however, have been proven vulnerable to Trojan attacks. Trojan attacks have
been widely explored in supervised learning (SL) tasks (e.g., image
classification), but rarely in sequential decision-making tasks solved by DRL.
Hence, in this paper, we explore Trojan attacks on DRL for AD tasks. First, we
propose a spatio-temporal DRL algorithm based on the recurrent neural network
and attention mechanism to prove that capturing spatio-temporal traffic
features is the key factor to the effectiveness and safety of a DRL-augment AD
system. We then design a spatial-temporal Trojan attack on DRL policies, where
the trigger is hidden in a sequence of spatial and temporal traffic features,
rather than a single instant state used in existing Trojan on SL and DRL tasks.
With our Trojan, the adversary acts as a surrounding normal vehicle and can
trigger attacks via specific spatial-temporal driving behaviors, rather than
physical or wireless access. Through extensive experiments, we show that while
capturing spatio-temporal traffic features can improve the performance of DRL
for different AD tasks, they suffer from Trojan attacks since our designed
Trojan shows high stealthy (various spatio-temporal trigger patterns),
effective (less than 3.1\% performance variance rate and more than 98.5\%
attack success rate), and sustainable to existing advanced defenses.


http://arxiv.org/abs/2211.11321
SPIN: Simulated Poisoning and Inversion Network for Federated Learning-Based 6G Vehicular Networks. (8%)
Sunder Ali Khowaja; Parus Khuwaja; Kapal Dev; Angelos Antonopoulos
  The applications concerning vehicular networks benefit from the vision of
beyond 5G and 6G technologies such as ultra-dense network topologies, low
latency, and high data rates. Vehicular networks have always faced data privacy
preservation concerns, which lead to the advent of distributed learning
techniques such as federated learning. Although federated learning has solved
data privacy preservation issues to some extent, the technique is quite
vulnerable to model inversion and model poisoning attacks. We assume that the
design of defense mechanism and attacks are two sides of the same coin.
Designing a method to reduce vulnerability requires the attack to be effective
and challenging with real-world implications. In this work, we propose
simulated poisoning and inversion network (SPIN) that leverages the
optimization approach for reconstructing data from a differential model trained
by a vehicular node and intercepted when transmitted to roadside unit (RSU). We
then train a generative adversarial network (GAN) to improve the generation of
data with each passing round and global update from the RSU, accordingly.
Evaluation results show the qualitative and quantitative effectiveness of the
proposed approach. The attack initiated by SPIN can reduce up to 22% accuracy
on publicly available datasets while just using a single attacker. We assume
that revealing the simulation of such attacks would help us find its defense
mechanism in an effective manner.


http://arxiv.org/abs/2211.11635
Understanding and Improving Visual Prompting: A Label-Mapping Perspective. (2%)
Aochuan Chen; Yuguang Yao; Pin-Yu Chen; Yihua Zhang; Sijia Liu
  We revisit and advance visual prompting (VP), an input prompting technique
for vision tasks. VP can reprogram a fixed, pre-trained source model to
accomplish downstream tasks in the target domain by simply incorporating
universal prompts (in terms of input perturbation patterns) into downstream
data points. Yet, it remains elusive why VP stays effective even given a
ruleless label mapping (LM) between the source classes and the target classes.
Inspired by the above, we ask: How is LM interrelated with VP? And how to
exploit such a relationship to improve its accuracy on target tasks? We peer
into the influence of LM on VP and provide an affirmative answer that a better
'quality' of LM (assessed by mapping precision and explanation) can
consistently improve the effectiveness of VP. This is in contrast to the prior
art where the factor of LM was missing. To optimize LM, we propose a new VP
framework, termed ILM-VP (iterative label mapping-based visual prompting),
which automatically re-maps the source labels to the target labels and
progressively improves the target task accuracy of VP. Further, when using a
contrastive language-image pretrained (CLIP) model, we propose to integrate an
LM process to assist the text prompt selection of CLIP and to improve the
target task accuracy. Extensive experiments demonstrate that our proposal
significantly outperforms state-of-the-art VP methods. As highlighted below, we
show that when reprogramming an ImageNet-pretrained ResNet-18 to 13 target
tasks, our method outperforms baselines by a substantial margin, e.g., 7.9% and
6.7% accuracy improvements in transfer learning to the target Flowers102 and
CIFAR100 datasets. Besides, our proposal on CLIP-based VP provides 13.7% and
7.1% accuracy improvements on Flowers102 and DTD respectively. Our code is
available at https://github.com/OPTML-Group/ILM-VP.


http://arxiv.org/abs/2211.11958
A Survey on Backdoor Attack and Defense in Natural Language Processing. (2%)
Xuan Sheng; Zhaoyang Han; Piji Li; Xiangmao Chang
  Deep learning is becoming increasingly popular in real-life applications,
especially in natural language processing (NLP). Users often choose training
outsourcing or adopt third-party data and models due to data and computation
resources being limited. In such a situation, training data and models are
exposed to the public. As a result, attackers can manipulate the training
process to inject some triggers into the model, which is called backdoor
attack. Backdoor attack is quite stealthy and difficult to be detected because
it has little inferior influence on the model's performance for the clean
samples. To get a precise grasp and understanding of this problem, in this
paper, we conduct a comprehensive review of backdoor attacks and defenses in
the field of NLP. Besides, we summarize benchmark datasets and point out the
open issues to design credible systems to defend against backdoor attacks.


http://arxiv.org/abs/2211.11434
Privacy in Practice: Private COVID-19 Detection in X-Ray Images. (1%)
Lucas Lange; Maja Schneider; Erhard Rahm
  Machine learning (ML) can help fight the COVID-19 pandemic by enabling rapid
screening of large volumes of chest X-ray images. To perform such data analysis
while maintaining patient privacy, we create ML models that satisfy
Differential Privacy (DP). Previous works exploring private COVID-19 ML models
are in part based on small or skewed datasets, are lacking in their privacy
guarantees, and do not investigate practical privacy. In this work, we
therefore suggest several improvements to address these open gaps. We account
for inherent class imbalances in the data and evaluate the utility-privacy
trade-off more extensively and over stricter privacy budgets than in previous
work. Our evaluation is supported by empirically estimating practical privacy
leakage through actual attacks. Based on theory, the introduced DP should help
limit and mitigate information leakage threats posed by black-box Membership
Inference Attacks (MIAs). Our practical privacy analysis is the first to test
this hypothesis on the COVID-19 detection task. In addition, we also re-examine
the evaluation on the MNIST database. Our results indicate that based on the
task-dependent threat from MIAs, DP does not always improve practical privacy,
which we show on the COVID-19 task. The results further suggest that with
increasing DP guarantees, empirical privacy leakage reaches an early plateau
and DP therefore appears to have a limited impact on MIA defense. Our findings
identify possibilities for better utility-privacy trade-offs, and we thus
believe that empirical attack-specific privacy estimation can play a vital role
in tuning for practical privacy.


http://arxiv.org/abs/2211.11357
A Tale of Frozen Clouds: Quantifying the Impact of Algorithmic Complexity Vulnerabilities in Popular Web Servers. (1%)
Masudul Hasan Masud Bhuiyan; Cristian-Alexandru Staicu
  Algorithmic complexity vulnerabilities are a class of security problems that
enables attackers to trigger the worst-case complexity of certain algorithms.
Such vulnerabilities can be leveraged to deploy low-volume, asymmetric,
CPU-based denial-of-service (DoS) attacks. Previous work speculates that these
vulnerabilities are more dangerous in certain web servers, like Node.js, than
in traditional ones, like Apache. We believe it is of utmost importance to
understand if this is indeed the case or if there are ways to compensate
against such problems using various deployment strategies. To this end, we
study the resilience of popular web servers against CPU-based DoS attacks in
four major cloud platforms under realistic deployment conditions. We find that
there are indeed significant differences in how various web servers react to an
attack. However, our results suggest a more nuanced landscape than previously
believed: while event-based systems tend to recover faster from DoS in certain
scenarios, they also suffer the worst performance degradation overall.
Nevertheless, in some setups, Apache performs worse than event-based systems,
and there are cloud platforms in which all the considered servers are seriously
exposed to the attack. We also find that developers can harden their servers
against CPU-based DoS attacks by increasing the number of server instances
running in parallel. This, in turn, can lead to an increased cost of operation
or a slight degradation of performance in non-DoS conditions.


http://arxiv.org/abs/2211.10896
Spectral Adversarial Training for Robust Graph Neural Network. (99%)
Jintang Li; Jiaying Peng; Liang Chen; Zibin Zheng; Tingting Liang; Qing Ling
  Recent studies demonstrate that Graph Neural Networks (GNNs) are vulnerable
to slight but adversarially designed perturbations, known as adversarial
examples. To address this issue, robust training methods against adversarial
examples have received considerable attention in the literature.
\emph{Adversarial Training (AT)} is a successful approach to learning a robust
model using adversarially perturbed training samples. Existing AT methods on
GNNs typically construct adversarial perturbations in terms of graph structures
or node features. However, they are less effective and fraught with challenges
on graph data due to the discreteness of graph structure and the relationships
between connected examples. In this work, we seek to address these challenges
and propose Spectral Adversarial Training (SAT), a simple yet effective
adversarial training approach for GNNs. SAT first adopts a low-rank
approximation of the graph structure based on spectral decomposition, and then
constructs adversarial perturbations in the spectral domain rather than
directly manipulating the original graph structure. To investigate its
effectiveness, we employ SAT on three widely used GNNs. Experimental results on
four public graph datasets demonstrate that SAT significantly improves the
robustness of GNNs against adversarial attacks without sacrificing
classification accuracy and training efficiency.


http://arxiv.org/abs/2211.10933
Invisible Backdoor Attack with Dynamic Triggers against Person Re-identification. (81%)
Wenli Sun; Xinyang Jiang; Shuguang Dou; Dongsheng Li; Duoqian Miao; Cheng Deng; Cairong Zhao
  In recent years, person Re-identification (ReID) has rapidly progressed with
wide real-world applications, but also poses significant risks of adversarial
attacks. In this paper, we focus on the backdoor attack on deep ReID models.
Existing backdoor attack methods follow an all-to-one/all attack scenario,
where all the target classes in the test set have already been seen in the
training set. However, ReID is a much more complex fine-grained open-set
recognition problem, where the identities in the test set are not contained in
the training set. Thus, previous backdoor attack methods for classification are
not applicable for ReID. To ameliorate this issue, we propose a novel backdoor
attack on deep ReID under a new all-to-unknown scenario, called Dynamic
Triggers Invisible Backdoor Attack (DT-IBA). Instead of learning fixed triggers
for the target classes from the training set, DT-IBA can dynamically generate
new triggers for any unknown identities. Specifically, an identity hashing
network is proposed to first extract target identity information from a
reference image, which is then injected into the benign images by image
steganography. We extensively validate the effectiveness and stealthiness of
the proposed attack on benchmark datasets, and evaluate the effectiveness of
several defense methods against our attack.


http://arxiv.org/abs/2211.11030
Adversarial Cheap Talk. (4%)
Chris Lu; Timon Willi; Alistair Letcher; Jakob Foerster
  Adversarial attacks in reinforcement learning (RL) often assume
highly-privileged access to the victim's parameters, environment, or data.
Instead, this paper proposes a novel adversarial setting called a Cheap Talk
MDP in which an Adversary can merely append deterministic messages to the
Victim's observation, resulting in a minimal range of influence. The Adversary
cannot occlude ground truth, influence underlying environment dynamics or
reward signals, introduce non-stationarity, add stochasticity, see the Victim's
actions, or access their parameters. Additionally, we present a simple
meta-learning algorithm called Adversarial Cheap Talk (ACT) to train
Adversaries in this setting. We demonstrate that an Adversary trained with ACT
can still significantly influence the Victim's training and testing
performance, despite the highly constrained setting. Affecting train-time
performance reveals a new attack vector and provides insight into the success
and failure modes of existing RL algorithms. More specifically, we show that an
ACT Adversary is capable of harming performance by interfering with the
learner's function approximation, or instead helping the Victim's performance
by outputting useful features. Finally, we show that an ACT Adversary can
manipulate messages during train-time to directly and arbitrarily control the
Victim at test-time.


http://arxiv.org/abs/2211.11039
Deep Composite Face Image Attacks: Generation, Vulnerability and Detection. (2%)
Jag Mohan Singh; Raghavendra Ramachandra
  Face manipulation attacks have drawn the attention of biometric researchers
because of their vulnerability to Face Recognition Systems (FRS). This paper
proposes a novel scheme to generate Composite Face Image Attacks (CFIA) based
on the Generative Adversarial Networks (GANs). Given the face images from
contributory data subjects, the proposed CFIA method will independently
generate the segmented facial attributes, then blend them using transparent
masks to generate the CFIA samples. { The primary motivation for CFIA is to
utilize deep learning to generate facial attribute-based composite attacks,
which has been explored relatively less in the current literature.} We generate
$14$ different combinations of facial attributes resulting in $14$ unique CFIA
samples for each pair of contributory data subjects. Extensive experiments are
carried out on our newly generated CFIA dataset consisting of 1000 unique
identities with 2000 bona fide samples and 14000 CFIA samples, thus resulting
in an overall 16000 face image samples. We perform a sequence of experiments to
benchmark the vulnerability of CFIA to automatic FRS (based on both
deep-learning and commercial-off-the-shelf (COTS). We introduced a new metric
named Generalized Morphing Attack Potential (GMAP) to benchmark the
vulnerability effectively. Additional experiments are performed to compute the
perceptual quality of the generated CFIA samples. Finally, the CFIA detection
performance is presented using three different Face Morphing Attack Detection
(MAD) algorithms. The proposed CFIA method indicates good perceptual quality
based on the obtained results. Further, { FRS is vulnerable to CFIA} (much
higher than SOTA), making it difficult to detect by human observers and
automatic detection algorithms. Lastly, we performed experiments to detect the
CFIA samples using three different detection techniques automatically.


http://arxiv.org/abs/2211.10938
AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation. (2%)
Hyungmin Kim; Sungho Suh; Sunghyun Baek; Daehwan Kim; Daun Jeong; Hansang Cho; Junmo Kim
  We present a novel adversarial penalized self-knowledge distillation method,
named adversarial learning and implicit regularization for self-knowledge
distillation (AI-KD), which regularizes the training procedure by adversarial
learning and implicit distillations. Our model not only distills the
deterministic and progressive knowledge which are from the pre-trained and
previous epoch predictive probabilities but also transfers the knowledge of the
deterministic predictive distributions using adversarial learning. The
motivation is that the self-knowledge distillation methods regularize the
predictive probabilities with soft targets, but the exact distributions may be
hard to predict. Our method deploys a discriminator to distinguish the
distributions between the pre-trained and student models while the student
model is trained to fool the discriminator in the trained procedure. Thus, the
student model not only can learn the pre-trained model's predictive
probabilities but also align the distributions between the pre-trained and
student models. We demonstrate the effectiveness of the proposed method with
network architectures on multiple datasets and show the proposed method
achieves better performance than state-of-the-art methods.


http://arxiv.org/abs/2211.10670
Towards Adversarial Robustness of Deep Vision Algorithms. (92%)
Hanshu Yan
  Deep learning methods have achieved great success in solving computer vision
tasks, and they have been widely utilized in artificially intelligent systems
for image processing, analysis, and understanding. However, deep neural
networks have been shown to be vulnerable to adversarial perturbations in input
data. The security issues of deep neural networks have thus come to the fore.
It is imperative to study the adversarial robustness of deep vision algorithms
comprehensively. This talk focuses on the adversarial robustness of image
classification models and image denoisers. We will discuss the robustness of
deep vision algorithms from three perspectives: 1) robustness evaluation (we
propose the ObsAtk to evaluate the robustness of denoisers), 2) robustness
improvement (HAT, TisODE, and CIFS are developed to robustify vision models),
and 3) the connection between adversarial robustness and generalization
capability to new domains (we find that adversarially robust denoisers can deal
with unseen types of real-world noise).


http://arxiv.org/abs/2211.10661
Phonemic Adversarial Attack against Audio Recognition in Real World. (87%)
Jiakai Wang; Zhendong Chen; Zixin Yin; Qinghong Yang; Xianglong Liu
  Recently, adversarial attacks for audio recognition have attracted much
attention. However, most of the existing studies mainly rely on the
coarse-grain audio features at the instance level to generate adversarial
noises, which leads to expensive generation time costs and weak universal
attacking ability. Motivated by the observations that all audio speech consists
of fundamental phonemes, this paper proposes a phonemic adversarial tack (PAT)
paradigm, which attacks the fine-grain audio features at the phoneme level
commonly shared across audio instances, to generate phonemic adversarial
noises, enjoying the more general attacking ability with fast generation speed.
Specifically, for accelerating the generation, a phoneme density balanced
sampling strategy is introduced to sample quantity less but phonemic features
abundant audio instances as the training data via estimating the phoneme
density, which substantially alleviates the heavy dependency on the large
training dataset. Moreover, for promoting universal attacking ability, the
phonemic noise is optimized in an asynchronous way with a sliding window, which
enhances the phoneme diversity and thus well captures the critical fundamental
phonemic patterns. By conducting extensive experiments, we comprehensively
investigate the proposed PAT framework and demonstrate that it outperforms the
SOTA baselines by large margins (i.e., at least 11X speed up and 78% attacking
ability improvement).


http://arxiv.org/abs/2211.10752
Towards Robust Dataset Learning. (82%)
Yihan Wu; Xinda Li; Florian Kerschbaum; Heng Huang; Hongyang Zhang
  Adversarial training has been actively studied in recent computer vision
research to improve the robustness of models. However, due to the huge
computational cost of generating adversarial samples, adversarial training
methods are often slow. In this paper, we study the problem of learning a
robust dataset such that any classifier naturally trained on the dataset is
adversarially robust. Such a dataset benefits the downstream tasks as natural
training is much faster than adversarial training, and demonstrates that the
desired property of robustness is transferable between models and data. In this
work, we propose a principled, tri-level optimization to formulate the robust
dataset learning problem. We show that, under an abstraction model that
characterizes robust vs. non-robust features, the proposed method provably
learns a robust dataset. Extensive experiments on MNIST, CIFAR10, and
TinyImageNet demostrate the effectiveness of our algorithm with different
network initializations and architectures.


http://arxiv.org/abs/2211.10782
Let Graph be the Go Board: Gradient-free Node Injection Attack for Graph Neural Networks via Reinforcement Learning. (80%)
Mingxuan Ju; Yujie Fan; Chuxu Zhang; Yanfang Ye
  Graph Neural Networks (GNNs) have drawn significant attentions over the years
and been broadly applied to essential applications requiring solid robustness
or vigorous security standards, such as product recommendation and user
behavior modeling. Under these scenarios, exploiting GNN's vulnerabilities and
further downgrading its performance become extremely incentive for adversaries.
Previous attackers mainly focus on structural perturbations or node injections
to the existing graphs, guided by gradients from the surrogate models. Although
they deliver promising results, several limitations still exist. For the
structural perturbation attack, to launch a proposed attack, adversaries need
to manipulate the existing graph topology, which is impractical in most
circumstances. Whereas for the node injection attack, though being more
practical, current approaches require training surrogate models to simulate a
white-box setting, which results in significant performance downgrade when the
surrogate architecture diverges from the actual victim model. To bridge these
gaps, in this paper, we study the problem of black-box node injection attack,
without training a potentially misleading surrogate model. Specifically, we
model the node injection attack as a Markov decision process and propose
Gradient-free Graph Advantage Actor Critic, namely G2A2C, a reinforcement
learning framework in the fashion of advantage actor critic. By directly
querying the victim model, G2A2C learns to inject highly malicious nodes with
extremely limited attacking budgets, while maintaining a similar node feature
distribution. Through our comprehensive experiments over eight acknowledged
benchmark datasets with different characteristics, we demonstrate the superior
performance of our proposed G2A2C over the existing state-of-the-art attackers.
Source code is publicly available at: https://github.com/jumxglhf/G2A2C}.


http://arxiv.org/abs/2211.10843
Mask Off: Analytic-based Malware Detection By Transfer Learning and Model Personalization. (9%)
Amirmohammad Pasdar; Young Choon Lee; Seok-Hee Hong
  The vulnerability of smartphones to cyberattacks has been a severe concern to
users arising from the integrity of installed applications (\textit{apps}).
Although applications are to provide legitimate and diversified on-the-go
services, harmful and dangerous ones have also uncovered the feasible way to
penetrate smartphones for malicious behaviors. Thorough application analysis is
key to revealing malicious intent and providing more insights into the
application behavior for security risk assessments. Such in-depth analysis
motivates employing deep neural networks (DNNs) for a set of features and
patterns extracted from applications to facilitate detecting potentially
dangerous applications independently. This paper presents an Analytic-based
deep neural network, Android Malware detection (ADAM), that employs a
fine-grained set of features to train feature-specific DNNs to have consensus
on the application labels when their ground truth is unknown. In addition, ADAM
leverages the transfer learning technique to obtain its adjustability to new
applications across smartphones for recycling the pre-trained model(s) and
making them more adaptable by model personalization and federated learning
techniques. This adjustability is also assisted by federated learning guards,
which protect ADAM against poisoning attacks through model analysis. ADAM
relies on a diverse dataset containing more than 153000 applications with over
41000 extracted features for DNNs training. The ADAM's feature-specific DNNs,
on average, achieved more than 98% accuracy, resulting in an outstanding
performance against data manipulation attacks.


http://arxiv.org/abs/2211.10033
Adversarial Stimuli: Attacking Brain-Computer Interfaces via Perturbed Sensory Events. (98%)
Bibek Upadhayay; Vahid Behzadan
  Machine learning models are known to be vulnerable to adversarial
perturbations in the input domain, causing incorrect predictions. Inspired by
this phenomenon, we explore the feasibility of manipulating EEG-based Motor
Imagery (MI) Brain Computer Interfaces (BCIs) via perturbations in sensory
stimuli. Similar to adversarial examples, these \emph{adversarial stimuli} aim
to exploit the limitations of the integrated brain-sensor-processing components
of the BCI system in handling shifts in participants' response to changes in
sensory stimuli. This paper proposes adversarial stimuli as an attack vector
against BCIs, and reports the findings of preliminary experiments on the impact
of visual adversarial stimuli on the integrity of EEG-based MI BCIs. Our
findings suggest that minor adversarial stimuli can significantly deteriorate
the performance of MI BCIs across all participants (p=0.0003). Additionally,
our results indicate that such attacks are more effective in conditions with
induced stress.


http://arxiv.org/abs/2211.10227
Adversarial Detection by Approximation of Ensemble Boundary. (75%)
T. Windeatt
  A spectral approximation of a Boolean function is proposed for approximating
the decision boundary of an ensemble of Deep Neural Networks (DNNs) solving
two-class pattern recognition problems. The Walsh combination of relatively
weak DNN classifiers is shown experimentally to be capable of detecting
adversarial attacks. By observing the difference in Walsh coefficient
approximation between clean and adversarial images, it appears that
transferability of attack may be used for detection. Approximating the decision
boundary may also aid in understanding the learning and transferability
properties of DNNs. While the experiments here use images, the proposed
approach of modelling two-class ensemble decision boundaries could in principle
be applied to any application area.


http://arxiv.org/abs/2211.10209
Leveraging Algorithmic Fairness to Mitigate Blackbox Attribute Inference Attacks. (68%)
Jan Aalmoes; Vasisht Duddu; Antoine Boutet
  Machine learning (ML) models have been deployed for high-stakes applications,
e.g., healthcare and criminal justice. Prior work has shown that ML models are
vulnerable to attribute inference attacks where an adversary, with some
background knowledge, trains an ML attack model to infer sensitive attributes
by exploiting distinguishable model predictions. However, some prior attribute
inference attacks have strong assumptions about adversary's background
knowledge (e.g., marginal distribution of sensitive attribute) and pose no more
privacy risk than statistical inference. Moreover, none of the prior attacks
account for class imbalance of sensitive attribute in datasets coming from
real-world applications (e.g., Race and Sex). In this paper, we propose an
practical and effective attribute inference attack that accounts for this
imbalance using an adaptive threshold over the attack model's predictions. We
exhaustively evaluate our proposed attack on multiple datasets and show that
the adaptive threshold over the model's predictions drastically improves the
attack accuracy over prior work. Finally, current literature lacks an effective
defence against attribute inference attacks. We investigate the impact of
fairness constraints (i.e., designed to mitigate unfairness in model
predictions) during model training on our attribute inference attack. We show
that constraint based fairness algorithms which enforces equalized odds acts as
an effective defense against attribute inference attacks without impacting the
model utility. Hence, the objective of algorithmic fairness and sensitive
attribute privacy are aligned.


http://arxiv.org/abs/2211.10370
Invariant Learning via Diffusion Dreamed Distribution Shifts. (10%)
Priyatham Kattakinda; Alexander Levine; Soheil Feizi
  Though the background is an important signal for image classification, over
reliance on it can lead to incorrect predictions when spurious correlations
between foreground and background are broken at test time. Training on a
dataset where these correlations are unbiased would lead to more robust models.
In this paper, we propose such a dataset called Diffusion Dreamed Distribution
Shifts (D3S). D3S consists of synthetic images generated through
StableDiffusion using text prompts and image guides obtained by pasting a
sample foreground image onto a background template image. Using this scalable
approach we generate 120K images of objects from all 1000 ImageNet classes in
10 diverse backgrounds. Due to the incredible photorealism of the diffusion
model, our images are much closer to natural images than previous synthetic
datasets. D3S contains a validation set of more than 17K images whose labels
are human-verified in an MTurk study. Using the validation set, we evaluate
several popular DNN image classifiers and find that the classification
performance of models generally suffers on our background diverse images. Next,
we leverage the foreground & background labels in D3S to learn a foreground
(background) representation that is invariant to changes in background
(foreground) by penalizing the mutual information between the foreground
(background) features and the background (foreground) labels. Linear
classifiers trained on these features to predict foreground (background) from
foreground (background) have high accuracies at 82.9% (93.8%), while
classifiers that predict these labels from background and foreground have a
much lower accuracy of 2.4% and 45.6% respectively. This suggests that our
foreground and background features are well disentangled. We further test the
efficacy of these representations by training classifiers on a task with strong
spurious correlations.


http://arxiv.org/abs/2211.10095
Improving Robustness of TCM-based Robust Steganography with Variable Robustness. (1%)
Jimin Zhang; Xianfeng Zhao; Xiaolei He
  Recent study has found out that after multiple times of recompression, the
DCT coefficients of JPEG image can form an embedding domain that is robust to
recompression, which is called transport channel matching (TCM) method. Because
the cost function of the adaptive steganography does not consider the impact of
modification on the robustness, the modified DCT coefficients of the stego
image after TCM will change after recompression. To reduce the number of
changed coefficients after recompression, this paper proposes a robust
steganography algorithm which dynamically updates the robustness cost of every
DCT coefficient. The robustness cost proposed is calculated by testing whether
the modified DCT coefficient can resist recompression in every step of STC
embedding process. By adding robustness cost to the distortion cost and using
the framework of STC embedding algorithm to embed the message, the stego images
have good performance both in robustness and security. The experimental results
show that the proposed algorithm can significantly enhance the robustness of
stego images, and the embedded messages could be extracted correctly at almost
all cases when recompressing with a lower quality factor and recompression
process is known to the user of proposed algorithm.


http://arxiv.org/abs/2211.10062
Intrusion Detection in Internet of Things using Convolutional Neural Networks. (1%)
Martin Kodys; Zhi Lu; Kar Wai Fok; Vrizlynn L. L. Thing
  Internet of Things (IoT) has become a popular paradigm to fulfil needs of the
industry such as asset tracking, resource monitoring and automation. As
security mechanisms are often neglected during the deployment of IoT devices,
they are more easily attacked by complicated and large volume intrusion attacks
using advanced techniques. Artificial Intelligence (AI) has been used by the
cyber security community in the past decade to automatically identify such
attacks. However, deep learning methods have yet to be extensively explored for
Intrusion Detection Systems (IDS) specifically for IoT. Most recent works are
based on time sequential models like LSTM and there is short of research in
CNNs as they are not naturally suited for this problem. In this article, we
propose a novel solution to the intrusion attacks against IoT devices using
CNNs. The data is encoded as the convolutional operations to capture the
patterns from the sensors data along time that are useful for attacks detection
by CNNs. The proposed method is integrated with two classical CNNs: ResNet and
EfficientNet, where the detection performance is evaluated. The experimental
results show significant improvement in both true positive rate and false
positive rate compared to the baseline using LSTM.


http://arxiv.org/abs/2211.10530
Provable Defense against Backdoor Policies in Reinforcement Learning. (1%)
Shubham Kumar Bharti; Xuezhou Zhang; Adish Singla; Xiaojin Zhu
  We propose a provable defense mechanism against backdoor policies in
reinforcement learning under subspace trigger assumption. A backdoor policy is
a security threat where an adversary publishes a seemingly well-behaved policy
which in fact allows hidden triggers. During deployment, the adversary can
modify observed states in a particular way to trigger unexpected actions and
harm the agent. We assume the agent does not have the resources to re-train a
good policy. Instead, our defense mechanism sanitizes the backdoor policy by
projecting observed states to a 'safe subspace', estimated from a small number
of interactions with a clean (non-triggered) environment. Our sanitized policy
achieves $\epsilon$ approximate optimality in the presence of triggers,
provided the number of clean interactions is $O\left(\frac{D}{(1-\gamma)^4
\epsilon^2}\right)$ where $\gamma$ is the discounting factor and $D$ is the
dimension of state space. Empirically, we show that our sanitization defense
performs well on two Atari game environments.


http://arxiv.org/abs/2211.10024
Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks. (99%)
Stephen Casper; Kaivalya Hariharan; Dylan Hadfield-Menell
  Deep neural networks (DNNs) are powerful, but they can make mistakes that
pose significant risks. A model performing well on a test set does not imply
safety in deployment, so it is important to have additional tools to understand
its flaws. Adversarial examples can help reveal weaknesses, but they are often
difficult for a human to interpret or draw generalizable, actionable
conclusions from. Some previous works have addressed this by studying
human-interpretable attacks. We build on these with three contributions. First,
we introduce a method termed Search for Natural Adversarial Features Using
Embeddings (SNAFUE) which offers a fully-automated method for finding
"copy/paste" attacks in which one natural image can be pasted into another in
order to induce an unrelated misclassification. Second, we use this to red team
an ImageNet classifier and identify hundreds of easily-describable sets of
vulnerabilities. Third, we compare this approach with other interpretability
tools by attempting to rediscover trojans. Our results suggest that SNAFUE can
be useful for interpreting DNNs and generating adversarial data for them. Code
is available at https://github.com/thestephencasper/snafue


http://arxiv.org/abs/2211.09565
Towards Good Practices in Evaluating Transfer Adversarial Attacks. (93%)
Zhengyu Zhao; Hanwei Zhang; Renjue Li; Ronan Sicre; Laurent Amsaleg; Michael Backes
  Transfer adversarial attacks raise critical security concerns in real-world,
black-box scenarios. However, the actual progress of attack methods is
difficult to assess due to two main limitations in existing evaluations. First,
existing evaluations are unsystematic and sometimes unfair since new methods
are often directly added to old ones without complete comparisons to similar
methods. Second, existing evaluations mainly focus on transferability but
overlook another key attack property: stealthiness. In this work, we design
good practices to address these limitations. We first introduce a new attack
categorization, which enables our systematic analyses of similar attacks in
each specific category. Our analyses lead to new findings that complement or
even challenge existing knowledge. Furthermore, we comprehensively evaluate 23
representative attacks against 9 defenses on ImageNet. We pay particular
attention to stealthiness, by adopting diverse imperceptibility metrics and
looking into new, finer-grained characteristics. Our evaluation reveals new
important insights: 1) Transferability is highly contextual, and some white-box
defenses may give a false sense of security since they are actually vulnerable
to (black-box) transfer attacks; 2) All transfer attacks are less stealthy, and
their stealthiness can vary dramatically under the same $L_{\infty}$ bound.


http://arxiv.org/abs/2211.09782
Assessing Neural Network Robustness via Adversarial Pivotal Tuning. (92%)
Peter Ebert Christensen; Vésteinn Snæbjarnarson; Andrea Dittadi; Serge Belongie; Sagie Benaim
  The ability to assess the robustness of image classifiers to a diverse set of
manipulations is essential to their deployment in the real world. Recently,
semantic manipulations of real images have been considered for this purpose, as
they may not arise using standard adversarial settings. However, such semantic
manipulations are often limited to style, color or attribute changes. While
expressive, these manipulations do not consider the full capacity of a
pretrained generator to affect adversarial image manipulations. In this work,
we aim at leveraging the full capacity of a pretrained image generator to
generate highly detailed, diverse and photorealistic image manipulations.
Inspired by recent GAN-based image inversion methods, we propose a method
called Adversarial Pivotal Tuning (APT). APT first finds a pivot latent space
input to a pretrained generator that best reconstructs an input image. It then
adjusts the weights of the generator to create small, but semantic,
manipulations which fool a pretrained classifier. Crucially, APT changes both
the input and the weights of the pretrained generator, while preserving its
expressive latent editing capability, thus allowing the use of its full
capacity in creating semantic adversarial manipulations. We demonstrate that
APT generates a variety of semantic image manipulations, which preserve the
input image class, but which fool a variety of pretrained classifiers. We
further demonstrate that classifiers trained to be robust to other robustness
benchmarks, are not robust to our generated manipulations and propose an
approach to improve the robustness towards our generated manipulations. Code
available at: https://captaine.github.io/apt/


http://arxiv.org/abs/2211.09717
UPTON: Unattributable Authorship Text via Data Poisoning. (86%)
Ziyao Wang; Thai Le; Dongwon Lee
  In online medium such as opinion column in Bloomberg, The Guardian and
Western Journal, aspiring writers post their writings for various reasons with
their names often proudly open. However, it may occur that such a writer wants
to write in other venues anonymously or under a pseudonym (e.g., activist,
whistle-blower). However, if an attacker has already built an accurate
authorship attribution (AA) model based off of the writings from such
platforms, attributing an anonymous writing to the known authorship is
possible. Therefore, in this work, we ask a question "can one make the writings
and texts, T, in the open spaces such as opinion sharing platforms
unattributable so that AA models trained from T cannot attribute authorship
well?" Toward this question, we present a novel solution, UPTON, that exploits
textual data poisoning method to disturb the training process of AA models.
UPTON uses data poisoning to destroy the authorship feature only in training
samples by perturbing them, and try to make released textual data unlearnable
on deep neuron networks. It is different from previous obfuscation works, that
use adversarial attack to modify the test samples and mislead an AA model, and
also the backdoor works, which use trigger words both in test and training
samples and only change the model output when trigger words occur. Using four
authorship datasets (e.g., IMDb10, IMDb64, Enron and WJO), then, we present
empirical validation where: (1)UPTON is able to downgrade the test accuracy to
about 30% with carefully designed target-selection methods. (2)UPTON poisoning
is able to preserve most of the original semantics. The BERTSCORE between the
clean and UPTON poisoned texts are higher than 0.95. The number is very closed
to 1.00, which means no sematic change. (3)UPTON is also robust towards
spelling correction systems.


http://arxiv.org/abs/2211.09363
Generalizable Deepfake Detection with Phase-Based Motion Analysis. (50%)
Ekta Prashnani; Michael Goebel; B. S. Manjunath
  We propose PhaseForensics, a DeepFake (DF) video detection method that
leverages a phase-based motion representation of facial temporal dynamics.
Existing methods relying on temporal inconsistencies for DF detection present
many advantages over the typical frame-based methods. However, they still show
limited cross-dataset generalization and robustness to common distortions.
These shortcomings are partially due to error-prone motion estimation and
landmark tracking, or the susceptibility of the pixel intensity-based features
to spatial distortions and the cross-dataset domain shifts. Our key insight to
overcome these issues is to leverage the temporal phase variations in the
band-pass components of the Complex Steerable Pyramid on face sub-regions. This
not only enables a robust estimate of the temporal dynamics in these regions,
but is also less prone to cross-dataset variations. Furthermore, the band-pass
filters used to compute the local per-frame phase form an effective defense
against the perturbations commonly seen in gradient-based adversarial attacks.
Overall, with PhaseForensics, we show improved distortion and adversarial
robustness, and state-of-the-art cross-dataset generalization, with 91.2%
video-level AUC on the challenging CelebDFv2 (a recent state-of-the-art
compares at 86.9%).


http://arxiv.org/abs/2211.09345
More Effective Centrality-Based Attacks on Weighted Networks. (15%)
Balume Mburano; Weisheng Si; Qing Cao; Wei Xing Zheng
  Only when understanding hackers' tactics, can we thwart their attacks. With
this spirit, this paper studies how hackers can effectively launch the
so-called 'targeted node attacks', in which iterative attacks are staged on a
network, and in each iteration the most important node is removed. In the
existing attacks for weighted networks, the node importance is typically
measured by the centralities related to shortest-path lengths, and the attack
effectiveness is also measured mostly by length-related metrics. However, this
paper argues that flows can better reflect network functioning than
shortest-path lengths for those networks with carrying traffic as the main
functionality. Thus, this paper proposes metrics based on flows for measuring
the node importance and the attack effectiveness, respectively. Our node
importance metrics include three flow-based centralities (flow betweenness,
current-flow betweenness and current-flow closeness), which have not been
proposed for use in the attacks on weighted networks yet. Our attack
effectiveness metric is a new one proposed by us based on average network flow.
Extensive experiments on both artificial and real-world networks show that the
attack methods with our three suggested centralities are more effective than
the existing attack methods when evaluated under our proposed attack
effectiveness metric.


http://arxiv.org/abs/2211.09959
Potential Auto-driving Threat: Universal Rain-removal Attack. (2%)
Jinchegn Hu; Jihao Li; Zhuoran Hou; Jingjing Jiang; Cunjia Liu; Yuanjian Zhang
  The problem of robustness in adverse weather conditions is considered a
significant challenge for computer vision algorithms in the applicants of
autonomous driving. Image rain removal algorithms are a general solution to
this problem. They find a deep connection between raindrops/rain-streaks and
images by mining the hidden features and restoring information about the
rain-free environment based on the powerful representation capabilities of
neural networks. However, previous research has focused on architecture
innovations and has yet to consider the vulnerability issues that already exist
in neural networks. This research gap hints at a potential security threat
geared toward the intelligent perception of autonomous driving in the rain. In
this paper, we propose a universal rain-removal attack (URA) on the
vulnerability of image rain-removal algorithms by generating a non-additive
spatial perturbation that significantly reduces the similarity and image
quality of scene restoration. Notably, this perturbation is difficult to
recognise by humans and is also the same for different target images. Thus, URA
could be considered a critical tool for the vulnerability detection of image
rain-removal algorithms. It also could be developed as a real-world artificial
intelligence attack method. Experimental results show that URA can reduce the
scene repair capability by 39.5% and the image generation quality by 26.4%,
targeting the state-of-the-art (SOTA) single-image rain-removal algorithms
currently available.


http://arxiv.org/abs/2211.09859
Data-Centric Debugging: mitigating model failures via targeted data collection. (1%)
Sahil Singla; Atoosa Malemir Chegini; Mazda Moayeri; Soheil Feiz
  Deep neural networks can be unreliable in the real world when the training
set does not adequately cover all the settings where they are deployed.
Focusing on image classification, we consider the setting where we have an
error distribution $\mathcal{E}$ representing a deployment scenario where the
model fails. We have access to a small set of samples $\mathcal{E}_{sample}$
from $\mathcal{E}$ and it can be expensive to obtain additional samples. In the
traditional model development framework, mitigating failures of the model in
$\mathcal{E}$ can be challenging and is often done in an ad hoc manner. In this
paper, we propose a general methodology for model debugging that can
systemically improve model performance on $\mathcal{E}$ while maintaining its
performance on the original test set. Our key assumption is that we have access
to a large pool of weakly (noisily) labeled data $\mathcal{F}$. However,
naively adding $\mathcal{F}$ to the training would hurt model performance due
to the large extent of label noise. Our Data-Centric Debugging (DCD) framework
carefully creates a debug-train set by selecting images from $\mathcal{F}$ that
are perceptually similar to the images in $\mathcal{E}_{sample}$. To do this,
we use the $\ell_2$ distance in the feature space (penultimate layer
activations) of various models including ResNet, Robust ResNet and DINO where
we observe DINO ViTs are significantly better at discovering similar images
compared to Resnets. Compared to LPIPS, we find that our method reduces compute
and storage requirements by 99.58\%. Compared to the baselines that maintain
model performance on the test set, we achieve significantly (+9.45\%) improved
results on the debug-heldout sets.


http://arxiv.org/abs/2211.10012
A Tale of Two Cities: Data and Configuration Variances in Robust Deep Learning. (1%)
Guanqin Zhang; Jiankun Sun; Feng Xu; H. M. N. Dilum Bandara; Shiping Chen; Yulei Sui; Tim Menzies
  Deep neural networks (DNNs), are widely used in many industries such as image
recognition, supply chain, medical diagnosis, and autonomous driving. However,
prior work has shown the high accuracy of a DNN model does not imply high
robustness (i.e., consistent performances on new and future datasets) because
the input data and external environment (e.g., software and model
configurations) for a deployed model are constantly changing. Hence, ensuring
the robustness of deep learning is not an option but a priority to enhance
business and consumer confidence. Previous studies mostly focus on the data
aspect of model variance. In this article, we systematically summarize DNN
robustness issues and formulate them in a holistic view through two important
aspects, i.e., data and software configuration variances in DNNs. We also
provide a predictive framework to generate representative variances
(counterexamples) by considering both data and configurations for robust
learning through the lens of search-based optimization.


http://arxiv.org/abs/2211.09773
T-SEA: Transfer-based Self-Ensemble Attack on Object Detection. (99%)
Hao Huang; Ziyan Chen; Huanran Chen; Yongtao Wang; Kevin Zhang
  Compared to query-based black-box attacks, transfer-based black-box attacks
do not require any information of the attacked models, which ensures their
secrecy. However, most existing transfer-based approaches rely on ensembling
multiple models to boost the attack transferability, which is time- and
resource-intensive, not to mention the difficulty of obtaining diverse models
on the same task. To address this limitation, in this work, we focus on the
single-model transfer-based black-box attack on object detection, utilizing
only one model to achieve a high-transferability adversarial attack on multiple
black-box detectors. Specifically, we first make observations on the patch
optimization process of the existing method and propose an enhanced attack
framework by slightly adjusting its training strategies. Then, we analogize
patch optimization with regular model optimization, proposing a series of
self-ensemble approaches on the input data, the attacked model, and the
adversarial patch to efficiently make use of the limited information and
prevent the patch from overfitting. The experimental results show that the
proposed framework can be applied with multiple classical base attack methods
(e.g., PGD and MIM) to greatly improve the black-box transferability of the
well-optimized patch on multiple mainstream detectors, meanwhile boosting
white-box performance. Our code is available at
https://github.com/VDIGPKU/T-SEA.


http://arxiv.org/abs/2211.08706
Efficiently Finding Adversarial Examples with DNN Preprocessing. (99%)
Avriti Chauhan; Mohammad Afzal; Hrishikesh Karmarkar; Yizhak Elboher; Kumar Madhukar; Guy Katz
  Deep Neural Networks (DNNs) are everywhere, frequently performing a fairly
complex task that used to be unimaginable for machines to carry out. In doing
so, they do a lot of decision making which, depending on the application, may
be disastrous if gone wrong. This necessitates a formal argument that the
underlying neural networks satisfy certain desirable properties. Robustness is
one such key property for DNNs, particularly if they are being deployed in
safety or business critical applications. Informally speaking, a DNN is not
robust if very small changes to its input may affect the output in a
considerable way (e.g. changes the classification for that input). The task of
finding an adversarial example is to demonstrate this lack of robustness,
whenever applicable. While this is doable with the help of constrained
optimization techniques, scalability becomes a challenge due to large-sized
networks. This paper proposes the use of information gathered by preprocessing
the DNN to heavily simplify the optimization problem. Our experiments
substantiate that this is effective, and does significantly better than the
state-of-the-art.


http://arxiv.org/abs/2211.08686
Improving Interpretability via Regularization of Neural Activation Sensitivity. (92%)
Ofir Moshe; Gil Fidel; Ron Bitton; Asaf Shabtai
  State-of-the-art deep neural networks (DNNs) are highly effective at tackling
many real-world tasks. However, their wide adoption in mission-critical
contexts is hampered by two major weaknesses - their susceptibility to
adversarial attacks and their opaqueness. The former raises concerns about the
security and generalization of DNNs in real-world conditions, whereas the
latter impedes users' trust in their output. In this research, we (1) examine
the effect of adversarial robustness on interpretability and (2) present a
novel approach for improving the interpretability of DNNs that is based on
regularization of neural activation sensitivity. We evaluate the
interpretability of models trained using our method to that of standard models
and models trained using state-of-the-art adversarial robustness techniques.
Our results show that adversarially robust models are superior to standard
models and that models trained using our proposed method are even better than
adversarially robust models in terms of interpretability.


http://arxiv.org/abs/2211.08859
Attacking Object Detector Using A Universal Targeted Label-Switch Patch. (86%)
Avishag Shapira; Ron Bitton; Dan Avraham; Alon Zolfi; Yuval Elovici; Asaf Shabtai
  Adversarial attacks against deep learning-based object detectors (ODs) have
been studied extensively in the past few years. These attacks cause the model
to make incorrect predictions by placing a patch containing an adversarial
pattern on the target object or anywhere within the frame. However, none of
prior research proposed a misclassification attack on ODs, in which the patch
is applied on the target object. In this study, we propose a novel, universal,
targeted, label-switch attack against the state-of-the-art object detector,
YOLO. In our attack, we use (i) a tailored projection function to enable the
placement of the adversarial patch on multiple target objects in the image
(e.g., cars), each of which may be located a different distance away from the
camera or have a different view angle relative to the camera, and (ii) a unique
loss function capable of changing the label of the attacked objects. The
proposed universal patch, which is trained in the digital domain, is
transferable to the physical domain. We performed an extensive evaluation using
different types of object detectors, different video streams captured by
different cameras, and various target classes, and evaluated different
configurations of the adversarial patch in the physical domain.


http://arxiv.org/abs/2211.08942
Differentially Private Optimizers Can Learn Adversarially Robust Models. (83%)
Yuan Zhang; Zhiqi Bu
  Machine learning models have shone in a variety of domains and attracted
increasing attention from both the security and the privacy communities. One
important yet worrying question is: will training models under the differential
privacy (DP) constraint unfavorably impact on the adversarial robustness? While
previous works have postulated that privacy comes at the cost of worse
robustness, we give the first theoretical analysis to show that DP models can
indeed be robust and accurate, even sometimes more robust than their
naturally-trained non-private counterparts. We observe three key factors that
influence the privacy-robustness-accuracy tradeoff: (1) hyperparameters for DP
optimizers are critical; (2) pre-training on public data significantly
mitigates the accuracy and robustness drop; (3) choice of DP optimizers makes a
difference. With these factors set properly, we achieve 90\% natural accuracy,
72\% robust accuracy ($+9\%$ than the non-private model) under $l_2(0.5)$
attack, and 69\% robust accuracy ($+16\%$ than the non-private model) with
pre-trained SimCLRv2 model under $l_\infty(4/255)$ attack on CIFAR10 with
$\epsilon=2$. In fact, we show both theoretically and empirically that DP
models are Pareto optimal on the accuracy-robustness tradeoff. Empirically, the
robustness of DP models is consistently observed on MNIST, Fashion MNIST and
CelebA datasets, with ResNet and Vision Transformer. We believe our encouraging
results are a significant step towards training models that are private as well
as robust.


http://arxiv.org/abs/2211.09321
Interpretable Dimensionality Reduction by Feature Preserving Manifold Approximation and Projection. (56%)
Yang Yang; Hongjian Sun; Jialei Gong; Yali Du; Di Yu
  Nonlinear dimensionality reduction lacks interpretability due to the absence
of source features in low-dimensional embedding space. We propose an
interpretable method featMAP to preserve source features by tangent space
embedding. The core of our proposal is to utilize local singular value
decomposition (SVD) to approximate the tangent space which is embedded to
low-dimensional space by maintaining the alignment. Based on the embedding
tangent space, featMAP enables the interpretability by locally demonstrating
the source features and feature importance. Furthermore, featMAP embeds the
data points by anisotropic projection to preserve the local similarity and
original density. We apply featMAP to interpreting digit classification, object
detection and MNIST adversarial examples. FeatMAP uses source features to
explicitly distinguish the digits and objects and to explain the
misclassification of adversarial examples. We also compare featMAP with other
state-of-the-art methods on local and global metrics.


http://arxiv.org/abs/2211.09273
Privacy against Real-Time Speech Emotion Detection via Acoustic Adversarial Evasion of Machine Learning. (38%)
Brian Testa; Yi Xiao; Avery Gump; Asif Salekin
  Emotional Surveillance is an emerging area with wide-reaching privacy
concerns. These concerns are exacerbated by ubiquitous IoT devices with
multiple sensors that can support these surveillance use cases. The work
presented here considers one such use case: the use of a speech emotion
recognition (SER) classifier tied to a smart speaker. This work demonstrates
the ability to evade black-box SER classifiers tied to a smart speaker without
compromising the utility of the smart speaker. This privacy concern is
considered through the lens of adversarial evasion of machine learning. Our
solution, Defeating Acoustic Recognition of Emotion via Genetic Programming
(DARE-GP), uses genetic programming to generate non-invasive additive audio
perturbations (AAPs). By constraining the evolution of these AAPs,
transcription accuracy can be protected while simultaneously degrading SER
classifier performance. The additive nature of these AAPs, along with an
approach that generates these AAPs for a fixed set of users in an utterance and
user location-independent manner, supports real-time, real-world evasion of SER
classifiers. DARE-GP's use of spectral features, which underlay the emotional
content of speech, allows the transferability of AAPs to previously unseen
black-box SER classifiers. Further, DARE-GP outperforms state-of-the-art SER
evasion techniques and is robust against defenses employed by a knowledgeable
adversary. The evaluations in this work culminate with acoustic evaluations
against two off-the-shelf commercial smart speakers, where a single AAP could
evade a black box classifier over 70% of the time. The final evaluation
deployed AAP playback on a small-form-factor system (raspberry pi) integrated
with a wake-word system to evaluate the efficacy of a real-world, real-time
deployment where DARE-GP is automatically invoked with the smart speaker's wake
word.


http://arxiv.org/abs/2211.09110
Holistic Evaluation of Language Models. (2%)
Percy Liang; Rishi Bommasani; Tony Lee; Dimitris Tsipras; Dilara Soylu; Michihiro Yasunaga; Yian Zhang; Deepak Narayanan; Yuhuai Wu; Ananya Kumar; Benjamin Newman; Binhang Yuan; Bobby Yan; Ce Zhang; Christian Cosgrove; Christopher D. Manning; Christopher Ré; Diana Acosta-Navas; Drew A. Hudson; Eric Zelikman; Esin Durmus; Faisal Ladhak; Frieda Rong; Hongyu Ren; Huaxiu Yao; Jue Wang; Keshav Santhanam; Laurel Orr; Lucia Zheng; Mert Yuksekgonul; Mirac Suzgun; Nathan Kim; Neel Guha; Niladri Chatterji; Omar Khattab; Peter Henderson; Qian Huang; Ryan Chi; Sang Michael Xie; Shibani Santurkar; Surya Ganguli; Tatsunori Hashimoto; Thomas Icard; Tianyi Zhang; Vishrav Chaudhary; William Wang; Xuechen Li; Yifan Mai; Yuhui Zhang; Yuta Koreeda
  Language models (LMs) are becoming the foundation for almost all major
language technologies, but their capabilities, limitations, and risks are not
well understood. We present Holistic Evaluation of Language Models (HELM) to
improve the transparency of language models. First, we taxonomize the vast
space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata)
that are of interest for LMs. Then we select a broad subset based on coverage
and feasibility, noting what's missing or underrepresented (e.g. question
answering for neglected English dialects, metrics for trustworthiness). Second,
we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration,
robustness, fairness, bias, toxicity, and efficiency) for each of 16 core
scenarios when possible (87.5% of the time). This ensures metrics beyond
accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We
also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze
specific aspects (e.g. reasoning, disinformation). Third, we conduct a
large-scale evaluation of 30 prominent language models (spanning open,
limited-access, and closed models) on all 42 scenarios, 21 of which were not
previously used in mainstream LM evaluation. Prior to HELM, models on average
were evaluated on just 17.9% of the core HELM scenarios, with some prominent
models not sharing a single scenario in common. We improve this to 96.0%: now
all 30 models have been densely benchmarked on the same core scenarios and
metrics under standardized conditions. Our evaluation surfaces 25 top-level
findings. For full transparency, we release all raw model prompts and
completions publicly for further analysis, as well as a general modular
toolkit. We intend for HELM to be a living benchmark for the community,
continuously updated with new scenarios, metrics, and models.


http://arxiv.org/abs/2211.08804
Analysis and Detectability of Offline Data Poisoning Attacks on Linear Systems. (1%)
Alessio Russo; Alexandre Proutiere
  A recent body of literature has investigated the effect of data poisoning
attacks on data-driven control methods. Data poisoning attacks are well-known
to the Machine Learning community, which, however, make use of assumptions,
such as cross-sample independence, that in general do not hold for dynamical
systems. As a consequence, attacks, and detection methods, operate differently
from the i.i.d. setting studied in classical supervised problems. In
particular, data poisoning attacks against data-driven control methods can be
fundamentally seen as changing the behavior of the dynamical system described
by the data. In this work, we study this phenomenon through the lens of
statistical testing, and verify the detectability of different attacks for a
linear dynamical system. On the basis of the arguments hereby presented, we
propose a stealthy data poisoning attack that can escape classical detection
tests, and conclude by showing the efficiency of the proposed attack.


http://arxiv.org/abs/2211.08068
Resisting Graph Adversarial Attack via Cooperative Homophilous Augmentation. (99%)
Zhihao Zhu; Chenwang Wu; Min Zhou; Hao Liao; Defu Lian; Enhong Chen
  Recent studies show that Graph Neural Networks(GNNs) are vulnerable and
easily fooled by small perturbations, which has raised considerable concerns
for adapting GNNs in various safety-critical applications. In this work, we
focus on the emerging but critical attack, namely, Graph Injection Attack(GIA),
in which the adversary poisons the graph by injecting fake nodes instead of
modifying existing structures or node attributes. Inspired by findings that the
adversarial attacks are related to the increased heterophily on perturbed
graphs (the adversary tends to connect dissimilar nodes), we propose a general
defense framework CHAGNN against GIA through cooperative homophilous
augmentation of graph data and model. Specifically, the model generates
pseudo-labels for unlabeled nodes in each round of training to reduce
heterophilous edges of nodes with distinct labels. The cleaner graph is fed
back to the model, producing more informative pseudo-labels. In such an
iterative manner, model robustness is then promisingly enhanced. We present the
theoretical analysis of the effect of homophilous augmentation and provide the
guarantee of the proposal's validity. Experimental results empirically
demonstrate the effectiveness of CHAGNN in comparison with recent
state-of-the-art defense methods on diverse real-world datasets.


http://arxiv.org/abs/2211.08384
Universal Distributional Decision-based Black-box Adversarial Attack with Reinforcement Learning. (99%)
Yiran Huang; Yexu Zhou; Michael Hefenbrock; Till Riedel; Likun Fang; Michael Beigl
  The vulnerability of the high-performance machine learning models implies a
security risk in applications with real-world consequences. Research on
adversarial attacks is beneficial in guiding the development of machine
learning models on the one hand and finding targeted defenses on the other.
However, most of the adversarial attacks today leverage the gradient or logit
information from the models to generate adversarial perturbation. Works in the
more realistic domain: decision-based attacks, which generate adversarial
perturbation solely based on observing the output label of the targeted model,
are still relatively rare and mostly use gradient-estimation strategies. In
this work, we propose a pixel-wise decision-based attack algorithm that finds a
distribution of adversarial perturbation through a reinforcement learning
algorithm. We call this method Decision-based Black-box Attack with
Reinforcement learning (DBAR). Experiments show that the proposed approach
outperforms state-of-the-art decision-based attacks with a higher attack
success rate and greater transferability.


http://arxiv.org/abs/2211.08008
MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack. (99%)
Yunrui Yu; Xitong Gao; Cheng-Zhong Xu
  Adversarial attacks can deceive neural networks by adding tiny perturbations
to their input data. Ensemble defenses, which are trained to minimize attack
transferability among sub-models, offer a promising research direction to
improve robustness against such attacks while maintaining a high accuracy on
natural inputs. We discover, however, that recent state-of-the-art (SOTA)
adversarial attack strategies cannot reliably evaluate ensemble defenses,
sizeably overestimating their robustness. This paper identifies the two factors
that contribute to this behavior. First, these defenses form ensembles that are
notably difficult for existing gradient-based method to attack, due to gradient
obfuscation. Second, ensemble defenses diversify sub-model gradients,
presenting a challenge to defeat all sub-models simultaneously, simply summing
their contributions may counteract the overall attack objective; yet, we
observe that ensemble may still be fooled despite most sub-models being
correct. We therefore introduce MORA, a model-reweighing attack to steer
adversarial example synthesis by reweighing the importance of sub-model
gradients. MORA finds that recent ensemble defenses all exhibit varying degrees
of overestimated robustness. Comparing it against recent SOTA white-box
attacks, it can converge orders of magnitude faster while achieving higher
attack success rates across all ensemble models examined with three different
ensemble modes (i.e., ensembling by either softmax, voting or logits). In
particular, most ensemble defenses exhibit near or exactly 0% robustness
against MORA with $\ell^\infty$ perturbation within 0.02 on CIFAR-10, and 0.01
on CIFAR-100. We make MORA open source with reproducible results and
pre-trained models; and provide a leaderboard of ensemble defenses under
various attack strategies.


http://arxiv.org/abs/2211.08657
Person Text-Image Matching via Text-Featur Interpretability Embedding and External Attack Node Implantation. (92%)
Fan Li; Hang Zhou; Huafeng Li; Yafei Zhang; Zhengtao Yu
  Person text-image matching, also known as textbased person search, aims to
retrieve images of specific pedestrians using text descriptions. Although
person text-image matching has made great research progress, existing methods
still face two challenges. First, the lack of interpretability of text features
makes it challenging to effectively align them with their corresponding image
features. Second, the same pedestrian image often corresponds to multiple
different text descriptions, and a single text description can correspond to
multiple different images of the same identity. The diversity of text
descriptions and images makes it difficult for a network to extract robust
features that match the two modalities. To address these problems, we propose a
person text-image matching method by embedding text-feature interpretability
and an external attack node. Specifically, we improve the interpretability of
text features by providing them with consistent semantic information with image
features to achieve the alignment of text and describe image region features.To
address the challenges posed by the diversity of text and the corresponding
person images, we treat the variation caused by diversity to features as caused
by perturbation information and propose a novel adversarial attack and defense
method to solve it. In the model design, graph convolution is used as the basic
framework for feature representation and the adversarial attacks caused by text
and image diversity on feature extraction is simulated by implanting an
additional attack node in the graph convolution layer to improve the robustness
of the model against text and image diversity. Extensive experiments
demonstrate the effectiveness and superiority of text-pedestrian image matching
over existing methods. The source code of the method is published at


http://arxiv.org/abs/2211.07915
Backdoor Attacks on Time Series: A Generative Approach. (70%)
Yujing Jiang; Xingjun Ma; Sarah Monazam Erfani; James Bailey
  Backdoor attacks have emerged as one of the major security threats to deep
learning models as they can easily control the model's test-time predictions by
pre-injecting a backdoor trigger into the model at training time. While
backdoor attacks have been extensively studied on images, few works have
investigated the threat of backdoor attacks on time series data. To fill this
gap, in this paper we present a novel generative approach for time series
backdoor attacks against deep learning based time series classifiers. Backdoor
attacks have two main goals: high stealthiness and high attack success rate. We
find that, compared to images, it can be more challenging to achieve the two
goals on time series. This is because time series have fewer input dimensions
and lower degrees of freedom, making it hard to achieve a high attack success
rate without compromising stealthiness. Our generative approach addresses this
challenge by generating trigger patterns that are as realistic as real-time
series patterns while achieving a high attack success rate without causing a
significant drop in clean accuracy. We also show that our proposed attack is
resistant to potential backdoor defenses. Furthermore, we propose a novel
universal generator that can poison any type of time series with a single
generator that allows universal attacks without the need to fine-tune the
generative model for new time series datasets.


http://arxiv.org/abs/2211.08453
Improved techniques for deterministic l2 robustness. (22%)
Sahil Singla; Soheil Feizi
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz
constraint under the $l_{2}$ norm is useful for adversarial robustness,
interpretable gradients and stable training. 1-Lipschitz CNNs are usually
designed by enforcing each layer to have an orthogonal Jacobian matrix (for all
inputs) to prevent the gradients from vanishing during backpropagation.
However, their performance often significantly lags behind that of heuristic
methods to enforce Lipschitz constraints where the resulting CNN is not
\textit{provably} 1-Lipschitz. In this work, we reduce this gap by introducing
(a) a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last
linear layer with a 1-hidden layer MLP that significantly improves their
performance for both standard and provably robust accuracy, (b) a method to
significantly reduce the training time per epoch for Skew Orthogonal
Convolution (SOC) layers (>30\% reduction for deeper networks) and (c) a class
of pooling layers using the mathematical property that the $l_{2}$ distance of
an input to a manifold is 1-Lipschitz. Using these methods, we significantly
advance the state-of-the-art for standard and provable robust accuracies on
CIFAR-10 (gains of +1.79\% and +3.82\%) and similarly on CIFAR-100 (+3.78\% and
+4.75\%) across all networks. Code is available at
\url{https://github.com/singlasahil14/improved_l2_robustness}.


http://arxiv.org/abs/2211.08229
CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learning. (22%)
Jinghuai Zhang; Hongbin Liu; Jinyuan Jia; Neil Zhenqiang Gong
  Contrastive learning (CL) pre-trains general-purpose encoders using an
unlabeled pre-training dataset, which consists of images (called single-modal
CL) or image-text pairs (called multi-modal CL). CL is vulnerable to data
poisoning based backdoor attacks (DPBAs), in which an attacker injects poisoned
inputs into the pre-training dataset so the encoder is backdoored. However,
existing DPBAs achieve limited effectiveness. In this work, we propose new
DPBAs called CorruptEncoder to CL. Our experiments show that CorruptEncoder
substantially outperforms existing DPBAs for both single-modal and multi-modal
CL. CorruptEncoder is the first DPBA that achieves more than 90% attack success
rates on single-modal CL with only a few (3) reference images and a small
poisoning ratio (0.5%). Moreover, we also propose a defense, called localized
cropping, to defend single-modal CL against DPBAs. Our results show that our
defense can reduce the effectiveness of DPBAs, but it sacrifices the utility of
the encoder, highlighting the needs of new defenses.


http://arxiv.org/abs/2211.08044
Backdoor Attacks for Remote Sensing Data with Wavelet Transform. (12%)
Nikolaus Dräger; Yonghao Xu; Pedram Ghamisi
  Recent years have witnessed the great success of deep learning algorithms in
the geoscience and remote sensing realm. Nevertheless, the security and
robustness of deep learning models deserve special attention when addressing
safety-critical remote sensing tasks. In this paper, we provide a systematic
analysis of backdoor attacks for remote sensing data, where both scene
classification and semantic segmentation tasks are considered. While most of
the existing backdoor attack algorithms rely on visible triggers like squared
patches with well-designed patterns, we propose a novel wavelet transform-based
attack (WABA) method, which can achieve invisible attacks by injecting the
trigger image into the poisoned image in the low-frequency domain. In this way,
the high-frequency information in the trigger image can be filtered out in the
attack, resulting in stealthy data poisoning. Despite its simplicity, the
proposed method can significantly cheat the current state-of-the-art deep
learning models with a high attack success rate. We further analyze how
different trigger images and the hyper-parameters in the wavelet transform
would influence the performance of the proposed method. Extensive experiments
on four benchmark remote sensing datasets demonstrate the effectiveness of the
proposed method for both scene classification and semantic segmentation tasks
and thus highlight the importance of designing advanced backdoor defense
algorithms to address this threat in remote sensing scenarios. The code will be
available online at \url{https://github.com/ndraeger/waba}.


http://arxiv.org/abs/2211.07263
Efficient Adversarial Training with Robust Early-Bird Tickets. (92%)
Zhiheng Xi; Rui Zheng; Tao Gui; Qi Zhang; Xuanjing Huang
  Adversarial training is one of the most powerful methods to improve the
robustness of pre-trained language models (PLMs). However, this approach is
typically more expensive than traditional fine-tuning because of the necessity
to generate adversarial examples via gradient descent. Delving into the
optimization process of adversarial training, we find that robust connectivity
patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs),
far before parameters converge. Inspired by this finding, we dig out robust
early-bird tickets (i.e., subnetworks) to develop an efficient adversarial
training method: (1) searching for robust tickets with structured sparsity in
the early stage; (2) fine-tuning robust tickets in the remaining time. To
extract the robust tickets as early as possible, we design a ticket convergence
metric to automatically terminate the searching process. Experiments show that
the proposed efficient adversarial training method can achieve up to $7\times
\sim 13 \times$ training speedups while maintaining comparable or even better
robustness compared to the most competitive state-of-the-art adversarial
training methods.


http://arxiv.org/abs/2211.07383
Attacking Face Recognition with T-shirts: Database, Vulnerability Assessment and Detection. (13%)
M. Ibsen; C. Rathgeb; F. Brechtel; R. Klepp; K. Pöppelmann; A. George; S. Marcel; C. Busch
  Face recognition systems are widely deployed for biometric authentication.
Despite this, it is well-known that, without any safeguards, face recognition
systems are highly vulnerable to presentation attacks. In response to this
security issue, several promising methods for detecting presentation attacks
have been proposed which show high performance on existing benchmarks. However,
an ongoing challenge is the generalization of presentation attack detection
methods to unseen and new attack types. To this end, we propose a new T-shirt
Face Presentation Attack (TFPA) database of 1,608 T-shirt attacks using 100
unique presentation attack instruments. In an extensive evaluation, we show
that this type of attack can compromise the security of face recognition
systems and that some state-of-the-art attack detection mechanisms trained on
popular benchmarks fail to robustly generalize to the new attacks. Further, we
propose three new methods for detecting T-shirt attack images, one which relies
on the statistical differences between depth maps of bona fide images and
T-shirt attacks, an anomaly detection approach trained on features only
extracted from bona fide RGB images, and a fusion approach which achieves
competitive detection performance.


http://arxiv.org/abs/2211.07455
Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems. (5%)
Jialiang Xu; Mengyu Zhou; Xinyi He; Shi Han; Dongmei Zhang
  Numerical Question Answering is the task of answering questions that require
numerical capabilities. Previous works introduce general adversarial attacks to
Numerical Question Answering, while not systematically exploring numerical
capabilities specific to the topic. In this paper, we propose to conduct
numerical capability diagnosis on a series of Numerical Question Answering
systems and datasets. A series of numerical capabilities are highlighted, and
corresponding dataset perturbations are designed. Empirical results indicate
that existing systems are severely challenged by these perturbations. E.g.,
Graph2Tree experienced a 53.83% absolute accuracy drop against the ``Extra''
perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the
``Language'' perturbation on the numerical subset of DROP. As a counteracting
approach, we also investigate the effectiveness of applying perturbations as
data augmentation to relieve systems' lack of robust numerical capabilities.
With experiment analysis and empirical studies, it is demonstrated that
Numerical Question Answering with robust numerical capabilities is still to a
large extent an open question. We discuss future directions of Numerical
Question Answering and summarize guidelines on future dataset collection and
system design.


http://arxiv.org/abs/2211.07650
Explainer Divergence Scores (EDS): Some Post-Hoc Explanations May be Effective for Detecting Unknown Spurious Correlations. (5%)
Shea Cardozo; Gabriel Islas Montero; Dmitry Kazhdan; Botty Dimanov; Maleakhi Wijaya; Mateja Jamnik; Pietro Lio
  Recent work has suggested post-hoc explainers might be ineffective for
detecting spurious correlations in Deep Neural Networks (DNNs). However, we
show there are serious weaknesses with the existing evaluation frameworks for
this setting. Previously proposed metrics are extremely difficult to interpret
and are not directly comparable between explainer methods. To alleviate these
constraints, we propose a new evaluation methodology, Explainer Divergence
Scores (EDS), grounded in an information theory approach to evaluate
explainers. EDS is easy to interpret and naturally comparable across
explainers. We use our methodology to compare the detection performance of
three different explainers - feature attribution methods, influential examples
and concept extraction, on two different image datasets. We discover post-hoc
explainers often contain substantial information about a DNN's dependence on
spurious artifacts, but in ways often imperceptible to human users. This
suggests the need for new techniques that can use this information to better
detect a DNN's reliance on spurious correlations.


http://arxiv.org/abs/2211.07277
Robustifying Deep Vision Models Through Shape Sensitization. (2%)
Aditay Tripathi; Rishubh Singh; Anirban Chakraborty; Pradeep Shenoy
  Recent work has shown that deep vision models tend to be overly dependent on
low-level or "texture" features, leading to poor generalization. Various data
augmentation strategies have been proposed to overcome this so-called texture
bias in DNNs. We propose a simple, lightweight adversarial augmentation
technique that explicitly incentivizes the network to learn holistic shapes for
accurate prediction in an object classification setting. Our augmentations
superpose edgemaps from one image onto another image with shuffled patches,
using a randomly determined mixing proportion, with the image label of the
edgemap image. To classify these augmented images, the model needs to not only
detect and focus on edges but distinguish between relevant and spurious edges.
We show that our augmentations significantly improve classification accuracy
and robustness measures on a range of datasets and neural architectures. As an
example, for ViT-S, We obtain absolute gains on classification accuracy gains
up to 6%. We also obtain gains of up to 28% and 8.5% on natural adversarial and
out-of-distribution datasets like ImageNet-A (for ViT-B) and ImageNet-R (for
ViT-S), respectively. Analysis using a range of probe datasets shows
substantially increased shape sensitivity in our trained models, explaining the
observed improvement in robustness and classification accuracy.


http://arxiv.org/abs/2211.09810
Certifying Robustness of Convolutional Neural Networks with Tight Linear Approximation. (26%)
Yuan Xiao; Tongtong Bai; Mingzheng Gu; Chunrong Fang; Zhenyu Chen
  The robustness of neural network classifiers is becoming important in the
safety-critical domain and can be quantified by robustness verification.
However, at present, efficient and scalable verification techniques are always
sound but incomplete. Therefore, the improvement of certified robustness bounds
is the key criterion to evaluate the superiority of robustness verification
approaches. In this paper, we present a Tight Linear approximation approach for
robustness verification of Convolutional Neural Networks(Ti-Lin). For general
CNNs, we first provide a new linear constraints for S-shaped activation
functions, which is better than both existing Neuron-wise Tightest and
Network-wise Tightest tools. We then propose Neuron-wise Tightest linear bounds
for Maxpool function. We implement Ti-Lin, the resulting verification method.
We evaluate it with 48 different CNNs trained on MNIST, CIFAR-10, and Tiny
ImageNet datasets. Experimental results show that Ti-Lin significantly
outperforms other five state-of-the-art methods(CNN-Cert, DeepPoly, DeepCert,
VeriNet, Newise). Concretely, Ti-Lin certifies much more precise robustness
bounds on pure CNNs with Sigmoid/Tanh/Arctan functions and CNNs with Maxpooling
function with at most 63.70% and 253.54% improvement, respectively.


http://arxiv.org/abs/2211.06788
Adversarial and Random Transformations for Robust Domain Adaptation and Generalization. (75%)
Liang Xiao; Jiaolong Xu; Dawei Zhao; Erke Shang; Qi Zhu; Bin Dai
  Data augmentation has been widely used to improve generalization in training
deep neural networks. Recent works show that using worst-case transformations
or adversarial augmentation strategies can significantly improve the accuracy
and robustness. However, due to the non-differentiable properties of image
transformations, searching algorithms such as reinforcement learning or
evolution strategy have to be applied, which are not computationally practical
for large scale problems. In this work, we show that by simply applying
consistency training with random data augmentation, state-of-the-art results on
domain adaptation (DA) and generalization (DG) can be obtained. To further
improve the accuracy and robustness with adversarial examples, we propose a
differentiable adversarial data augmentation method based on spatial
transformer networks (STN). The combined adversarial and random transformations
based method outperforms the state-of-the-art on multiple DA and DG benchmark
datasets. Besides, the proposed method shows desirable robustness to
corruption, which is also validated on commonly used datasets.


http://arxiv.org/abs/2211.06571
Generating Textual Adversaries with Minimal Perturbation. (98%)
Xingyi Zhao; Lu Zhang; Depeng Xu; Shuhan Yuan
  Many word-level adversarial attack approaches for textual data have been
proposed in recent studies. However, due to the massive search space consisting
of combinations of candidate words, the existing approaches face the problem of
preserving the semantics of texts when crafting adversarial counterparts. In
this paper, we develop a novel attack strategy to find adversarial texts with
high similarity to the original texts while introducing minimal perturbation.
The rationale is that we expect the adversarial texts with small perturbation
can better preserve the semantic meaning of original texts. Experiments show
that, compared with state-of-the-art attack approaches, our approach achieves
higher success rates and lower perturbation rates in four benchmark datasets.


http://arxiv.org/abs/2211.06508
On the robustness of non-intrusive speech quality model by adversarial examples. (98%)
Hsin-Yi Lin; Huan-Hsin Tseng; Yu Tsao
  It has been shown recently that deep learning based models are effective on
speech quality prediction and could outperform traditional metrics in various
perspectives. Although network models have potential to be a surrogate for
complex human hearing perception, they may contain instabilities in
predictions. This work shows that deep speech quality predictors can be
vulnerable to adversarial perturbations, where the prediction can be changed
drastically by unnoticeable perturbations as small as $-30$ dB compared with
speech inputs. In addition to exposing the vulnerability of deep speech quality
predictors, we further explore and confirm the viability of adversarial
training for strengthening robustness of models.


http://arxiv.org/abs/2211.06500
An investigation of security controls and MITRE ATT\&CK techniques. (47%)
Md Rayhanur Rahman; Laurie Williams
  Attackers utilize a plethora of adversarial techniques in cyberattacks to
compromise the confidentiality, integrity, and availability of the target
organizations and systems. Information security standards such as NIST, ISO/IEC
specify hundreds of security controls that organizations can enforce to protect
and defend the information systems from adversarial techniques. However,
implementing all the available controls at the same time can be infeasible and
security controls need to be investigated in terms of their mitigation ability
over adversarial techniques used in cyberattacks as well. The goal of this
research is to aid organizations in making informed choices on security
controls to defend against cyberthreats through an investigation of adversarial
techniques used in current cyberattacks. In this study, we investigated the
extent of mitigation of 298 NIST SP800-53 controls over 188 adversarial
techniques used in 669 cybercrime groups and malware cataloged in the MITRE
ATT\&CK framework based upon an existing mapping between the controls and
techniques. We identify that, based on the mapping, only 101 out of 298 control
are capable of mitigating adversarial techniques. However, we also identify
that 53 adversarial techniques cannot be mitigated by any existing controls,
and these techniques primarily aid adversaries in bypassing system defense and
discovering targeted system information. We identify a set of 20 critical
controls that can mitigate 134 adversarial techniques, and on average, can
mitigate 72\% of all techniques used by 98\% of the cataloged adversaries in
MITRE ATT\&CK. We urge organizations, that do not have any controls enforced in
place, to implement the top controls identified in the study.


http://arxiv.org/abs/2211.06495
Investigating co-occurrences of MITRE ATT\&CK Techniques. (12%)
Md Rayhanur Rahman; Laurie Williams
  Cyberattacks use adversarial techniques to bypass system defenses, persist,
and eventually breach systems. The MITRE ATT\&CK framework catalogs a set of
adversarial techniques and maps between adversaries and their used techniques
and tactics. Understanding how adversaries deploy techniques in conjunction is
pivotal for learning adversary behavior, hunting potential threats, and
formulating a proactive defense. The goal of this research is to aid
cybersecurity practitioners and researchers in choosing detection and
mitigation strategies through co-occurrence analysis of adversarial techniques
reported in MITRE ATT&CK. We collect the adversarial techniques of 115
cybercrime groups and 484 malware from the MITRE ATT\&CK. We apply association
rule mining and network analysis to investigate how adversarial techniques
co-occur. We identify that adversaries pair T1059: Command and scripting
interface and T1105: Ingress tool transfer techniques with a relatively large
number of ATT\&CK techniques. We also identify adversaries using the T1082:
System Information Discovery technique to determine their next course of
action. We observe adversaries deploy the highest number of techniques from the
TA0005: Defense evasion and TA0007: Discovery tactics. Based on our findings on
co-occurrence, we identify six detection, six mitigation strategies, and twelve
adversary behaviors. We urge defenders to prioritize primarily the detection of
TA0007: Discovery and mitigation of TA0005: Defense evasion techniques.
Overall, this study approximates how adversaries leverage techniques based on
publicly reported documents. We advocate organizations investigate adversarial
techniques in their environment and make the findings available for a more
precise and actionable understanding.


http://arxiv.org/abs/2211.06056
Remapped Cache Layout: Thwarting Cache-Based Side-Channel Attacks with a Hardware Defense. (9%)
Wei Song; Rui Hou; Peng Liu; Xiaoxin Li; Peinan Li; Lutan Zhao; Xiaofei Fu; Yifei Sun; Dan Meng
  As cache-based side-channel attacks become serious security problems, various
defenses have been proposed and deployed in both software and hardware.
Consequently, cache-based side-channel attacks on processes co-residing on the
same core are becoming extremely difficult. Most of recent attacks then shift
their focus to the last-level cache (LLC). Although cache partitioning is
currently the most promising defense against the attacks abusing LLC, it is
ineffective in thwarting the side-channel attacks that automatically create
eviction sets or bypass the user address space layout randomization. In fact,
these attacks are largely undefended in current computer systems.
  We propose Remapped Cache Layout (\textsf{RCL}) -- a pure hardware defense
against a broad range of conflict-based side-channel attacks. \textsf{RCL}
obfuscates the mapping from address to cache sets; therefore, an attacker
cannot accurately infer the location of her data in caches or using a cache set
to infer her victim's data. To our best knowledge, it is the first defense to
thwart the aforementioned largely undefended side-channel attacks .
\textsf{RCL} has been implemented in a superscalar processor and detailed
evaluation results show that \textsf{RCL} incurs only small costs in area,
frequency and execution time.


http://arxiv.org/abs/2211.05854
Test-time adversarial detection and robustness for localizing humans using ultra wide band channel impulse responses. (99%)
Abhiram Kolli; Muhammad Jehanzeb Mirza; Horst Possegger; Horst Bischof
  Keyless entry systems in cars are adopting neural networks for localizing its
operators. Using test-time adversarial defences equip such systems with the
ability to defend against adversarial attacks without prior training on
adversarial samples. We propose a test-time adversarial example detector which
detects the input adversarial example through quantifying the localized
intermediate responses of a pre-trained neural network and confidence scores of
an auxiliary softmax layer. Furthermore, in order to make the network robust,
we extenuate the non-relevant features by non-iterative input sample clipping.
Using our approach, mean performance over 15 levels of adversarial
perturbations is increased by 55.33% for the fast gradient sign method (FGSM)
and 6.3% for both the basic iterative method (BIM) and the projected gradient
method (PGD).


http://arxiv.org/abs/2211.05523
Impact of Adversarial Training on Robustness and Generalizability of Language Models. (99%)
Enes Altinisik; Hassan Sajjad; Husrev Taha Sencar; Safa Messaoud; Sanjay Chawla
  Adversarial training is widely acknowledged as the most effective defense
against adversarial attacks. However, it is also well established that
achieving both robustness and generalization in adversarially trained models
involves a trade-off. The goal of this work is to provide an in depth
comparison of different approaches for adversarial training in language models.
Specifically, we study the effect of pre-training data augmentation as well as
training time input perturbations vs. embedding space perturbations on the
robustness and generalization of BERT-like language models. Our findings
suggest that better robustness can be achieved by pre-training data
augmentation or by training with input space perturbation. However, training
with embedding space perturbation significantly improves generalization. A
linguistic correlation analysis of neurons of the learned models reveal that
the improved generalization is due to `more specialized' neurons. To the best
of our knowledge, this is the first work to carry out a deep qualitative
analysis of different methods of generating adversarial examples in adversarial
training of language models.


http://arxiv.org/abs/2211.05446
Privacy-Utility Balanced Voice De-Identification Using Adversarial Examples. (98%)
Meng Chen; Li Lu; Jiadi Yu; Yingying Chen; Zhongjie Ba; Feng Lin; Kui Ren
  Faced with the threat of identity leakage during voice data publishing, users
are engaged in a privacy-utility dilemma when enjoying convenient voice
services. Existing studies employ direct modification or text-based
re-synthesis to de-identify users' voices, but resulting in inconsistent
audibility in the presence of human participants. In this paper, we propose a
voice de-identification system, which uses adversarial examples to balance the
privacy and utility of voice services. Instead of typical additive examples
inducing perceivable distortions, we design a novel convolutional adversarial
example that modulates perturbations into real-world room impulse responses.
Benefit from this, our system could preserve user identity from exposure by
Automatic Speaker Identification (ASI) while remaining the voice perceptual
quality for non-intrusive de-identification. Moreover, our system learns a
compact speaker distribution through a conditional variational auto-encoder to
sample diverse target embeddings on demand. Combining diverse target generation
and input-specific perturbation construction, our system enables any-to-any
identify transformation for adaptive de-identification. Experimental results
show that our system could achieve 98% and 79% successful de-identification on
mainstream ASIs and commercial systems with an objective Mel cepstral
distortion of 4.31dB and a subjective mean opinion score of 4.48.


http://arxiv.org/abs/2211.05410
Stay Home Safe with Starving Federated Data. (80%)
Jaechul Roh; Yajun Fang
  Over the past few years, the field of adversarial attack received numerous
attention from various researchers with the help of successful attack success
rate against well-known deep neural networks that were acknowledged to achieve
high classification ability in various tasks. However, majority of the
experiments were completed under a single model, which we believe it may not be
an ideal case in a real-life situation. In this paper, we introduce a novel
federated adversarial training method for smart home face recognition, named
FLATS, where we observed some interesting findings that may not be easily
noticed in a traditional adversarial attack to federated learning experiments.
By applying different variations to the hyperparameters, we have spotted that
our method can make the global model to be robust given a starving federated
environment. Our code can be found on https://github.com/jcroh0508/FLATS.


http://arxiv.org/abs/2211.05371
MSDT: Masked Language Model Scoring Defense in Text Domain. (38%)
Jaechul Roh; Minhao Cheng; Yajun Fang
  Pre-trained language models allowed us to process downstream tasks with the
help of fine-tuning, which aids the model to achieve fairly high accuracy in
various Natural Language Processing (NLP) tasks. Such easily-downloaded
language models from various websites empowered the public users as well as
some major institutions to give a momentum to their real-life application.
However, it was recently proven that models become extremely vulnerable when
they are backdoor attacked with trigger-inserted poisoned datasets by malicious
users. The attackers then redistribute the victim models to the public to
attract other users to use them, where the models tend to misclassify when
certain triggers are detected within the training sample. In this paper, we
will introduce a novel improved textual backdoor defense method, named MSDT,
that outperforms the current existing defensive algorithms in specific
datasets. The experimental results illustrate that our method can be effective
and constructive in terms of defending against backdoor attack in text domain.
Code is available at https://github.com/jcroh0508/MSDT.


http://arxiv.org/abs/2211.09954
Robust DNN Surrogate Models with Uncertainty Quantification via Adversarial Training. (3%)
Lixiang Zhang; Jia Li
  For computational efficiency, surrogate models have been used to emulate
mathematical simulators for physical or biological processes. High-speed
simulation is crucial for conducting uncertainty quantification (UQ) when the
simulation is repeated over many randomly sampled input points (aka, the Monte
Carlo method). In some cases, UQ is only feasible with a surrogate model.
Recently, Deep Neural Network (DNN) surrogate models have gained popularity for
their hard-to-match emulation accuracy. However, it is well-known that DNN is
prone to errors when input data are perturbed in particular ways, the very
motivation for adversarial training. In the usage scenario of surrogate models,
the concern is less of a deliberate attack but more of the high sensitivity of
the DNN's accuracy to input directions, an issue largely ignored by researchers
using emulation models. In this paper, we show the severity of this issue
through empirical studies and hypothesis testing. Furthermore, we adopt methods
in adversarial training to enhance the robustness of DNN surrogate models.
Experiments demonstrate that our approaches significantly improve the
robustness of the surrogate models without compromising emulation accuracy.


http://arxiv.org/abs/2211.05347
Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations. (1%)
Sheng-Feng Yu; Wei-Chen Chiu
  Online continual learning (OCL) aims to enable model learning from a
non-stationary data stream to continuously acquire new knowledge as well as
retain the learnt one, under the constraints of having limited system size and
computational cost, in which the main challenge comes from the "catastrophic
forgetting" issue -- the inability to well remember the learnt knowledge while
learning the new ones. With the specific focus on the class-incremental OCL
scenario, i.e. OCL for classification, the recent advance incorporates the
contrastive learning technique for learning more generalised feature
representation to achieve the state-of-the-art performance but is still unable
to fully resolve the catastrophic forgetting. In this paper, we follow the
strategy of adopting contrastive learning but further introduce the
semantically distinct augmentation technique, in which it leverages strong
augmentation to generate more data samples, and we show that considering these
samples semantically different from their original classes (thus being related
to the out-of-distribution samples) in the contrastive learning mechanism
contributes to alleviate forgetting and facilitate model stability. Moreover,
in addition to contrastive learning, the typical classification mechanism and
objective (i.e. softmax classifier and cross-entropy loss) are included in our
model design for faster convergence and utilising the label information, but
particularly equipped with a sampling strategy to tackle the tendency of
favouring the new classes (i.e. model bias towards the recently learnt
classes). Upon conducting extensive experiments on CIFAR-10, CIFAR-100, and
Mini-Imagenet datasets, our proposed method is shown to achieve superior
performance against various baselines.


http://arxiv.org/abs/2211.04780
On the Robustness of Explanations of Deep Neural Network Models: A Survey. (50%)
Amlan Jyoti; Karthik Balaji Ganesh; Manoj Gayala; Nandita Lakshmi Tunuguntla; Sandesh Kamath; Vineeth N Balasubramanian
  Explainability has been widely stated as a cornerstone of the responsible and
trustworthy use of machine learning models. With the ubiquitous use of Deep
Neural Network (DNN) models expanding to risk-sensitive and safety-critical
domains, many methods have been proposed to explain the decisions of these
models. Recent years have also seen concerted efforts that have shown how such
explanations can be distorted (attacked) by minor input perturbations. While
there have been many surveys that review explainability methods themselves,
there has been no effort hitherto to assimilate the different methods and
metrics proposed to study the robustness of explanations of DNN models. In this
work, we present a comprehensive survey of methods that study, understand,
attack, and defend explanations of DNN models. We also present a detailed
review of different metrics used to evaluate explanation methods, as well as
describe attributional attack and defense methods. We conclude with lessons and
take-aways for the community towards ensuring robust explanations of DNN model
predictions.


http://arxiv.org/abs/2211.05184
Are All Edges Necessary? A Unified Framework for Graph Purification. (5%)
Zishan Gu; Jintang Li; Liang Chen
  Graph Neural Networks (GNNs) as deep learning models working on
graph-structure data have achieved advanced performance in many works. However,
it has been proved repeatedly that, not all edges in a graph are necessary for
the training of machine learning models. In other words, some of the
connections between nodes may bring redundant or even misleading information to
downstream tasks. In this paper, we try to provide a method to drop edges in
order to purify the graph data from a new perspective. Specifically, it is a
framework to purify graphs with the least loss of information, under which the
core problems are how to better evaluate the edges and how to delete the
relatively redundant edges with the least loss of information. To address the
above two problems, we propose several measurements for the evaluation and
different judges and filters for the edge deletion. We also introduce a
residual-iteration strategy and a surrogate model for measurements requiring
unknown information. The experimental results show that our proposed
measurements for KL divergence with constraints to maintain the connectivity of
the graph and delete edges in an iterative way can find out the most edges
while keeping the performance of GNNs. What's more, further experiments show
that this method also achieves the best defense performance against adversarial
attacks.


http://arxiv.org/abs/2211.05249
QuerySnout: Automating the Discovery of Attribute Inference Attacks against Query-Based Systems. (3%)
Ana-Maria Cretu; Florimond Houssiau; Antoine Cully; Montjoye Yves-Alexandre de
  Although query-based systems (QBS) have become one of the main solutions to
share data anonymously, building QBSes that robustly protect the privacy of
individuals contributing to the dataset is a hard problem. Theoretical
solutions relying on differential privacy guarantees are difficult to implement
correctly with reasonable accuracy, while ad-hoc solutions might contain
unknown vulnerabilities. Evaluating the privacy provided by QBSes must thus be
done by evaluating the accuracy of a wide range of privacy attacks. However,
existing attacks require time and expertise to develop, need to be manually
tailored to the specific systems attacked, and are limited in scope. In this
paper, we develop QuerySnout (QS), the first method to automatically discover
vulnerabilities in QBSes. QS takes as input a target record and the QBS as a
black box, analyzes its behavior on one or more datasets, and outputs a
multiset of queries together with a rule to combine answers to them in order to
reveal the sensitive attribute of the target record. QS uses evolutionary
search techniques based on a novel mutation operator to find a multiset of
queries susceptible to lead to an attack, and a machine learning classifier to
infer the sensitive attribute from answers to the queries selected. We showcase
the versatility of QS by applying it to two attack scenarios, three real-world
datasets, and a variety of protection mechanisms. We show the attacks found by
QS to consistently equate or outperform, sometimes by a large margin, the best
attacks from the literature. We finally show how QS can be extended to QBSes
that require a budget, and apply QS to a simple QBS based on the Laplace
mechanism. Taken together, our results show how powerful and accurate attacks
against QBSes can already be found by an automated system, allowing for highly
complex QBSes to be automatically tested "at the pressing of a button".


http://arxiv.org/abs/2211.04946
Accountable and Explainable Methods for Complex Reasoning over Text. (2%)
Pepa Atanasova
  A major concern of Machine Learning (ML) models is their opacity. They are
deployed in an increasing number of applications where they often operate as
black boxes that do not provide explanations for their predictions. Among
others, the potential harms associated with the lack of understanding of the
models' rationales include privacy violations, adversarial manipulations, and
unfair discrimination. As a result, the accountability and transparency of ML
models have been posed as critical desiderata by works in policy and law,
philosophy, and computer science.
  In computer science, the decision-making process of ML models has been
studied by developing accountability and transparency methods. Accountability
methods, such as adversarial attacks and diagnostic datasets, expose
vulnerabilities of ML models that could lead to malicious manipulations or
systematic faults in their predictions. Transparency methods explain the
rationales behind models' predictions gaining the trust of relevant
stakeholders and potentially uncovering mistakes and unfairness in models'
decisions. To this end, transparency methods have to meet accountability
requirements as well, e.g., being robust and faithful to the underlying
rationales of a model.
  This thesis presents my research that expands our collective knowledge in the
areas of accountability and transparency of ML models developed for complex
reasoning tasks over text.


http://arxiv.org/abs/2211.04205
Preserving Semantics in Textual Adversarial Attacks. (99%)
David Herel; Hugo Cisneros; Tomas Mikolov
  Adversarial attacks in NLP challenge the way we look at language models. The
goal of this kind of adversarial attack is to modify the input text to fool a
classifier while maintaining the original meaning of the text. Although most
existing adversarial attacks claim to fulfill the constraint of semantics
preservation, careful scrutiny shows otherwise. We show that the problem lies
in the text encoders used to determine the similarity of adversarial examples,
specifically in the way they are trained. Unsupervised training methods make
these encoders more susceptible to problems with antonym recognition. To
overcome this, we introduce a simple, fully supervised sentence embedding
technique called Semantics-Preserving-Encoder (SPE). The results show that our
solution minimizes the variation in the meaning of the adversarial examples
generated. It also significantly improves the overall quality of adversarial
examples, as confirmed by human evaluators. Furthermore, it can be used as a
component in any existing attack to speed up its execution while maintaining
similar attack success.


http://arxiv.org/abs/2211.04364
NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries? (98%)
Saadia Gabriel; Hamid Palangi; Yejin Choi
  While a substantial body of prior work has explored adversarial example
generation for natural language understanding tasks, these examples are often
unrealistic and diverge from the real-world data distributions. In this work,
we introduce a two-stage adversarial example generation framework
(NaturalAdversaries), for designing adversaries that are effective at fooling a
given classifier and demonstrate natural-looking failure cases that could
plausibly occur during in-the-wild deployment of the models.
  At the first stage a token attribution method is used to summarize a given
classifier's behaviour as a function of the key tokens in the input. In the
second stage a generative model is conditioned on the key tokens from the first
stage. NaturalAdversaries is adaptable to both black-box and white-box
adversarial attacks based on the level of access to the model parameters. Our
results indicate these adversaries generalize across domains, and offer
insights for future research on improving robustness of neural text
classification models.


http://arxiv.org/abs/2211.11534
How Fraudster Detection Contributes to Robust Recommendation. (54%)
Yuni Lai; Kai Zhou
  The adversarial robustness of recommendation systems under node injection
attacks has received considerable research attention. Recently, a robust
recommendation system GraphRfi was proposed, and it was shown that GraphRfi
could successfully mitigate the effects of injected fake users in the system.
Unfortunately, we demonstrate that GraphRfi is still vulnerable to attacks due
to the supervised nature of its fraudster detection component. Specifically, we
propose a new attack metaC against GraphRfi, and further analyze why GraphRfi
fails under such an attack. Based on the insights we obtained from the
vulnerability analysis, we build a new robust recommendation system PDR by
re-designing the fraudster detection component. Comprehensive experiments show
that our defense approach outperforms other benchmark methods under attacks.
Overall, our research demonstrates an effective framework of integrating
fraudster detection into recommendation to achieve adversarial robustness.


http://arxiv.org/abs/2211.04674
Lipschitz Continuous Algorithms for Graph Problems. (16%)
Soh Kumabe; Yuichi Yoshida
  It has been widely observed in the machine learning community that a small
perturbation to the input can cause a large change in the prediction of a
trained model, and such phenomena have been intensively studied in the machine
learning community under the name of adversarial attacks. Because graph
algorithms also are widely used for decision making and knowledge discovery, it
is important to design graph algorithms that are robust against adversarial
attacks. In this study, we consider the Lipschitz continuity of algorithms as a
robustness measure and initiate a systematic study of the Lipschitz continuity
of algorithms for (weighted) graph problems.
  Depending on how we embed the output solution to a metric space, we can think
of several Lipschitzness notions. We mainly consider the one that is invariant
under scaling of weights, and we provide Lipschitz continuous algorithms and
lower bounds for the minimum spanning tree problem, the shortest path problem,
and the maximum weight matching problem. In particular, our shortest path
algorithm is obtained by first designing an algorithm for unweighted graphs
that are robust against edge contractions and then applying it to the
unweighted graph constructed from the original weighted graph.
  Then, we consider another Lipschitzness notion induced by a natural mapping
that maps the output solution to its characteristic vector. It turns out that
no Lipschitz continuous algorithm exists for this Lipschitz notion, and we
instead design algorithms with bounded pointwise Lipschitz constants for the
minimum spanning tree problem and the maximum weight bipartite matching
problem. Our algorithm for the latter problem is based on an LP relaxation with
entropy regularization.


http://arxiv.org/abs/2211.04177
Learning advisor networks for noisy image classification. (1%)
Simone Ricci; Tiberio Uricchio; Bimbo Alberto Del
  In this paper, we introduced the novel concept of advisor network to address
the problem of noisy labels in image classification. Deep neural networks (DNN)
are prone to performance reduction and overfitting problems on training data
with noisy annotations. Weighting loss methods aim to mitigate the influence of
noisy labels during the training, completely removing their contribution. This
discarding process prevents DNNs from learning wrong associations between
images and their correct labels but reduces the amount of data used, especially
when most of the samples have noisy labels. Differently, our method weighs the
feature extracted directly from the classifier without altering the loss value
of each data. The advisor helps to focus only on some part of the information
present in mislabeled examples, allowing the classifier to leverage that data
as well. We trained it with a meta-learning strategy so that it can adapt
throughout the training of the main model. We tested our method on CIFAR10 and
CIFAR100 with synthetic noise, and on Clothing1M which contains real-world
noise, reporting state-of-the-art results.


http://arxiv.org/abs/2211.03769
Are AlphaZero-like Agents Robust to Adversarial Perturbations? (99%)
Li-Cheng Lan; Huan Zhang; Ti-Rong Wu; Meng-Yu Tsai; I-Chen Wu; Cho-Jui Hsieh
  The success of AlphaZero (AZ) has demonstrated that neural-network-based Go
AIs can surpass human performance by a large margin. Given that the state space
of Go is extremely large and a human player can play the game from any legal
state, we ask whether adversarial states exist for Go AIs that may lead them to
play surprisingly wrong actions. In this paper, we first extend the concept of
adversarial examples to the game of Go: we generate perturbed states that are
``semantically'' equivalent to the original state by adding meaningless moves
to the game, and an adversarial state is a perturbed state leading to an
undoubtedly inferior action that is obvious even for Go beginners. However,
searching the adversarial state is challenging due to the large, discrete, and
non-differentiable search space. To tackle this challenge, we develop the first
adversarial attack on Go AIs that can efficiently search for adversarial states
by strategically reducing the search space. This method can also be extended to
other board games such as NoGo. Experimentally, we show that the actions taken
by both Policy-Value neural network (PV-NN) and Monte Carlo tree search (MCTS)
can be misled by adding one or two meaningless stones; for example, on 58\% of
the AlphaGo Zero self-play games, our method can make the widely used KataGo
agent with 50 simulations of MCTS plays a losing action by adding two
meaningless stones. We additionally evaluated the adversarial examples found by
our algorithm with amateur human Go players and 90\% of examples indeed lead
the Go agent to play an obviously inferior action. Our code is available at
\url{https://PaperCode.cc/GoAttack}.


http://arxiv.org/abs/2211.03509
Black-Box Attack against GAN-Generated Image Detector with Contrastive Perturbation. (82%)
Zijie Lou; Gang Cao; Man Lin
  Visually realistic GAN-generated facial images raise obvious concerns on
potential misuse. Many effective forensic algorithms have been developed to
detect such synthetic images in recent years. It is significant to assess the
vulnerability of such forensic detectors against adversarial attacks. In this
paper, we propose a new black-box attack method against GAN-generated image
detectors. A novel contrastive learning strategy is adopted to train the
encoder-decoder network based anti-forensic model under a contrastive loss
function. GAN images and their simulated real counterparts are constructed as
positive and negative samples, respectively. Leveraging on the trained attack
model, imperceptible contrastive perturbation could be applied to input
synthetic images for removing GAN fingerprint to some extent. As such, existing
GAN-generated image detectors are expected to be deceived. Extensive
experimental results verify that the proposed attack effectively reduces the
accuracy of three state-of-the-art detectors on six popular GANs. High visual
quality of the attacked images is also achieved. The source code will be
available at https://github.com/ZXMMD/BAttGAND.


http://arxiv.org/abs/2211.03714
Deviations in Representations Induced by Adversarial Attacks. (70%)
Daniel Steinberg; Paul Munro
  Deep learning has been a popular topic and has achieved success in many
areas. It has drawn the attention of researchers and machine learning
practitioners alike, with developed models deployed to a variety of settings.
Along with its achievements, research has shown that deep learning models are
vulnerable to adversarial attacks. This finding brought about a new direction
in research, whereby algorithms were developed to attack and defend vulnerable
networks. Our interest is in understanding how these attacks effect change on
the intermediate representations of deep learning models. We present a method
for measuring and analyzing the deviations in representations induced by
adversarial attacks, progressively across a selected set of layers. Experiments
are conducted using an assortment of attack algorithms, on the CIFAR-10
dataset, with plots created to visualize the impact of adversarial attacks
across different layers in a network.


http://arxiv.org/abs/2211.03637
Interpreting deep learning output for out-of-distribution detection. (1%)
Damian Matuszewski; Ida-Maria Sintorn
  Commonly used AI networks are very self-confident in their predictions, even
when the evidence for a certain decision is dubious. The investigation of a
deep learning model output is pivotal for understanding its decision processes
and assessing its capabilities and limitations. By analyzing the distributions
of raw network output vectors, it can be observed that each class has its own
decision boundary and, thus, the same raw output value has different support
for different classes. Inspired by this fact, we have developed a new method
for out-of-distribution detection. The method offers an explanatory step beyond
simple thresholding of the softmax output towards understanding and
interpretation of the model learning process and its output. Instead of
assigning the class label of the highest logit to each new sample presented to
the network, it takes the distributions over all classes into consideration. A
probability score interpreter (PSI) is created based on the joint logit values
in relation to their respective correct vs wrong class distributions. The PSI
suggests whether the sample is likely to belong to a specific class, whether
the network is unsure, or whether the sample is likely an outlier or unknown
type for the network. The simple PSI has the benefit of being applicable on
already trained networks. The distributions for correct vs wrong class for each
output node are established by simply running the training examples through the
trained network. We demonstrate our OOD detection method on a challenging
transmission electron microscopy virus image dataset. We simulate a real-world
application in which images of virus types unknown to a trained virus
classifier, yet acquired with the same procedures and instruments, constitute
the OOD samples.


http://arxiv.org/abs/2211.03489
Resilience of Wireless Ad Hoc Federated Learning against Model Poisoning Attacks. (1%)
Naoya Tezuka; Hideya Ochiai; Yuwei Sun; Hiroshi Esaki
  Wireless ad hoc federated learning (WAFL) is a fully decentralized
collaborative machine learning framework organized by opportunistically
encountered mobile nodes. Compared to conventional federated learning, WAFL
performs model training by weakly synchronizing the model parameters with
others, and this shows great resilience to a poisoned model injected by an
attacker. In this paper, we provide our theoretical analysis of the WAFL's
resilience against model poisoning attacks, by formulating the force balance
between the poisoned model and the legitimate model. According to our
experiments, we confirmed that the nodes directly encountered the attacker has
been somehow compromised to the poisoned model but other nodes have shown great
resilience. More importantly, after the attacker has left the network, all the
nodes have finally found stronger model parameters combined with the poisoned
model. Most of the attack-experienced cases achieved higher accuracy than the
no-attack-experienced cases.


http://arxiv.org/abs/2211.03073
Contrastive Weighted Learning for Near-Infrared Gaze Estimation. (31%)
Adam Lee
  Appearance-based gaze estimation has been very successful with the use of
deep learning. Many following works improved domain generalization for gaze
estimation. However, even though there has been much progress in domain
generalization for gaze estimation, most of the recent work have been focused
on cross-dataset performance -- accounting for different distributions in
illuminations, head pose, and lighting. Although improving gaze estimation in
different distributions of RGB images is important, near-infrared image based
gaze estimation is also critical for gaze estimation in dark settings. Also
there are inherent limitations relying solely on supervised learning for
regression tasks. This paper contributes to solving these problems and proposes
GazeCWL, a novel framework for gaze estimation with near-infrared images using
contrastive learning. This leverages adversarial attack techniques for data
augmentation and a novel contrastive loss function specifically for regression
tasks that effectively clusters the features of different samples in the latent
space. Our model outperforms previous domain generalization models in infrared
image based gaze estimation and outperforms the baseline by 45.6\% while
improving the state-of-the-art by 8.6\%, we demonstrate the efficacy of our
method.


http://arxiv.org/abs/2211.02878
Textual Manifold-based Defense Against Natural Language Adversarial Examples. (99%)
Dang Minh Nguyen; Luu Anh Tuan
  Recent studies on adversarial images have shown that they tend to leave the
underlying low-dimensional data manifold, making them significantly more
challenging for current models to make correct predictions. This so-called
off-manifold conjecture has inspired a novel line of defenses against
adversarial attacks on images. In this study, we find a similar phenomenon
occurs in the contextualized embedding space induced by pretrained language
models, in which adversarial texts tend to have their embeddings diverge from
the manifold of natural ones. Based on this finding, we propose Textual
Manifold-based Defense (TMD), a defense mechanism that projects text embeddings
onto an approximated embedding manifold before classification. It reduces the
complexity of potential adversarial examples, which ultimately enhances the
robustness of the protected model. Through extensive experiments, our method
consistently and significantly outperforms previous defenses under various
attack settings without trading off clean accuracy. To the best of our
knowledge, this is the first NLP defense that leverages the manifold structure
against adversarial attacks. Our code is available at
\url{https://github.com/dangne/tmd}.


http://arxiv.org/abs/2211.02885
Stateful Detection of Adversarial Reprogramming. (96%)
Yang Zheng; Xiaoyi Feng; Zhaoqiang Xia; Xiaoyue Jiang; Maura Pintor; Ambra Demontis; Battista Biggio; Fabio Roli
  Adversarial reprogramming allows stealing computational resources by
repurposing machine learning models to perform a different task chosen by the
attacker. For example, a model trained to recognize images of animals can be
reprogrammed to recognize medical images by embedding an adversarial program in
the images provided as inputs. This attack can be perpetrated even if the
target model is a black box, supposed that the machine-learning model is
provided as a service and the attacker can query the model and collect its
outputs. So far, no defense has been demonstrated effective in this scenario.
We show for the first time that this attack is detectable using stateful
defenses, which store the queries made to the classifier and detect the
abnormal cases in which they are similar. Once a malicious query is detected,
the account of the user who made it can be blocked. Thus, the attacker must
create many accounts to perpetrate the attack. To decrease this number, the
attacker could create the adversarial program against a surrogate classifier
and then fine-tune it by making few queries to the target model. In this
scenario, the effectiveness of the stateful defense is reduced, but we show
that it is still effective.


http://arxiv.org/abs/2211.03013
Robust Lottery Tickets for Pre-trained Language Models. (83%)
Rui Zheng; Rong Bao; Yuhao Zhou; Di Liang; Sirui Wang; Wei Wu; Tao Gui; Qi Zhang; Xuanjing Huang
  Recent works on Lottery Ticket Hypothesis have shown that pre-trained
language models (PLMs) contain smaller matching subnetworks(winning tickets)
which are capable of reaching accuracy comparable to the original models.
However, these tickets are proved to be notrobust to adversarial examples, and
even worse than their PLM counterparts. To address this problem, we propose a
novel method based on learning binary weight masks to identify robust tickets
hidden in the original PLMs. Since the loss is not differentiable for the
binary mask, we assign the hard concrete distribution to the masks and
encourage their sparsity using a smoothing approximation of L0
regularization.Furthermore, we design an adversarial loss objective to guide
the search for robust tickets and ensure that the tickets perform well bothin
accuracy and robustness. Experimental results show the significant improvement
of the proposed method over previous work on adversarial robustness evaluation.


http://arxiv.org/abs/2211.02468
Improving Adversarial Robustness to Sensitivity and Invariance Attacks with Deep Metric Learning. (99%)
Anaelia Ovalle; Evan Czyzycki; Cho-Jui Hsieh
  Intentionally crafted adversarial samples have effectively exploited
weaknesses in deep neural networks. A standard method in adversarial robustness
assumes a framework to defend against samples crafted by minimally perturbing a
sample such that its corresponding model output changes. These sensitivity
attacks exploit the model's sensitivity toward task-irrelevant features.
Another form of adversarial sample can be crafted via invariance attacks, which
exploit the model underestimating the importance of relevant features. Previous
literature has indicated a tradeoff in defending against both attack types
within a strictly L_p bounded defense. To promote robustness toward both types
of attacks beyond Euclidean distance metrics, we use metric learning to frame
adversarial regularization as an optimal transport problem. Our preliminary
results indicate that regularizing over invariant perturbations in our
framework improves both invariant and sensitivity defense.


http://arxiv.org/abs/2211.02272
Logits are predictive of network type. (68%)
Ali Borji
  We show that it is possible to predict which deep network has generated a
given logit vector with accuracy well above chance. We utilize a number of
networks on a dataset, initialized with random weights or pretrained weights,
as well as fine-tuned networks. A classifier is then trained on the logit
vectors of the trained set of this dataset to map the logit vector to the
network index that has generated it. The classifier is then evaluated on the
test set of the dataset. Results are better with randomly initialized networks,
but also generalize to pretrained networks as well as fine-tuned ones.
Classification accuracy is higher using unnormalized logits than normalized
ones. We find that there is little transfer when applying a classifier to the
same networks but with different sets of weights. In addition to help better
understand deep networks and the way they encode uncertainty, we anticipate our
finding to be useful in some applications (e.g. tailoring an adversarial attack
for a certain type of network). Code is available at
https://github.com/aliborji/logits.


http://arxiv.org/abs/2211.02675
An Adversarial Robustness Perspective on the Topology of Neural Networks. (64%)
Morgane Goibert; Thomas Ricatte; Elvis Dohmatob
  In this paper, we investigate the impact of neural networks (NNs) topology on
adversarial robustness. Specifically, we study the graph produced when an input
traverses all the layers of a NN, and show that such graphs are different for
clean and adversarial inputs. We find that graphs from clean inputs are more
centralized around highway edges, whereas those from adversaries are more
diffuse, leveraging under-optimized edges. Through experiments on a variety of
datasets and architectures, we show that these under-optimized edges are a
source of adversarial vulnerability and that they can be used to detect
adversarial inputs.


http://arxiv.org/abs/2211.04449
Fairness-aware Regression Robust to Adversarial Attacks. (38%)
Yulu Jin; Lifeng Lai
  In this paper, we take a first step towards answering the question of how to
design fair machine learning algorithms that are robust to adversarial attacks.
Using a minimax framework, we aim to design an adversarially robust fair
regression model that achieves optimal performance in the presence of an
attacker who is able to add a carefully designed adversarial data point to the
dataset or perform a rank-one attack on the dataset. By solving the proposed
nonsmooth nonconvex-nonconcave minimax problem, the optimal adversary as well
as the robust fairness-aware regression model are obtained. For both synthetic
data and real-world datasets, numerical results illustrate that the proposed
adversarially robust fair models have better performance on poisoned datasets
than other fair machine learning models in both prediction accuracy and
group-based fairness measure.


http://arxiv.org/abs/2211.02755
Extension of Simple Algorithms to the Matroid Secretary Problem. (9%)
Simon Park
  Whereas there are simple algorithms that are proven to be optimal for the
Classical and the Multiple Choice Secretary Problem, the Matroid Secretary
Problem is less thoroughly understood. This paper proposes the generalization
of some simple algorithms from the Classical and Multiple Choice versions on
the Matroid Secretary Problem. Out of two algorithms that make decisions based
on samples, like the Dynkin's algorithm, one is proven to be an instance of
Greedy Algorithm (Bahrani et al., 2022), while the other is not. A generalized
version of the Virtual Algorithm (Babaioff et al., 2018) obtains a constant
competitive ratio for the Hat Graph, the adversarial example for Greedy
Algorithms, but fails to do so when a slight modificiation is introduced to the
graph. We show that there is no algorithm with Strong Forbidden Sets (Soto et
al., 2021) of size 1 on all graphic matroids.


http://arxiv.org/abs/2211.02646
Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions. (3%)
Gaurav Verma; Vishwa Vinay; Ryan A. Rossi; Srijan Kumar
  As multimodal learning finds applications in a wide variety of high-stakes
societal tasks, investigating their robustness becomes important. Existing work
has focused on understanding the robustness of vision-and-language models to
imperceptible variations on benchmark tasks. In this work, we investigate the
robustness of multimodal classifiers to cross-modal dilutions - a plausible
variation. We develop a model that, given a multimodal (image + text) input,
generates additional dilution text that (a) maintains relevance and topical
coherence with the image and existing text, and (b) when added to the original
text, leads to misclassification of the multimodal input. Via experiments on
Crisis Humanitarianism and Sentiment Detection tasks, we find that the
performance of task-specific fusion-based multimodal classifiers drops by 23.3%
and 22.5%, respectively, in the presence of dilutions generated by our model.
Metric-based comparisons with several baselines and human evaluations indicate
that our dilutions show higher relevance and topical coherence, while
simultaneously being more effective at demonstrating the brittleness of the
multimodal classifiers. Our work aims to highlight and encourage further
research on the robustness of deep multimodal models to realistic variations,
especially in human-facing societal applications. The code and other resources
are available at https://claws-lab.github.io/multimodal-robustness/.


http://arxiv.org/abs/2211.02578
Data Models for Dataset Drift Controls in Machine Learning With Images. (1%)
Luis Oala; Marco Aversa; Gabriel Nobis; Kurt Willis; Yoan Neuenschwander; Michèle Buck; Christian Matek; Jerome Extermann; Enrico Pomarico; Wojciech Samek; Roderick Murray-Smith; Christoph Clausen; Bruno Sanguinetti
  Camera images are ubiquitous in machine learning research. They also play a
central role in the delivery of important services spanning medicine and
environmental surveying. However, the application of machine learning models in
these domains has been limited because of robustness concerns. A primary
failure mode are performance drops due to differences between the training and
deployment data. While there are methods to prospectively validate the
robustness of machine learning models to such dataset drifts, existing
approaches do not account for explicit models of the primary object of
interest: the data. This makes it difficult to create physically faithful drift
test cases or to provide specifications of data models that should be avoided
when deploying a machine learning model. In this study, we demonstrate how
these shortcomings can be overcome by pairing machine learning robustness
validation with physical optics. We examine the role raw sensor data and
differentiable data models can play in controlling performance risks related to
image dataset drift. The findings are distilled into three applications. First,
drift synthesis enables the controlled generation of physically faithful drift
test cases. The experiments presented here show that the average decrease in
model performance is ten to four times less severe than under post-hoc
augmentation testing. Second, the gradient connection between task and data
models allows for drift forensics that can be used to specify
performance-sensitive data models which should be avoided during deployment of
a machine learning model. Third, drift adjustment opens up the possibility for
processing adjustments in the face of drift. This can lead to speed up and
stabilization of classifier training at a margin of up to 20% in validation
accuracy. A guide to access the open code and datasets is available at
https://github.com/aiaudit-org/raw2logit.


http://arxiv.org/abs/2211.01671
Physically Adversarial Attacks and Defenses in Computer Vision: A Survey. (99%)
Xingxing Wei; Bangzheng Pu; Jiefan Lu; Baoyuan Wu
  Although Deep Neural Networks (DNNs) have been widely applied in various
real-world scenarios, they are vulnerable to adversarial examples. The current
adversarial attacks in computer vision can be divided into digital attacks and
physical attacks according to their different attack forms. Compared with
digital attacks, which generate perturbations in the digital pixels, physical
attacks are more practical in the real world. Owing to the serious security
problem caused by physically adversarial examples, many works have been
proposed to evaluate the physically adversarial robustness of DNNs in the past
years. In this paper, we summarize a survey versus the current physically
adversarial attacks and physically adversarial defenses in computer vision. To
establish a taxonomy, we organize the current physical attacks from attack
tasks, attack forms, and attack methods, respectively. Thus, readers can have a
systematic knowledge about this topic from different aspects. For the physical
defenses, we establish the taxonomy from pre-processing, in-processing, and
post-processing for the DNN models to achieve a full coverage of the
adversarial defenses. Based on the above survey, we finally discuss the
challenges of this research field and further outlook the future direction.


http://arxiv.org/abs/2211.02223
Adversarial Defense via Neural Oscillation inspired Gradient Masking. (98%)
Chunming Jiang; Yilei Zhang
  Spiking neural networks (SNNs) attract great attention due to their low power
consumption, low latency, and biological plausibility. As they are widely
deployed in neuromorphic devices for low-power brain-inspired computing,
security issues become increasingly important. However, compared to deep neural
networks (DNNs), SNNs currently lack specifically designed defense methods
against adversarial attacks. Inspired by neural membrane potential oscillation,
we propose a novel neural model that incorporates the bio-inspired oscillation
mechanism to enhance the security of SNNs. Our experiments show that SNNs with
neural oscillation neurons have better resistance to adversarial attacks than
ordinary SNNs with LIF neurons on kinds of architectures and datasets.
Furthermore, we propose a defense method that changes model's gradients by
replacing the form of oscillation, which hides the original training gradients
and confuses the attacker into using gradients of 'fake' neurons to generate
invalid adversarial samples. Our experiments suggest that the proposed defense
method can effectively resist both single-step and iterative attacks with
comparable defense effectiveness and much less computational costs than
adversarial training methods on DNNs. To the best of our knowledge, this is the
first work that establishes adversarial defense through masking surrogate
gradients on SNNs.


http://arxiv.org/abs/2211.01875
M-to-N Backdoor Paradigm: A Stealthy and Fuzzy Attack to Deep Learning Models. (98%)
Linshan Hou; Zhongyun Hua; Yuhong Li; Leo Yu Zhang
  Recent studies show that deep neural networks (DNNs) are vulnerable to
backdoor attacks. A backdoor DNN model behaves normally with clean inputs,
whereas outputs attacker's expected behaviors when the inputs contain a
pre-defined pattern called a trigger. However, in some tasks, the attacker
cannot know the exact target that shows his/her expected behavior, because the
task may contain a large number of classes and the attacker does not have full
access to know the semantic details of these classes. Thus, the attacker is
willing to attack multiple suspected targets to achieve his/her purpose. In
light of this, in this paper, we propose the M-to-N backdoor attack, a new
attack paradigm that allows an attacker to launch a fuzzy attack by
simultaneously attacking N suspected targets, and each of the N targets can be
activated by any one of its M triggers. To achieve a better stealthiness, we
randomly select M clean images from the training dataset as our triggers for
each target. Since the triggers used in our attack have the same distribution
as the clean images, the inputs poisoned by the triggers are difficult to be
detected by the input-based defenses, and the backdoor models trained on the
poisoned training dataset are also difficult to be detected by the model-based
defenses. Thus, our attack is stealthier and has a higher probability of
achieving the attack purpose by attacking multiple suspected targets
simultaneously in contrast to prior backdoor attacks. Extensive experiments
show that our attack is effective against different datasets with various
models and achieves high attack success rates (e.g., 99.43% for attacking 2
targets and 98.23% for attacking 4 targets on the CIFAR-10 dataset) when
poisoning only an extremely small portion of the training dataset (e.g., less
than 2%). Besides, it is robust to pre-processing operations and can resist
state-of-the-art defenses.


http://arxiv.org/abs/2211.01598
Robust Few-shot Learning Without Using any Adversarial Samples. (89%)
Gaurav Kumar Nayak; Ruchit Rawal; Inder Khatri; Anirban Chakraborty
  The high cost of acquiring and annotating samples has made the `few-shot'
learning problem of prime importance. Existing works mainly focus on improving
performance on clean data and overlook robustness concerns on the data
perturbed with adversarial noise. Recently, a few efforts have been made to
combine the few-shot problem with the robustness objective using sophisticated
Meta-Learning techniques. These methods rely on the generation of adversarial
samples in every episode of training, which further adds a computational
burden. To avoid such time-consuming and complicated procedures, we propose a
simple but effective alternative that does not require any adversarial samples.
Inspired by the cognitive decision-making process in humans, we enforce
high-level feature matching between the base class data and their corresponding
low-frequency samples in the pretraining stage via self distillation. The model
is then fine-tuned on the samples of novel classes where we additionally
improve the discriminability of low-frequency query set features via cosine
similarity. On a 1-shot setting of the CIFAR-FS dataset, our method yields a
massive improvement of $60.55\%$ & $62.05\%$ in adversarial accuracy on the PGD
and state-of-the-art Auto Attack, respectively, with a minor drop in clean
accuracy compared to the baseline. Moreover, our method only takes $1.69\times$
of the standard training time while being $\approx$ $5\times$ faster than
state-of-the-art adversarial meta-learning methods. The code is available at
https://github.com/vcl-iisc/robust-few-shot-learning.


http://arxiv.org/abs/2211.01579
Data-free Defense of Black Box Models Against Adversarial Attacks. (84%)
Gaurav Kumar Nayak; Inder Khatri; Shubham Randive; Ruchit Rawal; Anirban Chakraborty
  Several companies often safeguard their trained deep models (i.e. details of
architecture, learnt weights, training details etc.) from third-party users by
exposing them only as black boxes through APIs. Moreover, they may not even
provide access to the training data due to proprietary reasons or sensitivity
concerns. We make the first attempt to provide adversarial robustness to the
black box models in a data-free set up. We construct synthetic data via
generative model and train surrogate network using model stealing techniques.
To minimize adversarial contamination on perturbed samples, we propose `wavelet
noise remover' (WNR) that performs discrete wavelet decomposition on input
images and carefully select only a few important coefficients determined by our
`wavelet coefficient selection module' (WCSM). To recover the high-frequency
content of the image after noise removal via WNR, we further train a
`regenerator' network with an objective to retrieve the coefficients such that
the reconstructed image yields similar to original predictions on the surrogate
model. At test time, WNR combined with trained regenerator network is prepended
to the black box network, resulting in a high boost in adversarial accuracy.
Our method improves the adversarial accuracy on CIFAR-10 by 38.98% and 32.01%
on state-of-the-art Auto Attack compared to baseline, even when the attacker
uses surrogate architecture (Alexnet-half and Alexnet) similar to the black box
architecture (Alexnet) with same model stealing strategy as defender. The code
is available at https://github.com/vcl-iisc/data-free-black-box-defense


http://arxiv.org/abs/2211.01621
Leveraging Domain Features for Detecting Adversarial Attacks Against Deep Speech Recognition in Noise. (38%)
Christian Heider Nielsen; Zheng-Hua Tan
  In recent years, significant progress has been made in deep model-based
automatic speech recognition (ASR), leading to its widespread deployment in the
real world. At the same time, adversarial attacks against deep ASR systems are
highly successful. Various methods have been proposed to defend ASR systems
from these attacks. However, existing classification based methods focus on the
design of deep learning models while lacking exploration of domain specific
features. This work leverages filter bank-based features to better capture the
characteristics of attacks for improved detection. Furthermore, the paper
analyses the potentials of using speech and non-speech parts separately in
detecting adversarial attacks. In the end, considering adverse environments
where ASR systems may be deployed, we study the impact of acoustic noise of
various types and signal-to-noise ratios. Extensive experiments show that the
inverse filter bank features generally perform better in both clean and noisy
environments, the detection is effective using either speech or non-speech
part, and the acoustic noise can largely degrade the detection performance.


http://arxiv.org/abs/2211.01592
Try to Avoid Attacks: A Federated Data Sanitization Defense for Healthcare IoMT Systems. (33%)
Chong Chen; Ying Gao; Leyu Shi; Siquan Huang
  Healthcare IoMT systems are becoming intelligent, miniaturized, and more
integrated into daily life. As for the distributed devices in the IoMT,
federated learning has become a topical area with cloud-based training
procedures when meeting data security. However, the distribution of IoMT has
the risk of protection from data poisoning attacks. Poisoned data can be
fabricated by falsifying medical data, which urges a security defense to IoMT
systems. Due to the lack of specific labels, the filtering of malicious data is
a unique unsupervised scenario. One of the main challenges is finding robust
data filtering methods for various poisoning attacks. This paper introduces a
Federated Data Sanitization Defense, a novel approach to protect the system
from data poisoning attacks. To solve this unsupervised problem, we first use
federated learning to project all the data to the subspace domain, allowing
unified feature mapping to be established since the data is stored locally.
Then we adopt the federated clustering to re-group their features to clarify
the poisoned data. The clustering is based on the consistent association of
data and its semantics. After we get the clustering of the private data, we do
the data sanitization with a simple yet efficient strategy. In the end, each
device of distributed ImOT is enabled to filter malicious data according to
federated data sanitization. Extensive experiments are conducted to evaluate
the efficacy of the proposed defense method against data poisoning attacks.
Further, we consider our approach in the different poisoning ratios and achieve
a high Accuracy and a low attack success rate.


http://arxiv.org/abs/2211.02245
Unintended Memorization and Timing Attacks in Named Entity Recognition Models. (12%)
Rana Salal Ali; Benjamin Zi Hao Zhao; Hassan Jameel Asghar; Tham Nguyen; Ian David Wood; Dali Kaafar
  Named entity recognition models (NER), are widely used for identifying named
entities (e.g., individuals, locations, and other information) in text
documents. Machine learning based NER models are increasingly being applied in
privacy-sensitive applications that need automatic and scalable identification
of sensitive information to redact text for data sharing. In this paper, we
study the setting when NER models are available as a black-box service for
identifying sensitive information in user documents and show that these models
are vulnerable to membership inference on their training datasets. With updated
pre-trained NER models from spaCy, we demonstrate two distinct membership
attacks on these models. Our first attack capitalizes on unintended
memorization in the NER's underlying neural network, a phenomenon NNs are known
to be vulnerable to. Our second attack leverages a timing side-channel to
target NER models that maintain vocabularies constructed from the training
data. We show that different functional paths of words within the training
dataset in contrast to words not previously seen have measurable differences in
execution time. Revealing membership status of training samples has clear
privacy implications, e.g., in text redaction, sensitive words or phrases to be
found and removed, are at risk of being detected in the training dataset. Our
experimental evaluation includes the redaction of both password and health
data, presenting both security risks and privacy/regulatory issues. This is
exacerbated by results that show memorization with only a single phrase. We
achieved 70% AUC in our first attack on a text redaction use-case. We also show
overwhelming success in the timing attack with 99.23% AUC. Finally we discuss
potential mitigation approaches to realize the safe use of NER models in light
of the privacy and security implications of membership inference attacks.


http://arxiv.org/abs/2211.00887
Certified Robustness of Quantum Classifiers against Adversarial Examples through Quantum Noise. (99%)
Jhih-Cing Huang; Yu-Lin Tsai; Chao-Han Huck Yang; Cheng-Fang Su; Chia-Mu Yu; Pin-Yu Chen; Sy-Yen Kuo
  Recently, quantum classifiers have been known to be vulnerable to adversarial
attacks, where quantum classifiers are fooled by imperceptible noises to have
misclassification. In this paper, we propose one first theoretical study that
utilizing the added quantum random rotation noise can improve the robustness of
quantum classifiers against adversarial attacks. We connect the definition of
differential privacy and demonstrate the quantum classifier trained with the
natural presence of additive noise is differentially private. Lastly, we derive
a certified robustness bound to enable quantum classifiers to defend against
adversarial examples supported by experimental results.


http://arxiv.org/abs/2211.01182
Defending with Errors: Approximate Computing for Robustness of Deep Neural Networks. (99%)
Amira Guesmi; Ihsen Alouani; Khaled N. Khasawneh; Mouna Baklouti; Tarek Frikha; Mohamed Abid; Nael Abu-Ghazaleh
  Machine-learning architectures, such as Convolutional Neural Networks (CNNs)
are vulnerable to adversarial attacks: inputs crafted carefully to force the
system output to a wrong label. Since machine-learning is being deployed in
safety-critical and security-sensitive domains, such attacks may have
catastrophic security and safety consequences. In this paper, we propose for
the first time to use hardware-supported approximate computing to improve the
robustness of machine-learning classifiers. We show that successful adversarial
attacks against the exact classifier have poor transferability to the
approximate implementation. Surprisingly, the robustness advantages also apply
to white-box attacks where the attacker has unrestricted access to the
approximate classifier implementation: in this case, we show that substantially
higher levels of adversarial noise are needed to produce adversarial examples.
Furthermore, our approximate computing model maintains the same level in terms
of classification accuracy, does not require retraining, and reduces resource
utilization and energy consumption of the CNN. We conducted extensive
experiments on a set of strong adversarial attacks; We empirically show that
the proposed implementation increases the robustness of a LeNet-5, Alexnet and
VGG-11 CNNs considerably with up to 50% by-product saving in energy consumption
due to the simpler nature of the approximate logic.


http://arxiv.org/abs/2211.01093
Improving transferability of 3D adversarial attacks with scale and shear transformations. (99%)
Jinali Zhang; Yinpeng Dong; Jun Zhu; Jihong Zhu; Minchi Kuang; Xiaming Yuan
  Previous work has shown that 3D point cloud classifiers can be vulnerable to
adversarial examples. However, most of the existing methods are aimed at
white-box attacks, where the parameters and other information of the
classifiers are known in the attack, which is unrealistic for real-world
applications. In order to improve the attack performance of the black-box
classifiers, the research community generally uses the transfer-based black-box
attack. However, the transferability of current 3D attacks is still relatively
low. To this end, this paper proposes Scale and Shear (SS) Attack to generate
3D adversarial examples with strong transferability. Specifically, we randomly
scale or shear the input point cloud, so that the attack will not overfit the
white-box model, thereby improving the transferability of the attack. Extensive
experiments show that the SS attack proposed in this paper can be seamlessly
combined with the existing state-of-the-art (SOTA) 3D point cloud attack
methods to form more powerful attack methods, and the SS attack improves the
transferability over 3.6 times compare to the baseline. Moreover, while
substantially outperforming the baseline methods, the SS attack achieves SOTA
transferability under various defenses. Our code will be available online at
https://github.com/cuge1995/SS-attack


http://arxiv.org/abs/2211.01112
Adversarial Attack on Radar-based Environment Perception Systems. (99%)
Amira Guesmi; Ihsen Alouani
  Due to their robustness to degraded capturing conditions, radars are widely
used for environment perception, which is a critical task in applications like
autonomous vehicles. More specifically, Ultra-Wide Band (UWB) radars are
particularly efficient for short range settings as they carry rich information
on the environment. Recent UWB-based systems rely on Machine Learning (ML) to
exploit the rich signature of these sensors. However, ML classifiers are
susceptible to adversarial examples, which are created from raw data to fool
the classifier such that it assigns the input to the wrong class. These attacks
represent a serious threat to systems integrity, especially for safety-critical
applications. In this work, we present a new adversarial attack on UWB radars
in which an adversary injects adversarial radio noise in the wireless channel
to cause an obstacle recognition failure. First, based on signals collected in
real-life environment, we show that conventional attacks fail to generate
robust noise under realistic conditions. We propose a-RNA, i.e., Adversarial
Radio Noise Attack to overcome these issues. Specifically, a-RNA generates an
adversarial noise that is efficient without synchronization between the input
signal and the noise. Moreover, a-RNA generated noise is, by-design, robust
against pre-processing countermeasures such as filtering-based defenses.
Moreover, in addition to the undetectability objective by limiting the noise
magnitude budget, a-RNA is also efficient in the presence of sophisticated
defenses in the spectral domain by introducing a frequency budget. We believe
this work should alert about potentially critical implementations of
adversarial attacks on radar systems that should be taken seriously.


http://arxiv.org/abs/2211.01236
Isometric Representations in Neural Networks Improve Robustness. (62%)
Kosio Beshkov; Jonas Verhellen; Mikkel Elle Lepperød
  Artificial and biological agents cannon learn given completely random and
unstructured data. The structure of data is encoded in the metric relationships
between data points. In the context of neural networks, neuronal activity
within a layer forms a representation reflecting the transformation that the
layer implements on its inputs. In order to utilize the structure in the data
in a truthful manner, such representations should reflect the input distances
and thus be continuous and isometric. Supporting this statement, recent
findings in neuroscience propose that generalization and robustness are tied to
neural representations being continuously differentiable. In machine learning,
most algorithms lack robustness and are generally thought to rely on aspects of
the data that differ from those that humans use, as is commonly seen in
adversarial attacks. During cross-entropy classification, the metric and
structural properties of network representations are usually broken both
between and within classes. This side effect from training can lead to
instabilities under perturbations near locations where such structure is not
preserved. One of the standard solutions to obtain robustness is to add ad hoc
regularization terms, but to our knowledge, forcing representations to preserve
the metric structure of the input data as a stabilising mechanism has not yet
been studied. In this work, we train neural networks to perform classification
while simultaneously maintaining within-class metric structure, leading to
isometric within-class representations. Such network representations turn out
to be beneficial for accurate and robust inference. By stacking layers with
this property we create a network architecture that facilitates hierarchical
manipulation of internal neural representations. Finally, we verify that
isometric regularization improves the robustness to adversarial attacks on
MNIST.


http://arxiv.org/abs/2211.01806
BATT: Backdoor Attack with Transformation-based Triggers. (56%)
Tong Xu; Yiming Li; Yong Jiang; Shu-Tao Xia
  Deep neural networks (DNNs) are vulnerable to backdoor attacks. The backdoor
adversaries intend to maliciously control the predictions of attacked DNNs by
injecting hidden backdoors that can be activated by adversary-specified trigger
patterns during the training process. One recent research revealed that most of
the existing attacks failed in the real physical world since the trigger
contained in the digitized test samples may be different from that of the one
used for training. Accordingly, users can adopt spatial transformations as the
image pre-processing to deactivate hidden backdoors. In this paper, we explore
the previous findings from another side. We exploit classical spatial
transformations (i.e. rotation and translation) with the specific parameter as
trigger patterns to design a simple yet effective poisoning-based backdoor
attack. For example, only images rotated to a particular angle can activate the
embedded backdoor of attacked DNNs. Extensive experiments are conducted,
verifying the effectiveness of our attack under both digital and physical
settings and its resistance to existing backdoor defenses.


http://arxiv.org/abs/2211.05638
Untargeted Backdoor Attack against Object Detection. (50%)
Chengxiao Luo; Yiming Li; Yong Jiang; Shu-Tao Xia
  Recent studies revealed that deep neural networks (DNNs) are exposed to
backdoor threats when training with third-party resources (such as training
samples or backbones). The backdoored model has promising performance in
predicting benign samples, whereas its predictions can be maliciously
manipulated by adversaries based on activating its backdoors with pre-defined
trigger patterns. Currently, most of the existing backdoor attacks were
conducted on the image classification under the targeted manner. In this paper,
we reveal that these threats could also happen in object detection, posing
threatening risks to many mission-critical applications ($e.g.$, pedestrian
detection and intelligent surveillance systems). Specifically, we design a
simple yet effective poison-only backdoor attack in an untargeted manner, based
on task characteristics. We show that, once the backdoor is embedded into the
target model by our attack, it can trick the model to lose detection of any
object stamped with our trigger patterns. We conduct extensive experiments on
the benchmark dataset, showing its effectiveness in both digital and
physical-world settings and its resistance to potential defenses.


http://arxiv.org/abs/2211.09728
Generative Adversarial Training Can Improve Neural Language Models. (33%)
Sajad Movahedi; Azadeh Shakery
  While deep learning in the form of recurrent neural networks (RNNs) has
caused a significant improvement in neural language modeling, the fact that
they are extremely prone to overfitting is still a mainly unresolved issue. In
this paper we propose a regularization method based on generative adversarial
networks (GANs) and adversarial training (AT), that can prevent overfitting in
neural language models. Unlike common adversarial training methods such as the
fast gradient sign method (FGSM) that require a second back-propagation through
time, and therefore effectively require at least twice the amount of time for
regular training, the overhead of our method does not exceed more than 20% of
the training of the baselines.


http://arxiv.org/abs/2211.05631
Backdoor Defense via Suppressing Model Shortcuts. (3%)
Sheng Yang; Yiming Li; Yong Jiang; Shu-Tao Xia
  Recent studies have demonstrated that deep neural networks (DNNs) are
vulnerable to backdoor attacks during the training process. Specifically, the
adversaries intend to embed hidden backdoors in DNNs so that malicious model
predictions can be activated through pre-defined trigger patterns. In this
paper, we explore the backdoor mechanism from the angle of the model structure.
We select the skip connection for discussions, inspired by the understanding
that it helps the learning of model `shortcuts' where backdoor triggers are
usually easier to be learned. Specifically, we demonstrate that the attack
success rate (ASR) decreases significantly when reducing the outputs of some
key skip connections. Based on this observation, we design a simple yet
effective backdoor removal method by suppressing the skip connections in
critical layers selected by our method. We also implement fine-tuning on these
layers to recover high benign accuracy and to further reduce ASR. Extensive
experiments on benchmark datasets verify the effectiveness of our method.


http://arxiv.org/abs/2211.00825
LMD: A Learnable Mask Network to Detect Adversarial Examples for Speaker Verification. (99%)
Xing Chen; Jie Wang; Xiao-Lei Zhang; Wei-Qiang Zhang; Kunde Yang
  Although the security of automatic speaker verification (ASV) is seriously
threatened by recently emerged adversarial attacks, there have been some
countermeasures to alleviate the threat. However, many defense approaches not
only require the prior knowledge of the attackers but also possess weak
interpretability. To address this issue, in this paper, we propose an
attacker-independent and interpretable method, named learnable mask detector
(LMD), to separate adversarial examples from the genuine ones. It utilizes
score variation as an indicator to detect adversarial examples, where the score
variation is the absolute discrepancy between the ASV scores of an original
audio recording and its transformed audio synthesized from its masked complex
spectrogram. A core component of the score variation detector is to generate
the masked spectrogram by a neural network. The neural network needs only
genuine examples for training, which makes it an attacker-independent approach.
Its interpretability lies that the neural network is trained to minimize the
score variation of the targeted ASV, and maximize the number of the masked
spectrogram bins of the genuine training examples. Its foundation is based on
the observation that, masking out the vast majority of the spectrogram bins
with little speaker information will inevitably introduce a large score
variation to the adversarial example, and a small score variation to the
genuine example. Experimental results with 12 attackers and two representative
ASV systems show that our proposed method outperforms five state-of-the-art
baselines. The extensive experimental results can also be a benchmark for the
detection-based ASV defenses.


http://arxiv.org/abs/2211.00525
The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Training. (99%)
Junhao Dong; Seyed-Mohsen Moosavi-Dezfooli; Jianhuang Lai; Xiaohua Xie
  Although current deep learning techniques have yielded superior performance
on various computer vision tasks, yet they are still vulnerable to adversarial
examples. Adversarial training and its variants have been shown to be the most
effective approaches to defend against adversarial examples. These methods
usually regularize the difference between output probabilities for an
adversarial and its corresponding natural example. However, it may have a
negative impact if the model misclassifies a natural example. To circumvent
this issue, we propose a novel adversarial training scheme that encourages the
model to produce similar outputs for an adversarial example and its ``inverse
adversarial'' counterpart. These samples are generated to maximize the
likelihood in the neighborhood of natural examples. Extensive experiments on
various vision datasets and architectures demonstrate that our training method
achieves state-of-the-art robustness as well as natural accuracy. Furthermore,
using a universal version of inverse adversarial examples, we improve the
performance of single-step adversarial training techniques at a low
computational cost.


http://arxiv.org/abs/2211.00322
DensePure: Understanding Diffusion Models towards Adversarial Robustness. (98%)
Chaowei Xiao; Zhongzhu Chen; Kun Jin; Jiongxiao Wang; Weili Nie; Mingyan Liu; Anima Anandkumar; Bo Li; Dawn Song
  Diffusion models have been recently employed to improve certified robustness
through the process of denoising. However, the theoretical understanding of why
diffusion models are able to improve the certified robustness is still lacking,
preventing from further improvement. In this study, we close this gap by
analyzing the fundamental properties of diffusion models and establishing the
conditions under which they can enhance certified robustness. This deeper
understanding allows us to propose a new method DensePure, designed to improve
the certified robustness of a pretrained model (i.e. classifier). Given an
(adversarial) input, DensePure consists of multiple runs of denoising via the
reverse process of the diffusion model (with different random seeds) to get
multiple reversed samples, which are then passed through the classifier,
followed by majority voting of inferred labels to make the final prediction.
This design of using multiple runs of denoising is informed by our theoretical
analysis of the conditional distribution of the reversed sample. Specifically,
when the data density of a clean sample is high, its conditional density under
the reverse process in a diffusion model is also high; thus sampling from the
latter conditional distribution can purify the adversarial example and return
the corresponding clean sample with a high probability. By using the highest
density point in the conditional distribution as the reversed sample, we
identify the robust region of a given instance under the diffusion model's
reverse process. We show that this robust region is a union of multiple convex
sets, and is potentially much larger than the robust regions identified in
previous works. In practice, DensePure can approximate the label of the high
density region in the conditional distribution so that it can enhance certified
robustness.


http://arxiv.org/abs/2211.00269
Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks. (87%)
Jianan Zhou; Jianing Zhu; Jingfeng Zhang; Tongliang Liu; Gang Niu; Bo Han; Masashi Sugiyama
  Adversarial training (AT) with imperfect supervision is significant but
receives limited attention. To push AT towards more practical scenarios, we
explore a brand new yet challenging setting, i.e., AT with complementary labels
(CLs), which specify a class that a data sample does not belong to. However,
the direct combination of AT with existing methods for CLs results in
consistent failure, but not on a simple baseline of two-stage training. In this
paper, we further explore the phenomenon and identify the underlying challenges
of AT with CLs as intractable adversarial optimization and low-quality
adversarial examples. To address the above problems, we propose a new learning
strategy using gradually informative attacks, which consists of two critical
components: 1) Warm-up Attack (Warm-up) gently raises the adversarial
perturbation budgets to ease the adversarial optimization with CLs; 2)
Pseudo-Label Attack (PLA) incorporates the progressively informative model
predictions into a corrected complementary loss. Extensive experiments are
conducted to demonstrate the effectiveness of our method on a range of
benchmarked datasets. The code is publicly available at:
https://github.com/RoyalSkye/ATCL.


http://arxiv.org/abs/2211.00366
Universal Perturbation Attack on Differentiable No-Reference Image- and Video-Quality Metrics. (82%)
Ekaterina Shumitskaya; Anastasia Antsiferova; Dmitriy Vatolin
  Universal adversarial perturbation attacks are widely used to analyze image
classifiers that employ convolutional neural networks. Nowadays, some attacks
can deceive image- and video-quality metrics. So sustainability analysis of
these metrics is important. Indeed, if an attack can confuse the metric, an
attacker can easily increase quality scores. When developers of image- and
video-algorithms can boost their scores through detached processing, algorithm
comparisons are no longer fair. Inspired by the idea of universal adversarial
perturbation for classifiers, we suggest a new method to attack differentiable
no-reference quality metrics through universal perturbation. We applied this
method to seven no-reference image- and video-quality metrics (PaQ-2-PiQ,
Linearity, VSFA, MDTVSFA, KonCept512, Nima and SPAQ). For each one, we trained
a universal perturbation that increases the respective scores. We also propose
a method for assessing metric stability and identify the metrics that are the
most vulnerable and the most resistant to our attack. The existence of
successful universal perturbations appears to diminish the metric's ability to
provide reliable scores. We therefore recommend our proposed method as an
additional verification of metric reliability to complement traditional
subjective tests and benchmarks.


http://arxiv.org/abs/2211.00453
The Perils of Learning From Unlabeled Data: Backdoor Attacks on Semi-supervised Learning. (80%)
Virat Shejwalkar; Lingjuan Lyu; Amir Houmansadr
  Semi-supervised machine learning (SSL) is gaining popularity as it reduces
the cost of training ML models. It does so by using very small amounts of
(expensive, well-inspected) labeled data and large amounts of (cheap,
non-inspected) unlabeled data. SSL has shown comparable or even superior
performances compared to conventional fully-supervised ML techniques.
  In this paper, we show that the key feature of SSL that it can learn from
(non-inspected) unlabeled data exposes SSL to strong poisoning attacks. In
fact, we argue that, due to its reliance on non-inspected unlabeled data,
poisoning is a much more severe problem in SSL than in conventional
fully-supervised ML.
  Specifically, we design a backdoor poisoning attack on SSL that can be
conducted by a weak adversary with no knowledge of target SSL pipeline. This is
unlike prior poisoning attacks in fully-supervised settings that assume strong
adversaries with practically-unrealistic capabilities. We show that by
poisoning only 0.2% of the unlabeled training data, our attack can cause
misclassification of more than 80% of test inputs (when they contain the
adversary's backdoor trigger). Our attacks remain effective across twenty
combinations of benchmark datasets and SSL algorithms, and even circumvent the
state-of-the-art defenses against backdoor attacks. Our work raises significant
concerns about the practical utility of existing SSL algorithms.


http://arxiv.org/abs/2211.00748
Maximum Likelihood Distillation for Robust Modulation Classification. (69%)
Javier Maroto; Gérôme Bovet; Pascal Frossard
  Deep Neural Networks are being extensively used in communication systems and
Automatic Modulation Classification (AMC) in particular. However, they are very
susceptible to small adversarial perturbations that are carefully crafted to
change the network decision. In this work, we build on knowledge distillation
ideas and adversarial training in order to build more robust AMC systems. We
first outline the importance of the quality of the training data in terms of
accuracy and robustness of the model. We then propose to use the Maximum
Likelihood function, which could solve the AMC problem in offline settings, to
generate better training labels. Those labels teach the model to be uncertain
in challenging conditions, which permits to increase the accuracy, as well as
the robustness of the model when combined with adversarial training.
Interestingly, we observe that this increase in performance transfers to online
settings, where the Maximum Likelihood function cannot be used in practice.
Overall, this work highlights the potential of learning to be uncertain in
difficult scenarios, compared to directly removing label noise.


http://arxiv.org/abs/2211.00294
FRSUM: Towards Faithful Abstractive Summarization via Enhancing Factual Robustness. (45%)
Wenhao Wu; Wei Li; Jiachen Liu; Xinyan Xiao; Ziqiang Cao; Sujian Li; Hua Wu
  Despite being able to generate fluent and grammatical text, current Seq2Seq
summarization models still suffering from the unfaithful generation problem. In
this paper, we study the faithfulness of existing systems from a new
perspective of factual robustness which is the ability to correctly generate
factual information over adversarial unfaithful information. We first measure a
model's factual robustness by its success rate to defend against adversarial
attacks when generating factual information. The factual robustness analysis on
a wide range of current systems shows its good consistency with human judgments
on faithfulness. Inspired by these findings, we propose to improve the
faithfulness of a model by enhancing its factual robustness. Specifically, we
propose a novel training strategy, namely FRSUM, which teaches the model to
defend against both explicit adversarial samples and implicit factual
adversarial perturbations. Extensive automatic and human evaluation results
show that FRSUM consistently improves the faithfulness of various Seq2Seq
models, such as T5, BART.


http://arxiv.org/abs/2211.00463
Amplifying Membership Exposure via Data Poisoning. (22%)
Yufei Chen; Chao Shen; Yun Shen; Cong Wang; Yang Zhang
  As in-the-wild data are increasingly involved in the training stage, machine
learning applications become more susceptible to data poisoning attacks. Such
attacks typically lead to test-time accuracy degradation or controlled
misprediction. In this paper, we investigate the third type of exploitation of
data poisoning - increasing the risks of privacy leakage of benign training
samples. To this end, we demonstrate a set of data poisoning attacks to amplify
the membership exposure of the targeted class. We first propose a generic
dirty-label attack for supervised classification algorithms. We then propose an
optimization-based clean-label attack in the transfer learning scenario,
whereby the poisoning samples are correctly labeled and look "natural" to evade
human moderation. We extensively evaluate our attacks on computer vision
benchmarks. Our results show that the proposed attacks can substantially
increase the membership inference precision with minimum overall test-time
model performance degradation. To mitigate the potential negative impacts of
our attacks, we also investigate feasible countermeasures.


http://arxiv.org/abs/2211.00273
ActGraph: Prioritization of Test Cases Based on Deep Neural Network Activation Graph. (13%)
Jinyin Chen; Jie Ge; Haibin Zheng
  Widespread applications of deep neural networks (DNNs) benefit from DNN
testing to guarantee their quality. In the DNN testing, numerous test cases are
fed into the model to explore potential vulnerabilities, but they require
expensive manual cost to check the label. Therefore, test case prioritization
is proposed to solve the problem of labeling cost, e.g., activation-based and
mutation-based prioritization methods. However, most of them suffer from
limited scenarios (i.e. high confidence adversarial or false positive cases)
and high time complexity. To address these challenges, we propose the concept
of the activation graph from the perspective of the spatial relationship of
neurons. We observe that the activation graph of cases that triggers the
models' misbehavior significantly differs from that of normal cases. Motivated
by it, we design a test case prioritization method based on the activation
graph, ActGraph, by extracting the high-order node features of the activation
graph for prioritization. ActGraph explains the difference between the test
cases to solve the problem of scenario limitation. Without mutation operations,
ActGraph is easy to implement, leading to lower time complexity. Extensive
experiments on three datasets and four models demonstrate that ActGraph has the
following key characteristics. (i) Effectiveness and generalizability: ActGraph
shows competitive performance in all of the natural, adversarial and mixed
scenarios, especially in RAUC-100 improvement (~1.40). (ii) Efficiency:
ActGraph does not use complex mutation operations and runs in less time (~1/50)
than the state-of-the-art method.


http://arxiv.org/abs/2210.17140
Scoring Black-Box Models for Adversarial Robustness. (98%)
Jian Vora; Pranay Reddy Samala
  Deep neural networks are susceptible to adversarial inputs and various
methods have been proposed to defend these models against adversarial attacks
under different perturbation models. The robustness of models to adversarial
attacks has been analyzed by first constructing adversarial inputs for the
model, and then testing the model performance on the constructed adversarial
inputs. Most of these attacks require the model to be white-box, need access to
data labels, and finding adversarial inputs can be computationally expensive.
We propose a simple scoring method for black-box models which indicates their
robustness to adversarial input. We show that adversarially more robust models
have a smaller $l_1$-norm of LIME weights and sharper explanations.


http://arxiv.org/abs/2211.00239
ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation. (88%)
Tomokatsu Takahashi; Masanori Yamada; Yuuki Yamanaka; Tomoya Yamashita
  Adversarial training is the most promising method for learning robust models
against adversarial examples. A recent study has shown that knowledge
distillation between the same architectures is effective in improving the
performance of adversarial training. Exploiting knowledge distillation is a new
approach to improve adversarial training and has attracted much attention.
However, its performance is still insufficient. Therefore, we propose
Adversarial Robust Distillation with Internal Representation~(ARDIR) to utilize
knowledge distillation even more effectively. In addition to the output of the
teacher model, ARDIR uses the internal representation of the teacher model as a
label for adversarial training. This enables the student model to be trained
with richer, more informative labels. As a result, ARDIR can learn more robust
student models. We show that ARDIR outperforms previous methods in our
experiments.


http://arxiv.org/abs/2210.17546
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy. (16%)
Daphne Ippolito; Florian Tramèr; Milad Nasr; Chiyuan Zhang; Matthew Jagielski; Katherine Lee; Christopher A. Choquette-Choo; Nicholas Carlini
  Studying data memorization in neural language models helps us understand the
risks (e.g., to privacy or copyright) associated with models regurgitating
training data, and aids in the evaluation of potential countermeasures. Many
prior works -- and some recently deployed defenses -- focus on "verbatim
memorization", defined as a model generation that exactly matches a substring
from the training set. We argue that verbatim memorization definitions are too
restrictive and fail to capture more subtle forms of memorization.
Specifically, we design and implement an efficient defense based on Bloom
filters that perfectly prevents all verbatim memorization. And yet, we
demonstrate that this "perfect" filter does not prevent the leakage of training
data. Indeed, it is easily circumvented by plausible and minimally modified
"style-transfer" prompts -- and in some cases even the non-modified original
prompts -- to extract memorized information. For example, instructing the model
to output ALL-CAPITAL texts bypasses memorization checks based on verbatim
matching. We conclude by discussing potential alternative definitions and why
defining memorization is a difficult yet crucial open question for neural
language models.


http://arxiv.org/abs/2210.17029
Poison Attack and Defense on Deep Source Code Processing Models. (99%)
Jia Li; Zhuo Li; Huangzhao Zhang; Ge Li; Zhi Jin; Xing Hu; Xin Xia
  In the software engineering community, deep learning (DL) has recently been
applied to many source code processing tasks. Due to the poor interpretability
of DL models, their security vulnerabilities require scrutiny. Recently,
researchers have identified an emergent security threat, namely poison attack.
The attackers aim to inject insidious backdoors into models by poisoning the
training data with poison samples. Poisoned models work normally with clean
inputs but produce targeted erroneous results with poisoned inputs embedded
with triggers. By activating backdoors, attackers can manipulate the poisoned
models in security-related scenarios.
  To verify the vulnerability of existing deep source code processing models to
the poison attack, we present a poison attack framework for source code named
CodePoisoner as a strong imaginary enemy. CodePoisoner can produce compilable
even human-imperceptible poison samples and attack models by poisoning the
training data with poison samples. To defend against the poison attack, we
further propose an effective defense approach named CodeDetector to detect
poison samples in the training data. CodeDetector can be applied to many model
architectures and effectively defend against multiple poison attack approaches.
We apply our CodePoisoner and CodeDetector to three tasks, including defect
detection, clone detection, and code repair. The results show that (1)
CodePoisoner achieves a high attack success rate (max: 100%) in misleading
models to targeted erroneous behaviors. It validates that existing deep source
code processing models have a strong vulnerability to the poison attack. (2)
CodeDetector effectively defends against multiple poison attack approaches by
detecting (max: 100%) poison samples in the training data. We hope this work
can help practitioners notice the poison attack and inspire the design of more
advanced defense techniques.


http://arxiv.org/abs/2210.17004
Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitution. (99%)
Aiwei Liu; Honghai Yu; Xuming Hu; Shu'ang Li; Li Lin; Fukun Ma; Yawen Yang; Lijie Wen
  We propose the first character-level white-box adversarial attack method
against transformer models. The intuition of our method comes from the
observation that words are split into subtokens before being fed into the
transformer models and the substitution between two close subtokens has a
similar effect to the character modification. Our method mainly contains three
steps. First, a gradient-based method is adopted to find the most vulnerable
words in the sentence. Then we split the selected words into subtokens to
replace the origin tokenization result from the transformer tokenizer. Finally,
we utilize an adversarial loss to guide the substitution of attachable
subtokens in which the Gumbel-softmax trick is introduced to ensure gradient
propagation. Meanwhile, we introduce the visual and length constraint in the
optimization process to achieve minimum character modifications. Extensive
experiments on both sentence-level and token-level tasks demonstrate that our
method could outperform the previous attack methods in terms of success rate
and edit distance. Furthermore, human evaluation verifies our adversarial
examples could preserve their origin labels.


http://arxiv.org/abs/2210.16765
Benchmarking Adversarial Patch Against Aerial Detection. (99%)
Jiawei Lian; Shaohui Mei; Shun Zhang; Mingyang Ma
  DNNs are vulnerable to adversarial examples, which poses great security
concerns for security-critical systems. In this paper, a novel
adaptive-patch-based physical attack (AP-PA) framework is proposed, which aims
to generate adversarial patches that are adaptive in both physical dynamics and
varying scales, and by which the particular targets can be hidden from being
detected. Furthermore, the adversarial patch is also gifted with attack
effectiveness against all targets of the same class with a patch outside the
target (No need to smear targeted objects) and robust enough in the physical
world. In addition, a new loss is devised to consider more available
information of detected objects to optimize the adversarial patch, which can
significantly improve the patch's attack efficacy (Average precision drop up to
87.86% and 85.48% in white-box and black-box settings, respectively) and
optimizing efficiency. We also establish one of the first comprehensive,
coherent, and rigorous benchmarks to evaluate the attack efficacy of
adversarial patches on aerial detection tasks. Finally, several proportionally
scaled experiments are performed physically to demonstrate that the elaborated
adversarial patches can successfully deceive aerial detection algorithms in
dynamic physical circumstances. The code is available at
https://github.com/JiaweiLian/AP-PA.


http://arxiv.org/abs/2210.16777
Symmetric Saliency-based Adversarial Attack To Speaker Identification. (92%)
Jiadi Yao; Xing Chen; Xiao-Lei Zhang; Wei-Qiang Zhang; Kunde Yang
  Adversarial attack approaches to speaker identification either need high
computational cost or are not very effective, to our knowledge. To address this
issue, in this paper, we propose a novel generation-network-based approach,
called symmetric saliency-based encoder-decoder (SSED), to generate adversarial
voice examples to speaker identification. It contains two novel components.
First, it uses a novel saliency map decoder to learn the importance of speech
samples to the decision of a targeted speaker identification system, so as to
make the attacker focus on generating artificial noise to the important
samples. It also proposes an angular loss function to push the speaker
embedding far away from the source speaker. Our experimental results
demonstrate that the proposed SSED yields the state-of-the-art performance,
i.e. over 97% targeted attack success rate and a signal-to-noise level of over
39 dB on both the open-set and close-set speaker identification tasks, with a
low computational cost.


http://arxiv.org/abs/2210.16940
FI-ODE: Certified and Robust Forward Invariance in Neural ODEs. (61%)
Yujia Huang; Ivan Dario Jimenez Rodriguez; Huan Zhang; Yuanyuan Shi; Yisong Yue
  We study how to certifiably enforce forward invariance properties in neural
ODEs. Forward invariance implies that the hidden states of the ODE will stay in
a ``good'' region, and a robust version would hold even under adversarial
perturbations to the input. Such properties can be used to certify desirable
behaviors such as adversarial robustness (the hidden states stay in the region
that generates accurate classification even under input perturbations) and
safety in continuous control (the system never leaves some safe set). We
develop a general approach using tools from non-linear control theory and
sampling-based verification. Our approach empirically produces the strongest
adversarial robustness guarantees compared to prior work on certifiably robust
ODE-based models (including implicit-depth models).


http://arxiv.org/abs/2210.16915
Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Games. (9%)
The Viet Bui; Tien Mai; Thanh H. Nguyen
  Recent research on vulnerabilities of deep reinforcement learning (RL) has
shown that adversarial policies adopted by an adversary agent can influence a
target RL agent (victim agent) to perform poorly in a multi-agent environment.
In existing studies, adversarial policies are directly trained based on
experiences of interacting with the victim agent. There is a key shortcoming of
this approach; knowledge derived from historical interactions may not be
properly generalized to unexplored policy regions of the victim agent, making
the trained adversarial policy significantly less effective. In this work, we
design a new effective adversarial policy learning algorithm that overcomes
this shortcoming. The core idea of our new algorithm is to create a new
imitator to imitate the victim agent's policy while the adversarial policy will
be trained not only based on interactions with the victim agent but also based
on feedback from the imitator to forecast victim's intention. By doing so, we
can leverage the capability of imitation learning in well capturing underlying
characteristics of the victim policy only based on sample trajectories of the
victim. Our victim imitation learning model differs from prior models as the
environment's dynamics are driven by adversary's policy and will keep changing
during the adversarial policy training. We provide a provable bound to
guarantee a desired imitating policy when the adversary's policy becomes
stable. We further strengthen our adversarial policy learning by making our
imitator a stronger version of the victim. Finally, our extensive experiments
using four competitive MuJoCo game environments show that our proposed
adversarial policy learning algorithm outperforms state-of-the-art algorithms.


http://arxiv.org/abs/2210.16690
On the Need of Neuromorphic Twins to Detect Denial-of-Service Attacks on Communication Networks. (10%)
Holger Boche; Rafael F. Schaefer; H. Vincent Poor; Frank H. P. Fitzek
  As we are more and more dependent on the communication technologies,
resilience against any attacks on communication networks is important to
guarantee the digital sovereignty of our society. New developments of
communication networks tackle the problem of resilience by in-network computing
approaches for higher protocol layers, while the physical layer remains an open
problem. This is particularly true for wireless communication systems which are
inherently vulnerable to adversarial attacks due to the open nature of the
wireless medium. In denial-of-service (DoS) attacks, an active adversary is
able to completely disrupt the communication and it has been shown that Turing
machines are incapable of detecting such attacks. As Turing machines provide
the fundamental limits of digital information processing and therewith of
digital twins, this implies that even the most powerful digital twins that
preserve all information of the physical network error-free are not capable of
detecting such attacks. This stimulates the question of how powerful the
information processing hardware must be to enable the detection of DoS attacks.
Therefore, in the paper the need of neuromorphic twins is advocated and by the
use of Blum-Shub-Smale machines a first implementation that enables the
detection of DoS attacks is shown. This result holds for both cases of with and
without constraints on the input and jamming sequences of the adversary.


http://arxiv.org/abs/2210.15997
Universal Adversarial Directions. (99%)
Ching Lam Choi; Farzan Farnia
  Despite their great success in image recognition tasks, deep neural networks
(DNNs) have been observed to be susceptible to universal adversarial
perturbations (UAPs) which perturb all input samples with a single perturbation
vector. However, UAPs often struggle in transferring across DNN architectures
and lead to challenging optimization problems. In this work, we study the
transferability of UAPs by analyzing equilibrium in the universal adversarial
example game between the classifier and UAP adversary players. We show that
under mild assumptions the universal adversarial example game lacks a pure Nash
equilibrium, indicating UAPs' suboptimal transferability across DNN
classifiers. To address this issue, we propose Universal Adversarial Directions
(UADs) which only fix a universal direction for adversarial perturbations and
allow the perturbations' magnitude to be chosen freely across samples. We prove
that the UAD adversarial example game can possess a Nash equilibrium with a
pure UAD strategy, implying the potential transferability of UADs. We also
connect the UAD optimization problem to the well-known principal component
analysis (PCA) and develop an efficient PCA-based algorithm for optimizing
UADs. We evaluate UADs over multiple benchmark image datasets. Our numerical
results show the superior transferability of UADs over standard gradient-based
UAPs.


http://arxiv.org/abs/2210.16117
Improving Transferability of Adversarial Examples on Face Recognition with Beneficial Perturbation Feature Augmentation. (99%)
Fengfan Zhou; Hefei Ling; Yuxuan Shi; Jiazhong Chen; Zongyi Li; Qian Wang
  Face recognition (FR) models can be easily fooled by adversarial examples,
which are crafted by adding imperceptible perturbations on benign face images.
To improve the transferability of adversarial examples on FR models, we propose
a novel attack method called Beneficial Perturbation Feature Augmentation
Attack (BPFA), which reduces the overfitting of the adversarial examples to
surrogate FR models by the adversarial strategy. Specifically, in the
backpropagation step, BPFA records the gradients on pre-selected features and
uses the gradient on the input image to craft adversarial perturbation to be
added on the input image. In the next forward propagation step, BPFA leverages
the recorded gradients to add perturbations(i.e., beneficial perturbations)
that can be pitted against the adversarial perturbation added on the input
image on their corresponding features. The above two steps are repeated until
the last backpropagation step before the maximum number of iterations is
reached. The optimization process of the adversarial perturbation added on the
input image and the optimization process of the beneficial perturbations added
on the features correspond to a minimax two-player game. Extensive experiments
demonstrate that BPFA outperforms the state-of-the-art gradient-based
adversarial attacks on FR.


http://arxiv.org/abs/2210.16346
Improving Hyperspectral Adversarial Robustness using Ensemble Networks in the Presences of Multiple Attacks. (98%)
Nicholas Soucy; Salimeh Yasaei Sekeh
  Semantic segmentation of hyperspectral images (HSI) has seen great strides in
recent years by incorporating knowledge from deep learning RGB classification
models. Similar to their classification counterparts, semantic segmentation
models are vulnerable to adversarial examples and need adversarial training to
counteract them. Traditional approaches to adversarial robustness focus on
training or retraining a single network on attacked data, however, in the
presence of multiple attacks these approaches decrease the performance compared
to networks trained individually on each attack. To combat this issue we
propose an Adversarial Discriminator Ensemble Network (ADE-Net) which focuses
on attack type detection and adversarial robustness under a unified model to
preserve per data-type weight optimally while robustifiying the overall
network. In the proposed method, a discriminator network is used to separate
data by attack type into their specific attack-expert ensemble network. Our
approach allows for the presence of multiple attacks mixed together while also
labeling attack types during testing. We experimentally show that ADE-Net
outperforms the baseline, which is a single network adversarially trained under
a mix of multiple attacks, for HSI Indian Pines, Kennedy Space, and Houston
datasets.


http://arxiv.org/abs/2210.16371
Distributed Black-box Attack against Image Classification Cloud Services. (89%)
Han Wu; Sareh Rowlands; Johan Wahlstrom
  Black-box adversarial attacks can fool image classifiers into misclassifying
images without requiring access to model structure and weights. Recently
proposed black-box attacks can achieve a success rate of more than 95\% after
less than 1,000 queries. The question then arises of whether black-box attacks
have become a real threat against IoT devices that rely on cloud APIs to
achieve image classification. To shed some light on this, note that prior
research has primarily focused on increasing the success rate and reducing the
number of required queries. However, another crucial factor for black-box
attacks against cloud APIs is the time required to perform the attack. This
paper applies black-box attacks directly to cloud APIs rather than to local
models, thereby avoiding multiple mistakes made in prior research. Further, we
exploit load balancing to enable distributed black-box attacks that can reduce
the attack time by a factor of about five for both local search and gradient
estimation methods.


http://arxiv.org/abs/2210.15944
RoChBert: Towards Robust BERT Fine-tuning for Chinese. (75%)
Zihan Zhang; Jinfeng Li; Ning Shi; Bo Yuan; Xiangyu Liu; Rong Zhang; Hui Xue; Donghong Sun; Chao Zhang
  Despite of the superb performance on a wide range of tasks, pre-trained
language models (e.g., BERT) have been proved vulnerable to adversarial texts.
In this paper, we present RoChBERT, a framework to build more Robust BERT-based
models by utilizing a more comprehensive adversarial graph to fuse Chinese
phonetic and glyph features into pre-trained representations during
fine-tuning. Inspired by curriculum learning, we further propose to augment the
training dataset with adversarial texts in combination with intermediate
samples. Extensive experiments demonstrate that RoChBERT outperforms previous
methods in significant ways: (i) robust -- RoChBERT greatly improves the model
robustness without sacrificing accuracy on benign texts. Specifically, the
defense lowers the success rates of unlimited and limited attacks by 59.43% and
39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible --
RoChBERT can easily extend to various language models to solve different
downstream tasks with excellent performance; and (iii) efficient -- RoChBERT
can be directly applied to the fine-tuning stage without pre-training language
model from scratch, and the proposed data augmentation method is also low-cost.


http://arxiv.org/abs/2210.16451
Robust Boosting Forests with Richer Deep Feature Hierarchy. (56%)
Jianqiao Wangni
  We propose a robust variant of boosting forest to the various adversarial
defense methods, and apply it to enhance the robustness of the deep neural
network. We retain the deep network architecture, weights, and middle layer
features, then install gradient boosting forest to select the features from
each layer of the deep network, and predict the target. For training each
decision tree, we propose a novel conservative and greedy trade-off, with
consideration for less misprediction instead of pure gain functions, therefore
being suboptimal and conservative. We actively increase tree depth to remedy
the accuracy with splits in more features, being more greedy in growing tree
depth. We propose a new task on 3D face model, whose robustness has not been
carefully studied, despite the great security and privacy concerns related to
face analytics. We tried a simple attack method on a pure convolutional neural
network (CNN) face shape estimator, making it degenerate to only output average
face shape with invisible perturbation. Our conservative-greedy boosting forest
(CGBF) on face landmark datasets showed a great improvement over original pure
deep learning methods under the adversarial attacks.


http://arxiv.org/abs/2210.16140
Localized Randomized Smoothing for Collective Robustness Certification. (26%)
Jan Schuchardt; Tom Wollschläger; Aleksandar Bojchevski; Stephan Günnemann
  Models for image segmentation, node classification and many other tasks map a
single input to multiple labels. By perturbing this single shared input (e.g.
the image) an adversary can manipulate several predictions (e.g. misclassify
several pixels). Collective robustness certification is the task of provably
bounding the number of robust predictions under this threat model. The only
dedicated method that goes beyond certifying each output independently is
limited to strictly local models, where each prediction is associated with a
small receptive field. We propose a more general collective robustness
certificate for all types of models and further show that this approach is
beneficial for the larger class of softly local models, where each output is
dependent on the entire input but assigns different levels of importance to
different input regions (e.g. based on their proximity in the image). The
certificate is based on our novel localized randomized smoothing approach,
where the random perturbation strength for different input regions is
proportional to their importance for the outputs. Localized smoothing
Pareto-dominates existing certificates on both image segmentation and node
classification tasks, simultaneously offering higher accuracy and stronger
guarantees.


http://arxiv.org/abs/2210.16258
On the Vulnerability of Data Points under Multiple Membership Inference Attacks and Target Models. (1%)
Mauro Conti; Jiaxin Li; Stjepan Picek
  Membership Inference Attacks (MIAs) infer whether a data point is in the
training data of a machine learning model. It is a threat while being in the
training data is private information of a data point. MIA correctly infers some
data points as members or non-members of the training data. Intuitively, data
points that MIA accurately detects are vulnerable. Considering those data
points may exist in different target models susceptible to multiple MIAs, the
vulnerability of data points under multiple MIAs and target models is worth
exploring.
  This paper defines new metrics that can reflect the actual situation of data
points' vulnerability and capture vulnerable data points under multiple MIAs
and target models. From the analysis, MIA has an inference tendency to some
data points despite a low overall inference performance. Additionally, we
implement 54 MIAs, whose average attack accuracy ranges from 0.5 to 0.9, to
support our analysis with our scalable and flexible platform, Membership
Inference Attacks Platform (VMIAP). Furthermore, previous methods are
unsuitable for finding vulnerable data points under multiple MIAs and different
target models. Finally, we observe that the vulnerability is not characteristic
of the data point but related to the MIA and target model.


http://arxiv.org/abs/2210.16114
Toward Reliable Neural Specifications. (1%)
Chuqin Geng; Nham Le; Xiaojie Xu; Zhaoyue Wang; Arie Gurfinkel; Xujie Si
  Having reliable specifications is an unavoidable challenge in achieving
verifiable correctness, robustness, and interpretability of AI systems.
Existing specifications for neural networks are in the paradigm of data as
specification. That is, the local neighborhood centering around a reference
input is considered to be correct (or robust). However, our empirical study
shows that such a specification is extremely overfitted since usually no data
points from the testing set lie in the certified region of the reference input,
making them impractical for real-world applications. We propose a new family of
specifications called neural representation as specification, which uses the
intrinsic information of neural networks - neural activation patterns (NAP),
rather than input data to specify the correctness and/or robustness of neural
network predictions. We present a simple statistical approach to mining
dominant neural activation patterns. We analyze NAPs from a statistical point
of view and find that a single NAP can cover a large number of training and
testing data points whereas ad hoc data-as-specification only covers the given
reference data point. To show the effectiveness of discovered NAPs, we formally
verify several important properties, such as various types of
misclassifications will never happen for a given NAP, and there is no-ambiguity
between different NAPs. We show that by using NAP, we can verify the prediction
of the entire input space, while still recalling 84% of the data. Thus, we
argue that using NAPs is a more reliable and extensible specification for
neural network verification.


http://arxiv.org/abs/2210.15700
TAD: Transfer Learning-based Multi-Adversarial Detection of Evasion Attacks against Network Intrusion Detection Systems. (99%)
Islam Debicha; Richard Bauwens; Thibault Debatty; Jean-Michel Dricot; Tayeb Kenaza; Wim Mees
  Nowadays, intrusion detection systems based on deep learning deliver
state-of-the-art performance. However, recent research has shown that specially
crafted perturbations, called adversarial examples, are capable of
significantly reducing the performance of these intrusion detection systems.
The objective of this paper is to design an efficient transfer learning-based
adversarial detector and then to assess the effectiveness of using multiple
strategically placed adversarial detectors compared to a single adversarial
detector for intrusion detection systems. In our experiments, we implement
existing state-of-the-art models for intrusion detection. We then attack those
models with a set of chosen evasion attacks. In an attempt to detect those
adversarial attacks, we design and implement multiple transfer learning-based
adversarial detectors, each receiving a subset of the information passed
through the IDS. By combining their respective decisions, we illustrate that
combining multiple detectors can further improve the detectability of
adversarial traffic compared to a single detector in the case of a parallel IDS
design.


http://arxiv.org/abs/2210.15291
Isometric 3D Adversarial Examples in the Physical World. (99%)
Yibo Miao; Yinpeng Dong; Jun Zhu; Xiao-Shan Gao
  3D deep learning models are shown to be as vulnerable to adversarial examples
as 2D models. However, existing attack methods are still far from stealthy and
suffer from severe performance degradation in the physical world. Although 3D
data is highly structured, it is difficult to bound the perturbations with
simple metrics in the Euclidean space. In this paper, we propose a novel
$\epsilon$-isometric ($\epsilon$-ISO) attack to generate natural and robust 3D
adversarial examples in the physical world by considering the geometric
properties of 3D objects and the invariance to physical transformations. For
naturalness, we constrain the adversarial example to be $\epsilon$-isometric to
the original one by adopting the Gaussian curvature as a surrogate metric
guaranteed by a theoretical analysis. For invariance to physical
transformations, we propose a maxima over transformation (MaxOT) method that
actively searches for the most harmful transformations rather than random ones
to make the generated adversarial example more robust in the physical world.
Experiments on typical point cloud recognition models validate that our
approach can significantly improve the attack success rate and naturalness of
the generated 3D adversarial examples than the state-of-the-art attack methods.


http://arxiv.org/abs/2210.15392
LeNo: Adversarial Robust Salient Object Detection Networks with Learnable Noise. (92%)
He Tang; He Wang
  Pixel-wise predction with deep neural network has become an effective
paradigm for salient object detection (SOD) and achieved remakable performance.
However, very few SOD models are robust against adversarial attacks which are
visually imperceptible for human visual attention. The previous work robust
salient object detection against adversarial attacks (ROSA) shuffles the
pre-segmented superpixels and then refines the coarse saliency map by the
densely connected CRF. Different from ROSA that rely on various pre- and
post-processings, this paper proposes a light-weight Learnble Noise (LeNo) to
against adversarial attacks for SOD models. LeNo preserves accuracy of SOD
models on both adversarial and clean images, as well as inference speed. In
general, LeNo consists of a simple shallow noise and noise estimation that
embedded in the encoder and decoder of arbitrary SOD networks respectively.
Inspired by the center prior of human visual attention mechanism, we initialize
the shallow noise with a cross-shaped gaussian distribution for better defense
against adversarial attacks. Instead of adding additional network components
for post-processing, the proposed noise estimation modifies only one channel of
the decoder. With the deeply-supervised noise-decoupled training on
state-of-the-art RGB and RGB-D SOD networks, LeNo outperforms previous works
not only on adversarial images but also clean images, which contributes
stronger robustness for SOD.


http://arxiv.org/abs/2210.15221
TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack. (92%)
Yu Cao; Dianqi Li; Meng Fang; Tianyi Zhou; Jun Gao; Yibing Zhan; Dacheng Tao
  We present Twin Answer Sentences Attack (TASA), an adversarial attack method
for question answering (QA) models that produces fluent and grammatical
adversarial contexts while maintaining gold answers. Despite phenomenal
progress on general adversarial attacks, few works have investigated the
vulnerability and attack specifically for QA models. In this work, we first
explore the biases in the existing models and discover that they mainly rely on
keyword matching between the question and context, and ignore the relevant
contextual relations for answer prediction. Based on two biases above, TASA
attacks the target model in two folds: (1) lowering the model's confidence on
the gold answer with a perturbed answer sentence; (2) misguiding the model
towards a wrong answer with a distracting answer sentence. Equipped with
designed beam search and filtering methods, TASA can generate more effective
attacks than existing textual attack methods while sustaining the quality of
contexts, in extensive experiments on five QA datasets and human evaluations.


http://arxiv.org/abs/2210.15318
Efficient and Effective Augmentation Strategy for Adversarial Training. (56%)
Sravanti Addepalli; Samyak Jain; R. Venkatesh Babu
  Adversarial training of Deep Neural Networks is known to be significantly
more data-hungry when compared to standard training. Furthermore, complex data
augmentations such as AutoAugment, which have led to substantial gains in
standard training of image classifiers, have not been successful with
Adversarial Training. We first explain this contrasting behavior by viewing
augmentation during training as a problem of domain generalization, and further
propose Diverse Augmentation-based Joint Adversarial Training (DAJAT) to use
data augmentations effectively in adversarial training. We aim to handle the
conflicting goals of enhancing the diversity of the training dataset and
training with data that is close to the test distribution by using a
combination of simple and complex augmentations with separate batch
normalization layers during training. We further utilize the popular
Jensen-Shannon divergence loss to encourage the joint learning of the diverse
augmentations, thereby allowing simple augmentations to guide the learning of
complex ones. Lastly, to improve the computational efficiency of the proposed
method, we propose and utilize a two-step defense, Ascending Constraint
Adversarial Training (ACAT), that uses an increasing epsilon schedule and
weight-space smoothing to prevent gradient masking. The proposed method DAJAT
achieves substantially better robustness-accuracy trade-off when compared to
existing methods on the RobustBench Leaderboard on ResNet-18 and
WideResNet-34-10. The code for implementing DAJAT is available here:
https://github.com/val-iisc/DAJAT.


http://arxiv.org/abs/2210.15764
Noise Injection Node Regularization for Robust Learning. (2%)
Noam Levi; Itay M. Bloch; Marat Freytsis; Tomer Volansky
  We introduce Noise Injection Node Regularization (NINR), a method of
injecting structured noise into Deep Neural Networks (DNN) during the training
stage, resulting in an emergent regularizing effect. We present theoretical and
empirical evidence for substantial improvement in robustness against various
test data perturbations for feed-forward DNNs when trained under NINR. The
novelty in our approach comes from the interplay of adaptive noise injection
and initialization conditions such that noise is the dominant driver of
dynamics at the start of training. As it simply requires the addition of
external nodes without altering the existing network structure or optimization
algorithms, this method can be easily incorporated into many standard problem
specifications. We find improved stability against a number of data
perturbations, including domain shifts, with the most dramatic improvement
obtained for unstructured noise, where our technique outperforms other existing
methods such as Dropout or $L_2$ regularization, in some cases. We further show
that desirable generalization properties on clean data are generally
maintained.


http://arxiv.org/abs/2210.15176
Domain Adaptive Object Detection for Autonomous Driving under Foggy Weather. (1%)
Jinlong Li; Runsheng Xu; Jin Ma; Qin Zou; Jiaqi Ma; Hongkai Yu
  Most object detection methods for autonomous driving usually assume a
consistent feature distribution between training and testing data, which is not
always the case when weathers differ significantly. The object detection model
trained under clear weather might not be effective enough in foggy weather
because of the domain gap. This paper proposes a novel domain adaptive object
detection framework for autonomous driving under foggy weather. Our method
leverages both image-level and object-level adaptation to diminish the domain
discrepancy in image style and object appearance. To further enhance the
model's capabilities under challenging samples, we also come up with a new
adversarial gradient reversal layer to perform adversarial mining for the hard
examples together with domain adaptation. Moreover, we propose to generate an
auxiliary domain by data augmentation to enforce a new domain-level metric
regularization. Experimental results on public benchmarks show the
effectiveness and accuracy of the proposed method. The code is available at
https://github.com/jinlong17/DA-Detect.


http://arxiv.org/abs/2210.15068
Improving Adversarial Robustness with Self-Paced Hard-Class Pair Reweighting. (99%)
Pengyue Hou; Jie Han; Xingyu Li
  Deep Neural Networks are vulnerable to adversarial attacks. Among many
defense strategies, adversarial training with untargeted attacks is one of the
most effective methods. Theoretically, adversarial perturbation in untargeted
attacks can be added along arbitrary directions and the predicted labels of
untargeted attacks should be unpredictable. However, we find that the naturally
imbalanced inter-class semantic similarity makes those hard-class pairs become
virtual targets of each other. This study investigates the impact of such
closely-coupled classes on adversarial attacks and develops a self-paced
reweighting strategy in adversarial training accordingly. Specifically, we
propose to upweight hard-class pair losses in model optimization, which prompts
learning discriminative features from hard classes. We further incorporate a
term to quantify hard-class pair consistency in adversarial training, which
greatly boosts model robustness. Extensive experiments show that the proposed
adversarial training method achieves superior robustness performance over
state-of-the-art defenses against a wide range of adversarial attacks.


http://arxiv.org/abs/2210.17316
There is more than one kind of robustness: Fooling Whisper with adversarial examples. (98%)
Raphael Olivier; Bhiksha Raj
  Whisper is a recent Automatic Speech Recognition (ASR) model displaying
impressive robustness to both out-of-distribution inputs and random noise. In
this work, we show that this robustness does not carry over to adversarial
noise. We generate very small input perturbations with Signal Noise Ratio of up
to 45dB, with which we can degrade Whisper performance dramatically, or even
transcribe a target sentence of our choice. We also show that by fooling the
Whisper language detector we can very easily degrade the performance of
multilingual models. These vulnerabilities of a widely popular open-source
model have practical security implications, and emphasize the need for
adversarially robust ASR.


http://arxiv.org/abs/2210.14957
Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustness. (86%)
Jiahao Zhao; Wenji Mao
  Adversarial vulnerability remains a major obstacle to constructing reliable
NLP systems. When imperceptible perturbations are added to raw input text, the
performance of a deep learning model may drop dramatically under attacks.
Recent work argues the adversarial vulnerability of the model is caused by the
non-robust features in supervised training. Thus in this paper, we tackle the
adversarial robustness challenge from the view of disentangled representation
learning, which is able to explicitly disentangle robust and non-robust
features in text. Specifically, inspired by the variation of information (VI)
in information theory, we derive a disentangled learning objective composed of
mutual information to represent both the semantic representativeness of latent
embeddings and differentiation of robust and non-robust features. On the basis
of this, we design a disentangled learning network to estimate these mutual
information. Experiments on text classification and entailment tasks show that
our method significantly outperforms the representative methods under
adversarial attacks, indicating that discarding non-robust features is critical
for improving adversarial robustness.


http://arxiv.org/abs/2210.14814
BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examples. (75%)
Mohaddeseh Bastan; Mihai Surdeanu; Niranjan Balasubramanian
  Natural language inference (NLI) is critical for complex decision-making in
biomedical domain. One key question, for example, is whether a given biomedical
mechanism is supported by experimental evidence. This can be seen as an NLI
problem but there are no directly usable datasets to address this. The main
challenge is that manually creating informative negative examples for this task
is difficult and expensive. We introduce a novel semi-supervised procedure that
bootstraps an NLI dataset from existing biomedical dataset that pairs
mechanisms with experimental evidence in abstracts. We generate a range of
negative examples using nine strategies that manipulate the structure of the
underlying mechanisms both with rules, e.g., flip the roles of the entities in
the interaction, and, more importantly, as perturbations via logical
constraints in a neuro-logical decoding system. We use this procedure to create
a novel dataset for NLI in the biomedical domain, called BioNLI and benchmark
two state-of-the-art biomedical classifiers. The best result we obtain is
around mid 70s in F1, suggesting the difficulty of the task. Critically, the
performance on the different classes of negative examples varies widely, from
97% F1 on the simple role change negative examples, to barely better than
chance on the negative examples generated using neuro-logic decoding.


http://arxiv.org/abs/2210.14999
EIPSIM: Modeling Secure IP Address Allocation at Cloud Scale. (11%)
Eric University of Wisconsin-Madison Pauley; Kyle Pennsylvania State University Domico; Blaine University of Wisconsin-Madison Hoak; Ryan University of Wisconsin-Madison Sheatsley; Quinn University of Wisconsin-Madison Burke; Yohan University of Wisconsin-Madison Beugin; Patrick University of Wisconsin-Madison McDaniel
  Public clouds provide impressive capability through resource sharing.
However, recent works have shown that the reuse of IP addresses can allow
adversaries to exploit the latent configurations left by previous tenants. In
this work, we perform a comprehensive analysis of the effect of cloud IP
address allocation on exploitation of latent configuration. We first develop a
statistical model of cloud tenant behavior and latent configuration based on
literature and deployed systems. Through these, we analyze IP allocation
policies under existing and novel threat models. Our resulting framework,
EIPSim, simulates our models in representative public cloud scenarios,
evaluating adversarial objectives against pool policies. In response to our
stronger proposed threat model, we also propose IP scan segmentation, an IP
allocation policy that protects the IP pool against adversarial scanning even
when an adversary is not limited by number of cloud tenants. Our evaluation
shows that IP scan segmentation reduces latent configuration exploitability by
97.1% compared to policies proposed in literature and 99.8% compared to those
currently deployed by cloud providers. Finally, we evaluate our statistical
assumptions by analyzing real allocation and configuration data, showing that
results generalize to deployed cloud workloads. In this way, we show that
principled analysis of cloud IP address allocation can lead to substantial
security gains for tenants and their users.


http://arxiv.org/abs/2210.15140
V-Cloak: Intelligibility-, Naturalness- & Timbre-Preserving Real-Time Voice Anonymization. (10%)
Jiangyi Zhejiang University Deng; Fei Zhejiang University Teng; Yanjiao Zhejiang University Chen; Xiaofu Wuhan University Chen; Zhaohui Wuhan University Wang; Wenyuan Zhejiang University Xu
  Voice data generated on instant messaging or social media applications
contains unique user voiceprints that may be abused by malicious adversaries
for identity inference or identity theft. Existing voice anonymization
techniques, e.g., signal processing and voice conversion/synthesis, suffer from
degradation of perceptual quality. In this paper, we develop a voice
anonymization system, named V-Cloak, which attains real-time voice
anonymization while preserving the intelligibility, naturalness and timbre of
the audio. Our designed anonymizer features a one-shot generative model that
modulates the features of the original audio at different frequency levels. We
train the anonymizer with a carefully-designed loss function. Apart from the
anonymity loss, we further incorporate the intelligibility loss and the
psychoacoustics-based naturalness loss. The anonymizer can realize untargeted
and targeted anonymization to achieve the anonymity goals of unidentifiability
and unlinkability.
  We have conducted extensive experiments on four datasets, i.e., LibriSpeech
(English), AISHELL (Chinese), CommonVoice (French) and CommonVoice (Italian),
five Automatic Speaker Verification (ASV) systems (including two DNN-based, two
statistical and one commercial ASV), and eleven Automatic Speech Recognition
(ASR) systems (for different languages). Experiment results confirm that
V-Cloak outperforms five baselines in terms of anonymity performance. We also
demonstrate that V-Cloak trained only on the VoxCeleb1 dataset against
ECAPA-TDNN ASV and DeepSpeech2 ASR has transferable anonymity against other
ASVs and cross-language intelligibility for other ASRs. Furthermore, we verify
the robustness of V-Cloak against various de-noising techniques and adaptive
attacks. Hopefully, V-Cloak may provide a cloak for us in a prism world.


http://arxiv.org/abs/2210.15127
Rethinking the Reverse-engineering of Trojan Triggers. (5%)
Zhenting Wang; Kai Mei; Hailun Ding; Juan Zhai; Shiqing Ma
  Deep Neural Networks are vulnerable to Trojan (or backdoor) attacks.
Reverse-engineering methods can reconstruct the trigger and thus identify
affected models. Existing reverse-engineering methods only consider input space
constraints, e.g., trigger size in the input space. Expressly, they assume the
triggers are static patterns in the input space and fail to detect models with
feature space triggers such as image style transformations. We observe that
both input-space and feature-space Trojans are associated with feature space
hyperplanes. Based on this observation, we design a novel reverse-engineering
method that exploits the feature space constraint to reverse-engineer Trojan
triggers. Results on four datasets and seven different attacks demonstrate that
our solution effectively defends both input-space and feature-space Trojans. It
outperforms state-of-the-art reverse-engineering methods and other types of
defenses in both Trojaned model detection and mitigation tasks. On average, the
detection accuracy of our method is 93\%. For Trojan mitigation, our method can
reduce the ASR (attack success rate) to only 0.26\% with the BA (benign
accuracy) remaining nearly unchanged. Our code can be found at
https://github.com/RU-System-Software-and-Security/FeatureRE.


http://arxiv.org/abs/2210.14632
Cover Reproducible Steganography via Deep Generative Models. (1%)
Kejiang Chen; Hang Zhou; Yaofei Wang; Menghan Li; Weiming Zhang; Nenghai Yu
  Whereas cryptography easily arouses attacks by means of encrypting a secret
message into a suspicious form, steganography is advantageous for its
resilience to attacks by concealing the message in an innocent-looking cover
signal. Minimal distortion steganography, one of the mainstream steganography
frameworks, embeds messages while minimizing the distortion caused by the
modification on the cover elements. Due to the unavailability of the original
cover signal for the receiver, message embedding is realized by finding the
coset leader of the syndrome function of steganographic codes migrated from
channel coding, which is complex and has limited performance. Fortunately, deep
generative models and the robust semantic of generated data make it possible
for the receiver to perfectly reproduce the cover signal from the stego signal.
With this advantage, we propose cover-reproducible steganography where the
source coding, e.g., arithmetic coding, serves as the steganographic code.
Specifically, the decoding process of arithmetic coding is used for message
embedding and its encoding process is regarded as message extraction. Taking
text-to-speech and text-to-image synthesis tasks as two examples, we illustrate
the feasibility of cover-reproducible steganography. Steganalysis experiments
and theoretical analysis are conducted to demonstrate that the proposed methods
outperform the existing methods in most cases.


http://arxiv.org/abs/2210.15042
EW-Tune: A Framework for Privately Fine-Tuning Large Language Models with Differential Privacy. (1%)
Rouzbeh Behnia; Mohamamdreza Ebrahimi; Jason Pacheco; balaji Padmanabhan
  Pre-trained Large Language Models (LLMs) are an integral part of modern AI
that have led to breakthrough performances in complex AI tasks. Major AI
companies with expensive infrastructures are able to develop and train these
large models with billions and millions of parameters from scratch. Third
parties, researchers, and practitioners are increasingly adopting these
pre-trained models and fine-tuning them on their private data to accomplish
their downstream AI tasks. However, it has been shown that an adversary can
extract/reconstruct the exact training samples from these LLMs, which can lead
to revealing personally identifiable information. The issue has raised deep
concerns about the privacy of LLMs. Differential privacy (DP) provides a
rigorous framework that allows adding noise in the process of training or
fine-tuning LLMs such that extracting the training data becomes infeasible
(i.e., with a cryptographically small success probability). While the
theoretical privacy guarantees offered in most extant studies assume learning
models from scratch through many training iterations in an asymptotic setting,
this assumption does not hold in fine-tuning scenarios in which the number of
training iterations is significantly smaller. To address the gap, we present
\ewtune, a DP framework for fine-tuning LLMs based on Edgeworth accountant with
finite-sample privacy guarantees. Our results across four well-established
natural language understanding (NLU) tasks show that while \ewtune~adds privacy
guarantees to LLM fine-tuning process, it directly contributes to decreasing
the induced noise to up to 5.6\% and improves the state-of-the-art LLMs
performance by up to 1.1\% across all NLU tasks. We have open-sourced our
implementations for wide adoption and public testing purposes.


http://arxiv.org/abs/2210.14622
DEMIS: A Threat Model for Selectively Encrypted Visual Surveillance Data. (1%)
Ifeoluwapo Aribilola; Mamoona Naveed Asghar; Brian Lee
  The monitoring of individuals/objects has become increasingly possible in
recent years due to the convenience of integrated cameras in many devices. Due
to the important moments or activities of people captured by these devices, it
has made it a great asset for attackers to launch attacks against by exploiting
the weaknesses in these devices. Different studies proposed na\"ive/selective
encryption of the captured visual data for safety but despite the encryption,
an attacker can still access or manipulate such data. This paper proposed a
novel threat model, DEMIS which helps analyse the threats against such
encrypted videos. The paper also examines the attack vectors that can be used
for threats and the mitigation that will reduce or prevent the attack. For
experiments, firstly the data set is generated by applying selective encryption
on the Regions-of-interests (ROI) of the tested videos using the image
segmentation technique and Chacha20 cipher. Secondly, different types of
attacks, such as inverse, lowercase, uppercase, random insertion, and
malleability attacks were simulated in experiments to show the effects of the
attacks, the risk matrix, and the severity of these attacks. Our developed data
set with the original, selective encrypted, and attacked videos are available
on git-repository(https://github.com/Ifeoluwapoo/video-datasets) for future
researchers.


http://arxiv.org/abs/2210.14405
Adversarially Robust Medical Classification via Attentive Convolutional Neural Networks. (99%)
Isaac Wasserman
  Convolutional neural network-based medical image classifiers have been shown
to be especially susceptible to adversarial examples. Such instabilities are
likely to be unacceptable in the future of automated diagnoses. Though
statistical adversarial example detection methods have proven to be effective
defense mechanisms, additional research is necessary that investigates the
fundamental vulnerabilities of deep-learning-based systems and how best to
build models that jointly maximize traditional and robust accuracy. This paper
presents the inclusion of attention mechanisms in CNN-based medical image
classifiers as a reliable and effective strategy for increasing robust accuracy
without sacrifice. This method is able to increase robust accuracy by up to 16%
in typical adversarial scenarios and up to 2700% in extreme cases.


http://arxiv.org/abs/2210.14018
A White-Box Adversarial Attack Against a Digital Twin. (99%)
Wilson Patterson; Ivan Fernandez; Subash Neupane; Milan Parmar; Sudip Mittal; Shahram Rahimi
  Recent research has shown that Machine Learning/Deep Learning (ML/DL) models
are particularly vulnerable to adversarial perturbations, which are small
changes made to the input data in order to fool a machine learning classifier.
The Digital Twin, which is typically described as consisting of a physical
entity, a virtual counterpart, and the data connections in between, is
increasingly being investigated as a means of improving the performance of
physical entities by leveraging computational techniques, which are enabled by
the virtual counterpart. This paper explores the susceptibility of Digital Twin
(DT), a virtual model designed to accurately reflect a physical object using
ML/DL classifiers that operate as Cyber Physical Systems (CPS), to adversarial
attacks. As a proof of concept, we first formulate a DT of a vehicular system
using a deep neural network architecture and then utilize it to launch an
adversarial attack. We attack the DT model by perturbing the input to the
trained model and show how easily the model can be broken with white-box
attacks.


http://arxiv.org/abs/2210.14404
Adaptive Test-Time Defense with the Manifold Hypothesis. (98%)
Zhaoyuan Yang; Zhiwei Xu; Jing Zhang; Richard Hartley; Peter Tu
  In this work, we formulate a novel framework of adversarial robustness using
the manifold hypothesis. Our framework provides sufficient conditions for
defending against adversarial examples. We develop a test-time defense method
with our formulation and variational inference. The developed approach combines
manifold learning with the Bayesian framework to provide adversarial robustness
without the need for adversarial training. We show that our proposed approach
can provide adversarial robustness even if attackers are aware of existence of
test-time defense. In additions, our approach can also serve as a test-time
defense mechanism for variational autoencoders.


http://arxiv.org/abs/2210.15446
LP-BFGS attack: An adversarial attack based on the Hessian with limited pixels. (98%)
Jiebao Zhang; Wenhua Qian; Rencan Nie; Jinde Cao; Dan Xu
  Deep neural networks are vulnerable to adversarial attacks. Most white-box
attacks are based on the gradient of models to the input. Since the computation
and memory budget, adversarial attacks based on the Hessian information are not
paid enough attention. In this work, we study the attack performance and
computation cost of the attack method based on the Hessian with a limited
perturbation pixel number. Specifically, we propose the Limited Pixel BFGS
(LP-BFGS) attack method by incorporating the BFGS algorithm. Some pixels are
selected as perturbation pixels by the Integrated Gradient algorithm, which are
regarded as optimization variables of the LP-BFGS attack. Experimental results
across different networks and datasets with various perturbation pixel numbers
demonstrate our approach has a comparable attack with an acceptable computation
compared with existing solutions.


http://arxiv.org/abs/2210.14410
Improving Adversarial Robustness via Joint Classification and Multiple Explicit Detection Classes. (98%)
Sina Baharlouei; Fatemeh Sheikholeslami; Meisam Razaviyayn; Zico Kolter
  This work concerns the development of deep networks that are certifiably
robust to adversarial attacks. Joint robust classification-detection was
recently introduced as a certified defense mechanism, where adversarial
examples are either correctly classified or assigned to the "abstain" class. In
this work, we show that such a provable framework can benefit by extension to
networks with multiple explicit abstain classes, where the adversarial examples
are adaptively assigned to those. We show that naively adding multiple abstain
classes can lead to "model degeneracy", then we propose a regularization
approach and a training method to counter this degeneracy by promoting full use
of the multiple abstain classes. Our experiments demonstrate that the proposed
approach consistently achieves favorable standard vs. robust verified accuracy
tradeoffs, outperforming state-of-the-art algorithms for various choices of
number of abstain classes.


http://arxiv.org/abs/2210.15429
Multi-view Representation Learning from Malware to Defend Against Adversarial Variants. (98%)
James Lee Hu; Mohammadreza Ebrahimi; Weifeng Li; Xin Li; Hsinchun Chen
  Deep learning-based adversarial malware detectors have yielded promising
results in detecting never-before-seen malware executables without relying on
expensive dynamic behavior analysis and sandbox. Despite their abilities, these
detectors have been shown to be vulnerable to adversarial malware variants -
meticulously modified, functionality-preserving versions of original malware
executables generated by machine learning. Due to the nature of these
adversarial modifications, these adversarial methods often use a \textit{single
view} of malware executables (i.e., the binary/hexadecimal view) to generate
adversarial malware variants. This provides an opportunity for the defenders
(i.e., malware detectors) to detect the adversarial variants by utilizing more
than one view of a malware file (e.g., source code view in addition to the
binary view). The rationale behind this idea is that while the adversary
focuses on the binary view, certain characteristics of the malware file in the
source code view remain untouched which leads to the detection of the
adversarial malware variants. To capitalize on this opportunity, we propose
Adversarially Robust Multiview Malware Defense (ARMD), a novel multi-view
learning framework to improve the robustness of DL-based malware detectors
against adversarial variants. Our experiments on three renowned open-source
deep learning-based malware detectors across six common malware categories show
that ARMD is able to improve the adversarial robustness by up to seven times on
these malware detectors.


http://arxiv.org/abs/2210.14283
Accelerating Certified Robustness Training via Knowledge Transfer. (73%)
Pratik Vaishnavi; Kevin Eykholt; Amir Rahmati
  Training deep neural network classifiers that are certifiably robust against
adversarial attacks is critical to ensuring the security and reliability of
AI-controlled systems. Although numerous state-of-the-art certified training
methods have been developed, they are computationally expensive and scale
poorly with respect to both dataset and network complexity. Widespread usage of
certified training is further hindered by the fact that periodic retraining is
necessary to incorporate new data and network improvements. In this paper, we
propose Certified Robustness Transfer (CRT), a general-purpose framework for
reducing the computational overhead of any certifiably robust training method
through knowledge transfer. Given a robust teacher, our framework uses a novel
training loss to transfer the teacher's robustness to the student. We provide
theoretical and empirical validation of CRT. Our experiments on CIFAR-10 show
that CRT speeds up certified robustness training by $8 \times$ on average
across three different architecture generations while achieving comparable
robustness to state-of-the-art methods. We also show that CRT can scale to
large-scale datasets like ImageNet.


http://arxiv.org/abs/2210.14229
Causal Information Bottleneck Boosts Adversarial Robustness of Deep Neural Network. (64%)
Huan Hua; Jun Yan; Xi Fang; Weiquan Huang; Huilin Yin; Wancheng Ge
  The information bottleneck (IB) method is a feasible defense solution against
adversarial attacks in deep learning. However, this method suffers from the
spurious correlation, which leads to the limitation of its further improvement
of adversarial robustness. In this paper, we incorporate the causal inference
into the IB framework to alleviate such a problem. Specifically, we divide the
features obtained by the IB method into robust features (content information)
and non-robust features (style information) via the instrumental variables to
estimate the causal effects. With the utilization of such a framework, the
influence of non-robust features could be mitigated to strengthen the
adversarial robustness. We make an analysis of the effectiveness of our
proposed method. The extensive experiments in MNIST, FashionMNIST, and CIFAR-10
show that our method exhibits the considerable robustness against multiple
adversarial attacks. Our code would be released.


http://arxiv.org/abs/2210.13762
Towards Robust Recommender Systems via Triple Cooperative Defense. (61%)
Qingyang Wang; Defu Lian; Chenwang Wu; Enhong Chen
  Recommender systems are often susceptible to well-crafted fake profiles,
leading to biased recommendations. The wide application of recommender systems
makes studying the defense against attack necessary. Among existing defense
methods, data-processing-based methods inevitably exclude normal samples, while
model-based methods struggle to enjoy both generalization and robustness.
Considering the above limitations, we suggest integrating data processing and
robust model and propose a general framework, Triple Cooperative Defense (TCD),
which cooperates to improve model robustness through the co-training of three
models. Specifically, in each round of training, we sequentially use the
high-confidence prediction ratings (consistent ratings) of any two models as
auxiliary training data for the remaining model, and the three models
cooperatively improve recommendation robustness. Notably, TCD adds pseudo label
data instead of deleting abnormal data, which avoids the cleaning of normal
data, and the cooperative training of the three models is also beneficial to
model generalization. Through extensive experiments with five poisoning attacks
on three real-world datasets, the results show that the robustness improvement
of TCD significantly outperforms baselines. It is worth mentioning that TCD is
also beneficial for model generalizations.


http://arxiv.org/abs/2210.13815
FocusedCleaner: Sanitizing Poisoned Graphs for Robust GNN-based Node Classification. (13%)
Yulin Zhu; Liang Tong; Kai Zhou
  Recently, a lot of research attention has been devoted to exploring Web
security, a most representative topic is the adversarial robustness of graph
mining algorithms. Especially, a widely deployed adversarial attacks
formulation is the graph manipulation attacks by modifying the relational data
to mislead the Graph Neural Networks' (GNNs) predictions. Naturally, an
intrinsic question one would ask is whether we can accurately identify the
manipulations over graphs - we term this problem as poisoned graph sanitation.
In this paper, we present FocusedCleaner, a poisoned graph sanitation framework
consisting of two modules: bi-level structural learning and victim node
detection. In particular, the structural learning module will reserve the
attack process to steadily sanitize the graph while the detection module
provides the "focus" - a narrowed and more accurate search region - to
structural learning. These two modules will operate in iterations and reinforce
each other to sanitize a poisoned graph step by step. Extensive experiments
demonstrate that FocusedCleaner outperforms the state-of-the-art baselines both
on poisoned graph sanitation and improving robustness.


http://arxiv.org/abs/2210.13915
Towards Formal Approximated Minimal Explanations of Neural Networks. (12%)
Shahaf Bassan; Guy Katz
  With the rapid growth of machine learning, deep neural networks (DNNs) are
now being used in numerous domains. Unfortunately, DNNs are "black-boxes", and
cannot be interpreted by humans, which is a substantial concern in
safety-critical systems. To mitigate this issue, researchers have begun working
on explainable AI (XAI) methods, which can identify a subset of input features
that are the cause of a DNN's decision for a given input. Most existing
techniques are heuristic, and cannot guarantee the correctness of the
explanation provided. In contrast, recent and exciting attempts have shown that
formal methods can be used to generate provably correct explanations. Although
these methods are sound, the computational complexity of the underlying
verification problem limits their scalability; and the explanations they
produce might sometimes be overly complex. Here, we propose a novel approach to
tackle these limitations. We (1) suggest an efficient, verification-based
method for finding minimal explanations, which constitute a provable
approximation of the global, minimum explanation; (2) show how DNN verification
can assist in calculating lower and upper bounds on the optimal explanation;
(3) propose heuristics that significantly improve the scalability of the
verification process; and (4) suggest the use of bundles, which allows us to
arrive at more succinct and interpretable explanations. Our evaluation shows
that our approach significantly outperforms state-of-the-art techniques, and
produces explanations that are more useful to humans. We thus regard this work
as a step toward leveraging verification technology in producing DNNs that are
more reliable and comprehensible.


http://arxiv.org/abs/2211.12851
A Streamlit-based Artificial Intelligence Trust Platform for Next-Generation Wireless Networks. (3%)
M. Kuzlu; F. O. Catak; S. Sarp; U. Cali; O Gueler
  With the rapid development and integration of artificial intelligence (AI)
methods in next-generation networks (NextG), AI algorithms have provided
significant advantages for NextG in terms of frequency spectrum usage,
bandwidth, latency, and security. A key feature of NextG is the integration of
AI, i.e., self-learning architecture based on self-supervised algorithms, to
improve the performance of the network. A secure AI-powered structure is also
expected to protect NextG networks against cyber-attacks. However, AI itself
may be attacked, i.e., model poisoning targeted by attackers, and it results in
cybersecurity violations. This paper proposes an AI trust platform using
Streamlit for NextG networks that allows researchers to evaluate, defend,
certify, and verify their AI models and applications against adversarial
threats of evasion, poisoning, extraction, and interference.


http://arxiv.org/abs/2210.14376
Robustness of Locally Differentially Private Graph Analysis Against Poisoning. (1%)
Jacob Imola; Amrita Roy Chowdhury; Kamalika Chaudhuri
  Locally differentially private (LDP) graph analysis allows private analysis
on a graph that is distributed across multiple users. However, such
computations are vulnerable to data poisoning attacks where an adversary can
skew the results by submitting malformed data. In this paper, we formally study
the impact of poisoning attacks for graph degree estimation protocols under
LDP. We make two key technical contributions. First, we observe LDP makes a
protocol more vulnerable to poisoning -- the impact of poisoning is worse when
the adversary can directly poison their (noisy) responses, rather than their
input data. Second, we observe that graph data is naturally redundant -- every
edge is shared between two users. Leveraging this data redundancy, we design
robust degree estimation protocols under LDP that can significantly reduce the
impact of data poisoning and compute degree estimates with high accuracy. We
evaluate our proposed robust degree estimation protocols under poisoning
attacks on real-world datasets to demonstrate their efficacy in practice.


http://arxiv.org/abs/2210.12952
Ares: A System-Oriented Wargame Framework for Adversarial ML. (99%)
Farhan Ahmed; Pratik Vaishnavi; Kevin Eykholt; Amir Rahmati
  Since the discovery of adversarial attacks against machine learning models
nearly a decade ago, research on adversarial machine learning has rapidly
evolved into an eternal war between defenders, who seek to increase the
robustness of ML models against adversarial attacks, and adversaries, who seek
to develop better attacks capable of weakening or defeating these defenses.
This domain, however, has found little buy-in from ML practitioners, who are
neither overtly concerned about these attacks affecting their systems in the
real world nor are willing to trade off the accuracy of their models in pursuit
of robustness against these attacks.
  In this paper, we motivate the design and implementation of Ares, an
evaluation framework for adversarial ML that allows researchers to explore
attacks and defenses in a realistic wargame-like environment. Ares frames the
conflict between the attacker and defender as two agents in a reinforcement
learning environment with opposing objectives. This allows the introduction of
system-level evaluation metrics such as time to failure and evaluation of
complex strategies such as moving target defenses. We provide the results of
our initial exploration involving a white-box attacker against an adversarially
trained defender.


http://arxiv.org/abs/2210.13660
SpacePhish: The Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learning. (99%)
Giovanni Apruzzese; Mauro Conti; Ying Yuan
  Existing literature on adversarial Machine Learning (ML) focuses either on
showing attacks that break every ML model, or defenses that withstand most
attacks. Unfortunately, little consideration is given to the actual
\textit{cost} of the attack or the defense. Moreover, adversarial samples are
often crafted in the "feature-space", making the corresponding evaluations of
questionable value. Simply put, the current situation does not allow to
estimate the actual threat posed by adversarial attacks, leading to a lack of
secure ML systems.
  We aim to clarify such confusion in this paper. By considering the
application of ML for Phishing Website Detection (PWD), we formalize the
"evasion-space" in which an adversarial perturbation can be introduced to fool
a ML-PWD -- demonstrating that even perturbations in the "feature-space" are
useful. Then, we propose a realistic threat model describing evasion attacks
against ML-PWD that are cheap to stage, and hence intrinsically more attractive
for real phishers. Finally, we perform the first statistically validated
assessment of state-of-the-art ML-PWD against 12 evasion attacks. Our
evaluation shows (i) the true efficacy of evasion attempts that are more likely
to occur; and (ii) the impact of perturbations crafted in different
evasion-spaces. Our realistic evasion attempts induce a statistically
significant degradation (3-10% at $p\!<$0.05), and their cheap cost makes them
a subtle threat. Notably, however, some ML-PWD are immune to our most realistic
attacks ($p$=0.22). Our contribution paves the way for a much needed
re-assessment of adversarial attacks against ML systems for cybersecurity.


http://arxiv.org/abs/2210.13710
Motif-Backdoor: Rethinking the Backdoor Attack on Graph Neural Networks via Motifs. (96%)
Haibin Zheng; Haiyang Xiong; Jinyin Chen; Haonan Ma; Guohan Huang
  Graph neural network (GNN) with a powerful representation capability has been
widely applied to various areas, such as biological gene prediction, social
recommendation, etc. Recent works have exposed that GNN is vulnerable to the
backdoor attack, i.e., models trained with maliciously crafted training samples
are easily fooled by patched samples. Most of the proposed studies launch the
backdoor attack using a trigger that either is the randomly generated subgraph
(e.g., erd\H{o}s-r\'enyi backdoor) for less computational burden, or the
gradient-based generative subgraph (e.g., graph trojaning attack) to enable a
more effective attack. However, the interpretation of how is the trigger
structure and the effect of the backdoor attack related has been overlooked in
the current literature. Motifs, recurrent and statistically significant
sub-graphs in graphs, contain rich structure information. In this paper, we are
rethinking the trigger from the perspective of motifs, and propose a
motif-based backdoor attack, denoted as Motif-Backdoor. It contributes from
three aspects. (i) Interpretation: it provides an in-depth explanation for
backdoor effectiveness by the validity of the trigger structure from motifs,
leading to some novel insights, e.g., using subgraphs that appear less
frequently in the graph as the trigger can achieve better attack performance.
(ii) Effectiveness: Motif-Backdoor reaches the state-of-the-art (SOTA) attack
performance in both black-box and defensive scenarios. (iii) Efficiency: based
on the graph motif distribution, Motif-Backdoor can quickly obtain an effective
trigger structure without target model feedback or subgraph model generation.
Extensive experimental results show that Motif-Backdoor realizes the SOTA
performance on three popular models and four public datasets compared with five
baselines.


http://arxiv.org/abs/2210.13631
On the Robustness of Dataset Inference. (81%)
Sebastian Szyller; Rui Zhang; Jian Liu; N. Asokan
  Machine learning (ML) models are costly to train as they can require a
significant amount of data, computational resources and technical expertise.
Thus, they constitute valuable intellectual property that needs protection from
adversaries wanting to steal them. Ownership verification techniques allow the
victims of model stealing attacks to demonstrate that a suspect model was in
fact stolen from theirs. Although a number of ownership verification techniques
based on watermarking or fingerprinting have been proposed, most of them fall
short either in terms of security guarantees (well-equipped adversaries can
evade verification) or computational cost. A fingerprinting technique
introduced at ICLR '21, Dataset Inference (DI), has been shown to offer better
robustness and efficiency than prior methods. The authors of DI provided a
correctness proof for linear (suspect) models. However, in the same setting, we
prove that DI suffers from high false positives (FPs) -- it can incorrectly
identify an independent model trained with non-overlapping data from the same
distribution as stolen. We further prove that DI also triggers FPs in
realistic, non-linear suspect models. We then confirm empirically that DI leads
to FPs, with high confidence. Second, we show that DI also suffers from false
negatives (FNs) -- an adversary can fool DI by regularising a stolen model's
decision boundaries using adversarial training, thereby leading to an FN. To
this end, we demonstrate that DI fails to identify a model adversarially
trained from a stolen dataset -- the setting where DI is the hardest to evade.
Finally, we discuss the implications of our findings, the viability of
fingerprinting-based ownership verification in general, and suggest directions
for future work.


http://arxiv.org/abs/2210.14225
Flexible Android Malware Detection Model based on Generative Adversarial Networks with Code Tensor. (16%)
Zhao Yang; Fengyang Deng; Linxi Han
  The behavior of malware threats is gradually increasing, heightened the need
for malware detection. However, existing malware detection methods only target
at the existing malicious samples, the detection of fresh malicious code and
variants of malicious code is limited. In this paper, we propose a novel scheme
that detects malware and its variants efficiently. Based on the idea of the
generative adversarial networks (GANs), we obtain the `true' sample
distribution that satisfies the characteristics of the real malware, use them
to deceive the discriminator, thus achieve the defense against malicious code
attacks and improve malware detection. Firstly, a new Android malware APK to
image texture feature extraction segmentation method is proposed, which is
called segment self-growing texture segmentation algorithm. Secondly, tensor
singular value decomposition (tSVD) based on the low-tubal rank transforms
malicious features with different sizes into a fixed third-order tensor
uniformly, which is entered into the neural network for training and learning.
Finally, a flexible Android malware detection model based on GANs with code
tensor (MTFD-GANs) is proposed. Experiments show that the proposed model can
generally surpass the traditional malware detection model, with a maximum
improvement efficiency of 41.6\%. At the same time, the newly generated samples
of the GANs generator greatly enrich the sample diversity. And retraining
malware detector can effectively improve the detection efficiency and
robustness of traditional models.


http://arxiv.org/abs/2210.12945
Revisiting Sparse Convolutional Model for Visual Recognition. (11%)
Xili Dai; Mingyang Li; Pengyuan Zhai; Shengbang Tong; Xingjian Gao; Shao-Lun Huang; Zhihui Zhu; Chong You; Yi Ma
  Despite strong empirical performance for image classification, deep neural
networks are often regarded as ``black boxes'' and they are difficult to
interpret. On the other hand, sparse convolutional models, which assume that a
signal can be expressed by a linear combination of a few elements from a
convolutional dictionary, are powerful tools for analyzing natural images with
good theoretical interpretability and biological plausibility. However, such
principled models have not demonstrated competitive performance when compared
with empirically designed deep networks. This paper revisits the sparse
convolutional modeling for image classification and bridges the gap between
good empirical performance (of deep learning) and good interpretability (of
sparse convolutional models). Our method uses differentiable optimization
layers that are defined from convolutional sparse coding as drop-in
replacements of standard convolutional layers in conventional deep neural
networks. We show that such models have equally strong empirical performance on
CIFAR-10, CIFAR-100, and ImageNet datasets when compared to conventional neural
networks. By leveraging stable recovery property of sparse modeling, we further
show that such models can be much more robust to input corruptions as well as
adversarial perturbations in testing through a simple proper trade-off between
sparse regularization and data reconstruction terms. Source code can be found
at https://github.com/Delay-Xili/SDNet.


http://arxiv.org/abs/2210.12873
FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning. (68%)
Kaiyuan Zhang; Guanhong Tao; Qiuling Xu; Siyuan Cheng; Shengwei An; Yingqi Liu; Shiwei Feng; Guangyu Shen; Pin-Yu Chen; Shiqing Ma; Xiangyu Zhang
  Federated Learning (FL) is a distributed learning paradigm that enables
different parties to train a model together for high quality and strong privacy
protection. In this scenario, individual participants may get compromised and
perform backdoor attacks by poisoning the data (or gradients). Existing work on
robust aggregation and certified FL robustness does not study how hardening
benign clients can affect the global model (and the malicious clients). In this
work, we theoretically analyze the connection among cross-entropy loss, attack
success rate, and clean accuracy in this setting. Moreover, we propose a
trigger reverse engineering based defense and show that our method can achieve
robustness improvement with guarantee (i.e., reducing the attack success rate)
without affecting benign accuracy. We conduct comprehensive experiments across
different datasets and attack settings. Our results on eight competing SOTA
defense methods show the empirical superiority of our method on both
single-shot and continuous FL backdoor attacks.


http://arxiv.org/abs/2210.13463
Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Future. (45%)
Guo-Jun Qi; Mubarak Shah
  In this paper, we review adversarial pretraining of self-supervised deep
networks including both convolutional neural networks and vision transformers.
Unlike the adversarial training with access to labeled examples, adversarial
pretraining is complicated as it only has access to unlabeled examples. To
incorporate adversaries into pretraining models on either input or feature
level, we find that existing approaches are largely categorized into two
groups: memory-free instance-wise attacks imposing worst-case perturbations on
individual examples, and memory-based adversaries shared across examples over
iterations. In particular, we review several representative adversarial
pretraining models based on Contrastive Learning (CL) and Masked Image Modeling
(MIM), respectively, two popular self-supervised pretraining methods in
literature. We also review miscellaneous issues about computing overheads,
input-/feature-level adversaries, as well as other adversarial pretraining
approaches beyond the above two groups. Finally, we discuss emerging trends and
future directions about the relations between adversarial and cooperative
pretraining, unifying adversarial CL and MIM pretraining, and the trade-off
between accuracy and robustness in adversarial pretraining.


http://arxiv.org/abs/2210.12396
ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation. (99%)
Fan Yin; Yao Li; Cho-Jui Hsieh; Kai-Wei Chang
  Adversarial Examples Detection (AED) is a crucial defense technique against
adversarial attacks and has drawn increasing attention from the Natural
Language Processing (NLP) community. Despite the surge of new AED methods, our
studies show that existing methods heavily rely on a shortcut to achieve good
performance. In other words, current search-based adversarial attacks in NLP
stop once model predictions change, and thus most adversarial examples
generated by those attacks are located near model decision boundaries. To
surpass this shortcut and fairly evaluate AED methods, we propose to test AED
methods with \textbf{F}ar \textbf{B}oundary (\textbf{FB}) adversarial examples.
Existing methods show worse than random guess performance under this scenario.
To overcome this limitation, we propose a new technique, \textbf{ADDMU},
\textbf{a}dversary \textbf{d}etection with \textbf{d}ata and \textbf{m}odel
\textbf{u}ncertainty, which combines two types of uncertainty estimation for
both regular and FB adversarial example detection. Our new method outperforms
previous methods by 3.6 and 6.0 \emph{AUC} points under each scenario. Finally,
our analysis shows that the two types of uncertainty provided by \textbf{ADDMU}
can be leveraged to characterize adversarial examples and identify the ones
that contribute most to model's robustness in adversarial training.


http://arxiv.org/abs/2210.13982
Hindering Adversarial Attacks with Implicit Neural Representations. (92%)
Andrei A. Rusu; Dan A. Calian; Sven Gowal; Raia Hadsell
  We introduce the Lossy Implicit Network Activation Coding (LINAC) defence, an
input transformation which successfully hinders several common adversarial
attacks on CIFAR-$10$ classifiers for perturbations up to $\epsilon = 8/255$ in
$L_\infty$ norm and $\epsilon = 0.5$ in $L_2$ norm. Implicit neural
representations are used to approximately encode pixel colour intensities in
$2\text{D}$ images such that classifiers trained on transformed data appear to
have robustness to small perturbations without adversarial training or large
drops in performance. The seed of the random number generator used to
initialise and train the implicit neural representation turns out to be
necessary information for stronger generic attacks, suggesting its role as a
private key. We devise a Parametric Bypass Approximation (PBA) attack strategy
for key-based defences, which successfully invalidates an existing method in
this category. Interestingly, our LINAC defence also hinders some transfer and
adaptive attacks, including our novel PBA strategy. Our results emphasise the
importance of a broad range of customised attacks despite apparent robustness
according to standard evaluations. LINAC source code and parameters of defended
classifier evaluated throughout this submission are available:
https://github.com/deepmind/linac


http://arxiv.org/abs/2210.12598
GANI: Global Attacks on Graph Neural Networks via Imperceptible Node Injections. (81%)
Junyuan Fang; Haixian Wen; Jiajing Wu; Qi Xuan; Zibin Zheng; Chi K. Tse
  Graph neural networks (GNNs) have found successful applications in various
graph-related tasks. However, recent studies have shown that many GNNs are
vulnerable to adversarial attacks. In a vast majority of existing studies,
adversarial attacks on GNNs are launched via direct modification of the
original graph such as adding/removing links, which may not be applicable in
practice. In this paper, we focus on a realistic attack operation via injecting
fake nodes. The proposed Global Attack strategy via Node Injection (GANI) is
designed under the comprehensive consideration of an unnoticeable perturbation
setting from both structure and feature domains. Specifically, to make the node
injections as imperceptible and effective as possible, we propose a sampling
operation to determine the degree of the newly injected nodes, and then
generate features and select neighbors for these injected nodes based on the
statistical information of features and evolutionary perturbations obtained
from a genetic algorithm, respectively. In particular, the proposed feature
generation mechanism is suitable for both binary and continuous node features.
Extensive experimental results on benchmark datasets against both general and
defended GNNs show strong attack performance of GANI. Moreover, the
imperceptibility analyses also demonstrate that GANI achieves a relatively
unnoticeable injection on benchmark datasets.


http://arxiv.org/abs/2210.12606
Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games. (26%)
Maria-Florina Balcan; Rattana Pukdee; Pradeep Ravikumar; Hongyang Zhang
  Adversarial training is a standard technique for training adversarially
robust models. In this paper, we study adversarial training as an alternating
best-response strategy in a 2-player zero-sum game. We prove that even in a
simple scenario of a linear classifier and a statistical model that abstracts
robust vs. non-robust features, the alternating best response strategy of such
game may not converge. On the other hand, a unique pure Nash equilibrium of the
game exists and is provably robust. We support our theoretical results with
experiments, showing the non-convergence of adversarial training and the
robustness of Nash equilibrium.


http://arxiv.org/abs/2210.12367
Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation. (4%)
Wenhao Wu; Wei Li; Jiachen Liu; Xinyan Xiao; Sujian Li; Yajuan Lyu
  Though model robustness has been extensively studied in language
understanding, the robustness of Seq2Seq generation remains understudied. In
this paper, we conduct the first quantitative analysis on the robustness of
pre-trained Seq2Seq models. We find that even current SOTA pre-trained Seq2Seq
model (BART) is still vulnerable, which leads to significant degeneration in
faithfulness and informativeness for text generation tasks. This motivated us
to further propose a novel adversarial augmentation framework, namely AdvSeq,
for generally improving faithfulness and informativeness of Seq2Seq models via
enhancing their robustness. AdvSeq automatically constructs two types of
adversarial augmentations during training, including implicit adversarial
samples by perturbing word representations and explicit adversarial samples by
word swapping, both of which effectively improve Seq2Seq robustness. Extensive
experiments on three popular text generation tasks demonstrate that AdvSeq
significantly improves both the faithfulness and informativeness of Seq2Seq
generation under both automatic and human evaluation settings.


http://arxiv.org/abs/2210.12030
Evolution of Neural Tangent Kernels under Benign and Adversarial Training. (99%)
Noel Loo; Ramin Hasani; Alexander Amini; Daniela Rus
  Two key challenges facing modern deep learning are mitigating deep networks'
vulnerability to adversarial attacks and understanding deep learning's
generalization capabilities. Towards the first issue, many defense strategies
have been developed, with the most common being Adversarial Training (AT).
Towards the second challenge, one of the dominant theories that has emerged is
the Neural Tangent Kernel (NTK) -- a characterization of neural network
behavior in the infinite-width limit. In this limit, the kernel is frozen, and
the underlying feature map is fixed. In finite widths, however, there is
evidence that feature learning happens at the earlier stages of the training
(kernel learning) before a second phase where the kernel remains fixed (lazy
training). While prior work has aimed at studying adversarial vulnerability
through the lens of the frozen infinite-width NTK, there is no work that
studies the adversarial robustness of the empirical/finite NTK during training.
In this work, we perform an empirical study of the evolution of the empirical
NTK under standard and adversarial training, aiming to disambiguate the effect
of adversarial training on kernel learning and lazy training. We find under
adversarial training, the empirical NTK rapidly converges to a different kernel
(and feature map) than standard training. This new kernel provides adversarial
robustness, even when non-robust training is performed on top of it.
Furthermore, we find that adversarial training on top of a fixed kernel can
yield a classifier with $76.1\%$ robust accuracy under PGD attacks with
$\varepsilon = 4/255$ on CIFAR-10.


http://arxiv.org/abs/2210.12179
The Dark Side of AutoML: Towards Architectural Backdoor Search. (68%)
Ren Pang; Changjiang Li; Zhaohan Xi; Shouling Ji; Ting Wang
  This paper asks the intriguing question: is it possible to exploit neural
architecture search (NAS) as a new attack vector to launch previously
improbable attacks? Specifically, we present EVAS, a new attack that leverages
NAS to find neural architectures with inherent backdoors and exploits such
vulnerability using input-aware triggers. Compared with existing attacks, EVAS
demonstrates many interesting properties: (i) it does not require polluting
training data or perturbing model parameters; (ii) it is agnostic to downstream
fine-tuning or even re-training from scratch; (iii) it naturally evades
defenses that rely on inspecting model parameters or training data. With
extensive evaluation on benchmark datasets, we show that EVAS features high
evasiveness, transferability, and robustness, thereby expanding the adversary's
design spectrum. We further characterize the mechanisms underlying EVAS, which
are possibly explainable by architecture-level ``shortcuts'' that recognize
trigger patterns. This work raises concerns about the current practice of NAS
and points to potential directions to develop effective countermeasures.


http://arxiv.org/abs/2210.11841
Diffusion Visual Counterfactual Explanations. (10%)
Maximilian Augustin; Valentyn Boreiko; Francesco Croce; Matthias Hein
  Visual Counterfactual Explanations (VCEs) are an important tool to understand
the decisions of an image classifier. They are 'small' but 'realistic' semantic
changes of the image changing the classifier decision. Current approaches for
the generation of VCEs are restricted to adversarially robust models and often
contain non-realistic artefacts, or are limited to image classification
problems with few classes. In this paper, we overcome this by generating
Diffusion Visual Counterfactual Explanations (DVCEs) for arbitrary ImageNet
classifiers via a diffusion process. Two modifications to the diffusion process
are key for our DVCEs: first, an adaptive parameterization, whose
hyperparameters generalize across images and models, together with distance
regularization and late start of the diffusion process, allow us to generate
images with minimal semantic changes to the original ones but different
classification. Second, our cone regularization via an adversarially robust
model ensures that the diffusion process does not converge to trivial
non-semantic changes, but instead produces realistic images of the target class
which achieve high confidence by the classifier.


http://arxiv.org/abs/2210.12233
TCAB: A Large-Scale Text Classification Attack Benchmark. (10%)
Kalyani Asthana; Zhouhang Xie; Wencong You; Adam Noack; Jonathan Brophy; Sameer Singh; Daniel Lowd
  We introduce the Text Classification Attack Benchmark (TCAB), a dataset for
analyzing, understanding, detecting, and labeling adversarial attacks against
text classifiers. TCAB includes 1.5 million attack instances, generated by
twelve adversarial attacks targeting three classifiers trained on six source
datasets for sentiment analysis and abuse detection in English. Unlike standard
text classification, text attacks must be understood in the context of the
target classifier that is being attacked, and thus features of the target
classifier are important as well. TCAB includes all attack instances that are
successful in flipping the predicted label; a subset of the attacks are also
labeled by human annotators to determine how frequently the primary semantics
are preserved. The process of generating attacks is automated, so that TCAB can
easily be extended to incorporate new text attacks and better classifiers as
they are developed. In addition to the primary tasks of detecting and labeling
attacks, TCAB can also be used for attack localization, attack target labeling,
and attack characterization. TCAB code and dataset are available at
https://react-nlp.github.io/tcab/.


http://arxiv.org/abs/2210.11726
A critical review of cyber-physical security for building automation systems. (2%)
Guowen Li; Lingyu Ren; Yangyang Fu; Zhiyao Yang; Veronica Adetola; Jin Wen; Qi Zhu; Teresa Wu; K. Selcuk Candanf; Zheng O'Neill
  Modern Building Automation Systems (BASs), as the brain that enables the
smartness of a smart building, often require increased connectivity both among
system components as well as with outside entities, such as optimized
automation via outsourced cloud analytics and increased building-grid
integrations. However, increased connectivity and accessibility come with
increased cyber security threats. BASs were historically developed as closed
environments with limited cyber-security considerations. As a result, BASs in
many buildings are vulnerable to cyber-attacks that may cause adverse
consequences, such as occupant discomfort, excessive energy usage, and
unexpected equipment downtime. Therefore, there is a strong need to advance the
state-of-the-art in cyber-physical security for BASs and provide practical
solutions for attack mitigation in buildings. However, an inclusive and
systematic review of BAS vulnerabilities, potential cyber-attacks with impact
assessment, detection & defense approaches, and cyber-secure resilient control
strategies is currently lacking in the literature. This review paper fills the
gap by providing a comprehensive up-to-date review of cyber-physical security
for BASs at three levels in commercial buildings: management level, automation
level, and field level. The general BASs vulnerabilities and protocol-specific
vulnerabilities for the four dominant BAS protocols are reviewed, followed by a
discussion on four attack targets and seven potential attack scenarios. The
impact of cyber-attacks on BASs is summarized as signal corruption, signal
delaying, and signal blocking. The typical cyber-attack detection and defense
approaches are identified at the three levels. Cyber-secure resilient control
strategies for BASs under attack are categorized into passive and active
resilient control schemes. Open challenges and future opportunities are finally
discussed.


http://arxiv.org/abs/2210.11735
Extracted BERT Model Leaks More Information than You Think! (1%)
Xuanli He; Chen Chen; Lingjuan Lyu; Qiongkai Xu
  The collection and availability of big data, combined with advances in
pre-trained models (e.g. BERT), have revolutionized the predictive performance
of natural language processing tasks. This allows corporations to provide
machine learning as a service (MLaaS) by encapsulating fine-tuned BERT-based
models as APIs. Due to significant commercial interest, there has been a surge
of attempts to steal re mote services via model extraction. Although previous
works have made progress in defending against model extraction attacks, there
has been little discussion on their performance in preventing privacy leakage.
This work bridges this gap by launching an attribute inference attack against
the extracted BERT model. Our extensive experiments reveal that model
extraction can cause severe privacy leakage even when victim models are
facilitated with advanced defensive strategies.


http://arxiv.org/abs/2210.11598
Identifying Human Strategies for Generating Word-Level Adversarial Examples. (98%)
Maximilian Mozes; Bennett Kleinberg; Lewis D. Griffin
  Adversarial examples in NLP are receiving increasing research attention. One
line of investigation is the generation of word-level adversarial examples
against fine-tuned Transformer models that preserve naturalness and
grammaticality. Previous work found that human- and machine-generated
adversarial examples are comparable in their naturalness and grammatical
correctness. Most notably, humans were able to generate adversarial examples
much more effortlessly than automated attacks. In this paper, we provide a
detailed analysis of exactly how humans create these adversarial examples. By
exploring the behavioural patterns of human workers during the generation
process, we identify statistically significant tendencies based on which words
humans prefer to select for adversarial replacement (e.g., word frequencies,
word saliencies, sentiment) as well as where and when words are replaced in an
input sequence. With our findings, we seek to inspire efforts that harness
human strategies for more robust NLP models.


http://arxiv.org/abs/2210.15427
Are You Stealing My Model? Sample Correlation for Fingerprinting Deep Neural Networks. (98%)
Jiyang Guan; Jian Liang; Ran He
  An off-the-shelf model as a commercial service could be stolen by model
stealing attacks, posing great threats to the rights of the model owner. Model
fingerprinting aims to verify whether a suspect model is stolen from the victim
model, which gains more and more attention nowadays. Previous methods always
leverage the transferable adversarial examples as the model fingerprint, which
is sensitive to adversarial defense or transfer learning scenarios. To address
this issue, we consider the pairwise relationship between samples instead and
propose a novel yet simple model stealing detection method based on SAmple
Correlation (SAC). Specifically, we present SAC-w that selects wrongly
classified normal samples as model inputs and calculates the mean correlation
among their model outputs. To reduce the training time, we further develop
SAC-m that selects CutMix Augmented samples as model inputs, without the need
for training the surrogate models or generating adversarial examples. Extensive
results validate that SAC successfully defends against various model stealing
attacks, even including adversarial training or transfer learning, and detects
the stolen models with the best performance in terms of AUC across different
datasets and model architectures. The codes are available at
https://github.com/guanjiyang/SAC.


http://arxiv.org/abs/2210.11498
Balanced Adversarial Training: Balancing Tradeoffs between Fickleness and Obstinacy in NLP Models. (98%)
Hannah Chen; Yangfeng Ji; David Evans
  Traditional (fickle) adversarial examples involve finding a small
perturbation that does not change an input's true label but confuses the
classifier into outputting a different prediction. Conversely, obstinate
adversarial examples occur when an adversary finds a small perturbation that
preserves the classifier's prediction but changes the true label of an input.
Adversarial training and certified robust training have shown some
effectiveness in improving the robustness of machine learnt models to fickle
adversarial examples. We show that standard adversarial training methods
focused on reducing vulnerability to fickle adversarial examples may make a
model more vulnerable to obstinate adversarial examples, with experiments for
both natural language inference and paraphrase identification tasks. To counter
this phenomenon, we introduce Balanced Adversarial Training, which incorporates
contrastive learning to increase robustness against both fickle and obstinate
adversarial examples.


http://arxiv.org/abs/2210.11513
Learning Sample Reweighting for Accuracy and Adversarial Robustness. (93%)
Chester Holtz; Tsui-Wei Weng; Gal Mishne
  There has been great interest in enhancing the robustness of neural network
classifiers to defend against adversarial perturbations through adversarial
training, while balancing the trade-off between robust accuracy and standard
accuracy. We propose a novel adversarial training framework that learns to
reweight the loss associated with individual training samples based on a notion
of class-conditioned margin, with the goal of improving robust generalization.
We formulate weighted adversarial training as a bilevel optimization problem
with the upper-level problem corresponding to learning a robust classifier, and
the lower-level problem corresponding to learning a parametric function that
maps from a sample's \textit{multi-class margin} to an importance weight.
Extensive experiments demonstrate that our approach consistently improves both
clean and robust accuracy compared to related methods and state-of-the-art
baselines.


http://arxiv.org/abs/2210.11592
New data poison attacks on machine learning classifiers for mobile exfiltration. (80%)
Miguel A. Ramirez; Sangyoung Yoon; Ernesto Damiani; Hussam Al Hamadi; Claudio Agostino Ardagna; Nicola Bena; Young-Ji Byon; Tae-Yeon Kim; Chung-Suk Cho; Chan Yeob Yeun
  Most recent studies have shown several vulnerabilities to attacks with the
potential to jeopardize the integrity of the model, opening in a few recent
years a new window of opportunity in terms of cyber-security. The main interest
of this paper is directed towards data poisoning attacks involving
label-flipping, this kind of attacks occur during the training phase, being the
aim of the attacker to compromise the integrity of the targeted machine
learning model by drastically reducing the overall accuracy of the model and/or
achieving the missclassification of determined samples. This paper is conducted
with intention of proposing two new kinds of data poisoning attacks based on
label-flipping, the targeted of the attack is represented by a variety of
machine learning classifiers dedicated for malware detection using mobile
exfiltration data. With that, the proposed attacks are proven to be
model-agnostic, having successfully corrupted a wide variety of machine
learning models; Logistic Regression, Decision Tree, Random Forest and KNN are
some examples. The first attack is performs label-flipping actions randomly
while the second attacks performs label flipping only one of the 2 classes in
particular. The effects of each attack are analyzed in further detail with
special emphasis on the accuracy drop and the misclassification rate. Finally,
this paper pursuits further research direction by suggesting the development of
a defense technique that could promise a feasible detection and/or mitigation
mechanisms; such technique should be capable of conferring a certain level of
robustness to a target model against potential attackers.


http://arxiv.org/abs/2210.11407
Similarity of Neural Architectures Based on Input Gradient Transferability. (62%)
Jaehui Hwang; Dongyoon Han; Byeongho Heo; Song Park; Sanghyuk Chun; Jong-Seok Lee
  In this paper, we aim to design a quantitative similarity function between
two neural architectures. Specifically, we define a model similarity using
input gradient transferability. We generate adversarial samples of two networks
and measure the average accuracy of the networks on adversarial samples of each
other. If two networks are highly correlated, then the attack transferability
will be high, resulting in high similarity. Using the similarity score, we
investigate two topics: (1) Which network component contributes to the model
diversity? (2) How does model diversity affect practical scenarios? We answer
the first question by providing feature importance analysis and clustering
analysis. The second question is validated by two different scenarios: model
ensemble and knowledge distillation. Our findings show that model diversity
takes a key role when interacting with different neural architectures. For
example, we found that more diversity leads to better ensemble performance. We
also observe that the relationship between teacher and student networks and
distillation performance depends on the choice of the base architecture of the
teacher and student networks. We expect our analysis tool helps a high-level
understanding of differences between various neural architectures as well as
practical guidance when using multiple architectures.


http://arxiv.org/abs/2210.11242
Attacking Motion Estimation with Adversarial Snow. (16%)
Jenny Schmalfuss; Lukas Mehl; Andrés Bruhn
  Current adversarial attacks for motion estimation (optical flow) optimize
small per-pixel perturbations, which are unlikely to appear in the real world.
In contrast, we exploit a real-world weather phenomenon for a novel attack with
adversarially optimized snow. At the core of our attack is a differentiable
renderer that consistently integrates photorealistic snowflakes with realistic
motion into the 3D scene. Through optimization we obtain adversarial snow that
significantly impacts the optical flow while being indistinguishable from
ordinary snow. Surprisingly, the impact of our novel attack is largest on
methods that previously showed a high robustness to small L_p perturbations.


http://arxiv.org/abs/2210.11049
How Does a Deep Learning Model Architecture Impact Its Privacy? (5%)
Guangsheng Zhang; Bo Liu; Huan Tian; Tianqing Zhu; Ming Ding; Wanlei Zhou
  As a booming research area in the past decade, deep learning technologies
have been driven by big data collected and processed on an unprecedented scale.
However, the sensitive information in the collected training data raises
privacy concerns. Recent research indicated that deep learning models are
vulnerable to various privacy attacks, including membership inference attacks,
attribute inference attacks, and gradient inversion attacks. It is noteworthy
that the performance of the attacks varies from model to model. In this paper,
we conduct empirical analyses to answer a fundamental question: Does model
architecture affect model privacy? We investigate several representative model
architectures from CNNs to Transformers, and show that Transformers are
generally more vulnerable to privacy attacks than CNNs. We further demonstrate
that the micro design of activation layers, stem layers, and bias parameters,
are the major reasons why CNNs are more resilient to privacy attacks than
Transformers. We also find that the presence of attention modules is another
reason why Transformers are more vulnerable to privacy attacks. We hope our
discovery can shed some new light on how to defend against the investigated
privacy attacks and help the community build privacy-friendly model
architectures.


http://arxiv.org/abs/2210.11061
Analyzing the Robustness of Decentralized Horizontal and Vertical Federated Learning Architectures in a Non-IID Scenario. (4%)
Pedro Miguel Sánchez Sánchez; Alberto Huertas Celdrán; Enrique Tomás Martínez Beltrán; Daniel Demeter; Gérôme Bovet; Gregorio Martínez Pérez; Burkhard Stiller
  Federated learning (FL) allows participants to collaboratively train machine
and deep learning models while protecting data privacy. However, the FL
paradigm still presents drawbacks affecting its trustworthiness since malicious
participants could launch adversarial attacks against the training process.
Related work has studied the robustness of horizontal FL scenarios under
different attacks. However, there is a lack of work evaluating the robustness
of decentralized vertical FL and comparing it with horizontal FL architectures
affected by adversarial attacks. Thus, this work proposes three decentralized
FL architectures, one for horizontal and two for vertical scenarios, namely
HoriChain, VertiChain, and VertiComb. These architectures present different
neural networks and training protocols suitable for horizontal and vertical
scenarios. Then, a decentralized, privacy-preserving, and federated use case
with non-IID data to classify handwritten digits is deployed to evaluate the
performance of the three architectures. Finally, a set of experiments computes
and compares the robustness of the proposed architectures when they are
affected by different data poisoning based on image watermarks and gradient
poisoning adversarial attacks. The experiments show that even though particular
configurations of both attacks can destroy the classification performance of
the architectures, HoriChain is the most robust one.


http://arxiv.org/abs/2210.11082
Apple of Sodom: Hidden Backdoors in Superior Sentence Embeddings via Contrastive Learning. (3%)
Xiaoyi Chen; Baisong Xin; Shengfang Zhai; Shiqing Ma; Qingni Shen; Zhonghai Wu
  This paper finds that contrastive learning can produce superior sentence
embeddings for pre-trained models but is also vulnerable to backdoor attacks.
We present the first backdoor attack framework, BadCSE, for state-of-the-art
sentence embeddings under supervised and unsupervised learning settings. The
attack manipulates the construction of positive and negative pairs so that the
backdoored samples have a similar embedding with the target sample (targeted
attack) or the negative embedding of its clean version (non-targeted attack).
By injecting the backdoor in sentence embeddings, BadCSE is resistant against
downstream fine-tuning. We evaluate BadCSE on both STS tasks and other
downstream tasks. The supervised non-targeted attack obtains a performance
degradation of 194.86%, and the targeted attack maps the backdoored samples to
the target embedding with a 97.70% success rate while maintaining the model
utility.


http://arxiv.org/abs/2210.11620
LOT: Layer-wise Orthogonal Training on Improving l2 Certified Robustness. (3%)
Xiaojun Xu; Linyi Li; Bo Li
  Recent studies show that training deep neural networks (DNNs) with Lipschitz
constraints are able to enhance adversarial robustness and other model
properties such as stability. In this paper, we propose a layer-wise orthogonal
training method (LOT) to effectively train 1-Lipschitz convolution layers via
parametrizing an orthogonal matrix with an unconstrained matrix. We then
efficiently compute the inverse square root of a convolution kernel by
transforming the input domain to the Fourier frequency domain. On the other
hand, as existing works show that semi-supervised training helps improve
empirical robustness, we aim to bridge the gap and prove that semi-supervised
learning also improves the certified robustness of Lipschitz-bounded models. We
conduct comprehensive evaluations for LOT under different settings. We show
that LOT significantly outperforms baselines regarding deterministic l2
certified robustness, and scales to deeper neural networks. Under the
supervised scenario, we improve the state-of-the-art certified robustness for
all architectures (e.g. from 59.04% to 63.50% on CIFAR-10 and from 32.57% to
34.59% on CIFAR-100 at radius rho = 36/255 for 40-layer networks). With
semi-supervised learning over unlabelled data, we are able to improve
state-of-the-art certified robustness on CIFAR-10 at rho = 108/255 from 36.04%
to 42.39%. In addition, LOT consistently outperforms baselines on different
model architectures with only 1/3 evaluation time.


http://arxiv.org/abs/2210.10485
Few-shot Transferable Robust Representation Learning via Bilevel Attacks. (93%)
Minseon Kim; Hyeonjeong Ha; Sung Ju Hwang
  Existing adversarial learning methods for enhancing the robustness of deep
neural networks assume the availability of a large amount of data from which we
can generate adversarial examples. However, in an adversarial meta-learning
setting, the model needs to train with only a few adversarial examples to learn
a robust model for unseen tasks, which is a very difficult goal to achieve.
Further, learning transferable robust representations for unseen domains is a
difficult problem even with a large amount of data. To tackle such a challenge,
we propose a novel adversarial self-supervised meta-learning framework with
bilevel attacks which aims to learn robust representations that can generalize
across tasks and domains. Specifically, in the inner loop, we update the
parameters of the given encoder by taking inner gradient steps using two
different sets of augmented samples, and generate adversarial examples for each
view by maximizing the instance classification loss. Then, in the outer loop,
we meta-learn the encoder parameter to maximize the agreement between the two
adversarial examples, which enables it to learn robust representations. We
experimentally validate the effectiveness of our approach on unseen domain
adaptation tasks, on which it achieves impressive performance. Specifically,
our method significantly outperforms the state-of-the-art meta-adversarial
learning methods on few-shot learning tasks, as well as self-supervised
learning baselines in standard learning settings with large-scale datasets.


http://arxiv.org/abs/2210.10482
Targeted Adversarial Self-Supervised Learning. (86%)
Minseon Kim; Hyeonjeong Ha; Sooel Son; Sung Ju Hwang
  Recently, unsupervised adversarial training (AT) has been extensively studied
to attain robustness with the models trained upon unlabeled data. To this end,
previous studies have applied existing supervised adversarial training
techniques to self-supervised learning (SSL) frameworks. However, all have
resorted to untargeted adversarial learning as obtaining targeted adversarial
examples is unclear in the SSL setting lacking of label information. In this
paper, we propose a novel targeted adversarial training method for the SSL
frameworks. Specifically, we propose a target selection algorithm for the
adversarial SSL frameworks; it is designed to select the most confusing sample
for each given instance based on similarity and entropy, and perturb the given
instance toward the selected target sample. Our method significantly enhances
the robustness of an SSL model without requiring large batches of images or
additional models, unlike existing works aimed at achieving the same goal.
Moreover, our method is readily applicable to general SSL frameworks that only
uses positive pairs. We validate our method on benchmark datasets, on which it
obtains superior robust accuracies, outperforming existing unsupervised
adversarial training methods.


http://arxiv.org/abs/2210.10886
Backdoor Attack and Defense in Federated Generative Adversarial Network-based Medical Image Synthesis. (83%)
Ruinan Jin; Xiaoxiao Li
  Deep Learning-based image synthesis techniques have been applied in
healthcare research for generating medical images to support open research and
augment medical datasets. Training generative adversarial neural networks
(GANs) usually require large amounts of training data. Federated learning (FL)
provides a way of training a central model using distributed data while keeping
raw data locally. However, given that the FL server cannot access the raw data,
it is vulnerable to backdoor attacks, an adversarial by poisoning training
data. Most backdoor attack strategies focus on classification models and
centralized domains. It is still an open question if the existing backdoor
attacks can affect GAN training and, if so, how to defend against the attack in
the FL setting. In this work, we investigate the overlooked issue of backdoor
attacks in federated GANs (FedGANs). The success of this attack is subsequently
determined to be the result of some local discriminators overfitting the
poisoned data and corrupting the local GAN equilibrium, which then further
contaminates other clients when averaging the generator's parameters and yields
high generator loss. Therefore, we proposed FedDetect, an efficient and
effective way of defending against the backdoor attack in the FL setting, which
allows the server to detect the client's adversarial behavior based on their
losses and block the malicious clients. Our extensive experiments on two
medical datasets with different modalities demonstrate the backdoor attack on
FedGANs can result in synthetic images with low fidelity. After detecting and
suppressing the detected malicious clients using the proposed defense strategy,
we show that FedGANs can synthesize high-quality medical datasets (with labels)
for data augmentation to improve classification models' performance.


http://arxiv.org/abs/2210.11237
Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Survey. (69%)
Hui Cao; Wenlong Zou; Yinkun Wang; Ting Song; Mengjun Liu
  Since the 2004 DARPA Grand Challenge, the autonomous driving technology has
witnessed nearly two decades of rapid development. Particularly, in recent
years, with the application of new sensors and deep learning technologies
extending to the autonomous field, the development of autonomous driving
technology has continued to make breakthroughs. Thus, many carmakers and
high-tech giants dedicated to research and system development of autonomous
driving. However, as the foundation of autonomous driving, the deep learning
technology faces many new security risks. The academic community has proposed
deep learning countermeasures against the adversarial examples and AI backdoor,
and has introduced them into the autonomous driving field for verification.
Deep learning security matters to autonomous driving system security, and then
matters to personal safety, which is an issue that deserves attention and
research.This paper provides an summary of the concepts, developments and
recent research in deep learning security technologies in autonomous driving.
Firstly, we briefly introduce the deep learning framework and pipeline in the
autonomous driving system, which mainly include the deep learning technologies
and algorithms commonly used in this field. Moreover, we focus on the potential
security threats of the deep learning based autonomous driving system in each
functional layer in turn. We reviews the development of deep learning attack
technologies to autonomous driving, investigates the State-of-the-Art
algorithms, and reveals the potential risks. At last, we provides an outlook on
deep learning security in the autonomous driving field and proposes
recommendations for building a safe and trustworthy autonomous driving system.


http://arxiv.org/abs/2210.10683
Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP. (64%)
Yangyi Chen; Hongcheng Gao; Ganqu Cui; Fanchao Qi; Longtao Huang; Zhiyuan Liu; Maosong Sun
  Textual adversarial samples play important roles in multiple subfields of NLP
research, including security, evaluation, explainability, and data
augmentation. However, most work mixes all these roles, obscuring the problem
definitions and research goals of the security role that aims to reveal the
practical concerns of NLP models. In this paper, we rethink the research
paradigm of textual adversarial samples in security scenarios. We discuss the
deficiencies in previous work and propose our suggestions that the research on
the Security-oriented adversarial NLP (SoadNLP) should: (1) evaluate their
methods on security tasks to demonstrate the real-world concerns; (2) consider
real-world attackers' goals, instead of developing impractical methods. To this
end, we first collect, process, and release a security datasets collection
Advbench. Then, we reformalize the task and adjust the emphasis on different
goals in SoadNLP. Next, we propose a simple method based on heuristic rules
that can easily fulfill the actual adversarial goals to simulate real-world
attack methods. We conduct experiments on both the attack and the defense sides
on Advbench. Experimental results show that our method has higher practical
value, indicating that the research paradigm in SoadNLP may start from our new
benchmark. All the code and data of Advbench can be obtained at
\url{https://github.com/thunlp/Advbench}.


http://arxiv.org/abs/2210.14164
Model-Free Prediction of Adversarial Drop Points in 3D Point Clouds. (54%)
Hanieh Naderi; Chinthaka Dinesh; Ivan V. Bajic; Shohreh Kasaei
  Adversarial attacks pose serious challenges for deep neural network
(DNN)-based analysis of various input signals. In the case of 3D point clouds,
methods have been developed to identify points that play a key role in the
network decision, and these become crucial in generating existing adversarial
attacks. For example, a saliency map approach is a popular method for
identifying adversarial drop points, whose removal would significantly impact
the network decision. Generally, methods for identifying adversarial points
rely on the deep model itself in order to determine which points are critically
important for the model's decision. This paper aims to provide a novel
viewpoint on this problem, in which adversarial points can be predicted
independently of the model. To this end, we define 14 point cloud features and
use multiple linear regression to examine whether these features can be used
for model-free adversarial point prediction, and which combination of features
is best suited for this purpose. Experiments show that a suitable combination
of features is able to predict adversarial points of three different networks
-- PointNet, PointNet++, and DGCNN -- significantly better than a random guess.
The results also provide further insight into DNNs for point cloud analysis, by
showing which features play key roles in their decision-making process.


http://arxiv.org/abs/2210.10936
FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Information. (41%)
Xiaoyu Cao; Jinyuan Jia; Zaixi Zhang; Neil Zhenqiang Gong
  Federated learning is vulnerable to poisoning attacks in which malicious
clients poison the global model via sending malicious model updates to the
server. Existing defenses focus on preventing a small number of malicious
clients from poisoning the global model via robust federated learning methods
and detecting malicious clients when there are a large number of them. However,
it is still an open challenge how to recover the global model from poisoning
attacks after the malicious clients are detected. A naive solution is to remove
the detected malicious clients and train a new global model from scratch, which
incurs large cost that may be intolerable for resource-constrained clients such
as smartphones and IoT devices.
  In this work, we propose FedRecover, which can recover an accurate global
model from poisoning attacks with small cost for the clients. Our key idea is
that the server estimates the clients' model updates instead of asking the
clients to compute and communicate them during the recovery process. In
particular, the server stores the global models and clients' model updates in
each round, when training the poisoned global model. During the recovery
process, the server estimates a client's model update in each round using its
stored historical information. Moreover, we further optimize FedRecover to
recover a more accurate global model using warm-up, periodic correction,
abnormality fixing, and final tuning strategies, in which the server asks the
clients to compute and communicate their exact model updates. Theoretically, we
show that the global model recovered by FedRecover is close to or the same as
that recovered by train-from-scratch under some assumptions. Empirically, our
evaluation on four datasets, three federated learning methods, as well as
untargeted and targeted poisoning attacks (e.g., backdoor attacks) shows that
FedRecover is both accurate and efficient.


http://arxiv.org/abs/2210.13235
Chaos Theory and Adversarial Robustness. (22%)
Jonathan S. Kent
  Neural Networks, being susceptible to adversarial attacks, should face a
strict level of scrutiny before being deployed in critical or adversarial
applications. This paper uses ideas from Chaos Theory to explain, analyze, and
quantify the degree to which Neural Networks are susceptible to or robust
against adversarial attacks. Our results show that susceptibility to attack
grows significantly with the depth of the model, which has significant safety
implications for the design of Neural Networks for production environments. We
also demonstrate how to quickly and easily approximate the certified robustness
radii for extremely large models, which until now has been computationally
infeasible to calculate directly, as well as show a clear relationship between
our new susceptibility metric and post-attack accuracy.


http://arxiv.org/abs/2210.10880
Learning to Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning. (16%)
Ruihan Wu; Xiangyu Chen; Chuan Guo; Kilian Q. Weinberger
  Gradient inversion attack enables recovery of training samples from model
updates in federated learning (FL) and constitutes a serious threat to data
privacy. To mitigate this vulnerability, prior work proposed both principled
defenses based on differential privacy, as well as heuristic defenses based on
gradient compression as countermeasures. These defenses have so far been very
effective, in particular those based on gradient compression that allow the
model to maintain high accuracy while greatly reducing the attack's
effectiveness. In this work, we argue that such findings do not accurately
reflect the privacy risk in FL, and show that existing defenses can be broken
by a simple adaptive attack that trains a model using auxiliary data to learn
how to invert gradients on both vision and language tasks.


http://arxiv.org/abs/2210.10378
Variational Model Perturbation for Source-Free Domain Adaptation. (1%)
Mengmeng Jing; Xiantong Zhen; Jingjing Li; Cees G. M. Snoek
  We aim for source-free domain adaptation, where the task is to deploy a model
pre-trained on source domains to target domains. The challenges stem from the
distribution shift from the source to the target domain, coupled with the
unavailability of any source data and labeled target data for optimization.
Rather than fine-tuning the model by updating the parameters, we propose to
perturb the source model to achieve adaptation to target domains. We introduce
perturbations into the model parameters by variational Bayesian inference in a
probabilistic framework. By doing so, we can effectively adapt the model to the
target domain while largely preserving the discriminative ability. Importantly,
we demonstrate the theoretical connection to learning Bayesian neural networks,
which proves the generalizability of the perturbed model to target domains. To
enable more efficient optimization, we further employ a parameter sharing
strategy, which substantially reduces the learnable parameters compared to a
fully Bayesian neural network. Our model perturbation provides a new
probabilistic way for domain adaptation which enables efficient adaptation to
target domains while maximally preserving knowledge in source models.
Experiments on several source-free benchmarks under three different evaluation
settings verify the effectiveness of the proposed variational model
perturbation for source-free domain adaptation.


http://arxiv.org/abs/2210.09852
Scaling Adversarial Training to Large Perturbation Bounds. (98%)
Sravanti Addepalli; Samyak Jain; Gaurang Sriramanan; R. Venkatesh Babu
  The vulnerability of Deep Neural Networks to Adversarial Attacks has fuelled
research towards building robust models. While most Adversarial Training
algorithms aim at defending attacks constrained within low magnitude Lp norm
bounds, real-world adversaries are not limited by such constraints. In this
work, we aim to achieve adversarial robustness within larger bounds, against
perturbations that may be perceptible, but do not change human (or Oracle)
prediction. The presence of images that flip Oracle predictions and those that
do not makes this a challenging setting for adversarial robustness. We discuss
the ideal goals of an adversarial defense algorithm beyond perceptual limits,
and further highlight the shortcomings of naively extending existing training
algorithms to higher perturbation bounds. In order to overcome these
shortcomings, we propose a novel defense, Oracle-Aligned Adversarial Training
(OA-AT), to align the predictions of the network with that of an Oracle during
adversarial training. The proposed approach achieves state-of-the-art
performance at large epsilon bounds (such as an L-inf bound of 16/255 on
CIFAR-10) while outperforming existing defenses (AWP, TRADES, PGD-AT) at
standard bounds (8/255) as well.


http://arxiv.org/abs/2210.09671
Not All Poisons are Created Equal: Robust Training against Data Poisoning. (97%)
Yu Yang; Tian Yu Liu; Baharan Mirzasoleiman
  Data poisoning causes misclassification of test time target examples by
injecting maliciously crafted samples in the training data. Existing defenses
are often effective only against a specific type of targeted attack,
significantly degrade the generalization performance, or are prohibitive for
standard deep learning pipelines.
  In this work, we propose an efficient defense mechanism that significantly
reduces the success rate of various data poisoning attacks, and provides
theoretical guarantees for the performance of the model. Targeted attacks work
by adding bounded perturbations to a randomly selected subset of training data
to match the targets' gradient or representation. We show that: (i) under
bounded perturbations, only a number of poisons can be optimized to have a
gradient that is close enough to that of the target and make the attack
successful; (ii) such effective poisons move away from their original class and
get isolated in the gradient space; (iii) dropping examples in low-density
gradient regions during training can successfully eliminate the effective
poisons, and guarantees similar training dynamics to that of training on full
data. Our extensive experiments show that our method significantly decreases
the success rate of state-of-the-art targeted attacks, including Gradient
Matching and Bullseye Polytope, and easily scales to large datasets.


http://arxiv.org/abs/2210.09658
ROSE: Robust Selective Fine-tuning for Pre-trained Language Models. (73%)
Lan Jiang; Hao Zhou; Yankai Lin; Peng Li; Jie Zhou; Rui Jiang
  Even though the large-scale language models have achieved excellent
performances, they suffer from various adversarial attacks. A large body of
defense methods has been proposed. However, they are still limited due to
redundant attack search spaces and the inability to defend against various
types of attacks. In this work, we present a novel fine-tuning approach called
\textbf{RO}bust \textbf{SE}letive fine-tuning (\textbf{ROSE}) to address this
issue. ROSE conducts selective updates when adapting pre-trained models to
downstream tasks, filtering out invaluable and unrobust updates of parameters.
Specifically, we propose two strategies: the first-order and second-order ROSE
for selecting target robust parameters. The experimental results show that ROSE
achieves significant improvements in adversarial robustness on various
downstream NLP tasks, and the ensemble method even surpasses both variants
above. Furthermore, ROSE can be easily incorporated into existing fine-tuning
methods to improve their adversarial robustness further. The empirical analysis
confirms that ROSE eliminates unrobust spurious updates during fine-tuning,
leading to solutions corresponding to flatter and wider optima than the
conventional method. Code is available at
\url{https://github.com/jiangllan/ROSE}.


http://arxiv.org/abs/2210.10667
Analysis of Master Vein Attacks on Finger Vein Recognition Systems. (56%)
Huy H. Nguyen; Trung-Nghia Le; Junichi Yamagishi; Isao Echizen
  Finger vein recognition (FVR) systems have been commercially used, especially
in ATMs, for customer verification. Thus, it is essential to measure their
robustness against various attack methods, especially when a hand-crafted FVR
system is used without any countermeasure methods. In this paper, we are the
first in the literature to introduce master vein attacks in which we craft a
vein-looking image so that it can falsely match with as many identities as
possible by the FVR systems. We present two methods for generating master veins
for use in attacking these systems. The first uses an adaptation of the latent
variable evolution algorithm with a proposed generative model (a multi-stage
combination of beta-VAE and WGAN-GP models). The second uses an adversarial
machine learning attack method to attack a strong surrogate CNN-based
recognition system. The two methods can be easily combined to boost their
attack ability. Experimental results demonstrated that the proposed methods
alone and together achieved false acceptance rates up to 73.29% and 88.79%,
respectively, against Miura's hand-crafted FVR system. We also point out that
Miura's system is easily compromised by non-vein-looking samples generated by a
WGAN-GP model with false acceptance rates up to 94.21%. The results raise the
alarm about the robustness of such systems and suggest that master vein attacks
should be considered an important security measure.


http://arxiv.org/abs/2210.10272
Training set cleansing of backdoor poisoning by self-supervised representation learning. (56%)
H. Wang; S. Karami; O. Dia; H. Ritter; E. Emamjomeh-Zadeh; J. Chen; Z. Xiang; D. J. Miller; G. Kesidis
  A backdoor or Trojan attack is an important type of data poisoning attack
against deep neural network (DNN) classifiers, wherein the training dataset is
poisoned with a small number of samples that each possess the backdoor pattern
(usually a pattern that is either imperceptible or innocuous) and which are
mislabeled to the attacker's target class. When trained on a backdoor-poisoned
dataset, a DNN behaves normally on most benign test samples but makes incorrect
predictions to the target class when the test sample has the backdoor pattern
incorporated (i.e., contains a backdoor trigger). Here we focus on image
classification tasks and show that supervised training may build stronger
association between the backdoor pattern and the associated target class than
that between normal features and the true class of origin. By contrast,
self-supervised representation learning ignores the labels of samples and
learns a feature embedding based on images' semantic content. %We thus propose
to use unsupervised representation learning to avoid emphasising
backdoor-poisoned training samples and learn a similar feature embedding for
samples of the same class. Using a feature embedding found by self-supervised
representation learning, a data cleansing method, which combines sample
filtering and re-labeling, is developed. Experiments on CIFAR-10 benchmark
datasets show that our method achieves state-of-the-art performance in
mitigating backdoor attacks.


http://arxiv.org/abs/2210.10253
On the Adversarial Robustness of Mixture of Experts. (13%)
Joan Puigcerver; Rodolphe Jenatton; Carlos Riquelme; Pranjal Awasthi; Srinadh Bhojanapalli
  Adversarial robustness is a key desirable property of neural networks. It has
been empirically shown to be affected by their sizes, with larger networks
being typically more robust. Recently, Bubeck and Sellke proved a lower bound
on the Lipschitz constant of functions that fit the training data in terms of
their number of parameters. This raises an interesting open question, do -- and
can -- functions with more parameters, but not necessarily more computational
cost, have better robustness? We study this question for sparse Mixture of
Expert models (MoEs), that make it possible to scale up the model size for a
roughly constant computational cost. We theoretically show that under certain
conditions on the routing and the structure of the data, MoEs can have
significantly smaller Lipschitz constants than their dense counterparts. The
robustness of MoEs can suffer when the highest weighted experts for an input
implement sufficiently different functions. We next empirically evaluate the
robustness of MoEs on ImageNet using adversarial attacks and show they are
indeed more robust than dense models with the same computational cost. We make
key observations showing the robustness of MoEs to the choice of experts,
highlighting the redundancy of experts in models trained in practice.


http://arxiv.org/abs/2210.10114
Transferable Unlearnable Examples. (8%)
Jie Ren; Han Xu; Yuxuan Wan; Xingjun Ma; Lichao Sun; Jiliang Tang
  With more people publishing their personal data online, unauthorized data
usage has become a serious concern. The unlearnable strategies have been
introduced to prevent third parties from training on the data without
permission. They add perturbations to the users' data before publishing, which
aims to make the models trained on the perturbed published dataset invalidated.
These perturbations have been generated for a specific training setting and a
target dataset. However, their unlearnable effects significantly decrease when
used in other training settings and datasets. To tackle this issue, we propose
a novel unlearnable strategy based on Classwise Separability Discriminant
(CSD), which aims to better transfer the unlearnable effects to other training
settings and datasets by enhancing the linear separability. Extensive
experiments demonstrate the transferability of the proposed unlearnable
examples across training settings and datasets.


http://arxiv.org/abs/2210.09940
Automatic Detection of Fake Key Attacks in Secure Messaging. (8%)
Tarun Kumar Yadav; Devashish Gosain; Amir Herzberg; Daniel Zappala; Kent Seamons
  Popular instant messaging applications such as WhatsApp and Signal provide
end-to-end encryption for billions of users. They rely on a centralized,
application-specific server to distribute public keys and relay encrypted
messages between the users. Therefore, they prevent passive attacks but are
vulnerable to some active attacks. A malicious or hacked server can distribute
fake keys to users to perform man-in-the-middle or impersonation attacks. While
typical secure messaging applications provide a manual method for users to
detect these attacks, this burdens users, and studies show it is ineffective in
practice. This paper presents KTACA, a completely automated approach for key
verification that is oblivious to users and easy to deploy. We motivate KTACA
by designing two approaches to automatic key verification. One approach uses
client auditing (KTCA) and the second uses anonymous key monitoring (AKM). Both
have relatively inferior security properties, leading to KTACA, which combines
these approaches to provide the best of both worlds. We provide a security
analysis of each defense, identifying which attacks they can automatically
detect. We implement the active attacks to demonstrate they are possible, and
we also create a prototype implementation of all the defenses to measure their
performance and confirm their feasibility. Finally, we discuss the strengths
and weaknesses of each defense, the overhead on clients and service providers,
and deployment considerations.


http://arxiv.org/abs/2210.09643
Improving Adversarial Robustness by Contrastive Guided Diffusion Process. (2%)
Yidong Ouyang; Liyan Xie; Guang Cheng
  Synthetic data generation has become an emerging tool to help improve the
adversarial robustness in classification tasks since robust learning requires a
significantly larger amount of training samples compared with standard
classification tasks. Among various deep generative models, the diffusion model
has been shown to produce high-quality synthetic images and has achieved good
performance in improving the adversarial robustness. However, diffusion-type
methods are typically slow in data generation as compared with other generative
models. Although different acceleration techniques have been proposed recently,
it is also of great importance to study how to improve the sample efficiency of
generated data for the downstream task. In this paper, we first analyze the
optimality condition of synthetic distribution for achieving non-trivial robust
accuracy. We show that enhancing the distinguishability among the generated
data is critical for improving adversarial robustness. Thus, we propose the
Contrastive-Guided Diffusion Process (Contrastive-DP), which adopts the
contrastive loss to guide the diffusion model in data generation. We verify our
theoretical results using simulations and demonstrate the good performance of
Contrastive-DP on image datasets.


http://arxiv.org/abs/2210.09405
Towards Generating Adversarial Examples on Mixed-type Data. (99%)
Han Xu; Menghai Pan; Zhimeng Jiang; Huiyuan Chen; Xiaoting Li; Mahashweta Das; Hao Yang
  The existence of adversarial attacks (or adversarial examples) brings huge
concern about the machine learning (ML) model's safety issues. For many
safety-critical ML tasks, such as financial forecasting, fraudulent detection,
and anomaly detection, the data samples are usually mixed-type, which contain
plenty of numerical and categorical features at the same time. However, how to
generate adversarial examples with mixed-type data is still seldom studied. In
this paper, we propose a novel attack algorithm M-Attack, which can effectively
generate adversarial examples in mixed-type data. Based on M-Attack, attackers
can attempt to mislead the targeted classification model's prediction, by only
slightly perturbing both the numerical and categorical features in the given
data samples. More importantly, by adding designed regularizations, our
generated adversarial examples can evade potential detection models, which
makes the attack indeed insidious. Through extensive empirical studies, we
validate the effectiveness and efficiency of our attack method and evaluate the
robustness of existing classification models against our proposed attack. The
experimental results highlight the feasibility of generating adversarial
examples toward machine learning models in real-world applications.


http://arxiv.org/abs/2210.08870
Differential Evolution based Dual Adversarial Camouflage: Fooling Human Eyes and Object Detectors. (99%)
Jialiang Sun; Tingsong Jiang; Wen Yao; Donghua Wang; Xiaoqian Chen
  Recent studies reveal that deep neural network (DNN) based object detectors
are vulnerable to adversarial attacks in the form of adding the perturbation to
the images, leading to the wrong output of object detectors. Most current
existing works focus on generating perturbed images, also called adversarial
examples, to fool object detectors. Though the generated adversarial examples
themselves can remain a certain naturalness, most of them can still be easily
observed by human eyes, which limits their further application in the real
world. To alleviate this problem, we propose a differential evolution based
dual adversarial camouflage (DE_DAC) method, composed of two stages to fool
human eyes and object detectors simultaneously. Specifically, we try to obtain
the camouflage texture, which can be rendered over the surface of the object.
In the first stage, we optimize the global texture to minimize the discrepancy
between the rendered object and the scene images, making human eyes difficult
to distinguish. In the second stage, we design three loss functions to optimize
the local texture, making object detectors ineffective. In addition, we
introduce the differential evolution algorithm to search for the near-optimal
areas of the object to attack, improving the adversarial performance under
certain attack area limitations. Besides, we also study the performance of
adaptive DE_DAC, which can be adapted to the environment. Experiments show that
our proposed method could obtain a good trade-off between the fooling human
eyes and object detectors under multiple specific scenes and objects.


http://arxiv.org/abs/2210.09364
Probabilistic Categorical Adversarial Attack & Adversarial Training. (99%)
Pengfei He; Han Xu; Jie Ren; Yuxuan Wan; Zitao Liu; Jiliang Tang
  The existence of adversarial examples brings huge concern for people to apply
Deep Neural Networks (DNNs) in safety-critical tasks. However, how to generate
adversarial examples with categorical data is an important problem but lack of
extensive exploration. Previously established methods leverage greedy search
method, which can be very time-consuming to conduct successful attack. This
also limits the development of adversarial training and potential defenses for
categorical data. To tackle this problem, we propose Probabilistic Categorical
Adversarial Attack (PCAA), which transfers the discrete optimization problem to
a continuous problem that can be solved efficiently by Projected Gradient
Descent. In our paper, we theoretically analyze its optimality and time
complexity to demonstrate its significant advantage over current greedy based
attacks. Moreover, based on our attack, we propose an efficient adversarial
training framework. Through a comprehensive empirical study, we justify the
effectiveness of our proposed attack and defense algorithms.


http://arxiv.org/abs/2210.09194
Marksman Backdoor: Backdoor Attacks with Arbitrary Target Class. (96%)
Khoa D. Doan; Yingjie Lao; Ping Li
  In recent years, machine learning models have been shown to be vulnerable to
backdoor attacks. Under such attacks, an adversary embeds a stealthy backdoor
into the trained model such that the compromised models will behave normally on
clean inputs but will misclassify according to the adversary's control on
maliciously constructed input with a trigger. While these existing attacks are
very effective, the adversary's capability is limited: given an input, these
attacks can only cause the model to misclassify toward a single pre-defined or
target class. In contrast, this paper exploits a novel backdoor attack with a
much more powerful payload, denoted as Marksman, where the adversary can
arbitrarily choose which target class the model will misclassify given any
input during inference. To achieve this goal, we propose to represent the
trigger function as a class-conditional generative model and to inject the
backdoor in a constrained optimization framework, where the trigger function
learns to generate an optimal trigger pattern to attack any target class at
will while simultaneously embedding this generative backdoor into the trained
model. Given the learned trigger-generation function, during inference, the
adversary can specify an arbitrary backdoor attack target class, and an
appropriate trigger causing the model to classify toward this target class is
created accordingly. We show empirically that the proposed framework achieves
high attack performance while preserving the clean-data performance in several
benchmark datasets, including MNIST, CIFAR10, GTSRB, and TinyImageNet. The
proposed Marksman backdoor attack can also easily bypass existing backdoor
defenses that were originally designed against backdoor attacks with a single
target class. Our work takes another significant step toward understanding the
extensive risks of backdoor attacks in practice.


http://arxiv.org/abs/2210.08929
DE-CROP: Data-efficient Certified Robustness for Pretrained Classifiers. (87%)
Gaurav Kumar Nayak; Ruchit Rawal; Anirban Chakraborty
  Certified defense using randomized smoothing is a popular technique to
provide robustness guarantees for deep neural networks against l2 adversarial
attacks. Existing works use this technique to provably secure a pretrained
non-robust model by training a custom denoiser network on entire training data.
However, access to the training set may be restricted to a handful of data
samples due to constraints such as high transmission cost and the proprietary
nature of the data. Thus, we formulate a novel problem of "how to certify the
robustness of pretrained models using only a few training samples". We observe
that training the custom denoiser directly using the existing techniques on
limited samples yields poor certification. To overcome this, our proposed
approach (DE-CROP) generates class-boundary and interpolated samples
corresponding to each training sample, ensuring high diversity in the feature
space of the pretrained classifier. We train the denoiser by maximizing the
similarity between the denoised output of the generated sample and the original
training sample in the classifier's logit space. We also perform distribution
level matching using domain discriminator and maximum mean discrepancy that
yields further benefit. In white box setup, we obtain significant improvements
over the baseline on multiple benchmark datasets and also report similar
performance under the challenging black box setup.


http://arxiv.org/abs/2210.08902
Beyond Model Interpretability: On the Faithfulness and Adversarial Robustness of Contrastive Textual Explanations. (78%)
Julia El Zini; Mariette Awad
  Contrastive explanation methods go beyond transparency and address the
contrastive aspect of explanations. Such explanations are emerging as an
attractive option to provide actionable change to scenarios adversely impacted
by classifiers' decisions. However, their extension to textual data is
under-explored and there is little investigation on their vulnerabilities and
limitations.
  This work motivates textual counterfactuals by laying the ground for a novel
evaluation scheme inspired by the faithfulness of explanations. Accordingly, we
extend the computation of three metrics, proximity,connectedness and stability,
to textual data and we benchmark two successful contrastive methods, POLYJUICE
and MiCE, on our suggested metrics. Experiments on sentiment analysis data show
that the connectedness of counterfactuals to their original counterparts is not
obvious in both models. More interestingly, the generated contrastive texts are
more attainable with POLYJUICE which highlights the significance of latent
representations in counterfactual search. Finally, we perform the first
semantic adversarial attack on textual recourse methods. The results
demonstrate the robustness of POLYJUICE and the role that latent input
representations play in robustness and reliability.


http://arxiv.org/abs/2210.09503
Towards Fair Classification against Poisoning Attacks. (76%)
Han Xu; Xiaorui Liu; Yuxuan Wan; Jiliang Tang
  Fair classification aims to stress the classification models to achieve the
equality (treatment or prediction quality) among different sensitive groups.
However, fair classification can be under the risk of poisoning attacks that
deliberately insert malicious training samples to manipulate the trained
classifiers' performance. In this work, we study the poisoning scenario where
the attacker can insert a small fraction of samples into training data, with
arbitrary sensitive attributes as well as other predictive features. We
demonstrate that the fairly trained classifiers can be greatly vulnerable to
such poisoning attacks, with much worse accuracy & fairness trade-off, even
when we apply some of the most effective defenses (originally proposed to
defend traditional classification tasks). As countermeasures to defend fair
classification tasks, we propose a general and theoretically guaranteed
framework which accommodates traditional defense methods to fair classification
against poisoning attacks. Through extensive experiments, the results validate
that the proposed defense framework obtains better robustness in terms of
accuracy and fairness than representative baseline methods.


http://arxiv.org/abs/2210.09421
Deepfake Text Detection: Limitations and Opportunities. (41%)
Jiameng Pu; Zain Sarwar; Sifat Muhammad Abdullah; Abdullah Rehman; Yoonjin Kim; Parantapa Bhattacharya; Mobin Javed; Bimal Viswanath
  Recent advances in generative models for language have enabled the creation
of convincing synthetic text or deepfake text. Prior work has demonstrated the
potential for misuse of deepfake text to mislead content consumers. Therefore,
deepfake text detection, the task of discriminating between human and
machine-generated text, is becoming increasingly critical. Several defenses
have been proposed for deepfake text detection. However, we lack a thorough
understanding of their real-world applicability. In this paper, we collect
deepfake text from 4 online services powered by Transformer-based tools to
evaluate the generalization ability of the defenses on content in the wild. We
develop several low-cost adversarial attacks, and investigate the robustness of
existing defenses against an adaptive attacker. We find that many defenses show
significant degradation in performance under our evaluation scenarios compared
to their original claimed performance. Our evaluation shows that tapping into
the semantic information in the text content is a promising approach for
improving the robustness and generalization performance of deepfake text
detection schemes.


http://arxiv.org/abs/2210.09482
You Can't See Me: Physical Removal Attacks on LiDAR-based Autonomous Vehicles Driving Frameworks. (15%)
Yulong Cao; S. Hrushikesh Bhupathiraju; Pirouz Naghavi; Takeshi Sugawara; Z. Morley Mao; Sara Rampazzi
  Autonomous Vehicles (AVs) increasingly use LiDAR-based object detection
systems to perceive other vehicles and pedestrians on the road. While existing
attacks on LiDAR-based autonomous driving architectures focus on lowering the
confidence score of AV object detection models to induce obstacle misdetection,
our research discovers how to leverage laser-based spoofing techniques to
selectively remove the LiDAR point cloud data of genuine obstacles at the
sensor level before being used as input to the AV perception. The ablation of
this critical LiDAR information causes autonomous driving obstacle detectors to
fail to identify and locate obstacles and, consequently, induces AVs to make
dangerous automatic driving decisions. In this paper, we present a method
invisible to the human eye that hides objects and deceives autonomous vehicles'
obstacle detectors by exploiting inherent automatic transformation and
filtering processes of LiDAR sensor data integrated with autonomous driving
frameworks. We call such attacks Physical Removal Attacks (PRA), and we
demonstrate their effectiveness against three popular AV obstacle detectors
(Apollo, Autoware, PointPillars), and we achieve 45{\deg} attack capability. We
evaluate the attack impact on three fusion models (Frustum-ConvNet, AVOD, and
Integrated-Semantic Level Fusion) and the consequences on the driving decision
using LGSVL, an industry-grade simulator. In our moving vehicle scenarios, we
achieve a 92.7% success rate removing 90% of a target obstacle's cloud points.
Finally, we demonstrate the attack's success against two popular defenses
against spoofing and object hiding attacks and discuss two enhanced defense
strategies to mitigate our attack.


http://arxiv.org/abs/2210.09545
Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models. (9%)
Zhiyuan Zhang; Lingjuan Lyu; Xingjun Ma; Chenguang Wang; Xu Sun
  Deep Neural Networks (DNNs) are known to be vulnerable to backdoor attacks.
In Natural Language Processing (NLP), DNNs are often backdoored during the
fine-tuning process of a large-scale Pre-trained Language Model (PLM) with
poisoned samples. Although the clean weights of PLMs are readily available,
existing methods have ignored this information in defending NLP models against
backdoor attacks. In this work, we take the first step to exploit the
pre-trained (unfine-tuned) weights to mitigate backdoors in fine-tuned language
models. Specifically, we leverage the clean pre-trained weights via two
complementary techniques: (1) a two-step Fine-mixing technique, which first
mixes the backdoored weights (fine-tuned on poisoned data) with the pre-trained
weights, then fine-tunes the mixed weights on a small subset of clean data; (2)
an Embedding Purification (E-PUR) technique, which mitigates potential
backdoors existing in the word embeddings. We compare Fine-mixing with typical
backdoor mitigation methods on three single-sentence sentiment classification
tasks and two sentence-pair classification tasks and show that it outperforms
the baselines by a considerable margin in all scenarios. We also show that our
E-PUR method can benefit existing mitigation methods. Our work establishes a
simple but strong baseline defense for secure fine-tuned NLP models against
backdoor attacks.


http://arxiv.org/abs/2210.09465
Understanding CNN Fragility When Learning With Imbalanced Data. (1%)
Damien Dablain; Kristen N. Jacobson; Colin Bellinger; Mark Roberts; Nitesh Chawla
  Convolutional neural networks (CNNs) have achieved impressive results on
imbalanced image data, but they still have difficulty generalizing to minority
classes and their decisions are difficult to interpret. These problems are
related because the method by which CNNs generalize to minority classes, which
requires improvement, is wrapped in a blackbox. To demystify CNN decisions on
imbalanced data, we focus on their latent features. Although CNNs embed the
pattern knowledge learned from a training set in model parameters, the effect
of this knowledge is contained in feature and classification embeddings (FE and
CE). These embeddings can be extracted from a trained model and their global,
class properties (e.g., frequency, magnitude and identity) can be analyzed. We
find that important information regarding the ability of a neural network to
generalize to minority classes resides in the class top-K CE and FE. We show
that a CNN learns a limited number of class top-K CE per category, and that
their number and magnitudes vary based on whether the same class is balanced or
imbalanced. This calls into question whether a CNN has learned intrinsic class
features, or merely frequently occurring ones that happen to exist in the
sampled class distribution. We also hypothesize that latent class diversity is
as important as the number of class examples, which has important implications
for re-sampling and cost-sensitive methods. These methods generally focus on
rebalancing model weights, class numbers and margins; instead of diversifying
class latent features through augmentation. We also demonstrate that a CNN has
difficulty generalizing to test data if the magnitude of its top-K latent
features do not match the training set. We use three popular image datasets and
two cost-sensitive algorithms commonly employed in imbalanced learning for our
experiments.


http://arxiv.org/abs/2210.08472
Object-Attentional Untargeted Adversarial Attack. (99%)
Chao Zhou; Yuan-Gen Wang; Guopu Zhu
  Deep neural networks are facing severe threats from adversarial attacks. Most
existing black-box attacks fool target model by generating either global
perturbations or local patches. However, both global perturbations and local
patches easily cause annoying visual artifacts in adversarial example. Compared
with some smooth regions of an image, the object region generally has more
edges and a more complex texture. Thus small perturbations on it will be more
imperceptible. On the other hand, the object region is undoubtfully the
decisive part of an image to classification tasks. Motivated by these two
facts, we propose an object-attentional adversarial attack method for
untargeted attack. Specifically, we first generate an object region by
intersecting the object detection region from YOLOv4 with the salient object
detection (SOD) region from HVPNet. Furthermore, we design an activation
strategy to avoid the reaction caused by the incomplete SOD. Then, we perform
an adversarial attack only on the detected object region by leveraging Simple
Black-box Adversarial Attack (SimBA). To verify the proposed method, we create
a unique dataset by extracting all the images containing the object defined by
COCO from ImageNet-1K, named COCO-Reduced-ImageNet in this paper. Experimental
results on ImageNet-1K and COCO-Reduced-ImageNet show that under various system
settings, our method yields the adversarial example with better perceptual
quality meanwhile saving the query budget up to 24.16\% compared to the
state-of-the-art approaches including SimBA.


http://arxiv.org/abs/2210.08579
Nowhere to Hide: A Lightweight Unsupervised Detector against Adversarial Examples. (99%)
Hui Liu; Bo Zhao; Kehuan Zhang; Peng Liu
  Although deep neural networks (DNNs) have shown impressive performance on
many perceptual tasks, they are vulnerable to adversarial examples that are
generated by adding slight but maliciously crafted perturbations to benign
images. Adversarial detection is an important technique for identifying
adversarial examples before they are entered into target DNNs. Previous studies
to detect adversarial examples either targeted specific attacks or required
expensive computation. How design a lightweight unsupervised detector is still
a challenging problem. In this paper, we propose an AutoEncoder-based
Adversarial Examples (AEAE) detector, that can guard DNN models by detecting
adversarial examples with low computation in an unsupervised manner. The AEAE
includes only a shallow autoencoder but plays two roles. First, a well-trained
autoencoder has learned the manifold of benign examples. This autoencoder can
produce a large reconstruction error for adversarial images with large
perturbations, so we can detect significantly perturbed adversarial examples
based on the reconstruction error. Second, the autoencoder can filter out the
small noise and change the DNN's prediction on adversarial examples with small
perturbations. It helps to detect slightly perturbed adversarial examples based
on the prediction distance. To cover these two cases, we utilize the
reconstruction error and prediction distance from benign images to construct a
two-tuple feature set and train an adversarial detector using the isolation
forest algorithm. We show empirically that the AEAE is unsupervised and
inexpensive against the most state-of-the-art attacks. Through the detection in
these two cases, there is nowhere to hide adversarial examples.


http://arxiv.org/abs/2210.08701
ODG-Q: Robust Quantization via Online Domain Generalization. (83%)
Chaofan Tao; Ngai Wong
  Quantizing neural networks to low-bitwidth is important for model deployment
on resource-limited edge hardware. Although a quantized network has a smaller
model size and memory footprint, it is fragile to adversarial attacks. However,
few methods study the robustness and training efficiency of quantized networks.
To this end, we propose a new method by recasting robust quantization as an
online domain generalization problem, termed ODG-Q, which generates diverse
adversarial data at a low cost during training. ODG-Q consistently outperforms
existing works against various adversarial attacks. For example, on CIFAR-10
dataset, ODG-Q achieves 49.2% average improvements under five common white-box
attacks and 21.7% average improvements under five common black-box attacks,
with a training cost similar to that of natural training (viz. without
adversaries). To our best knowledge, this work is the first work that trains
both quantized and binary neural networks on ImageNet that consistently improve
robustness under different attacks. We also provide a theoretical insight of
ODG-Q that accounts for the bound of model risk on attacked data.


http://arxiv.org/abs/2210.11235
Interpretable Machine Learning for Detection and Classification of Ransomware Families Based on API Calls. (1%)
Rawshan Ara Mowri; Madhuri Siddula; Kaushik Roy
  Ransomware has appeared as one of the major global threats in recent days The
alarming increasing rate of ransomware attacks and new ransomware variants
intrigue the researchers to constantly examine the distinguishing traits of
ransomware and refine their detection strategies Application Programming
Interface API is a way for one program to collaborate with another API calls
are the medium by which they communicate Ransomware uses this strategy to
interact with the OS and makes a significantly higher number of calls in
different sequences to ask for taking action This research work utilizes the
frequencies of different API calls to detect and classify ransomware families
First a WebCrawler is developed to automate collecting the Windows Portable
Executable PE files of 15 different ransomware families By extracting different
frequencies of 68 API calls we develop our dataset in the first phase of the
two phase feature engineering process After selecting the most significant
features in the second phase of the feature engineering process we deploy six
Supervised Machine Learning models Naive Bayes Logistic Regression Random
Forest Stochastic Gradient Descent K Nearest Neighbor and Support Vector
Machine Then the performances of all the classifiers are compared to select the
best model The results reveal that Logistic Regression can efficiently classify
ransomware into their corresponding families securing 9915 accuracy Finally
instead of relying on the Black box characteristic of the Machine Learning
models we present the interpretability of our best performing model using SHAP
values to ascertain the transparency and trustworthiness of the models
prediction


http://arxiv.org/abs/2210.08388
RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging. (2%)
Ajay Jaiswal; Kumar Ashutosh; Justin F Rousseau; Yifan Peng; Zhangyang Wang; Ying Ding
  AI-powered Medical Imaging has recently achieved enormous attention due to
its ability to provide fast-paced healthcare diagnoses. However, it usually
suffers from a lack of high-quality datasets due to high annotation cost,
inter-observer variability, human annotator error, and errors in
computer-generated labels. Deep learning models trained on noisy labelled
datasets are sensitive to the noise type and lead to less generalization on the
unseen samples. To address this challenge, we propose a Robust Stochastic
Knowledge Distillation (RoS-KD) framework which mimics the notion of learning a
topic from multiple sources to ensure deterrence in learning noisy information.
More specifically, RoS-KD learns a smooth, well-informed, and robust student
manifold by distilling knowledge from multiple teachers trained on overlapping
subsets of training data. Our extensive experiments on popular medical imaging
classification tasks (cardiopulmonary disease and lesion classification) using
real-world datasets, show the performance benefit of RoS-KD, its ability to
distill knowledge from many popular large networks (ResNet-50, DenseNet-121,
MobileNet-V2) in a comparatively small network, and its robustness to
adversarial attacks (PGD, FSGM). More specifically, RoS-KD achieves >2% and >4%
improvement on F1-score for lesion classification and cardiopulmonary disease
classification tasks, respectively, when the underlying student is ResNet-18
against recent competitive knowledge distillation baseline. Additionally, on
cardiopulmonary disease classification task, RoS-KD outperforms most of the
SOTA baselines by ~1% gain in AUC score.


http://arxiv.org/abs/2210.07540
When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture. (87%)
Yichuan Mo; Dongxian Wu; Yifei Wang; Yiwen Guo; Yisen Wang
  Vision Transformers (ViTs) have recently achieved competitive performance in
broad vision tasks. Unfortunately, on popular threat models, naturally trained
ViTs are shown to provide no more adversarial robustness than convolutional
neural networks (CNNs). Adversarial training is still required for ViTs to
defend against such adversarial attacks. In this paper, we provide the first
and comprehensive study on the adversarial training recipe of ViTs via
extensive evaluation of various training techniques across benchmark datasets.
We find that pre-training and SGD optimizer are necessary for ViTs' adversarial
training. Further considering ViT as a new type of model architecture, we
investigate its adversarial robustness from the perspective of its unique
architectural components. We find, when randomly masking gradients from some
attention blocks or masking perturbations on some patches during adversarial
training, the adversarial robustness of ViTs can be remarkably improved, which
may potentially open up a line of work to explore the architectural information
inside the newly designed models like ViTs. Our code is available at
https://github.com/mo666666/When-Adversarial-Training-Meets-Vision-Transformers.


http://arxiv.org/abs/2210.08159
Dynamics-aware Adversarial Attack of Adaptive Neural Networks. (86%)
An Tao; Yueqi Duan; Yingqi Wang; Jiwen Lu; Jie Zhou
  In this paper, we investigate the dynamics-aware adversarial attack problem
of adaptive neural networks. Most existing adversarial attack algorithms are
designed under a basic assumption -- the network architecture is fixed
throughout the attack process. However, this assumption does not hold for many
recently proposed adaptive neural networks, which adaptively deactivate
unnecessary execution units based on inputs to improve computational
efficiency. It results in a serious issue of lagged gradient, making the
learned attack at the current step ineffective due to the architecture change
afterward. To address this issue, we propose a Leaded Gradient Method (LGM) and
show the significant effects of the lagged gradient. More specifically, we
reformulate the gradients to be aware of the potential dynamic changes of
network architectures, so that the learned attack better "leads" the next step
than the dynamics-unaware methods when network architecture changes
dynamically. Extensive experiments on representative types of adaptive neural
networks for both 2D images and 3D point clouds show that our LGM achieves
impressive adversarial attack performance compared with the dynamic-unaware
attack methods.


http://arxiv.org/abs/2210.08178
Is Face Recognition Safe from Realizable Attacks? (84%)
Sanjay Saha; Terence Sim
  Face recognition is a popular form of biometric authentication and due to its
widespread use, attacks have become more common as well. Recent studies show
that Face Recognition Systems are vulnerable to attacks and can lead to
erroneous identification of faces. Interestingly, most of these attacks are
white-box, or they are manipulating facial images in ways that are not
physically realizable. In this paper, we propose an attack scheme where the
attacker can generate realistic synthesized face images with subtle
perturbations and physically realize that onto his face to attack black-box
face recognition systems. Comprehensive experiments and analyses show that
subtle perturbations realized on attackers face can create successful attacks
on state-of-the-art face recognition systems in black-box settings. Our study
exposes the underlying vulnerability posed by the Face Recognition Systems
against realizable black-box attacks.


http://arxiv.org/abs/2210.07907
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks. (76%)
Sishuo Chen; Wenkai Yang; Zhiyuan Zhang; Xiaohan Bi; Xu Sun
  Natural language processing (NLP) models are known to be vulnerable to
backdoor attacks, which poses a newly arisen threat to NLP models. Prior online
backdoor defense methods for NLP models only focus on the anomalies at either
the input or output level, still suffering from fragility to adaptive attacks
and high computational cost. In this work, we take the first step to
investigate the unconcealment of textual poisoned samples at the
intermediate-feature level and propose a feature-based efficient online defense
method. Through extensive experiments on existing attacking methods, we find
that the poisoned samples are far away from clean samples in the intermediate
feature space of a poisoned NLP model. Motivated by this observation, we devise
a distance-based anomaly score (DAN) to distinguish poisoned samples from clean
samples at the feature level. Experiments on sentiment analysis and offense
detection tasks demonstrate the superiority of DAN, as it substantially
surpasses existing online defense methods in terms of defending performance and
enjoys lower inference costs. Moreover, we show that DAN is also resistant to
adaptive attacks based on feature-level regularization. Our code is available
at https://github.com/lancopku/DAN.


http://arxiv.org/abs/2210.07714
Close the Gate: Detecting Backdoored Models in Federated Learning based on Client-Side Deep Layer Output Analysis. (67%)
Phillip Technical University Darmstadt Rieger; Torsten University of Würzburg Krauß; Markus Technical University Darmstadt Miettinen; Alexandra University of Würzburg Dmitrienko; Ahmad-Reza Technical University Darmstadt Sadeghi
  Federated Learning (FL) is a scheme for collaboratively training Deep Neural
Networks (DNNs) with multiple data sources from different clients. Instead of
sharing the data, each client trains the model locally, resulting in improved
privacy. However, recently so-called targeted poisoning attacks have been
proposed that allow individual clients to inject a backdoor into the trained
model. Existing defenses against these backdoor attacks either rely on
techniques like Differential Privacy to mitigate the backdoor, or analyze the
weights of the individual models and apply outlier detection methods that
restricts these defenses to certain data distributions. However, adding noise
to the models' parameters or excluding benign outliers might also reduce the
accuracy of the collaboratively trained model. Additionally, allowing the
server to inspect the clients' models creates a privacy risk due to existing
knowledge extraction methods.
  We propose \textit{CrowdGuard}, a model filtering defense, that mitigates
backdoor attacks by leveraging the clients' data to analyze the individual
models before the aggregation. To prevent data leaks, the server sends the
individual models to secure enclaves, running in client-located Trusted
Execution Environments. To effectively distinguish benign and poisoned models,
even if the data of different clients are not independently and identically
distributed (non-IID), we introduce a novel metric called \textit{HLBIM} to
analyze the outputs of the DNN's hidden layers. We show that the applied
significance-based detection algorithm combined can effectively detect poisoned
models, even in non-IID scenarios.


http://arxiv.org/abs/2210.06871
Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognition. (99%)
Shuai Jia; Bangjie Yin; Taiping Yao; Shouhong Ding; Chunhua Shen; Xiaokang Yang; Chao Ma
  Deep learning models have shown their vulnerability when dealing with
adversarial attacks. Existing attacks almost perform on low-level instances,
such as pixels and super-pixels, and rarely exploit semantic clues. For face
recognition attacks, existing methods typically generate the l_p-norm
perturbations on pixels, however, resulting in low attack transferability and
high vulnerability to denoising defense models. In this work, instead of
performing perturbations on the low-level pixels, we propose to generate
attacks through perturbing on the high-level semantics to improve attack
transferability. Specifically, a unified flexible framework, Adversarial
Attributes (Adv-Attribute), is designed to generate inconspicuous and
transferable attacks on face recognition, which crafts the adversarial noise
and adds it into different attributes based on the guidance of the difference
in face recognition features from the target. Moreover, the importance-aware
attribute selection and the multi-objective optimization strategy are
introduced to further ensure the balance of stealthiness and attacking
strength. Extensive experiments on the FFHQ and CelebA-HQ datasets show that
the proposed Adv-Attribute method achieves the state-of-the-art attacking
success rates while maintaining better visual effects against recent attack
methods.


http://arxiv.org/abs/2210.06888
AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient. (99%)
Farzad Nikfam; Alberto Marchisio; Maurizio Martina; Muhammad Shafique
  Adversarial training is exploited to develop a robust Deep Neural Network
(DNN) model against the malicious altered data. These attacks may have
catastrophic effects on DNN models but are indistinguishable for a human being.
For example, an external attack can modify an image adding noises invisible for
a human eye, but a DNN model misclassified the image. A key objective for
developing robust DNN models is to use a learning algorithm that is fast but
can also give model that is robust against different types of adversarial
attacks. Especially for adversarial training, enormously long training times
are needed for obtaining high accuracy under many different types of
adversarial samples generated using different adversarial attack techniques.
  This paper aims at accelerating the adversarial training to enable fast
development of robust DNN models against adversarial attacks. The general
method for improving the training performance is the hyperparameters
fine-tuning, where the learning rate is one of the most crucial
hyperparameters. By modifying its shape (the value over time) and value during
the training, we can obtain a model robust to adversarial attacks faster than
standard training.
  First, we conduct experiments on two different datasets (CIFAR10, CIFAR100),
exploring various techniques. Then, this analysis is leveraged to develop a
novel fast training methodology, AccelAT, which automatically adjusts the
learning rate for different epochs based on the accuracy gradient. The
experiments show comparable results with the related works, and in several
experiments, the adversarial training of DNNs using our AccelAT framework is
conducted up to 2 times faster than the existing techniques. Thus, our findings
boost the speed of adversarial training in an era in which security and
performance are fundamental optimization objectives in DNN-based applications.


http://arxiv.org/abs/2210.07346
Demystifying Self-supervised Trojan Attacks. (87%)
Changjiang Li; Ren Pang; Zhaohan Xi; Tianyu Du; Shouling Ji; Yuan Yao; Ting Wang
  As an emerging machine learning paradigm, self-supervised learning (SSL) is
able to learn high-quality representations for complex data without data
labels. Prior work shows that, besides obviating the reliance on labeling, SSL
also benefits adversarial robustness by making it more challenging for the
adversary to manipulate model prediction. However, whether this robustness
benefit generalizes to other types of attacks remains an open question.
  We explore this question in the context of trojan attacks by showing that SSL
is comparably vulnerable as supervised learning to trojan attacks.
Specifically, we design and evaluate CTRL, an extremely simple self-supervised
trojan attack. By polluting a tiny fraction of training data (less than 1%)
with indistinguishable poisoning samples, CTRL causes any trigger-embedded
input to be misclassified to the adversary's desired class with a high
probability (over 99%) at inference. More importantly, through the lens of
CTRL, we study the mechanisms underlying self-supervised trojan attacks. With
both empirical and analytical evidence, we reveal that the representation
invariance property of SSL, which benefits adversarial robustness, may also be
the very reason making SSL highly vulnerable to trojan attacks. We further
discuss the fundamental challenges to defending against self-supervised trojan
attacks, pointing to promising directions for future research.


http://arxiv.org/abs/2210.06807
Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors. (81%)
Qixun Wang; Yifei Wang; Hong Zhu; Yisen Wang
  Deep models often fail to generalize well in test domains when the data
distribution differs from that in the training domain. Among numerous
approaches to address this Out-of-Distribution (OOD) generalization problem,
there has been a growing surge of interest in exploiting Adversarial Training
(AT) to improve OOD performance. Recent works have revealed that the robust
model obtained by conducting sample-wise AT also retains transferability to
biased test domains. In this paper, we empirically show that sample-wise AT has
limited improvement on OOD performance. Specifically, we find that AT can only
maintain performance at smaller scales of perturbation while Universal AT (UAT)
is more robust to larger-scale perturbations. This provides us with clues that
adversarial perturbations with universal (low dimensional) structures can
enhance the robustness against large data distribution shifts that are common
in OOD scenarios. Inspired by this, we propose two AT variants with low-rank
structures to train OOD-robust models. Extensive experiments on DomainBed
benchmark show that our proposed approaches outperform Empirical Risk
Minimization (ERM) and sample-wise AT. Our code is available at
https://github.com/NOVAglow646/NIPS22-MAT-and-LDAT-for-OOD.


http://arxiv.org/abs/2210.07394
Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation. (13%)
Zhouxing Shi; Yihan Wang; Huan Zhang; Zico Kolter; Cho-Jui Hsieh
  Lipschitz constants are connected to many properties of neural networks, such
as robustness, fairness, and generalization. Existing methods for computing
Lipschitz constants either produce relatively loose upper bounds or are limited
to small networks. In this paper, we develop an efficient framework for
computing the $\ell_\infty$ local Lipschitz constant of a neural network by
tightly upper bounding the norm of Clarke Jacobian via linear bound
propagation. We formulate the computation of local Lipschitz constants with a
linear bound propagation process on a high-order backward graph induced by the
chain rule of Clarke Jacobian. To enable linear bound propagation, we derive
tight linear relaxations for specific nonlinearities in Clarke Jacobian. This
formulate unifies existing ad-hoc approaches such as RecurJac, which can be
seen as a special case of ours with weaker relaxations. The bound propagation
framework also allows us to easily borrow the popular Branch-and-Bound (BaB)
approach from neural network verification to further tighten Lipschitz
constants. Experiments show that on tiny models, our method produces comparable
bounds compared to exact methods that cannot scale to slightly larger models;
on larger models, our method efficiently produces tighter results than existing
relaxed or naive methods, and our method scales to much larger practical models
that previous works could not handle. We also demonstrate an application on
provable monotonicity analysis. Code is available at
https://github.com/shizhouxing/Local-Lipschitz-Constants.


http://arxiv.org/abs/2210.06789
Large-Scale Open-Set Classification Protocols for ImageNet. (2%)
Jesus Andres Palechor Anacona; Annesha Bhoumik; Manuel Günther
  Open-Set Classification (OSC) intends to adapt closed-set classification
models to real-world scenarios, where the classifier must correctly label
samples of known classes while rejecting previously unseen unknown samples.
Only recently, research started to investigate on algorithms that are able to
handle these unknown samples correctly. Some of these approaches address OSC by
including into the training set negative samples that a classifier learns to
reject, expecting that these data increase the robustness of the classifier on
unknown classes. Most of these approaches are evaluated on small-scale and
low-resolution image datasets like MNIST, SVHN or CIFAR, which makes it
difficult to assess their applicability to the real world, and to compare them
among each other. We propose three open-set protocols that provide rich
datasets of natural images with different levels of similarity between known
and unknown classes. The protocols consist of subsets of ImageNet classes
selected to provide training and testing data closer to real-world scenarios.
Additionally, we propose a new validation metric that can be employed to assess
whether the training of deep learning models addresses both the classification
of known samples and the rejection of unknown samples. We use the protocols to
compare the performance of two baseline open-set algorithms to the standard
SoftMax baseline and find that the algorithms work well on negative samples
that have been seen during training, and partially on out-of-distribution
detection tasks, but drop performance in the presence of samples from
previously unseen unknown classes.


http://arxiv.org/abs/2210.06792
SoK: How Not to Architect Your Next-Generation TEE Malware? (1%)
Kubilay Ahmet Küçük; Steve Moyle; Andrew Martin; Alexandru Mereacre; Nicholas Allott
  Besides Intel's SGX technology, there are long-running discussions on how
trusted computing technologies can be used to cloak malware. Past research
showed example methods of malicious activities utilising Flicker, Trusted
Platform Module, and recently integrating with enclaves. We observe two
ambiguous methodologies of malware development being associated with SGX, and
it is crucial to systematise their details. One methodology is to use the core
SGX ecosystem to cloak malware; potentially affecting a large number of
systems. The second methodology is to create a custom enclave not adhering to
base assumptions of SGX, creating a demonstration code of malware behaviour
with these incorrect assumptions; remaining local without any impact. We
examine what malware aims to do in real-world scenarios and state-of-art
techniques in malware evasion. We present multiple limitations of maintaining
the SGX-assisted malware and evading it from anti-malware mechanisms. The
limitations make SGX enclaves a poor choice for achieving a successful malware
campaign. We systematise twelve misconceptions (myths) outlining how an
overfit-malware using SGX weakens malware's existing abilities. We find the
differences by comparing SGX assistance for malware with non-SGX malware (i.e.,
malware in the wild in our paper). We conclude that the use of hardware
enclaves does not increase the preexisting attack surface, enables no new
infection vector, and does not contribute any new methods to the stealthiness
of malware.


http://arxiv.org/abs/2210.06771
Feature Reconstruction Attacks and Countermeasures of DNN training in Vertical Federated Learning. (1%)
Peng Ye; Zhifeng Jiang; Wei Wang; Bo Li; Baochun Li
  Federated learning (FL) has increasingly been deployed, in its vertical form,
among organizations to facilitate secure collaborative training over siloed
data. In vertical FL (VFL), participants hold disjoint features of the same set
of sample instances. Among them, only one has labels. This participant, known
as the active party, initiates the training and interacts with the other
participants, known as the passive parties. Despite the increasing adoption of
VFL, it remains largely unknown if and how the active party can extract feature
data from the passive party, especially when training deep neural network (DNN)
models.
  This paper makes the first attempt to study the feature security problem of
DNN training in VFL. We consider a DNN model partitioned between active and
passive parties, where the latter only holds a subset of the input layer and
exhibits some categorical features of binary values. Using a reduction from the
Exact Cover problem, we prove that reconstructing those binary features is
NP-hard. Through analysis, we demonstrate that, unless the feature dimension is
exceedingly large, it remains feasible, both theoretically and practically, to
launch a reconstruction attack with an efficient search-based algorithm that
prevails over current feature protection techniques. To address this problem,
we develop a novel feature protection scheme against the reconstruction attack
that effectively misleads the search to some pre-specified random values. With
an extensive set of experiments, we show that our protection scheme sustains
the feature reconstruction attack in various VFL applications at no expense of
accuracy loss.


http://arxiv.org/abs/2210.07441
Characterizing the Influence of Graph Elements. (1%)
Zizhang Chen; Peizhao Li; Hongfu Liu; Pengyu Hong
  Influence function, a method from robust statistics, measures the changes of
model parameters or some functions about model parameters concerning the
removal or modification of training instances. It is an efficient and useful
post-hoc method for studying the interpretability of machine learning models
without the need for expensive model re-training. Recently, graph convolution
networks (GCNs), which operate on graph data, have attracted a great deal of
attention. However, there is no preceding research on the influence functions
of GCNs to shed light on the effects of removing training nodes/edges from an
input graph. Since the nodes/edges in a graph are interdependent in GCNs, it is
challenging to derive influence functions for GCNs. To fill this gap, we
started with the simple graph convolution (SGC) model that operates on an
attributed graph and formulated an influence function to approximate the
changes in model parameters when a node or an edge is removed from an
attributed graph. Moreover, we theoretically analyzed the error bound of the
estimated influence of removing an edge. We experimentally validated the
accuracy and effectiveness of our influence estimation function. In addition,
we showed that the influence function of an SGC model could be used to estimate
the impact of removing training nodes/edges on the test performance of the SGC
without re-training the model. Finally, we demonstrated how to use influence
functions to guide the adversarial attacks on GCNs effectively.


http://arxiv.org/abs/2210.06670
A Game Theoretical vulnerability analysis of Adversarial Attack. (99%)
Khondker Fariha Hossain; Alireza Tavakkoli; Shamik Sengupta
  In recent times deep learning has been widely used for automating various
security tasks in Cyber Domains. However, adversaries manipulate data in many
situations and diminish the deployed deep learning model's accuracy. One
notable example is fooling CAPTCHA data to access the CAPTCHA-based Classifier
leading to the critical system being vulnerable to cybersecurity attacks. To
alleviate this, we propose a computational framework of game theory to analyze
the CAPTCHA-based Classifier's vulnerability, strategy, and outcomes by forming
a simultaneous two-player game. We apply the Fast Gradient Symbol Method (FGSM)
and One Pixel Attack on CAPTCHA Data to imitate real-life scenarios of possible
cyber-attack. Subsequently, to interpret this scenario from a Game theoretical
perspective, we represent the interaction in the Stackelberg Game in Kuhn tree
to study players' possible behaviors and actions by applying our Classifier's
actual predicted values. Thus, we interpret potential attacks in deep learning
applications while representing viable defense strategies in the game theory
prospect.


http://arxiv.org/abs/2210.06284
Visual Prompting for Adversarial Robustness. (99%)
Aochuan Chen; Peter Lorenz; Yuguang Yao; Pin-Yu Chen; Sijia Liu
  In this work, we leverage visual prompting (VP) to improve adversarial
robustness of a fixed, pre-trained model at testing time. Compared to
conventional adversarial defenses, VP allows us to design universal (i.e.,
data-agnostic) input prompting templates, which have plug-and-play capabilities
at testing time to achieve desired model performance without introducing much
computation overhead. Although VP has been successfully applied to improving
model generalization, it remains elusive whether and how it can be used to
defend against adversarial attacks. We investigate this problem and show that
the vanilla VP approach is not effective in adversarial defense since a
universal input prompt lacks the capacity for robust learning against
sample-specific adversarial perturbations. To circumvent it, we propose a new
VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate
class-wise visual prompts so as to not only leverage the strengths of ensemble
prompts but also optimize their interrelations to improve model robustness. Our
experiments show that C-AVP outperforms the conventional VP method, with 2.1X
standard accuracy gain and 2X robust accuracy gain. Compared to classical
test-time defenses, C-AVP also yields a 42X inference time speedup.


http://arxiv.org/abs/2210.05968
Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation. (99%)
Zeyu Qin; Yanbo Fan; Yi Liu; Li Shen; Yong Zhang; Jue Wang; Baoyuan Wu
  Deep neural networks (DNNs) have been shown to be vulnerable to adversarial
examples, which can produce erroneous predictions by injecting imperceptible
perturbations. In this work, we study the transferability of adversarial
examples, which is significant due to its threat to real-world applications
where model architecture or parameters are usually unknown. Many existing works
reveal that the adversarial examples are likely to overfit the surrogate model
that they are generated from, limiting its transfer attack performance against
different target models. To mitigate the overfitting of the surrogate model, we
propose a novel attack method, dubbed reverse adversarial perturbation (RAP).
Specifically, instead of minimizing the loss of a single adversarial point, we
advocate seeking adversarial example located at a region with unified low loss
value, by injecting the worst-case perturbation (the reverse adversarial
perturbation) for each step of the optimization procedure. The adversarial
attack with RAP is formulated as a min-max bi-level optimization problem. By
integrating RAP into the iterative process for attacks, our method can find
more stable adversarial examples which are less sensitive to the changes of
decision boundary, mitigating the overfitting of the surrogate model.
Comprehensive experimental comparisons demonstrate that RAP can significantly
boost adversarial transferability. Furthermore, RAP can be naturally combined
with many existing black-box attack techniques, to further boost the
transferability. When attacking a real-world image recognition system, Google
Cloud Vision API, we obtain 22% performance improvement of targeted attacks
over the compared method. Our codes are available at
https://github.com/SCLBD/Transfer_attack_RAP.


http://arxiv.org/abs/2210.05938
Robust Models are less Over-Confident. (96%)
Julia Grabinski; Paul Gavrikov; Janis Keuper; Margret Keuper
  Despite the success of convolutional neural networks (CNNs) in many academic
benchmarks for computer vision tasks, their application in the real-world is
still facing fundamental challenges. One of these open problems is the inherent
lack of robustness, unveiled by the striking effectiveness of adversarial
attacks. Current attack methods are able to manipulate the network's prediction
by adding specific but small amounts of noise to the input. In turn,
adversarial training (AT) aims to achieve robustness against such attacks and
ideally a better model generalization ability by including adversarial samples
in the trainingset. However, an in-depth analysis of the resulting robust
models beyond adversarial robustness is still pending. In this paper, we
empirically analyze a variety of adversarially trained models that achieve high
robust accuracies when facing state-of-the-art attacks and we show that AT has
an interesting side-effect: it leads to models that are significantly less
overconfident with their decisions, even on clean data than non-robust models.
Further, our analysis of robust models shows that not only AT but also the
model's building blocks (like activation functions and pooling) have a strong
influence on the models' prediction confidences. Data & Project website:
https://github.com/GeJulia/robustness_confidences_evaluation


http://arxiv.org/abs/2210.06077
Double Bubble, Toil and Trouble: Enhancing Certified Robustness through Transitivity. (86%)
Andrew C. Cullen; Paul Montague; Shijie Liu; Sarah M. Erfani; Benjamin I. P. Rubinstein
  In response to subtle adversarial examples flipping classifications of neural
network models, recent research has promoted certified robustness as a
solution. There, invariance of predictions to all norm-bounded attacks is
achieved through randomised smoothing of network inputs. Today's
state-of-the-art certifications make optimal use of the class output scores at
the input instance under test: no better radius of certification (under the
$L_2$ norm) is possible given only these score. However, it is an open question
as to whether such lower bounds can be improved using local information around
the instance under test. In this work, we demonstrate how today's "optimal"
certificates can be improved by exploiting both the transitivity of
certifications, and the geometry of the input space, giving rise to what we
term Geometrically-Informed Certified Robustness. By considering the smallest
distance to points on the boundary of a set of certifications this approach
improves certifications for more than $80\%$ of Tiny-Imagenet instances,
yielding an on average $5 \%$ increase in the associated certification. When
incorporating training time processes that enhance the certified radius, our
technique shows even more promising results, with a uniform $4$ percentage
point increase in the achieved certified radius.


http://arxiv.org/abs/2210.05927
Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning. (82%)
Yongyuan Liang; Yanchao Sun; Ruijie Zheng; Furong Huang
  Recent studies reveal that a well-trained deep reinforcement learning (RL)
policy can be particularly vulnerable to adversarial perturbations on input
observations. Therefore, it is crucial to train RL agents that are robust
against any attacks with a bounded budget. Existing robust training methods in
deep RL either treat correlated steps separately, ignoring the robustness of
long-term rewards, or train the agents and RL-based attacker together, doubling
the computational burden and sample complexity of the training process. In this
work, we propose a strong and efficient robust training framework for RL, named
Worst-case-aware Robust RL (WocaR-RL) that directly estimates and optimizes the
worst-case reward of a policy under bounded l_p attacks without requiring extra
samples for learning an attacker. Experiments on multiple environments show
that WocaR-RL achieves state-of-the-art performance under various strong
attacks, and obtains significantly higher training efficiency than prior
state-of-the-art robust training methods. The code of this work is available at
https://github.com/umd-huang-lab/WocaR-RL.


http://arxiv.org/abs/2210.06704
COLLIDER: A Robust Training Framework for Backdoor Data. (81%)
Hadi M. Dolatabadi; Sarah Erfani; Christopher Leckie
  Deep neural network (DNN) classifiers are vulnerable to backdoor attacks. An
adversary poisons some of the training data in such attacks by installing a
trigger. The goal is to make the trained DNN output the attacker's desired
class whenever the trigger is activated while performing as usual for clean
data. Various approaches have recently been proposed to detect malicious
backdoored DNNs. However, a robust, end-to-end training approach, like
adversarial training, is yet to be discovered for backdoor poisoned data. In
this paper, we take the first step toward such methods by developing a robust
training framework, COLLIDER, that selects the most prominent samples by
exploiting the underlying geometric structures of the data. Specifically, we
effectively filter out candidate poisoned data at each training epoch by
solving a geometrical coreset selection objective. We first argue how clean
data samples exhibit (1) gradients similar to the clean majority of data and
(2) low local intrinsic dimensionality (LID). Based on these criteria, we
define a novel coreset selection objective to find such samples, which are used
for training a DNN. We show the effectiveness of the proposed method for robust
training of DNNs on various poisoned datasets, reducing the backdoor success
rate significantly.


http://arxiv.org/abs/2210.06428
Trap and Replace: Defending Backdoor Attacks by Trapping Them into an Easy-to-Replace Subnetwork. (76%)
Haotao Wang; Junyuan Hong; Aston Zhang; Jiayu Zhou; Zhangyang Wang
  Deep neural networks (DNNs) are vulnerable to backdoor attacks. Previous
works have shown it extremely challenging to unlearn the undesired backdoor
behavior from the network, since the entire network can be affected by the
backdoor samples. In this paper, we propose a brand-new backdoor defense
strategy, which makes it much easier to remove the harmful influence of
backdoor samples from the model. Our defense strategy, \emph{Trap and Replace},
consists of two stages. In the first stage, we bait and trap the backdoors in a
small and easy-to-replace subnetwork. Specifically, we add an auxiliary image
reconstruction head on top of the stem network shared with a light-weighted
classification head. The intuition is that the auxiliary image reconstruction
task encourages the stem network to keep sufficient low-level visual features
that are hard to learn but semantically correct, instead of overfitting to the
easy-to-learn but semantically incorrect backdoor correlations. As a result,
when trained on backdoored datasets, the backdoors are easily baited towards
the unprotected classification head, since it is much more vulnerable than the
shared stem, leaving the stem network hardly poisoned. In the second stage, we
replace the poisoned light-weighted classification head with an untainted one,
by re-training it from scratch only on a small holdout dataset with clean
samples, while fixing the stem network. As a result, both the stem and the
classification head in the final network are hardly affected by backdoor
training samples. We evaluate our method against ten different backdoor
attacks. Our method outperforms previous state-of-the-art methods by up to
$20.57\%$, $9.80\%$, and $13.72\%$ attack success rate and on-average $3.14\%$,
$1.80\%$, and $1.21\%$ clean classification accuracy on CIFAR10, GTSRB, and
ImageNet-12, respectively. Code is available online.


http://arxiv.org/abs/2210.05929
Few-shot Backdoor Attacks via Neural Tangent Kernels. (62%)
Jonathan Hayase; Sewoong Oh
  In a backdoor attack, an attacker injects corrupted examples into the
training set. The goal of the attacker is to cause the final trained model to
predict the attacker's desired target label when a predefined trigger is added
to test inputs. Central to these attacks is the trade-off between the success
rate of the attack and the number of corrupted training examples injected. We
pose this attack as a novel bilevel optimization problem: construct strong
poison examples that maximize the attack success rate of the trained model. We
use neural tangent kernels to approximate the training dynamics of the model
being attacked and automatically learn strong poison examples. We experiment on
subclasses of CIFAR-10 and ImageNet with WideResNet-34 and ConvNeXt
architectures on periodic and patch trigger attacks and show that NTBA-designed
poisoned examples achieve, for example, an attack success rate of 90% with ten
times smaller number of poison examples injected compared to the baseline. We
provided an interpretation of the NTBA-designed attacks using the analysis of
kernel linear regression. We further demonstrate a vulnerability in
overparametrized deep neural networks, which is revealed by the shape of the
neural tangent kernel.


http://arxiv.org/abs/2210.06516
How to Sift Out a Clean Data Subset in the Presence of Data Poisoning? (9%)
Yi Zeng; Minzhou Pan; Himanshu Jahagirdar; Ming Jin; Lingjuan Lyu; Ruoxi Jia
  Given the volume of data needed to train modern machine learning models,
external suppliers are increasingly used. However, incorporating external data
poses data poisoning risks, wherein attackers manipulate their data to degrade
model utility or integrity. Most poisoning defenses presume access to a set of
clean data (or base set). While this assumption has been taken for granted,
given the fast-growing research on stealthy poisoning attacks, a question
arises: can defenders really identify a clean subset within a contaminated
dataset to support defenses?
  This paper starts by examining the impact of poisoned samples on defenses
when they are mistakenly mixed into the base set. We analyze five defenses and
find that their performance deteriorates dramatically with less than 1%
poisoned points in the base set. These findings suggest that sifting out a base
set with high precision is key to these defenses' performance. Motivated by
these observations, we study how precise existing automated tools and human
inspection are at identifying clean data in the presence of data poisoning.
Unfortunately, neither effort achieves the precision needed. Worse yet, many of
the outcomes are worse than random selection.
  In addition to uncovering the challenge, we propose a practical
countermeasure, Meta-Sift. Our method is based on the insight that existing
attacks' poisoned samples shifts from clean data distributions. Hence, training
on the clean portion of a dataset and testing on the corrupted portion will
result in high prediction loss. Leveraging the insight, we formulate a bilevel
optimization to identify clean data and further introduce a suite of techniques
to improve efficiency and precision. Our evaluation shows that Meta-Sift can
sift a clean base set with 100% precision under a wide range of poisoning
attacks. The selected base set is large enough to give rise to successful
defenses.


http://arxiv.org/abs/2210.06509
Understanding Impacts of Task Similarity on Backdoor Attack and Detection. (2%)
Di Tang; Rui Zhu; XiaoFeng Wang; Haixu Tang; Yi Chen
  With extensive studies on backdoor attack and detection, still fundamental
questions are left unanswered regarding the limits in the adversary's
capability to attack and the defender's capability to detect. We believe that
answers to these questions can be found through an in-depth understanding of
the relations between the primary task that a benign model is supposed to
accomplish and the backdoor task that a backdoored model actually performs. For
this purpose, we leverage similarity metrics in multi-task learning to formally
define the backdoor distance (similarity) between the primary task and the
backdoor task, and analyze existing stealthy backdoor attacks, revealing that
most of them fail to effectively reduce the backdoor distance and even for
those that do, still much room is left to further improve their stealthiness.
So we further design a new method, called TSA attack, to automatically generate
a backdoor model under a given distance constraint, and demonstrate that our
new attack indeed outperforms existing attacks, making a step closer to
understanding the attacker's limits. Most importantly, we provide both
theoretic results and experimental evidence on various datasets for the
positive correlation between the backdoor distance and backdoor detectability,
demonstrating that indeed our task similarity analysis help us better
understand backdoor risks and has the potential to identify more effective
mitigations.


http://arxiv.org/abs/2210.05577
What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness? (99%)
Nikolaos Tsilivis; Julia Kempe
  The adversarial vulnerability of neural nets, and subsequent techniques to
create robust models have attracted significant attention; yet we still lack a
full understanding of this phenomenon. Here, we study adversarial examples of
trained neural networks through analytical tools afforded by recent theory
advances connecting neural networks and kernel methods, namely the Neural
Tangent Kernel (NTK), following a growing body of work that leverages the NTK
approximation to successfully analyze important deep learning phenomena and
design algorithms for new applications. We show how NTKs allow to generate
adversarial examples in a ``training-free'' fashion, and demonstrate that they
transfer to fool their finite-width neural net counterparts in the ``lazy''
regime. We leverage this connection to provide an alternative view on robust
and non-robust features, which have been suggested to underlie the adversarial
brittleness of neural nets. Specifically, we define and study features induced
by the eigendecomposition of the kernel to better understand the role of robust
and non-robust features, the reliance on both for standard classification and
the robustness-accuracy trade-off. We find that such features are surprisingly
consistent across architectures, and that robust features tend to correspond to
the largest eigenvalues of the model, and thus are learned early during
training. Our framework allows us to identify and visualize non-robust yet
useful features. Finally, we shed light on the robustness mechanism underlying
adversarial training of neural nets used in practice: quantifying the evolution
of the associated empirical NTK, we demonstrate that its dynamics falls much
earlier into the ``lazy'' regime and manifests a much stronger form of the well
known bias to prioritize learning features within the top eigenspaces of the
kernel, compared to standard training.


http://arxiv.org/abs/2210.05373
Stable and Efficient Adversarial Training through Local Linearization. (91%)
Zhuorong Li; Daiwei Yu
  There has been a recent surge in single-step adversarial training as it shows
robustness and efficiency. However, a phenomenon referred to as ``catastrophic
overfitting" has been observed, which is prevalent in single-step defenses and
may frustrate attempts to use FGSM adversarial training. To address this issue,
we propose a novel method, Stable and Efficient Adversarial Training (SEAT),
which mitigates catastrophic overfitting by harnessing on local properties that
distinguish a robust model from that of a catastrophic overfitted model. The
proposed SEAT has strong theoretical justifications, in that minimizing the
SEAT loss can be shown to favour smooth empirical risk, thereby leading to
robustness. Experimental results demonstrate that the proposed method
successfully mitigates catastrophic overfitting, yielding superior performance
amongst efficient defenses. Our single-step method can reach 51% robust
accuracy for CIFAR-10 with $l_\infty$ perturbations of radius $8/255$ under a
strong PGD-50 attack, matching the performance of a 10-step iterative
adversarial training at merely 3% computational cost.


http://arxiv.org/abs/2210.05276
RoHNAS: A Neural Architecture Search Framework with Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks. (86%)
Alberto Marchisio; Vojtech Mrazek; Andrea Massa; Beatrice Bussolino; Maurizio Martina; Muhammad Shafique
  Neural Architecture Search (NAS) algorithms aim at finding efficient Deep
Neural Network (DNN) architectures for a given application under given system
constraints. DNNs are computationally-complex as well as vulnerable to
adversarial attacks. In order to address multiple design objectives, we propose
RoHNAS, a novel NAS framework that jointly optimizes for adversarial-robustness
and hardware-efficiency of DNNs executed on specialized hardware accelerators.
Besides the traditional convolutional DNNs, RoHNAS additionally accounts for
complex types of DNNs such as Capsule Networks. For reducing the exploration
time, RoHNAS analyzes and selects appropriate values of adversarial
perturbation for each dataset to employ in the NAS flow. Extensive evaluations
on multi - Graphics Processing Unit (GPU) - High Performance Computing (HPC)
nodes provide a set of Pareto-optimal solutions, leveraging the tradeoff
between the above-discussed design objectives. For example, a Pareto-optimal
DNN for the CIFAR-10 dataset exhibits 86.07% accuracy, while having an energy
of 38.63 mJ, a memory footprint of 11.85 MiB, and a latency of 4.47 ms.


http://arxiv.org/abs/2210.06589
Adversarial Attack Against Image-Based Localization Neural Networks. (78%)
Meir Brand; Itay Naeh; Daniel Teitelman
  In this paper, we present a proof of concept for adversarially attacking the
image-based localization module of an autonomous vehicle. This attack aims to
cause the vehicle to perform a wrong navigational decisions and prevent it from
reaching a desired predefined destination in a simulated urban environment. A
database of rendered images allowed us to train a deep neural network that
performs a localization task and implement, develop and assess the adversarial
pattern. Our tests show that using this adversarial attack we can prevent the
vehicle from turning at a given intersection. This is done by manipulating the
vehicle's navigational module to falsely estimate its current position and thus
fail to initialize the turning procedure until the vehicle misses the last
opportunity to perform a safe turn in a given intersection.


http://arxiv.org/abs/2210.11264
Detecting Backdoors in Deep Text Classifiers. (76%)
You Guo; Jun Wang; Trevor Cohn
  Deep neural networks are vulnerable to adversarial attacks, such as backdoor
attacks in which a malicious adversary compromises a model during training such
that specific behaviour can be triggered at test time by attaching a specific
word or phrase to an input. This paper considers the problem of diagnosing
whether a model has been compromised and if so, identifying the backdoor
trigger. We present the first robust defence mechanism that generalizes to
several backdoor attacks against text classification models, without prior
knowledge of the attack type, nor does our method require access to any
(potentially compromised) training resources. Our experiments show that our
technique is highly accurate at defending against state-of-the-art backdoor
attacks, including data poisoning and weight poisoning, across a range of text
classification tasks and model architectures. Our code will be made publicly
available upon acceptance.


http://arxiv.org/abs/2210.05667
Human Body Measurement Estimation with Adversarial Augmentation. (33%)
Nataniel Ruiz; Miriam Bellver; Timo Bolkart; Ambuj Arora; Ming C. Lin; Javier Romero; Raja Bala
  We present a Body Measurement network (BMnet) for estimating 3D
anthropomorphic measurements of the human body shape from silhouette images.
Training of BMnet is performed on data from real human subjects, and augmented
with a novel adversarial body simulator (ABS) that finds and synthesizes
challenging body shapes. ABS is based on the skinned multiperson linear (SMPL)
body model, and aims to maximize BMnet measurement prediction error with
respect to latent SMPL shape parameters. ABS is fully differentiable with
respect to these parameters, and trained end-to-end via backpropagation with
BMnet in the loop. Experiments show that ABS effectively discovers adversarial
examples, such as bodies with extreme body mass indices (BMI), consistent with
the rarity of extreme-BMI bodies in BMnet's training set. Thus ABS is able to
reveal gaps in training data and potential failures in predicting
under-represented body shapes. Results show that training BMnet with ABS
improves measurement prediction accuracy on real bodies by up to 10%, when
compared to no augmentation or random body shape sampling. Furthermore, our
method significantly outperforms SOTA measurement estimation methods by as much
as 3x. Finally, we release BodyM, the first challenging, large-scale dataset of
photo silhouettes and body measurements of real human subjects, to further
promote research in this area. Project website:
https://adversarialbodysim.github.io


http://arxiv.org/abs/2210.05742
Curved Representation Space of Vision Transformers. (10%)
Juyeop Kim; Junha Park; Songkuk Kim; Jong-Seok Lee
  Neural networks with self-attention (a.k.a. Transformers) like ViT and Swin
have emerged as a better alternative to traditional convolutional neural
networks (CNNs) for computer vision tasks. However, our understanding of how
the new architecture works is still limited. In this paper, we focus on the
phenomenon that Transformers show higher robustness against corruptions than
CNNs, while not being overconfident (in fact, we find Transformers are actually
underconfident). This is contrary to the intuition that robustness increases
with confidence. We resolve this contradiction by investigating how the output
of the penultimate layer moves in the representation space as the input data
moves within a small area. In particular, we show the following. (1) While CNNs
exhibit fairly linear relationship between the input and output movements,
Transformers show nonlinear relationship for some data. For those data, the
output of Transformers moves in a curved trajectory as the input moves
linearly. (2) When a data is located in a curved region, it is hard to move it
out of the decision region since the output moves along a curved trajectory
instead of a straight line to the decision boundary, resulting in high
robustness of Transformers. (3) If a data is slightly modified to jump out of
the curved region, the movements afterwards become linear and the output goes
to the decision boundary directly. Thus, Transformers can be attacked easily
after a small random jump and the perturbation in the final attacked data
remains imperceptible, i.e., there does exist a decision boundary near the
data. This also explains the underconfident prediction of Transformers. (4) The
curved regions in the representation space start to form at an early training
stage and grow throughout the training course. Some data are trapped in the
regions, obstructing Transformers from reducing the training loss.


http://arxiv.org/abs/2210.05279
Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity. (1%)
Vazelhes William de; Hualin Zhang; Huimin Wu; Xiao-Tong Yuan; Bin Gu
  $\ell_0$ constrained optimization is prevalent in machine learning,
particularly for high-dimensional problems, because it is a fundamental
approach to achieve sparse learning. Hard-thresholding gradient descent is a
dominant technique to solve this problem. However, first-order gradients of the
objective function may be either unavailable or expensive to calculate in a lot
of real-world problems, where zeroth-order (ZO) gradients could be a good
surrogate. Unfortunately, whether ZO gradients can work with the
hard-thresholding operator is still an unsolved problem. To solve this puzzle,
in this paper, we focus on the $\ell_0$ constrained black-box stochastic
optimization problems, and propose a new stochastic zeroth-order gradient
hard-thresholding (SZOHT) algorithm with a general ZO gradient estimator
powered by a novel random support sampling. We provide the convergence analysis
of SZOHT under standard assumptions. Importantly, we reveal a conflict between
the deviation of ZO estimators and the expansivity of the hard-thresholding
operator, and provide a theoretical minimal value of the number of random
directions in ZO gradients. In addition, we find that the query complexity of
SZOHT is independent or weakly dependent on the dimensionality under different
settings. Finally, we illustrate the utility of our method on a portfolio
optimization problem as well as black-box adversarial attacks.


http://arxiv.org/abs/2210.05177
Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach. (1%)
Peng Mi; Li Shen; Tianhe Ren; Yiyi Zhou; Xiaoshuai Sun; Rongrong Ji; Dacheng Tao
  Deep neural networks often suffer from poor generalization caused by complex
and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware
Minimization (SAM), which smooths the loss landscape via minimizing the
maximized change of training loss when adding a perturbation to the weight.
However, we find the indiscriminate perturbation of SAM on all parameters is
suboptimal, which also results in excessive computation, i.e., double the
overhead of common optimizers like Stochastic Gradient Descent (SGD). In this
paper, we propose an efficient and effective training scheme coined as Sparse
SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the
sparse mask, we provide two solutions which are based onFisher information and
dynamic sparse training, respectively. In addition, we theoretically prove that
SSAM can converge at the same rate as SAM, i.e., $O(\log T/\sqrt{T})$. Sparse
SAM not only has the potential for training acceleration but also smooths the
loss landscape effectively. Extensive experimental results on CIFAR10,
CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM,
and the performance is preserved or even better with a perturbation of merely
50% sparsity. Code is availiable at
https://github.com/Mi-Peng/Sparse-Sharpness-Aware-Minimization.


http://arxiv.org/abs/2210.05118
Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization. (92%)
Ziquan Liu; Antoni B. Chan
  The adversarial vulnerability of deep neural networks (DNNs) has been
actively investigated in the past several years. This paper investigates the
scale-variant property of cross-entropy loss, which is the most commonly used
loss function in classification tasks, and its impact on the effective margin
and adversarial robustness of deep neural networks. Since the loss function is
not invariant to logit scaling, increasing the effective weight norm will make
the loss approach zero and its gradient vanish while the effective margin is
not adequately maximized. On typical DNNs, we demonstrate that, if not properly
regularized, the standard training does not learn large effective margins and
leads to adversarial vulnerability. To maximize the effective margins and learn
a robust DNN, we propose to regularize the effective weight norm during
training. Our empirical study on feedforward DNNs demonstrates that the
proposed effective margin regularization (EMR) learns large effective margins
and boosts the adversarial robustness in both standard and adversarial
training. On large-scale models, we show that EMR outperforms basic adversarial
training, TRADES and two regularization baselines with substantial improvement.
Moreover, when combined with several strong adversarial defense methods (MART
and MAIL), our EMR further boosts the robustness.


http://arxiv.org/abs/2210.04886
Revisiting adapters with adversarial training. (88%)
Sylvestre-Alvise Rebuffi; Francesco Croce; Sven Gowal
  While adversarial training is generally used as a defense mechanism, recent
works show that it can also act as a regularizer. By co-training a neural
network on clean and adversarial inputs, it is possible to improve
classification accuracy on the clean, non-adversarial inputs. We demonstrate
that, contrary to previous findings, it is not necessary to separate batch
statistics when co-training on clean and adversarial inputs, and that it is
sufficient to use adapters with few domain-specific parameters for each type of
input. We establish that using the classification token of a Vision Transformer
(ViT) as an adapter is enough to match the classification performance of dual
normalization layers, while using significantly less additional parameters.
First, we improve upon the top-1 accuracy of a non-adversarially trained
ViT-B16 model by +1.12% on ImageNet (reaching 83.76% top-1 accuracy). Second,
and more importantly, we show that training with adapters enables model soups
through linear combinations of the clean and adversarial tokens. These model
soups, which we call adversarial model soups, allow us to trade-off between
clean and robust accuracy without sacrificing efficiency. Finally, we show that
we can easily adapt the resulting models in the face of distribution shifts.
Our ViT-B16 obtains top-1 accuracies on ImageNet variants that are on average
+4.00% better than those obtained with Masked Autoencoders.


http://arxiv.org/abs/2210.04591
Universal Adversarial Perturbations: Efficiency on a small image dataset. (81%)
Waris ENSEIRB-MATMECA, UB Radji
  Although neural networks perform very well on the image classification task,
they are still vulnerable to adversarial perturbations that can fool a neural
network without visibly changing an input image. A paper has shown the
existence of Universal Adversarial Perturbations which when added to any image
will fool the neural network with a very high probability. In this paper we
will try to reproduce the experience of the Universal Adversarial Perturbations
paper, but on a smaller neural network architecture and training set, in order
to be able to study the efficiency of the computed perturbation.


http://arxiv.org/abs/2210.04311
Pruning Adversarially Robust Neural Networks without Adversarial Examples. (99%)
Tong Jian; Zifeng Wang; Yanzhi Wang; Jennifer Dy; Stratis Ioannidis
  Adversarial pruning compresses models while preserving robustness. Current
methods require access to adversarial examples during pruning. This
significantly hampers training efficiency. Moreover, as new adversarial attacks
and training methods develop at a rapid rate, adversarial pruning methods need
to be modified accordingly to keep up. In this work, we propose a novel
framework to prune a previously trained robust neural network while maintaining
adversarial robustness, without further generating adversarial examples. We
leverage concurrent self-distillation and pruning to preserve knowledge in the
original model as well as regularizing the pruned model via the Hilbert-Schmidt
Information Bottleneck. We comprehensively evaluate our proposed framework and
show its superior performance in terms of both adversarial robustness and
efficiency when pruning architectures trained on the MNIST, CIFAR-10, and
CIFAR-100 datasets against five state-of-the-art attacks. Code is available at
https://github.com/neu-spiral/PwoA/.


http://arxiv.org/abs/2210.04213
Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective. (99%)
Yao Zhu; Yuefeng Chen; Xiaodan Li; Kejiang Chen; Yuan He; Xiang Tian; Bolun Zheng; Yaowu Chen; Qingming Huang
  Transferable adversarial attacks against Deep neural networks (DNNs) have
received broad attention in recent years. An adversarial example can be crafted
by a surrogate model and then attack the unknown target model successfully,
which brings a severe threat to DNNs. The exact underlying reasons for the
transferability are still not completely understood. Previous work mostly
explores the causes from the model perspective, e.g., decision boundary, model
architecture, and model capacity. adversarial attacks against Deep neural
networks (DNNs) have received broad attention in recent years. An adversarial
example can be crafted by a surrogate model and then attack the unknown target
model successfully, which brings a severe threat to DNNs. The exact underlying
reasons for the transferability are still not completely understood. Previous
work mostly explores the causes from the model perspective. Here, we
investigate the transferability from the data distribution perspective and
hypothesize that pushing the image away from its original distribution can
enhance the adversarial transferability. To be specific, moving the image out
of its original distribution makes different models hardly classify the image
correctly, which benefits the untargeted attack, and dragging the image into
the target distribution misleads the models to classify the image as the target
class, which benefits the targeted attack. Towards this end, we propose a novel
method that crafts adversarial examples by manipulating the distribution of the
image. We conduct comprehensive transferable attacks against multiple DNNs to
demonstrate the effectiveness of the proposed method. Our method can
significantly improve the transferability of the crafted attacks and achieves
state-of-the-art performance in both untargeted and targeted scenarios,
surpassing the previous best method by up to 40$\%$ in some cases.


http://arxiv.org/abs/2210.04195
Online Training Through Time for Spiking Neural Networks. (1%)
Mingqing Xiao; Qingyan Meng; Zongpeng Zhang; Di He; Zhouchen Lin
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient
models. Recent progress in training methods has enabled successful deep SNNs on
large-scale tasks with low latency. Particularly, backpropagation through time
(BPTT) with surrogate gradients (SG) is popularly used to achieve high
performance in a very small number of time steps. However, it is at the cost of
large memory consumption for training, lack of theoretical clarity for
optimization, and inconsistency with the online property of biological learning
and rules on neuromorphic hardware. Other works connect spike representations
of SNNs with equivalent artificial neural network formulation and train SNNs by
gradients from equivalent mappings to ensure descent directions. But they fail
to achieve low latency and are also not online. In this work, we propose online
training through time (OTTT) for SNNs, which is derived from BPTT to enable
forward-in-time learning by tracking presynaptic activities and leveraging
instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove
that gradients of OTTT can provide a similar descent direction for optimization
as gradients based on spike representations under both feedforward and
recurrent conditions. OTTT only requires constant training memory costs
agnostic to time steps, avoiding the significant memory costs of BPTT for GPU
training. Furthermore, the update rule of OTTT is in the form of three-factor
Hebbian learning, which could pave a path for online on-chip learning. With
OTTT, it is the first time that two mainstream supervised SNN training methods,
BPTT with SG and spike representation-based training, are connected, and
meanwhile in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100,
ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on
large-scale static and neuromorphic datasets in small time steps.


http://arxiv.org/abs/2210.04087
Symmetry Subgroup Defense Against Adversarial Attacks. (99%)
Blerta Lindqvist
  Adversarial attacks and defenses disregard the lack of invariance of
convolutional neural networks (CNNs), that is, the inability of CNNs to
classify samples and their symmetric transformations the same. The lack of
invariance of CNNs with respect to symmetry transformations is detrimental when
classifying transformed original samples but not necessarily detrimental when
classifying transformed adversarial samples. For original images, the lack of
invariance means that symmetrically transformed original samples are classified
differently from their correct labels. However, for adversarial images, the
lack of invariance means that symmetrically transformed adversarial images are
classified differently from their incorrect adversarial labels. Might the CNN
lack of invariance revert symmetrically transformed adversarial samples to the
correct classification? This paper answers this question affirmatively for a
threat model that ranges from zero-knowledge adversaries to perfect-knowledge
adversaries. We base our defense against perfect-knowledge adversaries on
devising a Klein four symmetry subgroup that incorporates an additional
artificial symmetry of pixel intensity inversion. The closure property of the
subgroup not only provides a framework for the accuracy evaluation but also
confines the transformations that an adaptive, perfect-knowledge adversary can
apply. We find that by using only symmetry defense, no adversarial samples, and
by changing nothing in the model architecture and parameters, we can defend
against white-box PGD adversarial attacks, surpassing the PGD adversarial
training defense by up to ~50% even against a perfect-knowledge adversary for
ImageNet. The proposed defense also maintains and surpasses the classification
accuracy for non-adversarial samples.


http://arxiv.org/abs/2210.04052
FedDef: Defense Against Gradient Leakage in Federated Learning-based Network Intrusion Detection Systems. (99%)
Jiahui Chen; Yi Zhao; Qi Li; Xuewei Feng; Ke Xu
  Deep learning (DL) methods have been widely applied to anomaly-based network
intrusion detection system (NIDS) to detect malicious traffic. To expand the
usage scenarios of DL-based methods, the federated learning (FL) framework
allows multiple users to train a global model on the basis of respecting
individual data privacy. However, it has not yet been systematically evaluated
how robust FL-based NIDSs are against existing privacy attacks under existing
defenses. To address this issue, we propose two privacy evaluation metrics
designed for FL-based NIDSs, including (1) privacy score that evaluates the
similarity between the original and recovered traffic features using
reconstruction attacks, and (2) evasion rate against NIDSs using Generative
Adversarial Network-based adversarial attack with the reconstructed benign
traffic. We conduct experiments to show that existing defenses provide little
protection that the corresponding adversarial traffic can even evade the SOTA
NIDS Kitsune. To defend against such attacks and build a more robust FL-based
NIDS, we further propose FedDef, a novel optimization-based input perturbation
defense strategy with theoretical guarantee. It achieves both high utility by
minimizing the gradient distance and strong privacy protection by maximizing
the input distance. We experimentally evaluate four existing defenses on four
datasets and show that our defense outperforms all the baselines in terms of
privacy protection with up to 7 times higher privacy score, while maintaining
model accuracy loss within 3% under optimal parameter combination.


http://arxiv.org/abs/2210.04076
Robustness of Unsupervised Representation Learning without Labels. (54%)
Aleksandar Petrov; Marta Kwiatkowska
  Unsupervised representation learning leverages large unlabeled datasets and
is competitive with supervised learning. But non-robust encoders may affect
downstream task robustness. Recently, robust representation encoders have
become of interest. Still, all prior work evaluates robustness using a
downstream classification task. Instead, we propose a family of unsupervised
robustness measures, which are model- and task-agnostic and label-free. We
benchmark state-of-the-art representation encoders and show that none dominates
the rest. We offer unsupervised extensions to the FGSM and PGD attacks. When
used in adversarial training, they improve most unsupervised robustness
measures, including certified robustness. We validate our results against a
linear probe and show that, for MOCOv2, adversarial training results in 3 times
higher certified accuracy, a 2-fold decrease in impersonation attack success
rate and considerable improvements in certified robustness.


http://arxiv.org/abs/2210.03429
Adversarially Robust Prototypical Few-shot Segmentation with Neural-ODEs. (99%)
Prashant Pandey; Aleti Vardhan; Mustafa Chasmai; Tanuj Sur; Brejesh Lall
  Few-shot Learning (FSL) methods are being adopted in settings where data is
not abundantly available. This is especially seen in medical domains where the
annotations are expensive to obtain. Deep Neural Networks have been shown to be
vulnerable to adversarial attacks. This is even more severe in the case of FSL
due to the lack of a large number of training examples. In this paper, we
provide a framework to make few-shot segmentation models adversarially robust
in the medical domain where such attacks can severely impact the decisions made
by clinicians who use them. We propose a novel robust few-shot segmentation
framework, Prototypical Neural Ordinary Differential Equation (PNODE), that
provides defense against gradient-based adversarial attacks. We show that our
framework is more robust compared to traditional adversarial defense mechanisms
such as adversarial training. Adversarial training involves increased training
time and shows robustness to limited types of attacks depending on the type of
adversarial examples seen during training. Our proposed framework generalises
well to common adversarial attacks like FGSM, PGD and SMIA while having the
model parameters comparable to the existing few-shot segmentation models. We
show the effectiveness of our proposed approach on three publicly available
multi-organ segmentation datasets in both in-domain and cross-domain settings
by attacking the support and query sets without the need for ad-hoc adversarial
training.


http://arxiv.org/abs/2210.03372
Pre-trained Adversarial Perturbations. (99%)
Yuanhao Ban; Yinpeng Dong
  Self-supervised pre-training has drawn increasing attention in recent years
due to its superior performance on numerous downstream tasks after fine-tuning.
However, it is well-known that deep learning models lack the robustness to
adversarial examples, which can also invoke security issues to pre-trained
models, despite being less explored. In this paper, we delve into the
robustness of pre-trained models by introducing Pre-trained Adversarial
Perturbations (PAPs), which are universal perturbations crafted for the
pre-trained models to maintain the effectiveness when attacking fine-tuned ones
without any knowledge of the downstream tasks. To this end, we propose a
Low-Level Layer Lifting Attack (L4A) method to generate effective PAPs by
lifting the neuron activations of low-level layers of the pre-trained models.
Equipped with an enhanced noise augmentation strategy, L4A is effective at
generating more transferable PAPs against fine-tuned models. Extensive
experiments on typical pre-trained vision models and ten downstream tasks
demonstrate that our method improves the attack success rate by a large margin
compared with state-of-the-art methods.


http://arxiv.org/abs/2210.03895
ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints. (93%)
Yinpeng Dong; Shouwei Ruan; Hang Su; Caixin Kang; Xingxing Wei; Jun Zhu
  Recent studies have demonstrated that visual recognition models lack
robustness to distribution shift. However, current work mainly considers model
robustness to 2D image transformations, leaving viewpoint changes in the 3D
world less explored. In general, viewpoint changes are prevalent in various
real-world applications (e.g., autonomous driving), making it imperative to
evaluate viewpoint robustness. In this paper, we propose a novel method called
ViewFool to find adversarial viewpoints that mislead visual recognition models.
By encoding real-world objects as neural radiance fields (NeRF), ViewFool
characterizes a distribution of diverse adversarial viewpoints under an
entropic regularizer, which helps to handle the fluctuations of the real camera
pose and mitigate the reality gap between the real objects and their neural
representations. Experiments validate that the common image classifiers are
extremely vulnerable to the generated adversarial viewpoints, which also
exhibit high cross-model transferability. Based on ViewFool, we introduce
ImageNet-V, a new out-of-distribution dataset for benchmarking viewpoint
robustness of image classifiers. Evaluation results on 40 classifiers with
diverse architectures, objective functions, and data augmentations reveal a
significant drop in model performance when tested on ImageNet-V, which provides
a possibility to leverage ViewFool as an effective data augmentation strategy
to improve viewpoint robustness.


http://arxiv.org/abs/2210.03349
Game-Theoretic Understanding of Misclassification. (47%)
Kosuke Sumiyasu; Kazuhiko Kawamoto; Hiroshi Kera
  This paper analyzes various types of image misclassification from a
game-theoretic view. Particularly, we consider the misclassification of clean,
adversarial, and corrupted images and characterize it through the distribution
of multi-order interactions. We discover that the distribution of multi-order
interactions varies across the types of misclassification. For example,
misclassified adversarial images have a higher strength of high-order
interactions than correctly classified clean images, which indicates that
adversarial perturbations create spurious features that arise from complex
cooperation between pixels. By contrast, misclassified corrupted images have a
lower strength of low-order interactions than correctly classified clean
images, which indicates that corruptions break the local cooperation between
pixels. We also provide the first analysis of Vision Transformers using
interactions. We found that Vision Transformers show a different tendency in
the distribution of interactions from that in CNNs, and this implies that they
exploit the features that CNNs do not use for the prediction. Our study
demonstrates that the recent game-theoretic analysis of deep learning models
can be broadened to analyze various malfunctions of deep learning models
including Vision Transformers by using the distribution, order, and sign of
interactions.


http://arxiv.org/abs/2210.03543
A2: Efficient Automated Attacker for Boosting Adversarial Training. (41%)
Zhuoer Xu; Guanghui Zhu; Changhua Meng; Shiwen Cui; Zhenzhe Ying; Weiqiang Wang; Ming GU; Yihua Huang
  Based on the significant improvement of model robustness by AT (Adversarial
Training), various variants have been proposed to further boost the
performance. Well-recognized methods have focused on different components of AT
(e.g., designing loss functions and leveraging additional unlabeled data). It
is generally accepted that stronger perturbations yield more robust models.
However, how to generate stronger perturbations efficiently is still missed. In
this paper, we propose an efficient automated attacker called A2 to boost AT by
generating the optimal perturbations on-the-fly during training. A2 is a
parameterized automated attacker to search in the attacker space for the best
attacker against the defense model and examples. Extensive experiments across
different datasets demonstrate that A2 generates stronger perturbations with
low extra cost and reliably improves the robustness of various AT methods
against different attacks.


http://arxiv.org/abs/2210.03696
NMTSloth: Understanding and Testing Efficiency Degradation of Neural Machine Translation Systems. (13%)
Simin Chen; Cong Liu; Mirazul Haque; Zihe Song; Wei Yang
  Neural Machine Translation (NMT) systems have received much recent attention
due to their human-level accuracy. While existing works mostly focus on either
improving accuracy or testing accuracy robustness, the computation efficiency
of NMT systems, which is of paramount importance due to often vast translation
demands and real-time requirements, has surprisingly received little attention.
In this paper, we make the first attempt to understand and test potential
computation efficiency robustness in state-of-the-art NMT systems. By analyzing
the working mechanism and implementation of 1455 public-accessible NMT systems,
we observe a fundamental property in NMT systems that could be manipulated in
an adversarial manner to reduce computation efficiency significantly. Our key
motivation is to generate test inputs that could sufficiently delay the
generation of EOS such that NMT systems would have to go through enough
iterations to satisfy the pre-configured threshold. We present NMTSloth, which
develops a gradient-guided technique that searches for a minimal and
unnoticeable perturbation at character-level, token-level, and structure-level,
which sufficiently delays the appearance of EOS and forces these inputs to
reach the naturally-unreachable threshold. To demonstrate the effectiveness of
NMTSloth, we conduct a systematic evaluation on three public-available NMT
systems: Google T5, AllenAI WMT14, and Helsinki-NLP translators. Experimental
results show that NMTSloth can increase NMT systems' response latency and
energy consumption by 85% to 3153% and 86% to 3052%, respectively, by
perturbing just one character or token in the input sentence. Our case study
shows that inputs generated by NMTSloth significantly affect the battery power
in real-world mobile devices (i.e., drain more than 30 times battery power than
normal inputs).


http://arxiv.org/abs/2210.03688
A Wolf in Sheep's Clothing: Spreading Deadly Pathogens Under the Disguise of Popular Music. (2%)
Anomadarshi Barua; Yonatan Gizachew Achamyeleh; Mohammad Abdullah Al Faruque
  A Negative Pressure Room (NPR) is an essential requirement by the Bio-Safety
Levels (BSLs) in biolabs or infectious-control hospitals to prevent deadly
pathogens from being leaked from the facility. An NPR maintains a negative
pressure inside with respect to the outside reference space so that microbes
are contained inside of an NPR. Nowadays, differential pressure sensors (DPSs)
are utilized by the Building Management Systems (BMSs) to control and monitor
the negative pressure in an NPR. This paper demonstrates a non-invasive and
stealthy attack on NPRs by spoofing a DPS at its resonant frequency. Our
contributions are: (1) We show that DPSs used in NPRs typically have resonant
frequencies in the audible range. (2) We use this finding to design malicious
music to create resonance in DPSs, resulting in an overshooting in the DPS's
normal pressure readings. (3) We show how the resonance in DPSs can fool the
BMSs so that the NPR turns its negative pressure to a positive one, causing a
potential \textit{leak} of deadly microbes from NPRs. We do experiments on 8
DPSs from 5 different manufacturers to evaluate their resonant frequencies
considering the sampling tube length and find resonance in 6 DPSs. We can
achieve a 2.5 Pa change in negative pressure from a $\sim$7 cm distance when a
sampling tube is not present and from a $\sim$2.5 cm distance for a 1 m
sampling tube length. We also introduce an interval-time variation approach for
an adversarial control over the negative pressure and show that the
\textit{forged} pressure can be varied within 12 - 33 Pa. Our attack is also
capable of attacking multiple NPRs simultaneously. Moreover, we demonstrate our
attack at a real-world NPR located in an anonymous bioresearch facility, which
is FDA approved and follows CDC guidelines. We also provide countermeasures to
prevent the attack.


http://arxiv.org/abs/2210.03879
Improving Fine-Grain Segmentation via Interpretable Modifications: A Case Study in Fossil Segmentation. (1%)
Indu Panigrahi; Ryan Manzuk; Adam Maloof; Ruth Fong
  Most interpretability research focuses on datasets containing thousands of
images of commonplace objects. However, many high-impact datasets, such as
those in medicine and the geosciences, contain fine-grain objects that require
domain-expert knowledge to recognize and are time-consuming to collect and
annotate. As a result, these datasets contain few annotated images, and current
machine vision models cannot train intensively on them. Thus, adapting
interpretability techniques to maximize the amount of information that models
can learn from small, fine-grain datasets is an important endeavor.
  Using a Mask R-CNN to segment ancient reef fossils in rock sample images, we
present a general paradigm for identifying and mitigating model weaknesses.
Specifically, we apply image perturbations to expose the Mask R-CNN's inability
to distinguish between different classes of fossils and its inconsistency in
segmenting fossils with different textures. To address these shortcomings, we
extend an existing model-editing method for correcting systematic mistakes in
image classification to image segmentation and introduce a novel application of
the technique: encouraging a greater separation between positive and negative
pixels for a given class. Through extensive experiments, we find that editing
the model by perturbing all pixels for a given class in one image is most
effective (compared to using multiple images and/or fewer pixels). Our paradigm
may also generalize to other segmentation models trained on small, fine-grain
datasets.


http://arxiv.org/abs/2210.04688
Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets. (1%)
Chen Gong; Zhou Yang; Yunpeng Bai; Junda He; Jieke Shi; Arunesh Sinha; Bowen Xu; Xinwen Hou; Guoliang Fan; David Lo
  A growing body of research works has focused on the Offline Reinforcement
Learning (RL) paradigm. Data providers share large pre-collected datasets on
which others can train high-quality agents without interacting with the
environments. Such an offline RL paradigm has demonstrated effectiveness in
many critical tasks, including robot control, autonomous driving, etc. A
well-trained agent can be regarded as a software system. However, less
attention is paid to investigating the security threats to the offline RL
system. In this paper, we focus on a critical security threat: backdoor
attacks. Given normal observations, an agent implanted with backdoors takes
actions leading to high rewards. However, the same agent takes actions that
lead to low rewards if the observations are injected with triggers that can
activate the backdoor. In this paper, we propose Baffle (Backdoor Attack for
Offline Reinforcement Learning) and evaluate how different Offline RL
algorithms react to this attack. Our experiments conducted on four tasks and
four offline RL algorithms expose a disquieting fact: none of the existing
offline RL algorithms is immune to such a backdoor attack. More specifically,
Baffle modifies $10\%$ of the datasets for four tasks (3 robotic controls and 1
autonomous driving). Agents trained on the poisoned datasets perform well in
normal settings. However, when triggers are presented, the agents' performance
decreases drastically by $63.6\%$, $57.8\%$, $60.8\%$ and $44.7\%$ in the four
tasks on average. The backdoor still persists after fine-tuning poisoned agents
on clean datasets. We further show that the inserted backdoor is also hard to
be detected by a popular defensive method. This paper calls attention to
developing more effective protection for the open-source offline RL dataset.


http://arxiv.org/abs/2210.03297
Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systems. (99%)
Chawin Sitawarin; Florian Tramèr; Nicholas Carlini
  Decision-based adversarial attacks construct inputs that fool a
machine-learning model into making targeted mispredictions by making only
hard-label queries. For the most part, these attacks have been applied directly
to isolated neural network models. However, in practice, machine learning
models are just a component of a much larger system. By adding just a single
preprocessor in front of a classifier, we find that state-of-the-art
query-based attacks are as much as seven times less effective at attacking a
prediction pipeline than attacking the machine learning model alone. Hence,
attacks that are unaware of this invariance inevitably waste a large number of
queries to re-discover or overcome it. We, therefore, develop techniques to
first reverse-engineer the preprocessor and then use this extracted information
to attack the end-to-end system. Our extraction method requires only a few
hundred queries to learn the preprocessors used by most publicly available
model pipelines, and our preprocessor-aware attacks recover the same efficacy
as just attacking the model alone. The code can be found at
https://github.com/google-research/preprocessor-aware-black-box-attack.


http://arxiv.org/abs/2210.03003
Enhancing Code Classification by Mixup-Based Data Augmentation. (96%)
Zeming Dong; Qiang Hu; Yuejun Guo; Maxime Cordy; Mike Papadakis; Yves Le Traon; Jianjun Zhao
  Recently, deep neural networks (DNNs) have been widely applied in programming
language understanding. Generally, training a DNN model with competitive
performance requires massive and high-quality labeled training data. However,
collecting and labeling such data is time-consuming and labor-intensive. To
tackle this issue, data augmentation has been a popular solution, which
delicately increases the training data size, e.g., adversarial example
generation. However, few works focus on employing it for programming
language-related tasks. In this paper, we propose a Mixup-based data
augmentation approach, MixCode, to enhance the source code classification task.
First, we utilize multiple code refactoring methods to generate
label-consistent code data. Second, the Mixup technique is employed to mix the
original code and transformed code to form the new training data to train the
model. We evaluate MixCode on two programming languages (JAVA and Python), two
code tasks (problem classification and bug detection), four datasets (JAVA250,
Python800, CodRep1, and Refactory), and 5 model architectures. Experimental
results demonstrate that MixCode outperforms the standard data augmentation
baseline by up to 6.24\% accuracy improvement and 26.06\% robustness
improvement.


http://arxiv.org/abs/2210.02840
Deep Reinforcement Learning based Evasion Generative Adversarial Network for Botnet Detection. (92%)
Rizwan Hamid Randhawa; Nauman Aslam; Mohammad Alauthman; Muhammad Khalid; Husnain Rafiq
  Botnet detectors based on machine learning are potential targets for
adversarial evasion attacks. Several research works employ adversarial training
with samples generated from generative adversarial nets (GANs) to make the
botnet detectors adept at recognising adversarial evasions. However, the
synthetic evasions may not follow the original semantics of the input samples.
This paper proposes a novel GAN model leveraged with deep reinforcement
learning (DRL) to explore semantic aware samples and simultaneously harden its
detection. A DRL agent is used to attack the discriminator of the GAN that acts
as a botnet detector. The discriminator is trained on the crafted perturbations
by the agent during the GAN training, which helps the GAN generator converge
earlier than the case without DRL. We name this model RELEVAGAN, i.e. ["relive
a GAN" or deep REinforcement Learning-based Evasion Generative Adversarial
Network] because, with the help of DRL, it minimises the GAN's job by letting
its generator explore the evasion samples within the semantic limits. During
the GAN training, the attacks are conducted to adjust the discriminator weights
for learning crafted perturbations by the agent. RELEVAGAN does not require
adversarial training for the ML classifiers since it can act as an adversarial
semantic-aware botnet detection model. Code will be available at
https://github.com/rhr407/RELEVAGAN.


http://arxiv.org/abs/2210.02713
On Optimal Learning Under Targeted Data Poisoning. (82%)
Steve Hanneke; Amin Karbasi; Mohammad Mahmoody; Idan Mehalel; Shay Moran
  Consider the task of learning a hypothesis class $\mathcal{H}$ in the
presence of an adversary that can replace up to an $\eta$ fraction of the
examples in the training set with arbitrary adversarial examples. The adversary
aims to fail the learner on a particular target test point $x$ which is known
to the adversary but not to the learner. In this work we aim to characterize
the smallest achievable error $\epsilon=\epsilon(\eta)$ by the learner in the
presence of such an adversary in both realizable and agnostic settings. We
fully achieve this in the realizable setting, proving that
$\epsilon=\Theta(\mathtt{VC}(\mathcal{H})\cdot \eta)$, where
$\mathtt{VC}(\mathcal{H})$ is the VC dimension of $\mathcal{H}$. Remarkably, we
show that the upper bound can be attained by a deterministic learner. In the
agnostic setting we reveal a more elaborate landscape: we devise a
deterministic learner with a multiplicative regret guarantee of $\epsilon \leq
C\cdot\mathtt{OPT} + O(\mathtt{VC}(\mathcal{H})\cdot \eta)$, where $C > 1$ is a
universal numerical constant. We complement this by showing that for any
deterministic learner there is an attack which worsens its error to at least
$2\cdot \mathtt{OPT}$. This implies that a multiplicative deterioration in the
regret is unavoidable in this case. Finally, the algorithms we develop for
achieving the optimal rates are inherently improper. Nevertheless, we show that
for a variety of natural concept classes, such as linear classifiers, it is
possible to retain the dependence $\epsilon=\Theta_{\mathcal{H}}(\eta)$ by a
proper algorithm in the realizable setting. Here $\Theta_{\mathcal{H}}$
conceals a polynomial dependence on $\mathtt{VC}(\mathcal{H})$.


http://arxiv.org/abs/2210.03150
Towards Out-of-Distribution Adversarial Robustness. (73%)
Adam Ibrahim; Charles Guille-Escuret; Ioannis Mitliagkas; Irina Rish; David Krueger; Pouya Bashivan
  Adversarial robustness continues to be a major challenge for deep learning. A
core issue is that robustness to one type of attack often fails to transfer to
other attacks. While prior work establishes a theoretical trade-off in
robustness against different $L_p$ norms, we show that there is potential for
improvement against many commonly used attacks by adopting a domain
generalisation approach. Concretely, we treat each type of attack as a domain,
and apply the Risk Extrapolation method (REx), which promotes similar levels of
robustness against all training attacks. Compared to existing methods, we
obtain similar or superior worst-case adversarial robustness on attacks seen
during training. Moreover, we achieve superior performance on families or
tunings of attacks only encountered at test time. On ensembles of attacks, our
approach improves the accuracy from 3.4% the best existing baseline to 25.9% on
MNIST, and from 16.9% to 23.5% on CIFAR10.


http://arxiv.org/abs/2210.03068
InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples. (61%)
Venelin Kovatchev; Mariona Taulé
  In this paper, we present InferES - an original corpus for Natural Language
Inference (NLI) in European Spanish. We propose, implement, and analyze a
variety of corpus-creating strategies utilizing expert linguists and crowd
workers. The objectives behind InferES are to provide high-quality data, and,
at the same time to facilitate the systematic evaluation of automated systems.
Specifically, we focus on measuring and improving the performance of machine
learning systems on negation-based adversarial examples and their ability to
generalize across out-of-distribution topics.
  We train two transformer models on InferES (8,055 gold examples) in a variety
of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for
improvement. The "hypothesis-only" baseline performs only 2%-5% higher than
majority, indicating much fewer annotation artifacts than prior work. We find
that models trained on InferES generalize very well across topics (both in- and
out-of-distribution) and perform moderately well on negation-based adversarial
examples.


http://arxiv.org/abs/2210.03250
Unsupervised Domain Adaptation for COVID-19 Information Service with Contrastive Adversarial Domain Mixup. (41%)
Huimin Zeng; Zhenrui Yue; Ziyi Kou; Lanyu Shang; Yang Zhang; Dong Wang
  In the real-world application of COVID-19 misinformation detection, a
fundamental challenge is the lack of the labeled COVID data to enable
supervised end-to-end training of the models, especially at the early stage of
the pandemic. To address this challenge, we propose an unsupervised domain
adaptation framework using contrastive learning and adversarial domain mixup to
transfer the knowledge from an existing source data domain to the target
COVID-19 data domain. In particular, to bridge the gap between the source
domain and the target domain, our method reduces a radial basis function (RBF)
based discrepancy between these two domains. Moreover, we leverage the power of
domain adversarial examples to establish an intermediate domain mixup, where
the latent representations of the input text from both domains could be mixed
during the training process. Extensive experiments on multiple real-world
datasets suggest that our method can effectively adapt misinformation detection
systems to the unseen COVID-19 target domain with significant improvements
compared to the state-of-the-art baselines.


http://arxiv.org/abs/2210.03205
Synthetic Dataset Generation for Privacy-Preserving Machine Learning. (2%)
Efstathia Soufleri; Gobinda Saha; Kaushik Roy
  Machine Learning (ML) has achieved enormous success in solving a variety of
problems in computer vision, speech recognition, object detection, to name a
few. The principal reason for this success is the availability of huge datasets
for training deep neural networks (DNNs). However, datasets cannot be publicly
released if they contain sensitive information such as medical records, and
data privacy becomes a major concern. Encryption methods could be a possible
solution, however their deployment on ML applications seriously impacts
classification accuracy and results in substantial computational overhead.
Alternatively, obfuscation techniques could be used, but maintaining a good
trade-off between visual privacy and accuracy is challenging. In this paper, we
propose a method to generate secure synthetic datasets from the original
private datasets. Given a network with Batch Normalization (BN) layers
pretrained on the original dataset, we first record the class-wise BN layer
statistics. Next, we generate the synthetic dataset by optimizing random noise
such that the synthetic data match the layer-wise statistical distribution of
original images. We evaluate our method on image classification datasets
(CIFAR10, ImageNet) and show that synthetic data can be used in place of the
original CIFAR10/ImageNet data for training networks from scratch, producing
comparable classification performance. Further, to analyze visual privacy
provided by our method, we use Image Quality Metrics and show high degree of
visual dissimilarity between the original and synthetic images. Moreover, we
show that our proposed method preserves data-privacy under various
privacy-leakage attacks including Gradient Matching Attack, Model Memorization
Attack, and GAN-based Attack.


http://arxiv.org/abs/2210.03123
Enhancing Mixup-Based Graph Learning for Language Processing via Hybrid Pooling. (1%)
Zeming Dong; Qiang Hu; Yuejun Guo; Maxime Cordy; Mike Papadakis; Yves Le Traon; Jianjun Zhao
  Graph neural networks (GNNs) have recently been popular in natural language
and programming language processing, particularly in text and source code
classification. Graph pooling which processes node representation into the
entire graph representation, which can be used for multiple downstream tasks,
e.g., graph classification, is a crucial component of GNNs. Recently, to
enhance graph learning, Manifold Mixup, a data augmentation strategy that mixes
the graph data vector after the pooling layer, has been introduced. However,
since there are a series of graph pooling methods, how they affect the
effectiveness of such a Mixup approach is unclear. In this paper, we take the
first step to explore the influence of graph pooling methods on the
effectiveness of the Mixup-based data augmentation approach. Specifically, 9
types of hybrid pooling methods are considered in the study, e.g.,
$\mathcal{M}_{sum}(\mathcal{P}_{att},\mathcal{P}_{max})$. The experimental
results on both natural language datasets (Gossipcop, Politifact) and
programming language datasets (Java250, Python800) demonstrate that hybrid
pooling methods are more suitable for Mixup than the standard max pooling and
the state-of-the-art graph multiset transformer (GMT) pooling, in terms of
metric accuracy and robustness.


http://arxiv.org/abs/2210.03239
Bad Citrus: Reducing Adversarial Costs with Model Distances. (1%)
Giorgio Severi; Will Pearce; Alina Oprea
  Recent work by Jia et al., showed the possibility of effectively computing
pairwise model distances in weight space, using a model explanation technique
known as LIME. This method requires query-only access to the two models under
examination. We argue this insight can be leveraged by an adversary to reduce
the net cost (number of queries) of launching an evasion campaign against a
deployed model. We show that there is a strong negative correlation between the
success rate of adversarial transfer and the distance between the victim model
and the surrogate used to generate the evasive samples. Thus, we propose and
evaluate a method to reduce adversarial costs by finding the closest surrogate
model for adversarial transfer.


http://arxiv.org/abs/2210.02041
Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks. (99%)
Shengming Yuan; Qilong Zhang; Lianli Gao; Yaya Cheng; Jingkuan Song
  Unrestricted color attacks, which manipulate semantically meaningful color of
an image, have shown their stealthiness and success in fooling both human eyes
and deep neural networks. However, current works usually sacrifice the
flexibility of the uncontrolled setting to ensure the naturalness of
adversarial examples. As a result, the black-box attack performance of these
methods is limited. To boost transferability of adversarial examples without
damaging image quality, we propose a novel Natural Color Fool (NCF) which is
guided by realistic color distributions sampled from a publicly available
dataset and optimized by our neighborhood search and initialization reset. By
conducting extensive experiments and visualizations, we convincingly
demonstrate the effectiveness of our proposed method. Notably, on average,
results show that our NCF can outperform state-of-the-art approaches by
15.0%$\sim$32.9% for fooling normally trained models and 10.0%$\sim$25.3% for
evading defense methods. Our code is available at
https://github.com/ylhz/Natural-Color-Fool.


http://arxiv.org/abs/2210.02618
Dynamic Stochastic Ensemble with Adversarial Robust Lottery Ticket Subnetworks. (98%)
Qi Peng; Wenlin Liu; Ruoxi Qin; Libin Hou; Bin Yan; Linyuan Wang
  Adversarial attacks are considered the intrinsic vulnerability of CNNs.
Defense strategies designed for attacks have been stuck in the adversarial
attack-defense arms race, reflecting the imbalance between attack and defense.
Dynamic Defense Framework (DDF) recently changed the passive safety status quo
based on the stochastic ensemble model. The diversity of subnetworks, an
essential concern in the DDF, can be effectively evaluated by the adversarial
transferability between different networks. Inspired by the poor adversarial
transferability between subnetworks of scratch tickets with various remaining
ratios, we propose a method to realize the dynamic stochastic ensemble defense
strategy. We discover the adversarial transferable diversity between robust
lottery ticket subnetworks drawn from different basic structures and sparsity.
The experimental results suggest that our method achieves better robust and
clean recognition accuracy by adversarial transferable diversity, which would
decrease the reliability of attacks.


http://arxiv.org/abs/2210.02502
On Adversarial Robustness of Deep Image Deblurring. (83%)
Kanchana Vaishnavi Gandikota; Paramanand Chandramouli; Michael Moeller
  Recent approaches employ deep learning-based solutions for the recovery of a
sharp image from its blurry observation. This paper introduces adversarial
attacks against deep learning-based image deblurring methods and evaluates the
robustness of these neural networks to untargeted and targeted attacks. We
demonstrate that imperceptible distortion can significantly degrade the
performance of state-of-the-art deblurring networks, even producing drastically
different content in the output, indicating the strong need to include
adversarially robust training not only in classification but also for image
recovery.


http://arxiv.org/abs/2210.02577
A Closer Look at Robustness to L-infinity and Spatial Perturbations and their Composition. (81%)
Luke Rowe; Benjamin Thérien; Krzysztof Czarnecki; Hongyang Zhang
  In adversarial machine learning, the popular $\ell_\infty$ threat model has
been the focus of much previous work. While this mathematical definition of
imperceptibility successfully captures an infinite set of additive image
transformations that a model should be robust to, this is only a subset of all
transformations which leave the semantic label of an image unchanged. Indeed,
previous work also considered robustness to spatial attacks as well as other
semantic transformations; however, designing defense methods against the
composition of spatial and $\ell_{\infty}$ perturbations remains relatively
underexplored. In the following, we improve the understanding of this seldom
investigated compositional setting. We prove theoretically that no linear
classifier can achieve more than trivial accuracy against a composite adversary
in a simple statistical setting, illustrating its difficulty. We then
investigate how state-of-the-art $\ell_{\infty}$ defenses can be adapted to
this novel threat model and study their performance against compositional
attacks. We find that our newly proposed TRADES$_{\text{All}}$ strategy
performs the strongest of all. Analyzing its logit's Lipschitz constant for RT
transformations of different sizes, we find that TRADES$_{\text{All}}$ remains
stable over a wide range of RT transformations with and without $\ell_\infty$
perturbations.


http://arxiv.org/abs/2210.02082
Jitter Does Matter: Adapting Gaze Estimation to New Domains. (78%)
Ruicong Liu; Yiwei Bao; Mingjie Xu; Haofei Wang; Yunfei Liu; Feng Lu
  Deep neural networks have demonstrated superior performance on
appearance-based gaze estimation tasks. However, due to variations in person,
illuminations, and background, performance degrades dramatically when applying
the model to a new domain. In this paper, we discover an interesting gaze
jitter phenomenon in cross-domain gaze estimation, i.e., the gaze predictions
of two similar images can be severely deviated in target domain. This is
closely related to cross-domain gaze estimation tasks, but surprisingly, it has
not been noticed yet previously. Therefore, we innovatively propose to utilize
the gaze jitter to analyze and optimize the gaze domain adaptation task. We
find that the high-frequency component (HFC) is an important factor that leads
to jitter. Based on this discovery, we add high-frequency components to input
images using the adversarial attack and employ contrastive learning to
encourage the model to obtain similar representations between original and
perturbed data, which reduces the impacts of HFC. We evaluate the proposed
method on four cross-domain gaze estimation tasks, and experimental results
demonstrate that it significantly reduces the gaze jitter and improves the gaze
estimation performance in target domains.


http://arxiv.org/abs/2210.02357
Image Masking for Robust Self-Supervised Monocular Depth Estimation. (38%)
Hemang Chawla; Kishaan Jeeveswaran; Elahe Arani; Bahram Zonooz
  Self-supervised monocular depth estimation is a salient task for 3D scene
understanding. Learned jointly with monocular ego-motion estimation, several
methods have been proposed to predict accurate pixel-wise depth without using
labeled data. Nevertheless, these methods focus on improving performance under
ideal conditions without natural or digital corruptions. A general absence of
occlusions is assumed even for object-specific depth estimation. These methods
are also vulnerable to adversarial attacks, which is a pertinent concern for
their reliable deployment on robots and autonomous driving systems. We propose
MIMDepth, a method that adapts masked image modeling (MIM) for self-supervised
monocular depth estimation. While MIM has been used to learn generalizable
features during pre-training, we show how it could be adapted for direct
training of monocular depth estimation. Our experiments show that MIMDepth is
more robust to noise, blur, weather conditions, digital artifacts, occlusions,
as well as untargeted and targeted adversarial attacks.


http://arxiv.org/abs/2210.02235
Over-the-Air Federated Learning with Privacy Protection via Correlated Additive Perturbations. (38%)
Jialing Liao; Zheng Chen; Erik G. Larsson
  In this paper, we consider privacy aspects of wireless federated learning
(FL) with Over-the-Air (OtA) transmission of gradient updates from multiple
users/agents to an edge server. By exploiting the waveform superposition
property of multiple access channels, OtA FL enables the users to transmit
their updates simultaneously with linear processing techniques, which improves
resource efficiency. However, this setting is vulnerable to privacy leakage
since an adversary node can hear directly the uncoded message. Traditional
perturbation-based methods provide privacy protection while sacrificing the
training accuracy due to the reduced signal-to-noise ratio. In this work, we
aim at minimizing privacy leakage to the adversary and the degradation of model
accuracy at the edge server at the same time. More explicitly, spatially
correlated perturbations are added to the gradient vectors at the users before
transmission. Using the zero-sum property of the correlated perturbations, the
side effect of the added perturbation on the aggregated gradients at the edge
server can be minimized. In the meanwhile, the added perturbation will not be
canceled out at the adversary, which prevents privacy leakage. Theoretical
analysis of the perturbation covariance matrix, differential privacy, and model
convergence is provided, based on which an optimization problem is formulated
to jointly design the covariance matrix and the power scaling factor to balance
between privacy protection and convergence performance. Simulation results
validate the correlated perturbation approach can provide strong defense
ability while guaranteeing high learning accuracy.


http://arxiv.org/abs/2210.01787
Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective. (97%)
Bohang Zhang; Du Jiang; Di He; Liwei Wang
  Designing neural networks with bounded Lipschitz constant is a promising way
to obtain certifiably robust classifiers against adversarial examples. However,
the relevant progress for the important $\ell_\infty$ perturbation setting is
rather limited, and a principled understanding of how to design expressive
$\ell_\infty$ Lipschitz networks is still lacking. In this paper, we bridge the
gap by studying certified $\ell_\infty$ robustness from a novel perspective of
representing Boolean functions. We derive two fundamental impossibility results
that hold for any standard Lipschitz network: one for robust classification on
finite datasets, and the other for Lipschitz function approximation. These
results identify that networks built upon norm-bounded affine layers and
Lipschitz activations intrinsically lose expressive power even in the
two-dimensional case, and shed light on how recently proposed Lipschitz
networks (e.g., GroupSort and $\ell_\infty$-distance nets) bypass these
impossibilities by leveraging order statistic functions. Finally, based on
these insights, we develop a unified Lipschitz network that generalizes prior
works, and design a practical version that can be efficiently trained (making
certified robust training free). Extensive experiments show that our approach
is scalable, efficient, and consistently yields better certified robustness
across multiple datasets and perturbation radii than prior Lipschitz networks.
Our code is available at https://github.com/zbh2047/SortNet.


http://arxiv.org/abs/2210.01953
Robust Fair Clustering: A Novel Fairness Attack and Defense Framework. (93%)
Anshuman Chhabra; Peizhao Li; Prasant Mohapatra; Hongfu Liu
  Clustering algorithms are widely used in many societal resource allocation
applications, such as loan approvals and candidate recruitment, among others,
and hence, biased or unfair model outputs can adversely impact individuals that
rely on these applications. To this end, many fair clustering approaches have
been recently proposed to counteract this issue. Due to the potential for
significant harm, it is essential to ensure that fair clustering algorithms
provide consistently fair outputs even under adversarial influence. However,
fair clustering algorithms have not been studied from an adversarial attack
perspective. In contrast to previous research, we seek to bridge this gap and
conduct a robustness analysis against fair clustering by proposing a novel
black-box fairness attack. Through comprehensive experiments, we find that
state-of-the-art models are highly susceptible to our attack as it can reduce
their fairness performance significantly. Finally, we propose Consensus Fair
Clustering (CFC), the first robust fair clustering approach that transforms
consensus clustering into a fair graph partitioning problem, and iteratively
learns to generate fair cluster outputs. Experimentally, we observe that CFC is
highly robust to the proposed attack and is thus a truly robust fair clustering
alternative.


http://arxiv.org/abs/2210.01371
A Study on the Efficiency and Generalization of Light Hybrid Retrievers. (86%)
Man Luo; Shashank Jain; Anchit Gupta; Arash Einolghozati; Barlas Oguz; Debojeet Chatterjee; Xilun Chen; Chitta Baral; Peyman Heidari
  Existing hybrid retrievers which integrate sparse and dense retrievers, are
indexing-heavy, limiting their applicability in real-world on-devices settings.
We ask the question "Is it possible to reduce the indexing memory of hybrid
retrievers without sacrificing performance?" Driven by this question, we
leverage an indexing-efficient dense retriever (i.e. DrBoost) to obtain a light
hybrid retriever. Moreover, to further reduce the memory, we introduce a
lighter dense retriever (LITE) which is jointly trained on contrastive learning
and knowledge distillation from DrBoost. Compared to previous heavy hybrid
retrievers, our Hybrid-LITE retriever saves 13 memory while maintaining 98.0
performance.
  In addition, we study the generalization of light hybrid retrievers along two
dimensions, out-of-domain (OOD) generalization and robustness against
adversarial attacks. We evaluate models on two existing OOD benchmarks and
create six adversarial attack sets for robustness evaluation. Experiments show
that our light hybrid retrievers achieve better robustness performance than
both sparse and dense retrievers. Nevertheless there is a large room to improve
the robustness of retrievers, and our datasets can aid future research.


http://arxiv.org/abs/2210.02447
Practical Adversarial Attacks on Spatiotemporal Traffic Forecasting Models. (81%)
Fan Liu; Hao Liu; Wenzhao Jiang
  Machine learning based traffic forecasting models leverage sophisticated
spatiotemporal auto-correlations to provide accurate predictions of city-wide
traffic states. However, existing methods assume a reliable and unbiased
forecasting environment, which is not always available in the wild. In this
work, we investigate the vulnerability of spatiotemporal traffic forecasting
models and propose a practical adversarial spatiotemporal attack framework.
Specifically, instead of simultaneously attacking all geo-distributed data
sources, an iterative gradient-guided node saliency method is proposed to
identify the time-dependent set of victim nodes. Furthermore, we devise a
spatiotemporal gradient descent based scheme to generate real-valued
adversarial traffic states under a perturbation constraint. Meanwhile, we
theoretically demonstrate the worst performance bound of adversarial traffic
forecasting attacks. Extensive experiments on two real-world datasets show that
the proposed two-step framework achieves up to $67.8\%$ performance degradation
on various advanced spatiotemporal forecasting models. Remarkably, we also show
that adversarial training with our proposed attacks can significantly improve
the robustness of spatiotemporal traffic forecasting models. Our code is
available in \url{https://github.com/luckyfan-cs/ASTFA}.


http://arxiv.org/abs/2210.01940
On the Robustness of Deep Clustering Models: Adversarial Attacks and Defenses. (75%)
Anshuman Chhabra; Ashwin Sekhari; Prasant Mohapatra
  Clustering models constitute a class of unsupervised machine learning methods
which are used in a number of application pipelines, and play a vital role in
modern data science. With recent advancements in deep learning -- deep
clustering models have emerged as the current state-of-the-art over traditional
clustering approaches, especially for high-dimensional image datasets. While
traditional clustering approaches have been analyzed from a robustness
perspective, no prior work has investigated adversarial attacks and robustness
for deep clustering models in a principled manner. To bridge this gap, we
propose a blackbox attack using Generative Adversarial Networks (GANs) where
the adversary does not know which deep clustering model is being used, but can
query it for outputs. We analyze our attack against multiple state-of-the-art
deep clustering models and real-world datasets, and find that it is highly
successful. We then employ some natural unsupervised defense approaches, but
find that these are unable to mitigate our attack. Finally, we attack Face++, a
production-level face clustering API service, and find that we can
significantly reduce its performance as well. Through this work, we thus aim to
motivate the need for truly robust deep clustering models.


http://arxiv.org/abs/2210.04625
Robustness Certification of Visual Perception Models via Camera Motion Smoothing. (70%)
Hanjiang Hu; Zuxin Liu; Linyi Li; Jiacheng Zhu; Ding Zhao
  A vast literature shows that the learning-based visual perception model is
sensitive to adversarial noises but few works consider the robustness of
robotic perception models under widely-existing camera motion perturbations. To
this end, we study the robustness of the visual perception model under camera
motion perturbations to investigate the influence of camera motion on robotic
perception. Specifically, we propose a motion smoothing technique for arbitrary
image classification models, whose robustness under camera motion perturbations
could be certified. The proposed robustness certification framework based on
camera motion smoothing provides tight and scalable robustness guarantees for
visual perception modules so that they are applicable to wide robotic
applications. As far as we are aware, this is the first work to provide the
robustness certification for the deep perception module against camera motions,
which improves the trustworthiness of robotic perception. A realistic indoor
robotic dataset with the dense point cloud map for the entire room, MetaRoom,
is introduced for the challenging certifiable robust perception task. We
conduct extensive experiments to validate the certification approach via motion
smoothing against camera motion perturbations. Our framework guarantees the
certified accuracy of 81.7% against camera translation perturbation along depth
direction within -0.1m ` 0.1m. We also validate the effectiveness of our method
on the real-world robot by conducting hardware experiment on the robotic arm
with an eye-in-hand camera. The code is available on
https://github.com/HanjiangHu/camera-motion-smoothing.


http://arxiv.org/abs/2210.01632
Backdoor Attacks in the Supply Chain of Masked Image Modeling. (68%)
Xinyue Shen; Xinlei He; Zheng Li; Yun Shen; Michael Backes; Yang Zhang
  Masked image modeling (MIM) revolutionizes self-supervised learning (SSL) for
image pre-training. In contrast to previous dominating self-supervised methods,
i.e., contrastive learning, MIM attains state-of-the-art performance by masking
and reconstructing random patches of the input image. However, the associated
security and privacy risks of this novel generative method are unexplored. In
this paper, we perform the first security risk quantification of MIM through
the lens of backdoor attacks. Different from previous work, we are the first to
systematically threat modeling on SSL in every phase of the model supply chain,
i.e., pre-training, release, and downstream phases. Our evaluation shows that
models built with MIM are vulnerable to existing backdoor attacks in release
and downstream phases and are compromised by our proposed method in
pre-training phase. For instance, on CIFAR10, the attack success rate can reach
99.62%, 96.48%, and 98.89% in the downstream phase, release phase, and
pre-training phase, respectively. We also take the first step to investigate
the success factors of backdoor attacks in the pre-training phase and find the
trigger number and trigger pattern play key roles in the success of backdoor
attacks while trigger location has only tiny effects. In the end, our empirical
study of the defense mechanisms across three detection-level on model supply
chain phases indicates that different defenses are suitable for backdoor
attacks in different phases. However, backdoor attacks in the release phase
cannot be detected by all three detection-level methods, calling for more
effective defenses in future research.


http://arxiv.org/abs/2210.01742
CADet: Fully Self-Supervised Anomaly Detection With Contrastive Learning. (54%)
Charles Guille-Escuret; Pau Rodriguez; David Vazquez; Ioannis Mitliagkas; Joao Monteiro
  Handling out-of-distribution (OOD) samples has become a major stake in the
real-world deployment of machine learning systems. This work explores the
application of self-supervised contrastive learning to the simultaneous
detection of two types of OOD samples: unseen classes and adversarial
perturbations. Since in practice the distribution of such samples is not known
in advance, we do not assume access to OOD examples. We show that similarity
functions trained with contrastive learning can be leveraged with the maximum
mean discrepancy (MMD) two-sample test to verify whether two independent sets
of samples are drawn from the same distribution. Inspired by this approach, we
introduce CADet (Contrastive Anomaly Detection), a method based on image
augmentations to perform anomaly detection on single samples. CADet compares
favorably to adversarial detection methods to detect adversarially perturbed
samples on ImageNet. Simultaneously, it achieves comparable performance to
unseen label detection methods on two challenging benchmarks: ImageNet-O and
iNaturalist. CADet is fully self-supervised and requires neither labels for
in-distribution samples nor access to OOD examples.


http://arxiv.org/abs/2210.01834
Invariant Aggregator for Defending Federated Backdoor Attacks. (22%)
Xiaoyang Wang; Dimitrios Dimitriadis; Sanmi Koyejo; Shruti Tople
  Federated learning is gaining popularity as it enables training of
high-utility models across several clients without directly sharing their
private data. As a downside, the federated setting makes the model vulnerable
to various adversarial attacks in the presence of malicious clients.
Specifically, an adversary can perform backdoor attacks to control model
predictions via poisoning the training dataset with a trigger. In this work, we
propose a mitigation for backdoor attacks in a federated learning setup. Our
solution forces the model optimization trajectory to focus on the invariant
directions that are generally useful for utility and avoid selecting directions
that favor few and possibly malicious clients. Concretely, we consider the sign
consistency of the pseudo-gradient (the client update) as an estimation of the
invariance. Following this, our approach performs dimension-wise filtering to
remove pseudo-gradient elements with low sign consistency. Then, a robust mean
estimator eliminates outliers among the remaining dimensions. Our theoretical
analysis further shows the necessity of the defense combination and illustrates
how our proposed solution defends the federated learning model. Empirical
results on three datasets with different modalities and varying number of
clients show that our approach mitigates backdoor attacks with a negligible
cost on the model utility.


http://arxiv.org/abs/2210.01111
MultiGuard: Provably Robust Multi-label Classification against Adversarial Examples. (99%)
Jinyuan Jia; Wenjie Qu; Neil Zhenqiang Gong
  Multi-label classification, which predicts a set of labels for an input, has
many applications. However, multiple recent studies showed that multi-label
classification is vulnerable to adversarial examples. In particular, an
attacker can manipulate the labels predicted by a multi-label classifier for an
input via adding carefully crafted, human-imperceptible perturbation to it.
Existing provable defenses for multi-class classification achieve sub-optimal
provable robustness guarantees when generalized to multi-label classification.
In this work, we propose MultiGuard, the first provably robust defense against
adversarial examples to multi-label classification. Our MultiGuard leverages
randomized smoothing, which is the state-of-the-art technique to build provably
robust classifiers. Specifically, given an arbitrary multi-label classifier,
our MultiGuard builds a smoothed multi-label classifier via adding random noise
to the input. We consider isotropic Gaussian noise in this work. Our major
theoretical contribution is that we show a certain number of ground truth
labels of an input are provably in the set of labels predicted by our
MultiGuard when the $\ell_2$-norm of the adversarial perturbation added to the
input is bounded. Moreover, we design an algorithm to compute our provable
robustness guarantees. Empirically, we evaluate our MultiGuard on VOC 2007,
MS-COCO, and NUS-WIDE benchmark datasets. Our code is available at:
\url{https://github.com/quwenjie/MultiGuard}


http://arxiv.org/abs/2210.00753
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection. (97%)
Xuanjun Chen; Haibin Wu; Helen Meng; Hung-yi Lee; Jyh-Shing Roger Jang
  Audio-visual active speaker detection (AVASD) is well-developed, and now is
an indispensable front-end for several multi-modal applications. However, to
the best of our knowledge, the adversarial robustness of AVASD models hasn't
been investigated, not to mention the effective defense against such attacks.
In this paper, we are the first to reveal the vulnerability of AVASD models
under audio-only, visual-only, and audio-visual adversarial attacks through
extensive experiments. What's more, we also propose a novel audio-visual
interaction loss (AVIL) for making attackers difficult to find feasible
adversarial examples under an allocated attack budget. The loss aims at pushing
the inter-class embeddings to be dispersed, namely non-speech and speech
clusters, sufficiently disentangled, and pulling the intra-class embeddings as
close as possible to keep them compact. Experimental results show the AVIL
outperforms the adversarial training by 33.14 mAP (%) under multi-modal
attacks.


http://arxiv.org/abs/2210.00960
Stability Analysis and Generalization Bounds of Adversarial Training. (96%)
Jiancong Xiao; Yanbo Fan; Ruoyu Sun; Jue Wang; Zhi-Quan Luo
  In adversarial machine learning, deep neural networks can fit the adversarial
examples on the training dataset but have poor generalization ability on the
test set. This phenomenon is called robust overfitting, and it can be observed
when adversarially training neural nets on common datasets, including SVHN,
CIFAR-10, CIFAR-100, and ImageNet. In this paper, we study the robust
overfitting issue of adversarial training by using tools from uniform
stability. One major challenge is that the outer function (as a maximization of
the inner function) is nonsmooth, so the standard technique (e.g., hardt et
al., 2016) cannot be applied. Our approach is to consider $\eta$-approximate
smoothness: we show that the outer function satisfies this modified smoothness
assumption with $\eta$ being a constant related to the adversarial perturbation
$\epsilon$. Based on this, we derive stability-based generalization bounds for
stochastic gradient descent (SGD) on the general class of $\eta$-approximate
smooth functions, which covers the adversarial loss. Our results suggest that
robust test accuracy decreases in $\epsilon$ when $T$ is large, with a speed
between $\Omega(\epsilon\sqrt{T})$ and $\mathcal{O}(\epsilon T)$. This
phenomenon is also observed in practice. Additionally, we show that a few
popular techniques for adversarial training (e.g., early stopping, cyclic
learning rate, and stochastic weight averaging) are stability-promoting in
theory.


http://arxiv.org/abs/2210.02191
On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks. (92%)
Huimin Zeng; Zhenrui Yue; Yang Zhang; Ziyi Kou; Lanyu Shang; Dong Wang
  In many applications with real-world consequences, it is crucial to develop
reliable uncertainty estimation for the predictions made by the AI decision
systems. Targeting at the goal of estimating uncertainty, various deep neural
network (DNN) based uncertainty estimation algorithms have been proposed.
However, the robustness of the uncertainty returned by these algorithms has not
been systematically explored. In this work, to raise the awareness of the
research community on robust uncertainty estimation, we show that
state-of-the-art uncertainty estimation algorithms could fail catastrophically
under our proposed adversarial attack despite their impressive performance on
uncertainty estimation. In particular, we aim at attacking the out-domain
uncertainty estimation: under our attack, the uncertainty model would be fooled
to make high-confident predictions for the out-domain data, which they
originally would have rejected. Extensive experimental results on various
benchmark image datasets show that the uncertainty estimated by
state-of-the-art methods could be easily corrupted by our attack.


http://arxiv.org/abs/2210.01075
Decompiling x86 Deep Neural Network Executables. (83%)
Zhibo Liu; Yuanyuan Yuan; Shuai Wang; Xiaofei Xie; Lei Ma
  Due to their widespread use on heterogeneous hardware devices, deep learning
(DL) models are compiled into executables by DL compilers to fully leverage
low-level hardware primitives. This approach allows DL computations to be
undertaken at low cost across a variety of computing platforms, including CPUs,
GPUs, and various hardware accelerators.
  We present BTD (Bin to DNN), a decompiler for deep neural network (DNN)
executables. BTD takes DNN executables and outputs full model specifications,
including types of DNN operators, network topology, dimensions, and parameters
that are (nearly) identical to those of the input models. BTD delivers a
practical framework to process DNN executables compiled by different DL
compilers and with full optimizations enabled on x86 platforms. It employs
learning-based techniques to infer DNN operators, dynamic analysis to reveal
network architectures, and symbolic execution to facilitate inferring
dimensions and parameters of DNN operators.
  Our evaluation reveals that BTD enables accurate recovery of full
specifications of complex DNNs with millions of parameters (e.g., ResNet). The
recovered DNN specifications can be re-compiled into a new DNN executable
exhibiting identical behavior to the input executable. We show that BTD can
boost two representative attacks, adversarial example generation and knowledge
stealing, against DNN executables. We also demonstrate cross-architecture
legacy code reuse using BTD, and envision BTD being used for other critical
downstream tasks like DNN security hardening and patching.


http://arxiv.org/abs/2210.01288
Strength-Adaptive Adversarial Training. (80%)
Chaojian Yu; Dawei Zhou; Li Shen; Jun Yu; Bo Han; Mingming Gong; Nannan Wang; Tongliang Liu
  Adversarial training (AT) is proved to reliably improve network's robustness
against adversarial data. However, current AT with a pre-specified perturbation
budget has limitations in learning a robust network. Firstly, applying a
pre-specified perturbation budget on networks of various model capacities will
yield divergent degree of robustness disparity between natural and robust
accuracies, which deviates from robust network's desideratum. Secondly, the
attack strength of adversarial training data constrained by the pre-specified
perturbation budget fails to upgrade as the growth of network robustness, which
leads to robust overfitting and further degrades the adversarial robustness. To
overcome these limitations, we propose \emph{Strength-Adaptive Adversarial
Training} (SAAT). Specifically, the adversary employs an adversarial loss
constraint to generate adversarial training data. Under this constraint, the
perturbation budget will be adaptively adjusted according to the training state
of adversarial data, which can effectively avoid robust overfitting. Besides,
SAAT explicitly constrains the attack strength of training data through the
adversarial loss, which manipulates model capacity scheduling during training,
and thereby can flexibly control the degree of robustness disparity and adjust
the tradeoff between natural accuracy and robustness. Extensive experiments
show that our proposal boosts the robustness of adversarial training.


http://arxiv.org/abs/2210.01002
ASGNN: Graph Neural Networks with Adaptive Structure. (68%)
Zepeng Zhang; Songtao Lu; Zengfeng Huang; Ziping Zhao
  The graph neural network (GNN) models have presented impressive achievements
in numerous machine learning tasks. However, many existing GNN models are shown
to be vulnerable to adversarial attacks, which creates a stringent need to
build robust GNN architectures. In this work, we propose a novel interpretable
message passing scheme with adaptive structure (ASMP) to defend against
adversarial attacks on graph structure. Layers in ASMP are derived based on
optimization steps that minimize an objective function that learns the node
feature and the graph structure simultaneously. ASMP is adaptive in the sense
that the message passing process in different layers is able to be carried out
over dynamically adjusted graphs. Such property allows more fine-grained
handling of the noisy (or perturbed) graph structure and hence improves the
robustness. Convergence properties of the ASMP scheme are theoretically
established. Integrating ASMP with neural networks can lead to a new family of
GNN models with adaptive structure (ASGNN). Extensive experiments on
semi-supervised node classification tasks demonstrate that the proposed ASGNN
outperforms the state-of-the-art GNN architectures in terms of classification
performance under various adversarial attacks.


http://arxiv.org/abs/2210.00957
UnGANable: Defending Against GAN-based Face Manipulation. (2%)
Zheng Li; Ning Yu; Ahmed Salem; Michael Backes; Mario Fritz; Yang Zhang
  Deepfakes pose severe threats of visual misinformation to our society. One
representative deepfake application is face manipulation that modifies a
victim's facial attributes in an image, e.g., changing her age or hair color.
The state-of-the-art face manipulation techniques rely on Generative
Adversarial Networks (GANs). In this paper, we propose the first defense
system, namely UnGANable, against GAN-inversion-based face manipulation. In
specific, UnGANable focuses on defending GAN inversion, an essential step for
face manipulation. Its core technique is to search for alternative images
(called cloaked images) around the original images (called target images) in
image space. When posted online, these cloaked images can jeopardize the GAN
inversion process. We consider two state-of-the-art inversion techniques
including optimization-based inversion and hybrid inversion, and design five
different defenses under five scenarios depending on the defender's background
knowledge. Extensive experiments on four popular GAN models trained on two
benchmark face datasets show that UnGANable achieves remarkable effectiveness
and utility performance, and outperforms multiple baseline methods. We further
investigate four adaptive adversaries to bypass UnGANable and show that some of
them are slightly effective.


http://arxiv.org/abs/2210.00557
Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysis. (99%)
Jiancong Xiao; Zeyu Qin; Yanbo Fan; Baoyuan Wu; Jue Wang; Zhi-Quan Luo
  Adversarial Training (AT) has been demonstrated as one of the most effective
methods against adversarial examples. While most existing works focus on AT
with a single type of perturbation e.g., the $\ell_\infty$ attacks), DNNs are
facing threats from different types of adversarial examples. Therefore,
adversarial training for multiple perturbations (ATMP) is proposed to
generalize the adversarial robustness over different perturbation types (in
$\ell_1$, $\ell_2$, and $\ell_\infty$ norm-bounded perturbations). However, the
resulting model exhibits trade-off between different attacks. Meanwhile, there
is no theoretical analysis of ATMP, limiting its further development. In this
paper, we first provide the smoothness analysis of ATMP and show that $\ell_1$,
$\ell_2$, and $\ell_\infty$ adversaries give different contributions to the
smoothness of the loss function of ATMP. Based on this, we develop the
stability-based excess risk bounds and propose adaptive smoothness-weighted
adversarial training for multiple perturbations. Theoretically, our algorithm
yields better bounds. Empirically, our experiments on CIFAR10 and CIFAR100
achieve the state-of-the-art performance against the mixture of multiple
perturbations attacks.


http://arxiv.org/abs/2210.00430
Understanding Adversarial Robustness Against On-manifold Adversarial Examples. (99%)
Jiancong Xiao; Liusha Yang; Yanbo Fan; Jue Wang; Zhi-Quan Luo
  Deep neural networks (DNNs) are shown to be vulnerable to adversarial
examples. A well-trained model can be easily attacked by adding small
perturbations to the original data. One of the hypotheses of the existence of
the adversarial examples is the off-manifold assumption: adversarial examples
lie off the data manifold. However, recent research showed that on-manifold
adversarial examples also exist. In this paper, we revisit the off-manifold
assumption and want to study a question: at what level is the poor performance
of neural networks against adversarial attacks due to on-manifold adversarial
examples? Since the true data manifold is unknown in practice, we consider two
approximated on-manifold adversarial examples on both real and synthesis
datasets. On real datasets, we show that on-manifold adversarial examples have
greater attack rates than off-manifold adversarial examples on both
standard-trained and adversarially-trained models. On synthetic datasets,
theoretically, We prove that on-manifold adversarial examples are powerful, yet
adversarial training focuses on off-manifold directions and ignores the
on-manifold adversarial examples. Furthermore, we provide analysis to show that
the properties derived theoretically can also be observed in practice. Our
analysis suggests that on-manifold adversarial examples are important, and we
should pay more attention to on-manifold adversarial examples for training
robust models.


http://arxiv.org/abs/2210.00584
FLCert: Provably Secure Federated Learning against Poisoning Attacks. (74%)
Xiaoyu Cao; Zaixi Zhang; Jinyuan Jia; Neil Zhenqiang Gong
  Due to its distributed nature, federated learning is vulnerable to poisoning
attacks, in which malicious clients poison the training process via
manipulating their local training data and/or local model updates sent to the
cloud server, such that the poisoned global model misclassifies many
indiscriminate test inputs or attacker-chosen ones. Existing defenses mainly
leverage Byzantine-robust federated learning methods or detect malicious
clients. However, these defenses do not have provable security guarantees
against poisoning attacks and may be vulnerable to more advanced attacks. In
this work, we aim to bridge the gap by proposing FLCert, an ensemble federated
learning framework, that is provably secure against poisoning attacks with a
bounded number of malicious clients. Our key idea is to divide the clients into
groups, learn a global model for each group of clients using any existing
federated learning method, and take a majority vote among the global models to
classify a test input. Specifically, we consider two methods to group the
clients and propose two variants of FLCert correspondingly, i.e., FLCert-P that
randomly samples clients in each group, and FLCert-D that divides clients to
disjoint groups deterministically. Our extensive experiments on multiple
datasets show that the label predicted by our FLCert for a test input is
provably unaffected by a bounded number of malicious clients, no matter what
poisoning attacks they use.


http://arxiv.org/abs/2210.00621
Optimization for Robustness Evaluation beyond $\ell_p$ Metrics. (16%)
Hengyue Liang; Buyun Liang; Ying Cui; Tim Mitchell; Ju Sun
  Empirical evaluation of deep learning models against adversarial attacks
entails solving nontrivial constrained optimization problems. Popular
algorithms for solving these constrained problems rely on projected gradient
descent (PGD) and require careful tuning of multiple hyperparameters. Moreover,
PGD can only handle $\ell_1$, $\ell_2$, and $\ell_\infty$ attack models due to
the use of analytical projectors. In this paper, we introduce a novel
algorithmic framework that blends a general-purpose constrained-optimization
solver PyGRANSO, With Constraint-Folding (PWCF), to add reliability and
generality to robustness evaluation. PWCF 1) finds good-quality solutions
without the need of delicate hyperparameter tuning, and 2) can handle general
attack models, e.g., general $\ell_p$ ($p \geq 0$) and perceptual attacks,
which are inaccessible to PGD-based algorithms.


http://arxiv.org/abs/2210.00649
Automated Security Analysis of Exposure Notification Systems. (1%)
Kevin Morio; Ilkan Esiyok; Dennis Jackson; Robert Künnemann
  We present the first formal analysis and comparison of the security of the
two most widely deployed exposure notification systems, ROBERT and the Google
and Apple Exposure Notification (GAEN) framework. ROBERT is the most popular
instalment of the centralised approach to exposure notification, in which the
risk score is computed by a central server. GAEN, in contrast, follows the
decentralised approach, where the user's phone calculates the risk. The
relative merits of centralised and decentralised systems have proven to be a
controversial question. The majority of the previous analyses have focused on
the privacy implications of these systems, ours is the first formal analysis to
evaluate the security of the deployed systems -- the absence of false risk
alerts. We model the French deployment of ROBERT and the most widely deployed
GAEN variant, Germany's Corona-Warn-App. We isolate the precise conditions
under which these systems prevent false alerts. We determine exactly how an
adversary can subvert the system via network and Bluetooth sniffing, database
leakage or the compromise of phones, back-end systems and health authorities.
We also investigate the security of the original specification of the DP3T
protocol, in order to identify gaps between the proposed scheme and its
ultimate deployment. We find a total of 27 attack patterns, including many that
distinguish the centralised from the decentralised approach, as well as attacks
on the authorisation procedure that differentiate all three protocols. Our
results suggest that ROBERT's centralised design is more vulnerable against
both opportunistic and highly resourced attackers trying to perform
mass-notification attacks.


http://arxiv.org/abs/2210.00292
DeltaBound Attack: Efficient decision-based attack in low queries regime. (96%)
Lorenzo Rossi
  Deep neural networks and other machine learning systems, despite being
extremely powerful and able to make predictions with high accuracy, are
vulnerable to adversarial attacks. We proposed the DeltaBound attack: a novel,
powerful attack in the hard-label setting with $\ell_2$ norm bounded
perturbations. In this scenario, the attacker has only access to the top-1
predicted label of the model and can be therefore applied to real-world
settings such as remote API. This is a complex problem since the attacker has
very little information about the model. Consequently, most of the other
techniques present in the literature require a massive amount of queries for
attacking a single example. Oppositely, this work mainly focuses on the
evaluation of attack's power in the low queries regime $\leq 1000$ queries)
with $\ell_2$ norm in the hard-label settings. We find that the DeltaBound
attack performs as well and sometimes better than current state-of-the-art
attacks while remaining competitive across different kinds of models. Moreover,
we evaluate our method against not only deep neural networks, but also non-deep
learning models, such as Gradient Boosting Decision Trees and Multinomial Naive
Bayes.


http://arxiv.org/abs/2210.00008
Adversarial Attacks on Transformers-Based Malware Detectors. (91%)
Yash Jakhotiya; Heramb Patil; Jugal Rawlani; Dr. Sunil B. Mane
  Signature-based malware detectors have proven to be insufficient as even a
small change in malignant executable code can bypass these signature-based
detectors. Many machine learning-based models have been proposed to efficiently
detect a wide variety of malware. Many of these models are found to be
susceptible to adversarial attacks - attacks that work by generating
intentionally designed inputs that can force these models to misclassify. Our
work aims to explore vulnerabilities in the current state of the art malware
detectors to adversarial attacks. We train a Transformers-based malware
detector, carry out adversarial attacks resulting in a misclassification rate
of 23.9% and propose defenses that reduce this misclassification rate to half.
An implementation of our work can be found at
https://github.com/yashjakhotiya/Adversarial-Attacks-On-Transformers.


http://arxiv.org/abs/2210.00417
Voice Spoofing Countermeasures: Taxonomy, State-of-the-art, experimental analysis of generalizability, open challenges, and the way forward. (5%)
Awais Khan; Khalid Mahmood Malik; James Ryan; Mikul Saravanan
  Malicious actors may seek to use different voice-spoofing attacks to fool ASV
systems and even use them for spreading misinformation. Various countermeasures
have been proposed to detect these spoofing attacks. Due to the extensive work
done on spoofing detection in automated speaker verification (ASV) systems in
the last 6-7 years, there is a need to classify the research and perform
qualitative and quantitative comparisons on state-of-the-art countermeasures.
Additionally, no existing survey paper has reviewed integrated solutions to
voice spoofing evaluation and speaker verification, adversarial/antiforensics
attacks on spoofing countermeasures, and ASV itself, or unified solutions to
detect multiple attacks using a single model. Further, no work has been done to
provide an apples-to-apples comparison of published countermeasures in order to
assess their generalizability by evaluating them across corpora. In this work,
we conduct a review of the literature on spoofing detection using hand-crafted
features, deep learning, end-to-end, and universal spoofing countermeasure
solutions to detect speech synthesis (SS), voice conversion (VC), and replay
attacks. Additionally, we also review integrated solutions to voice spoofing
evaluation and speaker verification, adversarial and anti-forensics attacks on
voice countermeasures, and ASV. The limitations and challenges of the existing
spoofing countermeasures are also presented. We report the performance of these
countermeasures on several datasets and evaluate them across corpora. For the
experiments, we employ the ASVspoof2019 and VSDC datasets along with GMM, SVM,
CNN, and CNN-GRU classifiers. (For reproduceability of the results, the code of
the test bed can be found in our GitHub Repository.


http://arxiv.org/abs/2209.15246
Your Out-of-Distribution Detection Method is Not Robust! (99%)
Mohammad Azizmalayeri; Arshia Soltani Moakhar; Arman Zarei; Reihaneh Zohrabi; Mohammad Taghi Manzuri; Mohammad Hossein Rohban
  Out-of-distribution (OOD) detection has recently gained substantial attention
due to the importance of identifying out-of-domain samples in reliability and
safety. Although OOD detection methods have advanced by a great deal, they are
still susceptible to adversarial examples, which is a violation of their
purpose. To mitigate this issue, several defenses have recently been proposed.
Nevertheless, these efforts remained ineffective, as their evaluations are
based on either small perturbation sizes, or weak attacks. In this work, we
re-examine these defenses against an end-to-end PGD attack on in/out data with
larger perturbation sizes, e.g. up to commonly used $\epsilon=8/255$ for the
CIFAR-10 dataset. Surprisingly, almost all of these defenses perform worse than
a random detection under the adversarial setting. Next, we aim to provide a
robust OOD detection method. In an ideal defense, the training should expose
the model to almost all possible adversarial perturbations, which can be
achieved through adversarial training. That is, such training perturbations
should based on both in- and out-of-distribution samples. Therefore, unlike OOD
detection in the standard setting, access to OOD, as well as in-distribution,
samples sounds necessary in the adversarial training setup. These tips lead us
to adopt generative OOD detection methods, such as OpenGAN, as a baseline. We
subsequently propose the Adversarially Trained Discriminator (ATD), which
utilizes a pre-trained robust model to extract robust features, and a generator
model to create OOD samples. Using ATD with CIFAR-10 and CIFAR-100 as the
in-distribution data, we could significantly outperform all previous methods in
the robust AUROC while maintaining high standard AUROC and classification
accuracy. The code repository is available at https://github.com/rohban-lab/ATD .


http://arxiv.org/abs/2210.00062
Learning Robust Kernel Ensembles with Kernel Average Pooling. (99%)
Pouya Bashivan; Adam Ibrahim; Amirozhan Dehghani; Yifei Ren
  Model ensembles have long been used in machine learning to reduce the
variance in individual model predictions, making them more robust to input
perturbations. Pseudo-ensemble methods like dropout have also been commonly
used in deep learning models to improve generalization. However, the
application of these techniques to improve neural networks' robustness against
input perturbations remains underexplored. We introduce Kernel Average Pool
(KAP), a new neural network building block that applies the mean filter along
the kernel dimension of the layer activation tensor. We show that ensembles of
kernels with similar functionality naturally emerge in convolutional neural
networks equipped with KAP and trained with backpropagation. Moreover, we show
that when combined with activation noise, KAP models are remarkably robust
against various forms of adversarial attacks. Empirical evaluations on CIFAR10,
CIFAR100, TinyImagenet, and Imagenet datasets show substantial improvements in
robustness against strong adversarial attacks such as AutoAttack that are on
par with adversarially trained networks but are importantly obtained without
training on any adversarial examples.


http://arxiv.org/abs/2210.00122
Adversarial Robustness of Representation Learning for Knowledge Graphs. (95%)
Peru Bhardwaj
  Knowledge graphs represent factual knowledge about the world as relationships
between concepts and are critical for intelligent decision making in enterprise
applications. New knowledge is inferred from the existing facts in the
knowledge graphs by encoding the concepts and relations into low-dimensional
feature vector representations. The most effective representations for this
task, called Knowledge Graph Embeddings (KGE), are learned through neural
network architectures. Due to their impressive predictive performance, they are
increasingly used in high-impact domains like healthcare, finance and
education. However, are the black-box KGE models adversarially robust for use
in domains with high stakes? This thesis argues that state-of-the-art KGE
models are vulnerable to data poisoning attacks, that is, their predictive
performance can be degraded by systematically crafted perturbations to the
training knowledge graph. To support this argument, two novel data poisoning
attacks are proposed that craft input deletions or additions at training time
to subvert the learned model's performance at inference time. These adversarial
attacks target the task of predicting the missing facts in knowledge graphs
using KGE models, and the evaluation shows that the simpler attacks are
competitive with or outperform the computationally expensive ones. The thesis
contributions not only highlight and provide an opportunity to fix the security
vulnerabilities of KGE models, but also help to understand the black-box
predictive behaviour of KGE models.


http://arxiv.org/abs/2209.15304
Visual Privacy Protection Based on Type-I Adversarial Attack. (92%)
Zhigang Su; Dawei Zhou; Decheng Liu; Nannan Wang; Zhen Wang; Xinbo Gao
  With the development of online artificial intelligence systems, many deep
neural networks (DNNs) have been deployed in cloud environments. In practical
applications, developers or users need to provide their private data to DNNs,
such as faces. However, data transmitted and stored in the cloud is insecure
and at risk of privacy leakage. In this work, inspired by Type-I adversarial
attack, we propose an adversarial attack-based method to protect visual privacy
of data. Specifically, the method encrypts the visual information of private
data while maintaining them correctly predicted by DNNs, without modifying the
model parameters. The empirical results on face recognition tasks show that the
proposed method can deeply hide the visual information in face images and
hardly affect the accuracy of the recognition models. In addition, we further
extend the method to classification tasks and also achieve state-of-the-art
performance.


http://arxiv.org/abs/2210.00178
On the tightness of linear relaxation based robustness certification methods. (78%)
Cheng Tang
  There has been a rapid development and interest in adversarial training and
defenses in the machine learning community in the recent years. One line of
research focuses on improving the performance and efficiency of adversarial
robustness certificates for neural networks \cite{gowal:19, wong_zico:18,
raghunathan:18, WengTowardsFC:18, wong:scalable:18, singh:convex_barrier:19,
Huang_etal:19, single-neuron-relax:20, Zhang2020TowardsSA}. While each
providing a certification to lower (or upper) bound the true distortion under
adversarial attacks via relaxation, less studied was the tightness of
relaxation. In this paper, we analyze a family of linear outer approximation
based certificate methods via a meta algorithm, IBP-Lin. The aforementioned
works often lack quantitative analysis to answer questions such as how does the
performance of the certificate method depend on the network configuration and
the choice of approximation parameters. Under our framework, we make a first
attempt at answering these questions, which reveals that the tightness of
linear approximation based certification can depend heavily on the
configuration of the trained networks.


http://arxiv.org/abs/2209.15266
Data Poisoning Attacks Against Multimodal Encoders. (73%)
Ziqing Yang; Xinlei He; Zheng Li; Michael Backes; Mathias Humbert; Pascal Berrang; Yang Zhang
  Traditional machine learning (ML) models usually rely on large-scale labeled
datasets to achieve strong performance. However, such labeled datasets are
often challenging and expensive to obtain. Also, the predefined categories
limit the model's ability to generalize to other visual concepts as additional
labeled data is required. On the contrary, the newly emerged multimodal model,
which contains both visual and linguistic modalities, learns the concept of
images from the raw text. It is a promising way to solve the above problems as
it can use easy-to-collect image-text pairs to construct the training dataset
and the raw texts contain almost unlimited categories according to their
semantics. However, learning from a large-scale unlabeled dataset also exposes
the model to the risk of potential poisoning attacks, whereby the adversary
aims to perturb the model's training dataset to trigger malicious behaviors in
it. Previous work mainly focuses on the visual modality. In this paper, we
instead focus on answering two questions: (1) Is the linguistic modality also
vulnerable to poisoning attacks? and (2) Which modality is most vulnerable? To
answer the two questions, we conduct three types of poisoning attacks against
CLIP, the most representative multimodal contrastive learning framework.
Extensive evaluations on different datasets and model architectures show that
all three attacks can perform well on the linguistic modality with only a
relatively low poisoning rate and limited epochs. Also, we observe that the
poisoning effect differs between different modalities, i.e., with lower MinRank
in the visual modality and with higher Hit@K when K is small in the linguistic
modality. To mitigate the attacks, we propose both pre-training and
post-training defenses. We empirically show that both defenses can
significantly reduce the attack performance while preserving the model's
utility.


http://arxiv.org/abs/2210.00108
ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks. (70%)
Tim Clifford; Ilia Shumailov; Yiren Zhao; Ross Anderson; Robert Mullins
  Early backdoor attacks against machine learning set off an arms race in
attack and defence development. Defences have since appeared demonstrating some
ability to detect backdoors in models or even remove them. These defences work
by inspecting the training data, the model, or the integrity of the training
procedure. In this work, we show that backdoors can be added during
compilation, circumventing any safeguards in the data preparation and model
training stages. As an illustration, the attacker can insert weight-based
backdoors during the hardware compilation step that will not be detected by any
training or data-preparation process. Next, we demonstrate that some backdoors,
such as ImpNet, can only be reliably detected at the stage where they are
inserted and removing them anywhere else presents a significant challenge. We
conclude that machine-learning model security requires assurance of provenance
along the entire technical pipeline, including the data, model architecture,
compiler, and hardware specification.


http://arxiv.org/abs/2209.15179
Physical Adversarial Attack meets Computer Vision: A Decade Survey. (99%)
Hui Wei; Hao Tang; Xuemei Jia; Hanxun Yu; Zhubo Li; Zhixiang Wang; Shin'ichi Satoh; Zheng Wang
  Although Deep Neural Networks (DNNs) have achieved impressive results in
computer vision, their exposed vulnerability to adversarial attacks remains a
serious concern. A series of works has shown that by adding elaborate
perturbations to images, DNNs could have catastrophic degradation in
performance metrics. And this phenomenon does not only exist in the digital
space but also in the physical space. Therefore, estimating the security of
these DNNs-based systems is critical for safely deploying them in the real
world, especially for security-critical applications, e.g., autonomous cars,
video surveillance, and medical diagnosis. In this paper, we focus on physical
adversarial attacks and provide a comprehensive survey of over 150 existing
papers. We first clarify the concept of the physical adversarial attack and
analyze its characteristics. Then, we define the adversarial medium, essential
to perform attacks in the physical world. Next, we present the physical
adversarial attack methods in task order: classification, detection, and
re-identification, and introduce their performance in solving the trilemma:
effectiveness, stealthiness, and robustness. In the end, we discuss the current
challenges and potential future directions.


http://arxiv.org/abs/2209.14826
Towards Lightweight Black-Box Attacks against Deep Neural Networks. (99%)
Chenghao Sun; Yonggang Zhang; Wan Chaoqun; Qizhou Wang; Ya Li; Tongliang Liu; Bo Han; Xinmei Tian
  Black-box attacks can generate adversarial examples without accessing the
parameters of target model, largely exacerbating the threats of deployed deep
neural networks (DNNs). However, previous works state that black-box attacks
fail to mislead target models when their training data and outputs are
inaccessible. In this work, we argue that black-box attacks can pose practical
attacks in this extremely restrictive scenario where only several test samples
are available. Specifically, we find that attacking the shallow layers of DNNs
trained on a few test samples can generate powerful adversarial examples. As
only a few samples are required, we refer to these attacks as lightweight
black-box attacks. The main challenge to promoting lightweight attacks is to
mitigate the adverse impact caused by the approximation error of shallow
layers. As it is hard to mitigate the approximation error with few available
samples, we propose Error TransFormer (ETF) for lightweight attacks. Namely,
ETF transforms the approximation error in the parameter space into a
perturbation in the feature space and alleviates the error by disturbing
features. In experiments, lightweight black-box attacks with the proposed ETF
achieve surprising results. For example, even if only 1 sample per category
available, the attack success rate in lightweight black-box attacks is only
about 3% lower than that of the black-box attacks with complete training data.


http://arxiv.org/abs/2209.15042
Generalizability of Adversarial Robustness Under Distribution Shifts. (83%)
Kumail Alhamoud; Hasan Abed Al Kader Hammoud; Motasem Alfarra; Bernard Ghanem
  Recent progress in empirical and certified robustness promises to deliver
reliable and deployable Deep Neural Networks (DNNs). Despite that success, most
existing evaluations of DNN robustness have been done on images sampled from
the same distribution on which the model was trained. However, in the real
world, DNNs may be deployed in dynamic environments that exhibit significant
distribution shifts. In this work, we take a first step towards thoroughly
investigating the interplay between empirical and certified adversarial
robustness on one hand and domain generalization on another. To do so, we train
robust models on multiple domains and evaluate their accuracy and robustness on
an unseen domain. We observe that: (1) both empirical and certified robustness
generalize to unseen domains, and (2) the level of generalizability does not
correlate well with input visual similarity, measured by the FID between source
and target domains. We also extend our study to cover a real-world medical
application, in which adversarial augmentation significantly boosts the
generalization of robustness with minimal effect on clean data accuracy.


http://arxiv.org/abs/2209.14692
Digital and Physical Face Attacks: Reviewing and One Step Further. (2%)
Chenqi Kong; Shiqi Wang; Haoliang Li
  With the rapid progress over the past five years, face authentication has
become the most pervasive biometric recognition method. Thanks to the
high-accuracy recognition performance and user-friendly usage, automatic face
recognition (AFR) has exploded into a plethora of practical applications over
device unlocking, checking-in, and financial payment. In spite of the
tremendous success of face authentication, a variety of face presentation
attacks (FPA), such as print attacks, replay attacks, and 3D mask attacks, have
raised pressing mistrust concerns. Besides physical face attacks, face
videos/images are vulnerable to a wide variety of digital attack techniques
launched by malicious hackers, causing potential menace to the public at large.
Due to the unrestricted access to enormous digital face images/videos and
disclosed easy-to-use face manipulation tools circulating on the internet,
non-expert attackers without any prior professional skills are able to readily
create sophisticated fake faces, leading to numerous dangerous applications
such as financial fraud, impersonation, and identity theft. This survey aims to
build the integrity of face forensics by providing thorough analyses of
existing literature and highlighting the issues requiring further attention. In
this paper, we first comprehensively survey both physical and digital face
attack types and datasets. Then, we review the latest and most advanced
progress on existing counter-attack methodologies and highlight their current
limits. Moreover, we outline possible future research directions for existing
and upcoming challenges in the face forensics community. Finally, the necessity
of joint physical and digital face attack detection has been discussed, which
has never been studied in previous surveys.


http://arxiv.org/abs/2209.14673
Chameleon Cache: Approximating Fully Associative Caches with Random Replacement to Prevent Contention-Based Cache Attacks. (1%)
Thomas Unterluggauer; Austin Harris; Scott Constable; Fangfei Liu; Carlos Rozas
  Randomized, skewed caches (RSCs) such as CEASER-S have recently received much
attention to defend against contention-based cache side channels. By
randomizing and regularly changing the mapping(s) of addresses to cache sets,
these techniques are designed to obfuscate the leakage of memory access
patterns. However, new attack techniques, e.g., Prime+Prune+Probe, soon
demonstrated the limits of RSCs as they allow attackers to more quickly learn
which addresses contend in the cache and use this information to circumvent the
randomization. To yet maintain side-channel resilience, RSCs must change the
random mapping(s) more frequently with adverse effects on performance and
implementation complexity. This work aims to make randomization-based
approaches more robust to allow for reduced re-keying rates and presents
Chameleon Cache. Chameleon Cache extends RSCs with a victim cache (VC) to
decouple contention in the RSC from evictions observed by the user. The VC
allows Chameleon Cache to make additional use of the multiple mappings RSCs
provide to translate addresses to cache set indices: when a cache line is
evicted from the RSC to the VC under one of its mappings, the VC automatically
reinserts this evicted line back into the RSC by using a different mapping. As
a result, the effects of previous RSC set contention are hidden and Chameleon
Cache exhibits side-channel resistance and eviction patterns similar to fully
associative caches with random replacement. We show that Chameleon Cache has
performance overheads of < 1% and stress that VCs are more generically helpful
to increase side-channel resistance and re-keying intervals of randomized
caches.


http://arxiv.org/abs/2209.14262
A Survey on Physical Adversarial Attack in Computer Vision. (99%)
Donghua Wang; Wen Yao; Tingsong Jiang; Guijiang Tang; Xiaoqian Chen
  In the past decade, deep learning has dramatically changed the traditional
hand-craft feature manner with strong feature learning capability, resulting in
tremendous improvement of conventional tasks. However, deep neural networks
have recently been demonstrated vulnerable to adversarial examples, a kind of
malicious samples crafted by small elaborately designed noise, which mislead
the DNNs to make the wrong decisions while remaining imperceptible to humans.
Adversarial examples can be divided into digital adversarial attacks and
physical adversarial attacks. The digital adversarial attack is mostly
performed in lab environments, focusing on improving the performance of
adversarial attack algorithms. In contrast, the physical adversarial attack
focus on attacking the physical world deployed DNN systems, which is a more
challenging task due to the complex physical environment (i.e., brightness,
occlusion, and so on). Although the discrepancy between digital adversarial and
physical adversarial examples is small, the physical adversarial examples have
a specific design to overcome the effect of the complex physical environment.
In this paper, we review the development of physical adversarial attacks in
DNN-based computer vision tasks, including image recognition tasks, object
detection tasks, and semantic segmentation. For the sake of completeness of the
algorithm evolution, we will briefly introduce the works that do not involve
the physical adversarial attack. We first present a categorization scheme to
summarize the current physical adversarial attacks. Then discuss the advantages
and disadvantages of the existing physical adversarial attacks and focus on the
technique used to maintain the adversarial when applied into physical
environment. Finally, we point out the issues of the current physical
adversarial attacks to be solved and provide promising research directions.


http://arxiv.org/abs/2209.14105
Exploring the Relationship between Architecture and Adversarially Robust Generalization. (98%)
Shiyu Tang; Siyuan Liang; Ruihao Gong; Aishan Liu; Xianglong Liu; Dacheng Tao
  Adversarial training has been demonstrated to be one of the most effective
remedies for defending adversarial examples, yet it often suffers from the huge
robustness generalization gap on unseen testing adversaries, deemed as the
\emph{adversarially robust generalization problem}. Despite the preliminary
understandings devoted on adversarially robust generalization, little is known
from the architectural perspective. Thus, this paper tries to bridge the gap by
systematically examining the most representative architectures (e.g., Vision
Transformers and CNNs). In particular, we first comprehensively evaluated
\emph{20} adversarially trained architectures on ImageNette and CIFAR-10
datasets towards several adversaries (multiple $\ell_p$-norm adversarial
attacks), and found that Vision Transformers (e.g., PVT, CoAtNet) often yield
better adversarially robust generalization. To further understand what
architectural ingredients favor adversarially robust generalization, we delve
into several key building blocks and revealed the fact via the lens of
Rademacher complexity that the higher weight sparsity contributes significantly
towards the better adversarially robust generalization of Vision Transformers,
which can be often achieved by attention layers. Our extensive studies
discovered the close relationship between architectural design and
adversarially robust generalization, and instantiated several important
insights. We hope our findings could help to better understand the mechanism
towards designing robust deep learning architectures.


http://arxiv.org/abs/2209.14243
A Closer Look at Evaluating the Bit-Flip Attack Against Deep Neural Networks. (67%)
Kevin Hector; Mathieu Dumont; Pierre-Alain Moellic; Jean-Max Dutertre
  Deep neural network models are massively deployed on a wide variety of
hardware platforms. This results in the appearance of new attack vectors that
significantly extend the standard attack surface, extensively studied by the
adversarial machine learning community. One of the first attack that aims at
drastically dropping the performance of a model, by target