http://arxiv.org/abs/2110.14430
Adversarial Neuron Pruning Purifies Backdoored Deep Models. (15%)
Dongxian Wu; Yisen Wang
  As deep neural networks (DNNs) are growing larger, their requirements for
computational resources become huge, which makes outsourcing training more
popular. Training in a third-party platform, however, may introduce potential
risks that a malicious trainer will return backdoored DNNs, which behave
normally on clean samples but output targeted misclassifications whenever a
trigger appears at the test time. Without any knowledge of the trigger, it is
difficult to distinguish or recover benign DNNs from backdoored ones. In this
paper, we first identify an unexpected sensitivity of backdoored DNNs, that is,
they are much easier to collapse and tend to predict the target label on clean
samples when their neurons are adversarially perturbed. Based on these
observations, we propose a novel model repairing method, termed Adversarial
Neuron Pruning (ANP), which prunes some sensitive neurons to purify the
injected backdoor. Experiments show, even with only an extremely small amount
of clean data (e.g., 1%), ANP effectively removes the injected backdoor without
causing obvious performance degradation.


http://arxiv.org/abs/2110.14189
Robust Contrastive Learning Using Negative Samples with Diminished Semantics. (1%)
Songwei Ge; Shlok Mishra; Haohan Wang; Chun-Liang Li; David Jacobs
  Unsupervised learning has recently made exceptional progress because of the
development of more effective contrastive learning methods. However, CNNs are
prone to depend on low-level features that humans deem non-semantic. This
dependency has been conjectured to induce a lack of robustness to image
perturbations or domain shift. In this paper, we show that by generating
carefully designed negative samples, contrastive learning can learn more robust
representations with less dependence on such features. Contrastive learning
utilizes positive pairs that preserve semantic information while perturbing
superficial features in the training images. Similarly, we propose to generate
negative samples in a reversed way, where only the superfluous instead of the
semantic features are preserved. We develop two methods, texture-based and
patch-based augmentations, to generate negative samples. These samples achieve
better generalization, especially under out-of-domain settings. We also analyze
our method and the generated texture-based samples, showing that texture
features are indispensable in classifying particular ImageNet classes and
especially finer classes. We also show that model bias favors texture and shape
features differently under different test settings. Our code, trained models,
and ImageNet-Texture dataset can be found at
https://github.com/SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives.


http://arxiv.org/abs/2110.14188
RoMA: Robust Model Adaptation for Offline Model-based Optimization. (1%)
Sihyun Yu; Sungsoo Ahn; Le Song; Jinwoo Shin
  We consider the problem of searching an input maximizing a black-box
objective function given a static dataset of input-output queries. A popular
approach to solving this problem is maintaining a proxy model, e.g., a deep
neural network (DNN), that approximates the true objective function. Here, the
main challenge is how to avoid adversarially optimized inputs during the
search, i.e., the inputs where the DNN highly overestimates the true objective
function. To handle the issue, we propose a new framework, coined robust model
adaptation (RoMA), based on gradient-based optimization of inputs over the DNN.
Specifically, it consists of two steps: (a) a pre-training strategy to robustly
train the proxy model and (b) a novel adaptation procedure of the proxy model
to have robust estimates for a specific set of candidate solutions. At a high
level, our scheme utilizes the local smoothness prior to overcome the
brittleness of the DNN. Experiments under various tasks show the effectiveness
of RoMA compared with previous methods, obtaining state-of-the-art results,
e.g., RoMA outperforms all at 4 out of 6 tasks and achieves runner-up results
at the remaining tasks.


http://arxiv.org/abs/2110.14120
ScaleCert: Scalable Certified Defense against Adversarial Patches with Sparse Superficial Layers. (99%)
Husheng Han; Kaidi Xu; Xing Hu; Xiaobing Chen; Ling Liang; Zidong Du; Qi Guo; Yanzhi Wang; Yunji Chen
  Adversarial patch attacks that craft the pixels in a confined region of the
input images show their powerful attack effectiveness in physical environments
even with noises or deformations. Existing certified defenses towards
adversarial patch attacks work well on small images like MNIST and CIFAR-10
datasets, but achieve very poor certified accuracy on higher-resolution images
like ImageNet. It is urgent to design both robust and effective defenses
against such a practical and harmful attack in industry-level larger images. In
this work, we propose the certified defense methodology that achieves high
provable robustness for high-resolution images and largely improves the
practicality for real adoption of the certified defense. The basic insight of
our work is that the adversarial patch intends to leverage localized
superficial important neurons (SIN) to manipulate the prediction results.
Hence, we leverage the SIN-based DNN compression techniques to significantly
improve the certified accuracy, by reducing the adversarial region searching
overhead and filtering the prediction noises. Our experimental results show
that the certified accuracy is increased from 36.3% (the state-of-the-art
certified detection) to 60.4% on the ImageNet dataset, largely pushing the
certified defenses for practical use.


http://arxiv.org/abs/2110.13950
Can't Fool Me: Adversarially Robust Transformer for Video Understanding. (99%)
Divya Choudhary; Palash Goyal; Saurabh Sahu
  Deep neural networks have been shown to perform poorly on adversarial
examples. To address this, several techniques have been proposed to increase
robustness of a model for image classification tasks. However, in video
understanding tasks, developing adversarially robust models is still
unexplored. In this paper, we aim to bridge this gap. We first show that simple
extensions of image based adversarially robust models slightly improve the
worst-case performance. Further, we propose a temporal attention regularization
scheme in Transformer to improve the robustness of attention modules to
adversarial examples. We illustrate using a large-scale video data set
YouTube-8M that the final model (A-ART) achieves close to non-adversarial
performance on its adversarial example set. We achieve 91% GAP on adversarial
examples, whereas baseline Transformer and simple adversarial extensions
achieve 72.9% and 82% respectively, showing significant improvement in
robustness over the state-of-the-art.


http://arxiv.org/abs/2110.14068
Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks. (99%)
Yonggan Fu; Qixuan Yu; Yang Zhang; Shang Wu; Xu Ouyang; David Cox; Yingyan Lin
  Deep Neural Networks (DNNs) are known to be vulnerable to adversarial
attacks, i.e., an imperceptible perturbation to the input can mislead DNNs
trained on clean images into making erroneous predictions. To tackle this,
adversarial training is currently the most effective defense method, by
augmenting the training set with adversarial samples generated on the fly.
Interestingly, we discover for the first time that there exist subnetworks with
inborn robustness, matching or surpassing the robust accuracy of the
adversarially trained networks with comparable model sizes, within randomly
initialized networks without any model training, indicating that adversarial
training on model weights is not indispensable towards adversarial robustness.
We name such subnetworks Robust Scratch Tickets (RSTs), which are also by
nature efficient. Distinct from the popular lottery ticket hypothesis, neither
the original dense networks nor the identified RSTs need to be trained. To
validate and understand this fascinating finding, we further conduct extensive
experiments to study the existence and properties of RSTs under different
models, datasets, sparsity patterns, and attacks, drawing insights regarding
the relationship between DNNs' robustness and their
initialization/overparameterization. Furthermore, we identify the poor
adversarial transferability between RSTs of different sparsity ratios drawn
from the same randomly initialized dense network, and propose a Random RST
Switch (R2S) technique, which randomly switches between different RSTs, as a
novel defense method built on top of RSTs. We believe our findings about RSTs
have opened up a new perspective to study model robustness and extend the
lottery ticket hypothesis.


http://arxiv.org/abs/2110.13935
Frequency Centric Defense Mechanisms against Adversarial Examples. (99%)
Sanket B. Shah; Param Raval; Harin Khakhi; Mehul S. Raval
  Adversarial example (AE) aims at fooling a Convolution Neural Network by
introducing small perturbations in the input image.The proposed work uses the
magnitude and phase of the Fourier Spectrum and the entropy of the image to
defend against AE. We demonstrate the defense in two ways: by training an
adversarial detector and denoising the adversarial effect. Experiments were
conducted on the low-resolution CIFAR-10 and high-resolution ImageNet datasets.
The adversarial detector has 99% accuracy for FGSM and PGD attacks on the
CIFAR-10 dataset. However, the detection accuracy falls to 50% for
sophisticated DeepFool and Carlini & Wagner attacks on ImageNet. We overcome
the limitation by using autoencoder and show that 70% of AEs are correctly
classified after denoising.


http://arxiv.org/abs/2110.13864
FL-WBC: Enhancing Robustness against Model Poisoning Attacks in Federated Learning from a Client Perspective. (98%)
Jingwei Sun; Ang Li; Louis DiValentin; Amin Hassanzadeh; Yiran Chen; Hai Li
  Federated learning (FL) is a popular distributed learning framework that
trains a global model through iterative communications between a central server
and edge devices. Recent works have demonstrated that FL is vulnerable to model
poisoning attacks. Several server-based defense approaches (e.g. robust
aggregation), have been proposed to mitigate such attacks. However, we
empirically show that under extremely strong attacks, these defensive methods
fail to guarantee the robustness of FL. More importantly, we observe that as
long as the global model is polluted, the impact of attacks on the global model
will remain in subsequent rounds even if there are no subsequent attacks. In
this work, we propose a client-based defense, named White Blood Cell for
Federated Learning (FL-WBC), which can mitigate model poisoning attacks that
have already polluted the global model. The key idea of FL-WBC is to identify
the parameter space where long-lasting attack effect on parameters resides and
perturb that space during local training. Furthermore, we derive a certified
robustness guarantee against model poisoning attacks and a convergence
guarantee to FedAvg after applying our FL-WBC. We conduct experiments on
FasionMNIST and CIFAR10 to evaluate the defense against state-of-the-art model
poisoning attacks. The results demonstrate that our method can effectively
mitigate model poisoning attack impact on the global model within 5
communication rounds with nearly no accuracy drop under both IID and Non-IID
settings. Our defense is also complementary to existing server-based robust
aggregation approaches and can further improve the robustness of FL under
extremely strong attacks.


http://arxiv.org/abs/2110.13741
Disrupting Deep Uncertainty Estimation Without Harming Accuracy. (86%)
Ido Galil; Ran El-Yaniv
  Deep neural networks (DNNs) have proven to be powerful predictors and are
widely used for various tasks. Credible uncertainty estimation of their
predictions, however, is crucial for their deployment in many risk-sensitive
applications. In this paper we present a novel and simple attack, which unlike
adversarial attacks, does not cause incorrect predictions but instead cripples
the network's capacity for uncertainty estimation. The result is that after the
attack, the DNN is more confident of its incorrect predictions than about its
correct ones without having its accuracy reduced. We present two versions of
the attack. The first scenario focuses on a black-box regime (where the
attacker has no knowledge of the target network) and the second scenario
attacks a white-box setting. The proposed attack is only required to be of
minuscule magnitude for its perturbations to cause severe uncertainty
estimation damage, with larger magnitudes resulting in completely unusable
uncertainty estimations. We demonstrate successful attacks on three of the most
popular uncertainty estimation methods: the vanilla softmax score, Deep
Ensembles and MC-Dropout. Additionally, we show an attack on SelectiveNet, the
selective classification architecture. We test the proposed attack on several
contemporary architectures such as MobileNetV2 and EfficientNetB0, all trained
to classify ImageNet.


http://arxiv.org/abs/2110.14030
Improving Local Effectiveness for Global robust training. (83%)
Jingyue Lu; M. Pawan Kumar
  Despite its popularity, deep neural networks are easily fooled. To alleviate
this deficiency, researchers are actively developing new training strategies,
which encourage models that are robust to small input perturbations. Several
successful robust training methods have been proposed. However, many of them
rely on strong adversaries, which can be prohibitively expensive to generate
when the input dimension is high and the model structure is complicated. We
adopt a new perspective on robustness and propose a novel training algorithm
that allows a more effective use of adversaries. Our method improves the model
robustness at each local ball centered around an adversary and then, by
combining these local balls through a global term, achieves overall robustness.
We demonstrate that, by maximizing the use of adversaries via focusing on local
balls, we achieve high robust accuracy with weak adversaries. Specifically, our
method reaches a similar robust accuracy level to the state of the art
approaches trained on strong adversaries on MNIST, CIFAR-10 and CIFAR-100. As a
result, the overall training time is reduced. Furthermore, when trained with
strong adversaries, our method matches with the current state of the art on
MNIST and outperforms them on CIFAR-10 and CIFAR-100.


http://arxiv.org/abs/2110.14038
Robustness of Graph Neural Networks at Scale. (76%)
Simon Geisler; Tobias Schmidt; Hakan Şirin; Daniel Zügner; Aleksandar Bojchevski; Stephan Günnemann
  Graph Neural Networks (GNNs) are increasingly important given their
popularity and the diversity of applications. Yet, existing studies of their
vulnerability to adversarial attacks rely on relatively small graphs. We
address this gap and study how to attack and defend GNNs at scale. We propose
two sparsity-aware first-order optimization attacks that maintain an efficient
representation despite optimizing over a number of parameters which is
quadratic in the number of nodes. We show that common surrogate losses are not
well-suited for global attacks on GNNs. Our alternatives can double the attack
strength. Moreover, to improve GNNs' reliability we design a robust aggregation
function, Soft Median, resulting in an effective defense at all scales. We
evaluate our attacks and defense with standard GNNs on graphs more than 100
times larger compared to previous work. We even scale one order of magnitude
further by extending our techniques to a scalable GNN.


http://arxiv.org/abs/2110.13980
Adversarial Attacks and Defenses for Social Network Text Processing Applications: Techniques, Challenges and Future Research Directions. (75%)
Izzat Alsmadi; Kashif Ahmad; Mahmoud Nazzal; Firoj Alam; Ala Al-Fuqaha; Abdallah Khreishah; Abdulelah Algosaibi
  The growing use of social media has led to the development of several Machine
Learning (ML) and Natural Language Processing(NLP) tools to process the
unprecedented amount of social media content to make actionable decisions.
However, these MLand NLP algorithms have been widely shown to be vulnerable to
adversarial attacks. These vulnerabilities allow adversaries to launch a
diversified set of adversarial attacks on these algorithms in different
applications of social media text processing. In this paper, we provide a
comprehensive review of the main approaches for adversarial attacks and
defenses in the context of social media applications with a particular focus on
key challenges and future research directions. In detail, we cover literature
on six key applications, namely (i) rumors detection, (ii) satires detection,
(iii) clickbait & spams identification, (iv) hate speech detection,
(v)misinformation detection, and (vi) sentiment analysis. We then highlight the
concurrent and anticipated future research questions and provide
recommendations and directions for future work.


http://arxiv.org/abs/2110.13771
AugMax: Adversarial Composition of Random Augmentations for Robust Training. (56%)
Haotao Wang; Chaowei Xiao; Jean Kossaifi; Zhiding Yu; Anima Anandkumar; Zhangyang Wang
  Data augmentation is a simple yet effective way to improve the robustness of
deep neural networks (DNNs). Diversity and hardness are two complementary
dimensions of data augmentation to achieve robustness. For example, AugMix
explores random compositions of a diverse set of augmentations to enhance
broader coverage, while adversarial training generates adversarially hard
samples to spot the weakness. Motivated by this, we propose a data augmentation
framework, termed AugMax, to unify the two aspects of diversity and hardness.
AugMax first randomly samples multiple augmentation operators and then learns
an adversarial mixture of the selected operators. Being a stronger form of data
augmentation, AugMax leads to a significantly augmented input distribution
which makes model training more challenging. To solve this problem, we further
design a disentangled normalization module, termed DuBIN
(Dual-Batch-and-Instance Normalization), that disentangles the instance-wise
feature heterogeneity arising from AugMax. Experiments show that AugMax-DuBIN
leads to significantly improved out-of-distribution robustness, outperforming
prior arts by 3.03%, 3.49%, 1.82% and 0.71% on CIFAR10-C, CIFAR100-C, Tiny
ImageNet-C and ImageNet-C. Codes and pretrained models are available:
https://github.com/VITA-Group/AugMax.


http://arxiv.org/abs/2110.13541
Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes. (50%)
Sanghyun Hong; Michael-Andrei Panaitescu-Liess; Yiğitcan Kaya; Tudor Dumitraş
  Quantization is a popular technique that $transforms$ the parameter
representation of a neural network from floating-point numbers into
lower-precision ones ($e.g.$, 8-bit integers). It reduces the memory footprint
and the computational cost at inference, facilitating the deployment of
resource-hungry models. However, the parameter perturbations caused by this
transformation result in $behavioral$ $disparities$ between the model before
and after quantization. For example, a quantized model can misclassify some
test-time samples that are otherwise classified correctly. It is not known
whether such differences lead to a new security vulnerability. We hypothesize
that an adversary may control this disparity to introduce specific behaviors
that activate upon quantization. To study this hypothesis, we weaponize
quantization-aware training and propose a new training framework to implement
adversarial quantization outcomes. Following this framework, we present three
attacks we carry out with quantization: (i) an indiscriminate attack for
significant accuracy loss; (ii) a targeted attack against specific samples; and
(iii) a backdoor attack for controlling the model with an input trigger. We
further show that a single compromised model defeats multiple quantization
schemes, including robust quantization techniques. Moreover, in a federated
learning scenario, we demonstrate that a set of malicious participants who
conspire can inject our quantization-activated backdoor. Lastly, we discuss
potential counter-measures and show that only re-training consistently removes
the attack artifacts. Our code is available at
https://github.com/Secure-AI-Systems-Group/Qu-ANTI-zation


http://arxiv.org/abs/2110.13414
Semantic Host-free Trojan Attack. (10%)
Haripriya Harikumar; Kien Do; Santu Rana; Sunil Gupta; Svetha Venkatesh
  In this paper, we propose a novel host-free Trojan attack with triggers that
are fixed in the semantic space but not necessarily in the pixel space. In
contrast to existing Trojan attacks which use clean input images as hosts to
carry small, meaningless trigger patterns, our attack considers triggers as
full-sized images belonging to a semantically meaningful object class. Since in
our attack, the backdoored classifier is encouraged to memorize the abstract
semantics of the trigger images than any specific fixed pattern, it can be
later triggered by semantically similar but different looking images. This
makes our attack more practical to be applied in the real-world and harder to
defend against. Extensive experimental results demonstrate that with only a
small number of Trojan patterns for training, our attack can generalize well to
new patterns of the same Trojan class and can bypass state-of-the-art defense
methods.


http://arxiv.org/abs/2110.14032
MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge. (1%)
Geng Yuan; Xiaolong Ma; Wei Niu; Zhengang Li; Zhenglun Kong; Ning Liu; Yifan Gong; Zheng Zhan; Chaoyang He; Qing Jin; Siyue Wang; Minghai Qin; Bin Ren; Yanzhi Wang; Sijia Liu; Xue Lin
  Recently, a new trend of exploring sparsity for accelerating neural network
training has emerged, embracing the paradigm of training on the edge. This
paper proposes a novel Memory-Economic Sparse Training (MEST) framework
targeting for accurate and fast execution on edge devices. The proposed MEST
framework consists of enhancements by Elastic Mutation (EM) and Soft Memory
Bound (&S) that ensure superior accuracy at high sparsity ratios. Different
from the existing works for sparse training, this current work reveals the
importance of sparsity schemes on the performance of sparse training in terms
of accuracy as well as training speed on real edge devices. On top of that, the
paper proposes to employ data efficiency for further acceleration of sparse
training. Our results suggest that unforgettable examples can be identified
in-situ even during the dynamic exploration of sparsity masks in the sparse
training process, and therefore can be removed for further training speedup on
edge devices. Comparing with state-of-the-art (SOTA) works on accuracy, our
MEST increases Top-1 accuracy significantly on ImageNet when using the same
unstructured sparsity scheme. Systematical evaluation on accuracy, training
speed, and memory footprint are conducted, where the proposed MEST framework
consistently outperforms representative SOTA works. A reviewer strongly against
our work based on his false assumptions and misunderstandings. On top of the
previous submission, we employ data efficiency for further acceleration of
sparse training. And we explore the impact of model sparsity, sparsity schemes,
and sparse training algorithms on the number of removable training examples.
Our codes are publicly available at: https://github.com/boone891214/MEST.


http://arxiv.org/abs/2110.14019
Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection. (1%)
Chunjong Park; Anas Awadalla; Tadayoshi Kohno; Shwetak Patel
  Unpredictable ML model behavior on unseen data, especially in the health
domain, raises serious concerns about its safety as repercussions for mistakes
can be fatal. In this paper, we explore the feasibility of using
state-of-the-art out-of-distribution detectors for reliable and trustworthy
diagnostic predictions. We select publicly available deep learning models
relating to various health conditions (e.g., skin cancer, lung sound, and
Parkinson's disease) using various input data types (e.g., image, audio, and
motion data). We demonstrate that these models show unreasonable predictions on
out-of-distribution datasets. We show that Mahalanobis distance- and Gram
matrices-based out-of-distribution detection methods are able to detect
out-of-distribution data with high accuracy for the health models that operate
on different modalities. We then translate the out-of-distribution score into a
human interpretable CONFIDENCE SCORE to investigate its effect on the users'
interaction with health ML applications. Our user study shows that the
\textsc{confidence score} helped the participants only trust the results with a
high score to make a medical decision and disregard results with a low score.
Through this work, we demonstrate that dataset shift is a critical piece of
information for high-stake ML applications, such as medical diagnosis and
healthcare, to provide reliable and trustworthy predictions to the users.


http://arxiv.org/abs/2110.13859
Defensive Tensorization. (1%)
Adrian Bulat; Jean Kossaifi; Sourav Bhattacharya; Yannis Panagakis; Timothy Hospedales; Georgios Tzimiropoulos; Nicholas D Lane; Maja Pantic
  We propose defensive tensorization, an adversarial defence technique that
leverages a latent high-order factorization of the network. The layers of a
network are first expressed as factorized tensor layers. Tensor dropout is then
applied in the latent subspace, therefore resulting in dense reconstructed
weights, without the sparsity or perturbations typically induced by the
randomization.Our approach can be readily integrated with any arbitrary neural
architecture and combined with techniques like adversarial training. We
empirically demonstrate the effectiveness of our approach on standard image
classification benchmarks. We validate the versatility of our approach across
domains and low-precision architectures by considering an audio classification
task and binary networks. In all cases, we demonstrate improved performance
compared to prior works.


http://arxiv.org/abs/2110.13409
Task-Aware Meta Learning-based Siamese Neural Network for Classifying Obfuscated Malware. (1%)
Jinting Zhu; Julian Jang-Jaccard; Amardeep Singh; Paul A. Watters; Seyit Camtepe
  Malware authors apply different obfuscation techniques on the generic feature
of malware (i.e., unique malware signature) to create new variants to avoid
detection. Existing Siamese Neural Network (SNN) based malware detection
methods fail to correctly classify different malware families when similar
generic features are shared across multiple malware variants resulting in high
false-positive rates. To address this issue, we propose a novel Task-Aware Meta
Learning-based Siamese Neural Network resilient against obfuscated malware
while able to detect malware trained with one or a few training samples. Using
entropy features of each malware signature alongside image features as task
inputs, our task-aware meta leaner generates the parameters for the feature
layers to more accurately adjust the feature embedding for different malware
families. In addition, our model utilizes meta-learning with the extracted
features of a pre-trained network (e.g., VGG-16) to avoid the bias typically
associated with a model trained with a limited number of training samples. Our
proposed approach is highly effective in recognizing unique malware signatures,
thus correctly classifying malware samples that belong to the same malware
family even in the presence of obfuscation technique applied to malware. Our
experimental results, validated with N-way on N-shot learning, show that our
model is highly effective in classification accuracy exceeding the rate>91%
compared to other similar methods.


http://arxiv.org/abs/2110.12976
Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks. (99%)
Qiyu Kang; Yang Song; Qinxu Ding; Wee Peng Tay
  Deep neural networks (DNNs) are well-known to be vulnerable to adversarial
attacks, where malicious human-imperceptible perturbations are included in the
input to the deep network to fool it into making a wrong classification. Recent
studies have demonstrated that neural Ordinary Differential Equations (ODEs)
are intrinsically more robust against adversarial attacks compared to vanilla
DNNs. In this work, we propose a stable neural ODE with Lyapunov-stable
equilibrium points for defending against adversarial attacks (SODEF). By
ensuring that the equilibrium points of the ODE solution used as part of SODEF
is Lyapunov-stable, the ODE solution for an input with a small perturbation
converges to the same solution as the unperturbed input. We provide theoretical
results that give insights into the stability of SODEF as well as the choice of
regularizers to ensure its stability. Our analysis suggests that our proposed
regularizers force the extracted feature points to be within a neighborhood of
the Lyapunov-stable equilibrium points of the ODE. SODEF is compatible with
many defense methods and can be applied to any neural network's final regressor
layer to enhance its stability against adversarial attacks.


http://arxiv.org/abs/2110.12948
Generating Watermarked Adversarial Texts. (99%)
Mingjie Li; Hanzhou Wu; Xinpeng Zhang
  Adversarial example generation has been a hot spot in recent years because it
can cause deep neural networks (DNNs) to misclassify the generated adversarial
examples, which reveals the vulnerability of DNNs, motivating us to find good
solutions to improve the robustness of DNN models. Due to the extensiveness and
high liquidity of natural language over the social networks, various natural
language based adversarial attack algorithms have been proposed in the
literature. These algorithms generate adversarial text examples with high
semantic quality. However, the generated adversarial text examples may be
maliciously or illegally used. In order to tackle with this problem, we present
a general framework for generating watermarked adversarial text examples. For
each word in a given text, a set of candidate words are determined to ensure
that all the words in the set can be used to either carry secret bits or
facilitate the construction of adversarial example. By applying a word-level
adversarial text generation algorithm, the watermarked adversarial text example
can be finally generated. Experiments show that the adversarial text examples
generated by the proposed method not only successfully fool advanced DNN
models, but also carry a watermark that can effectively verify the ownership
and trace the source of the adversarial examples. Moreover, the watermark can
still survive after attacked with adversarial example generation algorithms,
which has shown the applicability and superiority.


http://arxiv.org/abs/2110.13250
Beyond $L_p$ clipping: Equalization-based Psychoacoustic Attacks against ASRs. (92%)
Hadi Abdullah; Muhammad Sajidur Rahman; Christian Peeters; Cassidy Gibson; Washington Garcia; Vincent Bindschaedler; Thomas Shrimpton; Patrick Traynor
  Automatic Speech Recognition (ASR) systems convert speech into text and can
be placed into two broad categories: traditional and fully end-to-end. Both
types have been shown to be vulnerable to adversarial audio examples that sound
benign to the human ear but force the ASR to produce malicious transcriptions.
Of these attacks, only the "psychoacoustic" attacks can create examples with
relatively imperceptible perturbations, as they leverage the knowledge of the
human auditory system. Unfortunately, existing psychoacoustic attacks can only
be applied against traditional models, and are obsolete against the newer,
fully end-to-end ASRs. In this paper, we propose an equalization-based
psychoacoustic attack that can exploit both traditional and fully end-to-end
ASRs. We successfully demonstrate our attack against real-world ASRs that
include DeepSpeech and Wav2Letter. Moreover, we employ a user study to verify
that our method creates low audible distortion. Specifically, 80 of the 100
participants voted in favor of all our attack audio samples as less noisier
than the existing state-of-the-art attack. Through this, we demonstrate both
types of existing ASR pipelines can be exploited with minimum degradation to
attack audio quality.


http://arxiv.org/abs/2110.12690
Scalable Lipschitz Residual Networks with Convex Potential Flows. (81%)
Laurent Meunier; Blaise Delattre; Alexandre Araujo; Alexandre Allauzen
  The Lipschitz constant of neural networks has been established as a key
property to enforce the robustness of neural networks to adversarial examples.
However, recent attempts to build $1$-Lipschitz Neural Networks have all shown
limitations and robustness have to be traded for accuracy and scalability or
vice versa. In this work, we first show that using convex potentials in a
residual network gradient flow provides a built-in $1$-Lipschitz
transformation. From this insight, we leverage the work on Input Convex Neural
Networks to parametrize efficient layers with this property. A comprehensive
set of experiments on CIFAR-10 demonstrates the scalability of our architecture
and the benefit of our approach for $\ell_2$ provable defenses. Indeed, we
train very deep and wide neural networks (up to $1000$ layers) and reach
state-of-the-art results in terms of standard and certified accuracy, along
with empirical robustness, in comparison with other $1$-Lipschitz
architectures.


http://arxiv.org/abs/2110.13650
GANash -- A GAN approach to steganography. (81%)
Venkatesh Subramaniyan; Vignesh Sivakumar; A. K. Vagheesan; S. Sakthivelan; K. J. Jegadish Kumar; K. K. Nagarajan
  Data security is of the utmost concern of a communication system. Since the
early days, many developments have been made to improve the performance of the
system. PSNR of the received signal, secure transmission channel, quality of
encoding used, etc. are some of the key attributes of a good system. To ensure
security, the most commonly used technique is cryptography in which the message
is altered with respect to a key and using the same, the encoded message is
decoded at the receiver side. A complementary technique that is popularly used
to insure security is steganography. The advancements in Artificial
Intelligence(AI) have paved way for performing steganography in an intelligent,
tamper-proof manner. The recent discovery by researchers in the field of Deep
Learning(DL), an unsupervised learning network known as the Generative
Adversarial Networks(GAN) has improved the performance of this technique
exponentially. It has been demonstrated that deep neural networks are highly
sensitive to tiny perturbations of input data, giving rise to adversarial
examples. Though this property is usually considered a weakness of learned
models, it could be beneficial if used appropriately. The work that has been
accomplished by MIT for this purpose, a deep-neural model by the name of
SteganoGAN, has shown obligation for using this technique for steganography. In
this work, we have proposed a novel approach to improve the performance of the
existing system using latent space compression on the encoded data. This
theoretically would improve the performance exponentially. Thus, the algorithms
used to improve the system's performance and the results obtained have been
enunciated in this work. The results indicate the level of dominance this
system could achieve to be able to diminish the difficulties in solving
real-time problems in terms of security, deployment and database management.


http://arxiv.org/abs/2110.12734
Fast Gradient Non-sign Methods. (74%)
Yaya Cheng; Xiaosu Zhu; Qilong Zhang; Lianli Gao; Jingkuan Song
  Adversarial attacks make their success in \enquote{fooling} DNNs and among
them, gradient-based algorithms become one of the mainstreams. Based on the
linearity hypothesis~\cite{fgsm}, under $\ell_\infty$ constraint, $sign$
operation applied to the gradients is a good choice for generating
perturbations. However, the side-effect from such operation exists since it
leads to the bias of direction between the real gradients and the
perturbations. In other words, current methods contain a gap between real
gradients and actual noises, which leads to biased and inefficient attacks.
Therefore in this paper, based on the Taylor expansion, the bias is analyzed
theoretically and the correction of $\sign$, \ie, Fast Gradient Non-sign Method
(FGNM), is further proposed. Notably, FGNM is a general routine, which can
seamlessly replace the conventional $sign$ operation in gradient-based attacks
with negligible extra computational cost. Extensive experiments demonstrate the
effectiveness of our methods. Specifically, ours outperform them by
\textbf{27.5\%} at most and \textbf{9.5\%} on average. Our anonymous code is
publicly available: \url{https://git.io/mm-fgnm}.


http://arxiv.org/abs/2110.12700
An Adaptive Structural Learning of Deep Belief Network for Image-based Crack Detection in Concrete Structures Using SDNET2018. (13%)
Shin Kamada; Takumi Ichimura; Takashi Iwasaki
  We have developed an adaptive structural Deep Belief Network (Adaptive DBN)
that finds an optimal network structure in a self-organizing manner during
learning. The Adaptive DBN is the hierarchical architecture where each layer
employs Adaptive Restricted Boltzmann Machine (Adaptive RBM). The Adaptive RBM
can find the appropriate number of hidden neurons during learning. The proposed
method was applied to a concrete image benchmark data set SDNET2018 for crack
detection. The dataset contains about 56,000 crack images for three types of
concrete structures: bridge decks, walls, and paved roads. The fine-tuning
method of the Adaptive DBN can show 99.7%, 99.7%, and 99.4% classification
accuracy for three types of structures. However, we found the database included
some wrong annotated data which cannot be judged from images by human experts.
This paper discusses consideration that purses the major factor for the wrong
cases and the removal of the adversarial examples from the dataset.


http://arxiv.org/abs/2110.12357
Towards A Conceptually Simple Defensive Approach for Few-shot classifiers Against Adversarial Support Samples. (80%)
Yi Xiang Marcus Tan; Penny Chong; Jiamei Sun; Ngai-man Cheung; Yuval Elovici; Alexander Binder
  Few-shot classifiers have been shown to exhibit promising results in use
cases where user-provided labels are scarce. These models are able to learn to
predict novel classes simply by training on a non-overlapping set of classes.
This can be largely attributed to the differences in their mechanisms as
compared to conventional deep networks. However, this also offers new
opportunities for novel attackers to induce integrity attacks against such
models, which are not present in other machine learning setups. In this work,
we aim to close this gap by studying a conceptually simple approach to defend
few-shot classifiers against adversarial attacks. More specifically, we propose
a simple attack-agnostic detection method, using the concept of self-similarity
and filtering, to flag out adversarial support sets which destroy the
understanding of a victim classifier for a certain class. Our extended
evaluation on the miniImagenet (MI) and CUB datasets exhibit good attack
detection performance, across three different few-shot classifiers and across
different attack strengths, beating baselines. Our observed results allow our
approach to establishing itself as a strong detection method for support set
poisoning attacks. We also show that our approach constitutes a generalizable
concept, as it can be paired with other filtering functions. Finally, we
provide an analysis of our results when we vary two components found in our
detection approach.


http://arxiv.org/abs/2110.12321
ADC: Adversarial attacks against object Detection that evade Context consistency checks. (99%)
Mingjun Yin; Shasha Li; Chengyu Song; M. Salman Asif; Amit K. Roy-Chowdhury; Srikanth V. Krishnamurthy
  Deep Neural Networks (DNNs) have been shown to be vulnerable to adversarial
examples, which are slightly perturbed input images which lead DNNs to make
wrong predictions. To protect from such examples, various defense strategies
have been proposed. A very recent defense strategy for detecting adversarial
examples, that has been shown to be robust to current attacks, is to check for
intrinsic context consistencies in the input data, where context refers to
various relationships (e.g., object-to-object co-occurrence relationships) in
images. In this paper, we show that even context consistency checks can be
brittle to properly crafted adversarial examples and to the best of our
knowledge, we are the first to do so. Specifically, we propose an adaptive
framework to generate examples that subvert such defenses, namely, Adversarial
attacks against object Detection that evade Context consistency checks (ADC).
In ADC, we formulate a joint optimization problem which has two attack goals,
viz., (i) fooling the object detector and (ii) evading the context consistency
check system, at the same time. Experiments on both PASCAL VOC and MS COCO
datasets show that examples generated with ADC fool the object detector with a
success rate of over 85% in most cases, and at the same time evade the recently
proposed context consistency checks, with a bypassing rate of over 80% in most
cases. Our results suggest that how to robustly model context and check its
consistency, is still an open problem.


http://arxiv.org/abs/2110.12308
A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness. (81%)
Chang Song; Riya Ranjan; Hai Li
  Neural networks are getting better accuracy with higher energy and
computational cost. After quantization, the cost can be greatly saved, and the
quantized models are more hardware friendly with acceptable accuracy loss. On
the other hand, recent research has found that neural networks are vulnerable
to adversarial attacks, and the robustness of a neural network model can only
be improved with defense methods, such as adversarial training. In this work,
we find that adversarially-trained neural networks are more vulnerable to
quantization loss than plain models. To minimize both the adversarial and the
quantization losses simultaneously and to make the quantized model robust, we
propose a layer-wise adversarial-aware quantization method, using the Lipschitz
constant to choose the best quantization parameter settings for a neural
network. We theoretically derive the losses and prove the consistency of our
metric selection. The experiment results show that our method can effectively
and efficiently improve the robustness of quantized adversarially-trained
neural networks.


http://arxiv.org/abs/2110.11987
Improving Robustness of Malware Classifiers using Adversarial Strings Generated from Perturbed Latent Representations. (99%)
Marek Galovic; Branislav Bosansky; Viliam Lisy
  In malware behavioral analysis, the list of accessed and created files very
often indicates whether the examined file is malicious or benign. However,
malware authors are trying to avoid detection by generating random filenames
and/or modifying used filenames with new versions of the malware. These changes
represent real-world adversarial examples. The goal of this work is to generate
realistic adversarial examples and improve the classifier's robustness against
these attacks. Our approach learns latent representations of input strings in
an unsupervised fashion and uses gradient-based adversarial attack methods in
the latent domain to generate adversarial examples in the input domain. We use
these examples to improve the classifier's robustness by training on the
generated adversarial set of strings. Compared to classifiers trained only on
perturbed latent vectors, our approach produces classifiers that are
significantly more robust without a large trade-off in standard accuracy.


http://arxiv.org/abs/2110.12072
How and When Adversarial Robustness Transfers in Knowledge Distillation? (91%)
Rulin Shao; Jinfeng Yi; Pin-Yu Chen; Cho-Jui Hsieh
  Knowledge distillation (KD) has been widely used in teacher-student training,
with applications to model compression in resource-constrained deep learning.
Current works mainly focus on preserving the accuracy of the teacher model.
However, other important model properties, such as adversarial robustness, can
be lost during distillation. This paper studies how and when the adversarial
robustness can be transferred from a teacher model to a student model in KD. We
show that standard KD training fails to preserve adversarial robustness, and we
propose KD with input gradient alignment (KDIGA) for remedy. Under certain
assumptions, we prove that the student model using our proposed KDIGA can
achieve at least the same certified robustness as the teacher model. Our
experiments of KD contain a diverse set of teacher and student models with
varying network architectures and sizes evaluated on ImageNet and CIFAR-10
datasets, including residual neural networks (ResNets) and vision transformers
(ViTs). Our comprehensive analysis shows several novel insights that (1) With
KDIGA, students can preserve or even exceed the adversarial robustness of the
teacher model, even when their models have fundamentally different
architectures; (2) KDIGA enables robustness to transfer to pre-trained
students, such as KD from an adversarially trained ResNet to a pre-trained ViT,
without loss of clean accuracy; and (3) Our derived local linearity bounds for
characterizing adversarial robustness in KD are consistent with the empirical
results.


http://arxiv.org/abs/2110.12020
Fairness Degrading Adversarial Attacks Against Clustering Algorithms. (86%)
Anshuman Chhabra; Adish Singla; Prasant Mohapatra
  Clustering algorithms are ubiquitous in modern data science pipelines, and
are utilized in numerous fields ranging from biology to facility location. Due
to their widespread use, especially in societal resource allocation problems,
recent research has aimed at making clustering algorithms fair, with great
success. Furthermore, it has also been shown that clustering algorithms, much
like other machine learning algorithms, are susceptible to adversarial attacks
where a malicious entity seeks to subvert the performance of the learning
algorithm. However, despite these known vulnerabilities, there has been no
research undertaken that investigates fairness degrading adversarial attacks
for clustering. We seek to bridge this gap by formulating a generalized attack
optimization problem aimed at worsening the group-level fairness of
centroid-based clustering algorithms. As a first step, we propose a fairness
degrading attack algorithm for k-median clustering that operates under a
whitebox threat model -- where the clustering algorithm, fairness notion, and
the input dataset are known to the adversary. We provide empirical results as
well as theoretical analysis for our simple attack algorithm, and find that the
addition of the generated adversarial samples can lead to significantly lower
fairness values. In this manner, we aim to motivate fairness degrading
adversarial attacks as a direction for future research in fair clustering.


http://arxiv.org/abs/2110.11950
Adversarial robustness for latent models: Revisiting the robust-standard accuracies tradeoff. (75%)
Adel Javanmard; Mohammad Mehrabi
  Over the past few years, several adversarial training methods have been
proposed to improve the robustness of machine learning models against
adversarial perturbations in the input. Despite remarkable progress in this
regard, adversarial training is often observed to drop the standard test
accuracy. This phenomenon has intrigued the research community to investigate
the potential tradeoff between standard and robust accuracy as two performance
measures. In this paper, we revisit this tradeoff for latent models and argue
that this tradeoff is mitigated when the data enjoys a low-dimensional
structure. In particular, we consider binary classification under two data
generative models, namely Gaussian mixture model and generalized linear model,
where the feature data lie on a low-dimensional manifold. We show that as the
manifold dimension to the ambient dimension decreases, one can obtain models
that are nearly optimal with respect to both, the standard accuracy and the
robust accuracy measures.


http://arxiv.org/abs/2110.11597
ProtoShotXAI: Using Prototypical Few-Shot Architecture for Explainable AI. (15%)
Samuel Hess; Gregory Ditzler
  Unexplainable black-box models create scenarios where anomalies cause
deleterious responses, thus creating unacceptable risks. These risks have
motivated the field of eXplainable Artificial Intelligence (XAI) to improve
trust by evaluating local interpretability in black-box neural networks.
Unfortunately, the ground truth is unavailable for the model's decision, so
evaluation is limited to qualitative assessment. Further, interpretability may
lead to inaccurate conclusions about the model or a false sense of trust. We
propose to improve XAI from the vantage point of the user's trust by exploring
a black-box model's latent feature space. We present an approach, ProtoShotXAI,
that uses a Prototypical few-shot network to explore the contrastive manifold
between nonlinear features of different classes. A user explores the manifold
by perturbing the input features of a query sample and recording the response
for a subset of exemplars from any class. Our approach is the first locally
interpretable XAI model that can be extended to, and demonstrated on, few-shot
networks. We compare ProtoShotXAI to the state-of-the-art XAI approaches on
MNIST, Omniglot, and ImageNet to demonstrate, both quantitatively and
qualitatively, that ProtoShotXAI provides more flexibility for model
exploration. Finally, ProtoShotXAI also demonstrates novel explainabilty and
detectabilty on adversarial samples.


http://arxiv.org/abs/2110.11578
PRECAD: Privacy-Preserving and Robust Federated Learning via Crypto-Aided Differential Privacy. (15%)
Xiaolan Gu; Ming Li; Li Xiong
  Federated Learning (FL) allows multiple participating clients to train
machine learning models collaboratively by keeping their datasets local and
only exchanging model updates. Existing FL protocol designs have been shown to
be vulnerable to attacks that aim to compromise data privacy and/or model
robustness. Recently proposed defenses focused on ensuring either privacy or
robustness, but not both. In this paper, we develop a framework called PRECAD,
which simultaneously achieves differential privacy (DP) and enhances robustness
against model poisoning attacks with the help of cryptography. Using secure
multi-party computation (MPC) techniques (e.g., secret sharing), noise is added
to the model updates by the honest-but-curious server(s) (instead of each
client) without revealing clients' inputs, which achieves the benefit of
centralized DP in terms of providing a better privacy-utility tradeoff than
local DP based solutions. Meanwhile, a crypto-aided secure validation protocol
is designed to verify that the contribution of model update from each client is
bounded without leaking privacy. We show analytically that the noise added to
ensure DP also provides enhanced robustness against malicious model
submissions. We experimentally demonstrate that our PRECAD framework achieves
higher privacy-utility tradeoff and enhances robustness for the trained models.


http://arxiv.org/abs/2110.12923
Spoofing Detection on Hand Images Using Quality Assessment. (1%)
Asish Bera; Ratnadeep Dey; Debotosh Bhattacharjee; Mita Nasipuri; Hubert P. H. Shum
  Recent research on biometrics focuses on achieving a high success rate of
authentication and addressing the concern of various spoofing attacks. Although
hand geometry recognition provides adequate security over unauthorized access,
it is susceptible to presentation attack. This paper presents an anti-spoofing
method toward hand biometrics. A presentation attack detection approach is
addressed by assessing the visual quality of genuine and fake hand images. A
threshold-based gradient magnitude similarity quality metric is proposed to
discriminate between the real and spoofed hand samples. The visual hand images
of 255 subjects from the Bogazici University hand database are considered as
original samples. Correspondingly, from each genuine sample, we acquire a
forged image using a Canon EOS 700D camera. Such fake hand images with natural
degradation are considered for electronic screen display based spoofing attack
detection. Furthermore, we create another fake hand dataset with artificial
degradation by introducing additional Gaussian blur, salt and pepper, and
speckle noises to original images. Ten quality metrics are measured from each
sample for classification between original and fake hand image. The
classification experiments are performed using the k-nearest neighbors, random
forest, and support vector machine classifiers, as well as deep convolutional
neural networks. The proposed gradient similarity-based quality metric achieves
1.5% average classification er ror using the k-nearest neighbors and random
forest classifiers. An average classification error of 2.5% is obtained using
the baseline evaluation with the MobileNetV2 deep network for discriminating
original and different types of fake hand samples.


http://arxiv.org/abs/2110.11891
On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning. (1%)
Anvith Thudi; Hengrui Jia; Ilia Shumailov; Nicolas Papernot
  Machine unlearning, i.e. having a model forget about some of its training
data, has become increasingly more important as privacy legislation promotes
variants of the right-to-be-forgotten. In the context of deep learning,
approaches for machine unlearning are broadly categorized into two classes:
exact unlearning methods, where an entity has formally removed the data point's
impact on the model by retraining the model from scratch, and approximate
unlearning, where an entity approximates the model parameters one would obtain
by exact unlearning to save on compute costs. In this paper we first show that
the definition that underlies approximate unlearning, which seeks to prove the
approximately unlearned model is close to an exactly retrained model, is
incorrect because one can obtain the same model using different datasets. Thus
one could unlearn without modifying the model at all. We then turn to exact
unlearning approaches and ask how to verify their claims of unlearning. Our
results show that even for a given training trajectory one cannot formally
prove the absence of certain data points used during training. We thus conclude
that unlearning is only well-defined at the algorithmic level, where an
entity's only possible auditable claim to unlearning is that they used a
particular algorithm designed to allow for external scrutiny during an audit.


http://arxiv.org/abs/2110.11589
Text Counterfactuals via Latent Optimization and Shapley-Guided Search. (1%)
Quintin Pope; Xiaoli Z. Fern
  We study the problem of generating counterfactual text for a classifier as a
means for understanding and debugging classification. Given a textual input and
a classification model, we aim to minimally alter the text to change the
model's prediction. White-box approaches have been successfully applied to
similar problems in vision where one can directly optimize the continuous
input. Optimization-based approaches become difficult in the language domain
due to the discrete nature of text. We bypass this issue by directly optimizing
in the latent space and leveraging a language model to generate candidate
modifications from optimized latent representations. We additionally use
Shapley values to estimate the combinatoric effect of multiple changes. We then
use these estimates to guide a beam search for the final counterfactual text.
We achieve favorable performance compared to recent white-box and black-box
baselines using human and automatic evaluations. Ablation studies show that
both latent optimization and the use of Shapley values improve success rate and
the quality of the generated counterfactuals.


http://arxiv.org/abs/2110.11736
MANDERA: Malicious Node Detection in Federated Learning via Ranking. (1%)
Wanchuang Zhu; Benjamin Zi Hao Zhao; Simon Luo; Ke Deng
  Federated learning is a distributed learning paradigm which seeks to preserve
the privacy of each participating node's data. However, federated learning is
vulnerable to attacks, specifically to our interest, model integrity attacks.
In this paper, we propose a novel method for malicious node detection called
MANDERA. By transferring the original message matrix into a ranking matrix
whose column shows the relative rankings of all local nodes along different
parameter dimensions, our approach seeks to distinguish the malicious nodes
from the benign ones with high efficiency based on key characteristics of the
rank domain. We have proved, under mild conditions, that MANDERA is guaranteed
to detect all malicious nodes under typical Byzantine attacks with no prior
knowledge or history about the participating nodes. The effectiveness of the
proposed approach is further confirmed by experiments on two classic datasets,
CIFAR-10 and MNIST. Compared to the state-of-art methods in the literature for
defending Byzantine attacks, MANDERA is unique in its way to identify the
malicious nodes by ranking and its robustness to effectively defense a wide
range of attacks.


http://arxiv.org/abs/2110.11459
CAPTIVE: Constrained Adversarial Perturbations to Thwart IC Reverse Engineering. (98%)
Amir Hosein Afandizadeh Zargari; Marzieh AshrafiAmiri; Minjun Seo; Sai Manoj Pudukotai Dinakarrao; Mohammed E. Fouda; Fadi Kurdahi
  Reverse engineering (RE) in Integrated Circuits (IC) is a process in which
one will attempt to extract the internals of an IC, extract the circuit
structure, and determine the gate-level information of an IC. In general, RE
process can be done for validation as well as intellectual property (IP)
stealing intentions. In addition, RE also facilitates different illicit
activities such as insertion of hardware Trojan, pirate, or counterfeit a
design, or develop an attack. In this work, we propose an approach to introduce
cognitive perturbations, with the aid of adversarial machine learning, to the
IC layout that could prevent the RE process from succeeding. We first construct
a layer-by-layer image dataset of 45nm predictive technology. With this
dataset, we propose a conventional neural network model called RecoG-Net to
recognize the logic gates, which is the first step in RE. RecoG-Net is
successfully to recognize the gates with more than 99.7% accuracy. Our
thwarting approach utilizes the concept of the adversarial attack generation
algorithms to generate perturbation. Unlike traditional adversarial attacks in
machine learning, the perturbation generation needs to be highly constrained to
meet the fab rules such as Design Rule Checking (DRC) Layout vs. Schematic
(LVS) checks. Hence, we propose CAPTIVE as an constrained perturbation
generation satisfying the DRC. The experiments shows that the accuracy of
reverse engineering using machine learning techniques can decrease from 100% to
approximately 30% based on the adversary generator.


http://arxiv.org/abs/2110.11411
PROVES: Establishing Image Provenance using Semantic Signatures. (93%)
Mingyang Xie; Manav Kulshrestha; Shaojie Wang; Jinghan Yang; Ayan Chakrabarti; Ning Zhang; Yevgeniy Vorobeychik
  Modern AI tools, such as generative adversarial networks, have transformed
our ability to create and modify visual data with photorealistic results.
However, one of the deleterious side-effects of these advances is the emergence
of nefarious uses in manipulating information in visual data, such as through
the use of deep fakes. We propose a novel architecture for preserving the
provenance of semantic information in images to make them less susceptible to
deep fake attacks. Our architecture includes semantic signing and verification
steps. We apply this architecture to verifying two types of semantic
information: individual identities (faces) and whether the photo was taken
indoors or outdoors. Verification accounts for a collection of common image
transformation, such as translation, scaling, cropping, and small rotations,
and rejects adversarial transformations, such as adversarially perturbed or, in
the case of face verification, swapped faces. Experiments demonstrate that in
the case of provenance of faces in an image, our approach is robust to
black-box adversarial transformations (which are rejected) as well as benign
transformations (which are accepted), with few false negatives and false
positives. Background verification, on the other hand, is susceptible to
black-box adversarial examples, but becomes significantly more robust after
adversarial training.


http://arxiv.org/abs/2110.11088
RoMA: a Method for Neural Network Robustness Measurement and Assessment. (92%)
Natan Levy; Guy Katz
  Neural network models have become the leading solution for a large variety of
tasks, such as classification, language processing, protein folding, and
others. However, their reliability is heavily plagued by adversarial inputs:
small input perturbations that cause the model to produce erroneous outputs.
Adversarial inputs can occur naturally when the system's environment behaves
randomly, even in the absence of a malicious adversary, and are a severe cause
for concern when attempting to deploy neural networks within critical systems.
In this paper, we present a new statistical method, called Robustness
Measurement and Assessment (RoMA), which can measure the expected robustness of
a neural network model. Specifically, RoMA determines the probability that a
random input perturbation might cause misclassification. The method allows us
to provide formal guarantees regarding the expected frequency of errors that a
trained model will encounter after deployment. Our approach can be applied to
large-scale, black-box neural networks, which is a significant advantage
compared to recently proposed verification methods. We apply our approach in
two ways: comparing the robustness of different models, and measuring how a
model's robustness is affected by the magnitude of input perturbation. One
interesting insight obtained through this work is that, in a classification
network, different output labels can exhibit very different robustness levels.
We term this phenomenon categorial robustness. Our ability to perform risk and
robustness assessments on a categorial basis opens the door to risk mitigation,
which may prove to be a significant step towards neural network certification
in safety-critical applications.


http://arxiv.org/abs/2110.11571
Anti-Backdoor Learning: Training Clean Models on Poisoned Data. (83%)
Yige Li; Xixiang Lyu; Nodens Koren; Lingjuan Lyu; Bo Li; Xingjun Ma
  Backdoor attack has emerged as a major security threat to deep neural
networks (DNNs). While existing defense methods have demonstrated promising
results on detecting and erasing backdoor triggers, it is still not clear if
measures can be taken to avoid the triggers from being learned into the model
in the first place. In this paper, we introduce the concept of
\emph{anti-backdoor learning}, of which the aim is to train clean models on
backdoor-poisoned data. We frame the overall learning process as a dual-task of
learning the clean portion of data and learning the backdoor portion of data.
From this view, we identify two inherent characteristics of backdoor attacks as
their weaknesses: 1) the models learn backdoored data at a much faster rate
than learning clean data, and the stronger the attack the faster the model
converges on backdoored data; and 2) the backdoor task is tied to a specific
class (the backdoor target class). Based on these two weaknesses, we propose a
general learning scheme, Anti-Backdoor Learning (ABL), to automatically prevent
backdoor attacks during training. ABL introduces a two-stage \emph{gradient
ascent} mechanism into standard training to 1) help isolate backdoor examples
at an early training stage, and 2) break the correlation between backdoor
examples and the target class at a later training stage. Through extensive
experiments on multiple benchmark datasets against 10 state-of-the-art attacks,
we empirically show that ABL-trained models on backdoor-poisoned data achieve
the same performance as they were trained on purely clean data. Code is
available at \underline{https://github.com/bboylyg/ABL}.


http://arxiv.org/abs/2110.10926
PipAttack: Poisoning Federated Recommender Systems forManipulating Item Promotion. (68%)
Shijie Zhang; Hongzhi Yin; Tong Chen; Zi Huang; Quoc Viet Hung Nguyen; Lizhen Cui
  Due to the growing privacy concerns, decentralization emerges rapidly in
personalized services, especially recommendation. Also, recent studies have
shown that centralized models are vulnerable to poisoning attacks, compromising
their integrity. In the context of recommender systems, a typical goal of such
poisoning attacks is to promote the adversary's target items by interfering
with the training dataset and/or process. Hence, a common practice is to
subsume recommender systems under the decentralized federated learning
paradigm, which enables all user devices to collaboratively learn a global
recommender while retaining all the sensitive data locally. Without exposing
the full knowledge of the recommender and entire dataset to end-users, such
federated recommendation is widely regarded `safe' towards poisoning attacks.
In this paper, we present a systematic approach to backdooring federated
recommender systems for targeted item promotion. The core tactic is to take
advantage of the inherent popularity bias that commonly exists in data-driven
recommenders. As popular items are more likely to appear in the recommendation
list, our innovatively designed attack model enables the target item to have
the characteristics of popular items in the embedding space. Then, by uploading
carefully crafted gradients via a small number of malicious users during the
model update, we can effectively increase the exposure rate of a target
(unpopular) item in the resulted federated recommender. Evaluations on two
real-world datasets show that 1) our attack model significantly boosts the
exposure rate of the target item in a stealthy way, without harming the
accuracy of the poisoned recommender; and 2) existing defenses are not
effective enough, highlighting the need for new defenses against our local
model poisoning attacks to federated recommender systems.


http://arxiv.org/abs/2110.10942
Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness. (61%)
Simon Geisler; Johanna Sommer; Jan Schuchardt; Aleksandar Bojchevski; Stephan Günnemann
  End-to-end (geometric) deep learning has seen first successes in
approximating the solution of combinatorial optimization problems. However,
generating data in the realm of NP-hard/-complete tasks brings practical and
theoretical challenges, resulting in evaluation protocols that are too
optimistic. Specifically, most datasets only capture a simpler subproblem and
likely suffer from spurious features. We investigate these effects by studying
adversarial robustness - a local generalization property - to reveal hard,
model-specific instances and spurious features. For this purpose, we derive
perturbation models for SAT and TSP. Unlike in other applications, where
perturbation models are designed around subjective notions of imperceptibility,
our perturbation models are efficient and sound, allowing us to determine the
true label of perturbed samples without a solver. Surprisingly, with such
perturbations, a sufficiently expressive neural solver does not suffer from the
limitations of the accuracy-robustness trade-off common in supervised learning.
Although such robust solvers exist, we show empirically that the assessed
neural solvers do not generalize well w.r.t. small perturbations of the problem
instance.


http://arxiv.org/abs/2110.11290
Physical Side-Channel Attacks on Embedded Neural Networks: A Survey. (8%)
Maria Méndez Real; Rubén Salvador
  During the last decade, Deep Neural Networks (DNN) have progressively been
integrated on all types of platforms, from data centers to embedded systems
including low-power processors and, recently, FPGAs. Neural Networks (NN) are
expected to become ubiquitous in IoT systems by transforming all sorts of
real-world applications, including applications in the safety-critical and
security-sensitive domains. However, the underlying hardware security
vulnerabilities of embedded NN implementations remain unaddressed. In
particular, embedded DNN implementations are vulnerable to Side-Channel
Analysis (SCA) attacks, which are especially important in the IoT and edge
computing contexts where an attacker can usually gain physical access to the
targeted device. A research field has therefore emerged and is rapidly growing
in terms of the use of SCA including timing, electromagnetic attacks and power
attacks to target NN embedded implementations. Since 2018, research papers have
shown that SCA enables an attacker to recover inference models architectures
and parameters, to expose industrial IP and endangers data confidentiality and
privacy. Without a complete review of this emerging field in the literature so
far, this paper surveys state-of-the-art physical SCA attacks relative to the
implementation of embedded DNNs on micro-controllers and FPGAs in order to
provide a thorough analysis on the current landscape. It provides a taxonomy
and a detailed classification of current attacks. It first discusses mitigation
techniques and then provides insights for future research leads.


http://arxiv.org/abs/2110.11024
Watermarking Graph Neural Networks based on Backdoor Attacks. (1%)
Jing Xu; Stjepan Picek
  Graph Neural Networks (GNNs) have achieved promising performance in various
real-world applications. Building a powerful GNN model is not a trivial task,
as it requires a large amount of training data, powerful computing resources,
and human expertise on fine-tuning the model. What is more, with the
development of adversarial attacks, e.g., model stealing attacks, GNNs raise
challenges to model authentication. To avoid copyright infringement on GNNs, it
is necessary to verify the ownership of the GNN models.
  In this paper, we present a watermarking framework for GNNs for both graph
and node classification tasks. We 1) design two strategies to generate
watermarked data for the graph classification and one for the node
classification task, 2) embed the watermark into the host model through
training to obtain the watermarked GNN model, and 3) verify the ownership of
the suspicious model in a black-box setting. The experiments show that our
framework can verify the ownership of GNN models with a very high probability
(around $100\%$) for both tasks. In addition, we experimentally show that our
watermarking approach is still effective even when considering suspicious
models obtained from different architectures than the owner's.


http://arxiv.org/abs/2110.10655
Adversarial Socialbot Learning via Multi-Agent Deep Hierarchical Reinforcement Learning. (75%)
Thai Le; Long Tran-Thanh; Dongwon Lee
  Socialbots are software-driven user accounts on social platforms, acting
autonomously (mimicking human behavior), with the aims to influence the
opinions of other users or spread targeted misinformation for particular goals.
As socialbots undermine the ecosystem of social platforms, they are often
considered harmful. As such, there have been several computational efforts to
auto-detect the socialbots. However, to our best knowledge, the adversarial
nature of these socialbots has not yet been studied. This begs a question "can
adversaries, controlling socialbots, exploit AI techniques to their advantage?"
To this question, we successfully demonstrate that indeed it is possible for
adversaries to exploit computational learning mechanism such as reinforcement
learning (RL) to maximize the influence of socialbots while avoiding being
detected. We first formulate the adversarial socialbot learning as a
cooperative game between two functional hierarchical RL agents. While one agent
curates a sequence of activities that can avoid the detection, the other agent
aims to maximize network influence by selectively connecting with right users.
Our proposed policy networks train with a vast amount of synthetic graphs and
generalize better than baselines on unseen real-life graphs both in terms of
maximizing network influence (up to +18%) and sustainable stealthiness (up to
+40% undetectability) under a strong bot detector (with 90% detection
accuracy). During inference, the complexity of our approach scales linearly,
independent of a network's structure and the virality of news. This makes our
approach a practical adversarial attack when deployed in a real-life setting.


http://arxiv.org/abs/2110.10482
Surrogate Representation Learning with Isometric Mapping for Gray-box Graph Adversarial Attacks. (62%)
Zihan Liul; Yun Luo; Zelin Zang; Stan Z. Li
  Gray-box graph attacks aim at disrupting the performance of the victim model
by using inconspicuous attacks with limited knowledge of the victim model. The
parameters of the victim model and the labels of the test nodes are invisible
to the attacker. To obtain the gradient on the node attributes or graph
structure, the attacker constructs an imaginary surrogate model trained under
supervision. However, there is a lack of discussion on the training of
surrogate models and the robustness of provided gradient information. The
general node classification model loses the topology of the nodes on the graph,
which is, in fact, an exploitable prior for the attacker. This paper
investigates the effect of representation learning of surrogate models on the
transferability of gray-box graph adversarial attacks. To reserve the topology
in the surrogate embedding, we propose Surrogate Representation Learning with
Isometric Mapping (SRLIM). By using Isometric mapping method, our proposed
SRLIM can constrain the topological structure of nodes from the input layer to
the embedding space, that is, to maintain the similarity of nodes in the
propagation process. Experiments prove the effectiveness of our approach
through the improvement in the performance of the adversarial attacks generated
by the gradient-based attacker in untargeted poisoning gray-box setups.


http://arxiv.org/abs/2110.10444
Moir\'e Attack (MA): A New Potential Risk of Screen Photos. (56%)
Dantong Niu; Ruohao Guo; Yisen Wang
  Images, captured by a camera, play a critical role in training Deep Neural
Networks (DNNs). Usually, we assume the images acquired by cameras are
consistent with the ones perceived by human eyes. However, due to the different
physical mechanisms between human-vision and computer-vision systems, the final
perceived images could be very different in some cases, for example shooting on
digital monitors. In this paper, we find a special phenomenon in digital image
processing, the moir\'e effect, that could cause unnoticed security threats to
DNNs. Based on it, we propose a Moir\'e Attack (MA) that generates the
physical-world moir\'e pattern adding to the images by mimicking the shooting
process of digital devices. Extensive experiments demonstrate that our proposed
digital Moir\'e Attack (MA) is a perfect camouflage for attackers to tamper
with DNNs with a high success rate ($100.0\%$ for untargeted and $97.0\%$ for
targeted attack with the noise budget $\epsilon=4$), high transferability rate
across different models, and high robustness under various defenses.
Furthermore, MA owns great stealthiness because the moir\'e effect is
unavoidable due to the camera's inner physical structure, which therefore
hardly attracts the awareness of humans. Our code is available at
https://github.com/Dantong88/Moire_Attack.


http://arxiv.org/abs/2110.10783
Adversarial attacks against Bayesian forecasting dynamic models. (13%)
Roi Naveiro
  The last decade has seen the rise of Adversarial Machine Learning (AML). This
discipline studies how to manipulate data to fool inference engines, and how to
protect those systems against such manipulation attacks. Extensive work on
attacks against regression and classification systems is available, while
little attention has been paid to attacks against time series forecasting
systems. In this paper, we propose a decision analysis based attacking strategy
that could be utilized against Bayesian forecasting dynamic models.


http://arxiv.org/abs/2110.12899
No One Representation to Rule Them All: Overlapping Features of Training Methods. (1%)
Raphael Gontijo-Lopes; Yann Dauphin; Ekin D. Cubuk
  Despite being able to capture a range of features of the data, high accuracy
models trained with supervision tend to make similar predictions. This
seemingly implies that high-performing models share similar biases regardless
of training methodology, which would limit ensembling benefits and render
low-accuracy models as having little practical use. Against this backdrop,
recent work has made very different training techniques, such as large-scale
contrastive learning, yield competitively-high accuracy on generalization and
robustness benchmarks. This motivates us to revisit the assumption that models
necessarily learn similar functions. We conduct a large-scale empirical study
of models across hyper-parameters, architectures, frameworks, and datasets. We
find that model pairs that diverge more in training methodology display
categorically different generalization behavior, producing increasingly
uncorrelated errors. We show these models specialize in subdomains of the data,
leading to higher ensemble performance: with just 2 models (each with ImageNet
accuracy ~76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly,
we find that even significantly low-accuracy models can be used to improve
high-accuracy models. Finally, we show diverging training methodology yield
representations that capture overlapping (but not supersetting) feature sets
which, when combined, lead to increased downstream performance.


http://arxiv.org/abs/2110.10287
Multi-concept adversarial attacks. (99%)
Vibha Belavadi; Yan Zhou; Murat Kantarcioglu; Bhavani M. Thuraisingham
  As machine learning (ML) techniques are being increasingly used in many
applications, their vulnerability to adversarial attacks becomes well-known.
Test time attacks, usually launched by adding adversarial noise to test
instances, have been shown effective against the deployed ML models. In
practice, one test input may be leveraged by different ML models. Test time
attacks targeting a single ML model often neglect their impact on other ML
models. In this work, we empirically demonstrate that naively attacking the
classifier learning one concept may negatively impact classifiers trained to
learn other concepts. For example, for the online image classification
scenario, when the Gender classifier is under attack, the (wearing) Glasses
classifier is simultaneously attacked with the accuracy dropped from 98.69 to
88.42. This raises an interesting question: is it possible to attack one set of
classifiers without impacting the other set that uses the same test instance?
Answers to the above research question have interesting implications for
protecting privacy against ML model misuse. Attacking ML models that pose
unnecessary risks of privacy invasion can be an important tool for protecting
individuals from harmful privacy exploitation. In this paper, we address the
above research question by developing novel attack techniques that can
simultaneously attack one set of ML models while preserving the accuracy of the
other. In the case of linear classifiers, we provide a theoretical framework
for finding an optimal solution to generate such adversarial examples. Using
this theoretical framework, we develop a multi-concept attack strategy in the
context of deep learning. Our results demonstrate that our techniques can
successfully attack the target classes while protecting the protected classes
in many different settings, which is not possible with the existing test-time
attack-single strategies.


http://arxiv.org/abs/2110.09759
A Regularization Method to Improve Adversarial Robustness of Neural Networks for ECG Signal Classification. (96%)
Linhai Ma; Liang Liang
  Electrocardiogram (ECG) is the most widely used diagnostic tool to monitor
the condition of the human heart. By using deep neural networks (DNNs),
interpretation of ECG signals can be fully automated for the identification of
potential abnormalities in a patient's heart in a fraction of a second. Studies
have shown that given a sufficiently large amount of training data, DNN
accuracy for ECG classification could reach human-expert cardiologist level.
However, despite of the excellent performance in classification accuracy, DNNs
are highly vulnerable to adversarial noises that are subtle changes in the
input of a DNN and may lead to a wrong class-label prediction. It is
challenging and essential to improve robustness of DNNs against adversarial
noises, which are a threat to life-critical applications. In this work, we
proposed a regularization method to improve DNN robustness from the perspective
of noise-to-signal ratio (NSR) for the application of ECG signal
classification. We evaluated our method on PhysioNet MIT-BIH dataset and
CPSC2018 ECG dataset, and the results show that our method can substantially
enhance DNN robustness against adversarial noises generated from adversarial
attacks, with a minimal change in accuracy on clean data.


http://arxiv.org/abs/2110.10108
TESSERACT: Gradient Flip Score to Secure Federated Learning Against Model Poisoning Attacks. (69%)
Atul Sharma; Wei Chen; Joshua Zhao; Qiang Qiu; Somali Chaterji; Saurabh Bagchi
  Federated learning---multi-party, distributed learning in a decentralized
environment---is vulnerable to model poisoning attacks, even more so than
centralized learning approaches. This is because malicious clients can collude
and send in carefully tailored model updates to make the global model
inaccurate. This motivated the development of Byzantine-resilient federated
learning algorithms, such as Krum, Bulyan, FABA, and FoolsGold. However, a
recently developed untargeted model poisoning attack showed that all prior
defenses can be bypassed. The attack uses the intuition that simply by changing
the sign of the gradient updates that the optimizer is computing, for a set of
malicious clients, a model can be diverted from the optima to increase the test
error rate. In this work, we develop TESSERACT---a defense against this
directed deviation attack, a state-of-the-art model poisoning attack. TESSERACT
is based on a simple intuition that in a federated learning setting, certain
patterns of gradient flips are indicative of an attack. This intuition is
remarkably stable across different learning algorithms, models, and datasets.
TESSERACT assigns reputation scores to the participating clients based on their
behavior during the training phase and then takes a weighted contribution of
the clients. We show that TESSERACT provides robustness against even a
white-box version of the attack.


http://arxiv.org/abs/2110.09902
Understanding Convolutional Neural Networks from Theoretical Perspective via Volterra Convolution. (61%)
Tenghui Li; Guoxu Zhou; Yuning Qiu; Qibin Zhao
  This study proposes a general and unified perspective of convolutional neural
networks by exploring the relationship between (deep) convolutional neural
networks and finite Volterra convolutions. It provides a novel approach to
explain and study the overall characteristics of neural networks without being
disturbed by the complex network architectures. Concretely, we examine the
basic structures of finite term Volterra convolutions and convolutional neural
networks. Our results show that convolutional neural network is an
approximation of the finite term Volterra convolution, whose order increases
exponentially with the number of layers and kernel size increases exponentially
with the strides. With this perspective, the specialized perturbations are
directly obtained from the approximated kernels rather than iterative generated
adversarial examples. Extensive experiments on synthetic and real-world data
sets show the correctness and effectiveness of our results.


http://arxiv.org/abs/2110.10354
Detecting Backdoor Attacks Against Point Cloud Classifiers. (26%)
Zhen Xiang; David J. Miller; Siheng Chen; Xi Li; George Kesidis
  Backdoor attacks (BA) are an emerging threat to deep neural network
classifiers. A classifier being attacked will predict to the attacker's target
class when a test sample from a source class is embedded with the backdoor
pattern (BP). Recently, the first BA against point cloud (PC) classifiers was
proposed, creating new threats to many important applications including
autonomous driving. Such PC BAs are not detectable by existing BA defenses due
to their special BP embedding mechanism. In this paper, we propose a
reverse-engineering defense that infers whether a PC classifier is backdoor
attacked, without access to its training set or to any clean classifiers for
reference. The effectiveness of our defense is demonstrated on the benchmark
ModeNet40 dataset for PCs.


http://arxiv.org/abs/2110.09814
Speech Pattern based Black-box Model Watermarking for Automatic Speech Recognition. (13%)
Haozhe Chen; Weiming Zhang; Kunlin Liu; Kejiang Chen; Han Fang; Nenghai Yu
  As an effective method for intellectual property (IP) protection, model
watermarking technology has been applied on a wide variety of deep neural
networks (DNN), including speech classification models. However, how to design
a black-box watermarking scheme for automatic speech recognition (ASR) models
is still an unsolved problem, which is a significant demand for protecting
remote ASR Application Programming Interface (API) deployed in cloud servers.
Due to conditional independence assumption and label-detection-based evasion
attack risk of ASR models, the black-box model watermarking scheme for speech
classification models cannot apply to ASR models. In this paper, we propose the
first black-box model watermarking framework for protecting the IP of ASR
models. Specifically, we synthesize trigger audios by spreading the speech
clips of model owners over the entire input audios and labeling the trigger
audios with the stego texts, which hides the authorship information with
linguistic steganography. Experiments on the state-of-the-art open-source ASR
system DeepSpeech demonstrate the feasibility of the proposed watermarking
scheme, which is robust against five kinds of attacks and has little impact on
accuracy.


http://arxiv.org/abs/2110.10291
A Deeper Look into RowHammer`s Sensitivities: Experimental Analysis of Real DRAM Chips and Implications on Future Attacks and Defenses. (5%)
Lois Orosa; Abdullah Giray Yağlıkçı; Haocong Luo; Ataberk Olgun; Jisung Park; Hasan Hassan; Minesh Patel; Jeremie S. Kim; Onur Mutlu
  RowHammer is a circuit-level DRAM vulnerability where repeatedly accessing
(i.e., hammering) a DRAM row can cause bit flips in physically nearby rows. The
RowHammer vulnerability worsens as DRAM cell size and cell-to-cell spacing
shrink. Recent studies demonstrate that modern DRAM chips, including chips
previously marketed as RowHammer-safe, are even more vulnerable to RowHammer
than older chips such that the required hammer count to cause a bit flip has
reduced by more than 10X in the last decade. Therefore, it is essential to
develop a better understanding and in-depth insights into the RowHammer
vulnerability of modern DRAM chips to more effectively secure current and
future systems.
  Our goal in this paper is to provide insights into fundamental properties of
the RowHammer vulnerability that are not yet rigorously studied by prior works,
but can potentially be $i$) exploited to develop more effective RowHammer
attacks or $ii$) leveraged to design more effective and efficient defense
mechanisms. To this end, we present an experimental characterization using
248~DDR4 and 24~DDR3 modern DRAM chips from four major DRAM manufacturers
demonstrating how the RowHammer effects vary with three fundamental properties:
1)~DRAM chip temperature, 2)~aggressor row active time, and 3)~victim DRAM
cell's physical location. Among our 16 new observations, we highlight that a
RowHammer bit flip 1)~is very likely to occur in a bounded range, specific to
each DRAM cell (e.g., 5.4% of the vulnerable DRAM cells exhibit errors in the
range 70C to 90C), 2)~is more likely to occur if the aggressor row is active
for longer time (e.g., RowHammer vulnerability increases by 36% if we keep a
DRAM row active for 15 column accesses), and 3)~is more likely to occur in
certain physical regions of the DRAM module under attack (e.g., 5% of the rows
are 2x more vulnerable than the remaining 95% of the rows).


http://arxiv.org/abs/2110.09714
Black-box Adversarial Attacks on Commercial Speech Platforms with Minimal Information. (99%)
Baolin Zheng; Peipei Jiang; Qian Wang; Qi Li; Chao Shen; Cong Wang; Yunjie Ge; Qingyang Teng; Shenyi Zhang
  Adversarial attacks against commercial black-box speech platforms, including
cloud speech APIs and voice control devices, have received little attention
until recent years. The current "black-box" attacks all heavily rely on the
knowledge of prediction/confidence scores to craft effective adversarial
examples, which can be intuitively defended by service providers without
returning these messages. In this paper, we propose two novel adversarial
attacks in more practical and rigorous scenarios. For commercial cloud speech
APIs, we propose Occam, a decision-only black-box adversarial attack, where
only final decisions are available to the adversary. In Occam, we formulate the
decision-only AE generation as a discontinuous large-scale global optimization
problem, and solve it by adaptively decomposing this complicated problem into a
set of sub-problems and cooperatively optimizing each one. Our Occam is a
one-size-fits-all approach, which achieves 100% success rates of attacks with
an average SNR of 14.23dB, on a wide range of popular speech and speaker
recognition APIs, including Google, Alibaba, Microsoft, Tencent, iFlytek, and
Jingdong, outperforming the state-of-the-art black-box attacks. For commercial
voice control devices, we propose NI-Occam, the first non-interactive physical
adversarial attack, where the adversary does not need to query the oracle and
has no access to its internal information and training data. We combine
adversarial attacks with model inversion attacks, and thus generate the
physically-effective audio AEs with high transferability without any
interaction with target devices. Our experimental results show that NI-Occam
can successfully fool Apple Siri, Microsoft Cortana, Google Assistant, iFlytek
and Amazon Echo with an average SRoA of 52% and SNR of 9.65dB, shedding light
on non-interactive physical attacks against voice control devices.


http://arxiv.org/abs/2110.09075
Boosting the Transferability of Video Adversarial Examples via Temporal Translation. (99%)
Zhipeng Wei; Jingjing Chen; Zuxuan Wu; Yu-Gang Jiang
  Although deep-learning based video recognition models have achieved
remarkable success, they are vulnerable to adversarial examples that are
generated by adding human-imperceptible perturbations on clean video samples.
As indicated in recent studies, adversarial examples are transferable, which
makes it feasible for black-box attacks in real-world applications.
Nevertheless, most existing adversarial attack methods have poor
transferability when attacking other video models and transfer-based attacks on
video models are still unexplored. To this end, we propose to boost the
transferability of video adversarial examples for black-box attacks on video
recognition models. Through extensive analysis, we discover that different
video recognition models rely on different discriminative temporal patterns,
leading to the poor transferability of video adversarial examples. This
motivates us to introduce a temporal translation attack method, which optimizes
the adversarial perturbations over a set of temporal translated video clips. By
generating adversarial examples over translated videos, the resulting
adversarial examples are less sensitive to temporal patterns existed in the
white-box model being attacked and thus can be better transferred. Extensive
experiments on the Kinetics-400 dataset and the UCF-101 dataset demonstrate
that our method can significantly boost the transferability of video
adversarial examples. For transfer-based attack against video recognition
models, it achieves a 61.56% average attack success rate on the Kinetics-400
and 48.60% on the UCF-101.


http://arxiv.org/abs/2110.09468
Improving Robustness using Generated Data. (97%)
Sven Gowal; Sylvestre-Alvise Rebuffi; Olivia Wiles; Florian Stimberg; Dan Andrei Calian; Timothy Mann
  Recent work argues that robust training requires substantially larger
datasets than those required for standard classification. On CIFAR-10 and
CIFAR-100, this translates into a sizable robust-accuracy gap between models
trained solely on data from the original training set and those trained with
additional data extracted from the "80 Million Tiny Images" dataset (TI-80M).
In this paper, we explore how generative models trained solely on the original
training set can be leveraged to artificially increase the size of the original
training set and improve adversarial robustness to $\ell_p$ norm-bounded
perturbations. We identify the sufficient conditions under which incorporating
additional generated data can improve robustness, and demonstrate that it is
possible to significantly reduce the robust-accuracy gap to models trained with
additional real data. Surprisingly, we even show that even the addition of
non-realistic random data (generated by Gaussian sampling) can improve
robustness. We evaluate our approach on CIFAR-10, CIFAR-100, SVHN and
TinyImageNet against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of
size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large
absolute improvements in robust accuracy compared to previous state-of-the-art
methods. Against $\ell_\infty$ norm-bounded perturbations of size $\epsilon =
8/255$, our models achieve 66.10% and 33.49% robust accuracy on CIFAR-10 and
CIFAR-100, respectively (improving upon the state-of-the-art by +8.96% and
+3.29%). Against $\ell_2$ norm-bounded perturbations of size $\epsilon =
128/255$, our model achieves 78.31% on CIFAR-10 (+3.81%). These results beat
most prior works that use external data.


http://arxiv.org/abs/2110.09506
MEMO: Test Time Robustness via Adaptation and Augmentation. (13%)
Marvin Zhang; Sergey Levine; Chelsea Finn
  While deep neural networks can attain good accuracy on in-distribution test
points, many applications require robustness even in the face of unexpected
perturbations in the input, changes in the domain, or other sources of
distribution shift. We study the problem of test time robustification, i.e.,
using the test input to improve model robustness. Recent prior works have
proposed methods for test time adaptation, however, they each introduce
additional assumptions, such as access to multiple test points, that prevent
widespread adoption. In this work, we aim to study and devise methods that make
no assumptions about the model training process and are broadly applicable at
test time. We propose a simple approach that can be used in any test setting
where the model is probabilistic and adaptable: when presented with a test
example, perform different data augmentations on the data point, and then adapt
(all of) the model parameters by minimizing the entropy of the model's average,
or marginal, output distribution across the augmentations. Intuitively, this
objective encourages the model to make the same prediction across different
augmentations, thus enforcing the invariances encoded in these augmentations,
while also maintaining confidence in its predictions. In our experiments, we
demonstrate that this approach consistently improves robust ResNet and vision
transformer models, achieving accuracy gains of 1-8% over standard model
evaluation and also generally outperforming prior augmentation and adaptation
strategies. We achieve state-of-the-art results for test shifts caused by image
corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and, among
ResNet-50 models, adversarially chosen natural examples (ImageNet-A).


http://arxiv.org/abs/2110.09929
Minimal Multi-Layer Modifications of Deep Neural Networks. (4%)
Idan Refaeli; Guy Katz
  Deep neural networks (DNNs) have become increasingly popular in recent years.
However, despite their many successes, DNNs may also err and produce incorrect
and potentially fatal outputs in safety-critical settings, such as autonomous
driving, medical diagnosis, and airborne collision avoidance systems. Much work
has been put into detecting such erroneous behavior in DNNs, e.g., via testing
or verification, but removing these errors after their detection has received
lesser attention. We present here a new tool, called 3M-DNN, for repairing a
given DNN, which is known to err on some set of inputs. The novel repair
procedure implemented in 3M-DNN computes a modification to the network's
weights that corrects its behavior, and attempts to minimize this change via a
sequence of calls to a backend, black-box DNN verification engine. To the best
of our knowledge, our method is the first one that allows repairing the network
by simultaneously modifying multiple layers. This is achieved by splitting the
network into sub-networks, and applying a single-layer repairing technique to
each component. We evaluated 3M-DNN tool on an extensive set of benchmarks,
obtaining promising results.


http://arxiv.org/abs/2110.09903
Unrestricted Adversarial Attacks on ImageNet Competition. (99%)
Yuefeng Chen; Xiaofeng Mao; Yuan He; Hui Xue; Chao Li; Yinpeng Dong; Qi-An Fu; Xiao Yang; Wenzhao Xiang; Tianyu Pang; Hang Su; Jun Zhu; Fangcheng Liu; Chao Zhang; Hongyang Zhang; Yichi Zhang; Shilong Liu; Chang Liu; Wenzhao Xiang; Yajie Wang; Huipeng Zhou; Haoran Lyu; Yidan Xu; Zixuan Xu; Taoyu Zhu; Wenjun Li; Xianfeng Gao; Guoqiu Wang; Huanqian Yan; Ying Guo; Chaoning Zhang; Zheng Fang; Yang Wang; Bingyang Fu; Yunfei Zheng; Yekui Wang; Haorong Luo; Zhen Yang
  Many works have investigated the adversarial attacks or defenses under the
settings where a bounded and imperceptible perturbation can be added to the
input. However in the real-world, the attacker does not need to comply with
this restriction. In fact, more threats to the deep model come from
unrestricted adversarial examples, that is, the attacker makes large and
visible modifications on the image, which causes the model classifying
mistakenly, but does not affect the normal observation in human perspective.
Unrestricted adversarial attack is a popular and practical direction but has
not been studied thoroughly. We organize this competition with the purpose of
exploring more effective unrestricted adversarial attack algorithm, so as to
accelerate the academical research on the model robustness under stronger
unbounded attacks. The competition is held on the TianChi platform
(\url{https://tianchi.aliyun.com/competition/entrance/531853/introduction}) as
one of the series of AI Security Challengers Program.


http://arxiv.org/abs/2110.08956
Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training. (99%)
Alexander Daniel Pan; Daniel Yongkyun; Lee; Huan Zhang; Yize Chen; Yuanyuan Shi
  Due to the proliferation of renewable energy and its intrinsic intermittency
and stochasticity, current power systems face severe operational challenges.
Data-driven decision-making algorithms from reinforcement learning (RL) offer a
solution towards efficiently operating a clean energy system. Although RL
algorithms achieve promising performance compared to model-based control
models, there has been limited investigation of RL robustness in
safety-critical physical systems. In this work, we first show that several
competition-winning, state-of-the-art RL agents proposed for power system
control are vulnerable to adversarial attacks. Specifically, we use an
adversary Markov Decision Process to learn an attack policy, and demonstrate
the potency of our attack by successfully attacking multiple winning agents
from the Learning To Run a Power Network (L2RPN) challenge, under both
white-box and black-box attack settings. We then propose to use adversarial
training to increase the robustness of RL agent against attacks and avoid
infeasible operational decisions. To the best of our knowledge, our work is the
first to highlight the fragility of grid control RL algorithms, and contribute
an effective defense scheme towards improving their robustness and security.


http://arxiv.org/abs/2110.09983
ECG-ATK-GAN: Robustness against Adversarial Attacks on ECG using Conditional Generative Adversarial Networks. (97%)
Khondker Fariha Hossain; Sharif Amit Kamran; Xingjun Ma; Alireza Tavakkoli
  Recently deep learning has reached human-level performance in classifying
arrhythmia from Electrocardiogram (ECG). However, deep neural networks (DNN)
are vulnerable to adversarial attacks, which can misclassify ECG signals by
decreasing the model's precision. Adversarial attacks are crafted perturbations
injected in data that manifest the conventional DNN models to misclassify the
correct class. Thus, safety concerns arise as it becomes challenging to
establish the system's reliability, given that clinical applications require
high levels of trust. To mitigate this problem and make DNN models more robust
in clinical and real-life settings, we introduce a novel Conditional Generative
Adversarial Network (GAN), robust against adversarial attacked ECG signals and
retaining high accuracy. Furthermore, we compared it with other state-of-art
models to detect cardiac abnormalities from indistinguishable adversarial
attacked ECGs. The experiment confirms, our model is more robust against
adversarial attacks compared to other architectures.


http://arxiv.org/abs/2110.08760
Adapting Membership Inference Attacks to GNN for Graph Classification: Approaches and Implications. (22%)
Bang Wu; Xiangwen Yang; Shirui Pan; Xingliang Yuan
  Graph Neural Networks (GNNs) are widely adopted to analyse non-Euclidean
data, such as chemical networks, brain networks, and social networks, modelling
complex relationships and interdependency between objects. Recently, Membership
Inference Attack (MIA) against GNNs raises severe privacy concerns, where
training data can be leaked from trained GNN models. However, prior studies
focus on inferring the membership of only the components in a graph, e.g., an
individual node or edge. How to infer the membership of an entire graph record
is yet to be explored.
  In this paper, we take the first step in MIA against GNNs for graph-level
classification. Our objective is to infer whether a graph sample has been used
for training a GNN model. We present and implement two types of attacks, i.e.,
training-based attacks and threshold-based attacks from different adversarial
capabilities. We perform comprehensive experiments to evaluate our attacks in
seven real-world datasets using five representative GNN models. Both our
attacks are shown effective and can achieve high performance, i.e., reaching
over 0.7 attack F1 scores in most cases. Furthermore, we analyse the
implications behind the MIA against GNNs. Our findings confirm that GNNs can be
even more vulnerable to MIA than the models with non-graph structures. And
unlike the node-level classifier, MIAs on graph-level classification tasks are
more co-related with the overfitting level of GNNs rather than the statistic
property of their training graphs.


http://arxiv.org/abs/2110.08932
Poisoning Attacks on Fair Machine Learning. (12%)
Minh-Hao Van; Wei Du; Xintao Wu; Aidong Lu
  Both fair machine learning and adversarial learning have been extensively
studied. However, attacking fair machine learning models has received less
attention. In this paper, we present a framework that seeks to effectively
generate poisoning samples to attack both model accuracy and algorithmic
fairness. Our attacking framework can target fair machine learning models
trained with a variety of group based fairness notions such as demographic
parity and equalized odds. We develop three online attacks, adversarial
sampling , adversarial labeling, and adversarial feature modification. All
three attacks effectively and efficiently produce poisoning samples via
sampling, labeling, or modifying a fraction of training data in order to reduce
the test accuracy. Our framework enables attackers to flexibly adjust the
attack's focus on prediction accuracy or fairness and accurately quantify the
impact of each candidate point to both accuracy loss and fairness violation,
thus producing effective poisoning samples. Experiments on two real datasets
demonstrate the effectiveness and efficiency of our framework.


http://arxiv.org/abs/2110.08712
Black-box Adversarial Attacks on Network-wide Multi-step Traffic State Prediction Models. (99%)
Bibek Poudel; Weizi Li
  Traffic state prediction is necessary for many Intelligent Transportation
Systems applications. Recent developments of the topic have focused on
network-wide, multi-step prediction, where state of the art performance is
achieved via deep learning models, in particular, graph neural network-based
models. While the prediction accuracy of deep learning models is high, these
models' robustness has raised many safety concerns, given that imperceptible
perturbations added to input can substantially degrade the model performance.
In this work, we propose an adversarial attack framework by treating the
prediction model as a black-box, i.e., assuming no knowledge of the model
architecture, training data, and (hyper)parameters. However, we assume that the
adversary can oracle the prediction model with any input and obtain
corresponding output. Next, the adversary can train a substitute model using
input-output pairs and generate adversarial signals based on the substitute
model. To test the attack effectiveness, two state of the art, graph neural
network-based models (GCGRNN and DCRNN) are examined. As a result, the
adversary can degrade the target model's prediction accuracy up to $54\%$. In
comparison, two conventional statistical models (linear regression and
historical average) are also examined. While these two models do not produce
high prediction accuracy, they are either influenced negligibly (less than
$3\%$) or are immune to the adversary's attack.


http://arxiv.org/abs/2110.08514
Analyzing Dynamic Adversarial Training Data in the Limit. (82%)
Eric Wallace; Adina Williams; Robin Jia; Douwe Kiela
  To create models that are robust across a wide range of test inputs, training
datasets should include diverse examples that span numerous phenomena. Dynamic
adversarial data collection (DADC), where annotators craft examples that
challenge continually improving models, holds promise as an approach for
generating such diverse training sets. Prior work has shown that running DADC
over 1-3 rounds can help models fix some error types, but it does not
necessarily lead to better generalization beyond adversarial test data. We
argue that running DADC over many rounds maximizes its training-time benefits,
as the different rounds can together cover many of the task-relevant phenomena.
We present the first study of longer-term DADC, where we collect 20 rounds of
NLI examples for a small set of premise paragraphs, with both adversarial and
non-adversarial approaches. Models trained on DADC examples make 26% fewer
errors on our expert-curated test set compared to models trained on
non-adversarial data. Our analysis shows that DADC yields examples that are
more difficult, more lexically and syntactically diverse, and contain fewer
annotation artifacts compared to non-adversarial examples.


http://arxiv.org/abs/2110.08517
Characterizing Improper Input Validation Vulnerabilities of Mobile Crowdsourcing Services. (5%)
Sojhal Ismail Khan; Dominika Woszczyk; Chengzeng You; Soteris Demetriou; Muhammad Naveed
  Mobile crowdsourcing services (MCS), enable fast and economical data
acquisition at scale and find applications in a variety of domains. Prior work
has shown that Foursquare and Waze (a location-based and a navigation MCS) are
vulnerable to different kinds of data poisoning attacks. Such attacks can be
upsetting and even dangerous especially when they are used to inject improper
inputs to mislead users. However, to date, there is no comprehensive study on
the extent of improper input validation (IIV) vulnerabilities and the
feasibility of their exploits in MCSs across domains. In this work, we leverage
the fact that MCS interface with their participants through mobile apps to
design tools and new methodologies embodied in an end-to-end feedback-driven
analysis framework which we use to study 10 popular and previously unexplored
services in five different domains. Using our framework we send tens of
thousands of API requests with automatically generated input values to
characterize their IIV attack surface. Alarmingly, we found that most of them
(8/10) suffer from grave IIV vulnerabilities which allow an adversary to launch
data poisoning attacks at scale: 7400 spoofed API requests were successful in
faking online posts for robberies, gunshots, and other dangerous incidents,
faking fitness activities with supernatural speeds and distances among many
others. Lastly, we discuss easy to implement and deploy mitigation strategies
which can greatly reduce the IIV attack surface and argue for their use as a
necessary complementary measure working toward trustworthy mobile crowdsourcing
services.


http://arxiv.org/abs/2110.08690
Tackling the Imbalance for GNNs. (4%)
Rui Wang; Weixuan Xiong; Qinghu Hou; Ou Wu
  Different from deep neural networks for non-graph data classification, graph
neural networks (GNNs) leverage the information exchange between nodes (or
samples) when representing nodes. The category distribution shows an imbalance
or even a highly-skewed trend on nearly all existing benchmark GNN data sets.
The imbalanced distribution will cause misclassification of nodes in the
minority classes, and even cause the classification performance on the entire
data set to decrease. This study explores the effects of the imbalance problem
on the performances of GNNs and proposes new methodologies to solve it. First,
a node-level index, namely, the label difference index ($LDI$), is defined to
quantitatively analyze the relationship between imbalance and
misclassification. The less samples in a class, the higher the value of its
average $LDI$; the higher the $LDI$ of a sample, the more likely the sample
will be misclassified. We define a new loss and propose four new methods based
on $LDI$. Experimental results indicate that the classification accuracies of
the three among our proposed four new methods are better in both transductive
and inductive settings. The $LDI$ can be applied to other GNNs.


http://arxiv.org/abs/2110.08449
Adversarial Attacks on Gaussian Process Bandits. (99%)
Eric Han; Jonathan Scarlett
  Gaussian processes (GP) are a widely-adopted tool used to sequentially
optimize black-box functions, where evaluations are costly and potentially
noisy. Recent works on GP bandits have proposed to move beyond random noise and
devise algorithms robust to adversarial attacks. In this paper, we study this
problem from the attacker's perspective, proposing various adversarial attack
methods with differing assumptions on the attacker's strength and prior
information. Our goal is to understand adversarial attacks on GP bandits from
both a theoretical and practical perspective. We focus primarily on targeted
attacks on the popular GP-UCB algorithm and a related elimination-based
algorithm, based on adversarially perturbing the function $f$ to produce
another function $\tilde{f}$ whose optima are in some region $\mathcal{R}_{\rm
target}$. Based on our theoretical analysis, we devise both white-box attacks
(known $f$) and black-box attacks (unknown $f$), with the former including a
Subtraction attack and Clipping attack, and the latter including an Aggressive
subtraction attack. We demonstrate that adversarial attacks on GP bandits can
succeed in forcing the algorithm towards $\mathcal{R}_{\rm target}$ even with a
low attack budget, and we compare our attacks' performance and efficiency on
several real and synthetic functions.


http://arxiv.org/abs/2110.08036
Generating Natural Language Adversarial Examples through An Improved Beam Search Algorithm. (99%)
Tengfei Zhao; Zhaocheng Ge; Hanping Hu; Dingmeng Shi
  The research of adversarial attacks in the text domain attracts many
interests in the last few years, and many methods with a high attack success
rate have been proposed. However, these attack methods are inefficient as they
require lots of queries for the victim model when crafting text adversarial
examples. In this paper, a novel attack model is proposed, its attack success
rate surpasses the benchmark attack methods, but more importantly, its attack
efficiency is much higher than the benchmark attack methods. The novel method
is empirically evaluated by attacking WordCNN, LSTM, BiLSTM, and BERT on four
benchmark datasets. For instance, it achieves a 100\% attack success rate
higher than the state-of-the-art method when attacking BERT and BiLSTM on IMDB,
but the number of queries for the victim models only is 1/4 and 1/6.5 of the
state-of-the-art method, respectively. Also, further experiments show the novel
method has a good transferability on the generated adversarial examples.


http://arxiv.org/abs/2110.08042
Adversarial Attacks on ML Defense Models Competition. (99%)
Yinpeng Dong; Qi-An Fu; Xiao Yang; Wenzhao Xiang; Tianyu Pang; Hang Su; Jun Zhu; Jiayu Tang; Yuefeng Chen; XiaoFeng Mao; Yuan He; Hui Xue; Chao Li; Ye Liu; Qilong Zhang; Lianli Gao; Yunrui Yu; Xitong Gao; Zhe Zhao; Daquan Lin; Jiadong Lin; Chuanbiao Song; Zihao Wang; Zhennan Wu; Yang Guo; Jiequan Cui; Xiaogang Xu; Pengguang Chen
  Due to the vulnerability of deep neural networks (DNNs) to adversarial
examples, a large number of defense techniques have been proposed to alleviate
this problem in recent years. However, the progress of building more robust
models is usually hampered by the incomplete or incorrect robustness
evaluation. To accelerate the research on reliable evaluation of adversarial
robustness of the current defense models in image classification, the TSAIL
group at Tsinghua University and the Alibaba Security group organized this
competition along with a CVPR 2021 workshop on adversarial machine learning
(https://aisecure-workshop.github.io/amlcvpr2021/). The purpose of this
competition is to motivate novel attack algorithms to evaluate adversarial
robustness more effectively and reliably. The participants were encouraged to
develop stronger white-box attack algorithms to find the worst-case robustness
of different defenses. This competition was conducted on an adversarial
robustness evaluation platform -- ARES (https://github.com/thu-ml/ares), and is
held on the TianChi platform
(https://tianchi.aliyun.com/competition/entrance/531847/introduction) as one of
the series of AI Security Challengers Program. After the competition, we
summarized the results and established a new adversarial robustness benchmark
at https://ml.cs.tsinghua.edu.cn/ares-bench/, which allows users to upload
adversarial attack algorithms and defense models for evaluation.


http://arxiv.org/abs/2110.08324
Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture. (76%)
Xinyu Tang; Saeed Mahloujifar; Liwei Song; Virat Shejwalkar; Milad Nasr; Amir Houmansadr; Prateek Mittal
  Membership inference attacks are a key measure to evaluate privacy leakage in
machine learning (ML) models. These attacks aim to distinguish training members
from non-members by exploiting differential behavior of the models on member
and non-member inputs. The goal of this work is to train ML models that have
high membership privacy while largely preserving their utility; we therefore
aim for an empirical membership privacy guarantee as opposed to the provable
privacy guarantees provided by techniques like differential privacy, as such
techniques are shown to deteriorate model utility. Specifically, we propose a
new framework to train privacy-preserving models that induces similar behavior
on member and non-member inputs to mitigate membership inference attacks. Our
framework, called SELENA, has two major components. The first component and the
core of our defense is a novel ensemble architecture for training. This
architecture, which we call Split-AI, splits the training data into random
subsets, and trains a model on each subset of the data. We use an adaptive
inference strategy at test time: our ensemble architecture aggregates the
outputs of only those models that did not contain the input sample in their
training data. We prove that our Split-AI architecture defends against a large
family of membership inference attacks, however, it is susceptible to new
adaptive attacks. Therefore, we use a second component in our framework called
Self-Distillation to protect against such stronger attacks. The
Self-Distillation component (self-)distills the training dataset through our
Split-AI ensemble, without using any external public datasets. Through
extensive experiments on major benchmark datasets we show that SELENA presents
a superior trade-off between membership privacy and utility compared to the
state of the art.


http://arxiv.org/abs/2110.08322
Robustness of different loss functions and their impact on networks learning capability. (76%)
Vishal Rajput
  Recent developments in AI have made it ubiquitous, every industry is trying
to adopt some form of intelligent processing of their data. Despite so many
advances in the field, AIs full capability is yet to be exploited by the
industry. Industries that involve some risk factors still remain cautious about
the usage of AI due to the lack of trust in such autonomous systems.
Present-day AI might be very good in a lot of things but it is very bad in
reasoning and this behavior of AI can lead to catastrophic results. Autonomous
cars crashing into a person or a drone getting stuck in a tree are a few
examples where AI decisions lead to catastrophic results. To develop insight
and generate an explanation about the learning capability of AI, we will try to
analyze the working of loss functions. For our case, we will use two sets of
loss functions, generalized loss functions like Binary cross-entropy or BCE and
specialized loss functions like Dice loss or focal loss. Through a series of
experiments, we will establish whether combining different loss functions is
better than using a single loss function and if yes, then what is the reason
behind it. In order to establish the difference between generalized loss and
specialized losses, we will train several models using the above-mentioned
losses and then compare their robustness on adversarial examples. In
particular, we will look at how fast the accuracy of different models decreases
when we change the pixels corresponding to the most salient gradients.


http://arxiv.org/abs/2110.08139
Chunked-Cache: On-Demand and Scalable Cache Isolation for Security Architectures. (22%)
Ghada Dessouky; Alexander Gruler; Pouya Mahmoody; Ahmad-Reza Sadeghi; Emmanuel Stapf
  Shared cache resources in multi-core processors are vulnerable to cache
side-channel attacks. Recently proposed defenses have their own caveats:
Randomization-based defenses are vulnerable to the evolving attack algorithms
besides relying on weak cryptographic primitives, because they do not
fundamentally address the root cause for cache side-channel attacks. Cache
partitioning defenses, on the other hand, provide the strict resource
partitioning and effectively block all side-channel threats. However, they
usually rely on way-based partitioning which is not fine-grained and cannot
scale to support a larger number of protection domains, e.g., in trusted
execution environment (TEE) security architectures, besides degrading
performance and often resulting in cache underutilization.
  To overcome the shortcomings of both approaches, we present a novel and
flexible set-associative cache partitioning design for TEE architectures,
called Chunked-Cache. Chunked-Cache enables an execution context to "carve" out
an exclusive configurable chunk of the cache if the execution requires
side-channel resilience. If side-channel resilience is not required, mainstream
cache resources are freely utilized. Hence, our solution addresses the
security-performance trade-off practically by enabling selective and on-demand
utilization of side-channel-resilient caches, while providing well-grounded
future-proof security guarantees. We show that Chunked-Cache provides
side-channel-resilient cache utilization for sensitive code execution, with
small hardware overhead, while incurring no performance overhead on the OS. We
also show that it outperforms conventional way-based cache partitioning by 43%,
while scaling significantly better to support a larger number of protection
domains.


http://arxiv.org/abs/2110.08247
Textual Backdoor Attacks Can Be More Harmful via Two Simple Tricks. (10%)
Yangyi Chen; Fanchao Qi; Zhiyuan Liu; Maosong Sun
  Backdoor attacks are a kind of emergent security threat in deep learning.
When a deep neural model is injected with a backdoor, it will behave normally
on standard inputs but give adversary-specified predictions once the input
contains specific backdoor triggers. Current textual backdoor attacks have poor
attack performance in some tough situations. In this paper, we find two simple
tricks that can make existing textual backdoor attacks much more harmful. The
first trick is to add an extra training task to distinguish poisoned and clean
data during the training of the victim model, and the second one is to use all
the clean training data rather than remove the original clean data
corresponding to the poisoned data. These two tricks are universally applicable
to different attack models. We conduct experiments in three tough situations
including clean data fine-tuning, low poisoning rate, and label-consistent
attacks. Experimental results show that the two tricks can significantly
improve attack performance. This paper exhibits the great potential harmfulness
of backdoor attacks. All the code and data will be made public to facilitate
further research.


http://arxiv.org/abs/2110.07858
Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation. (8%)
Yao Qin; Chiyuan Zhang; Ting Chen; Balaji Lakshminarayanan; Alex Beutel; Xuezhi Wang
  We investigate the robustness of vision transformers (ViTs) through the lens
of their special patch-based architectural structure, i.e., they process an
image as a sequence of image patches. We find that ViTs are surprisingly
insensitive to patch-based transformations, even when the transformation
largely destroys the original semantics and makes the image unrecognizable by
humans. This indicates that ViTs heavily use features that survived such
transformations but are generally not indicative of the semantic class to
humans. Further investigations show that these features are useful but
non-robust, as ViTs trained on them can achieve high in-distribution accuracy,
but break down under distribution shifts. From this understanding, we ask: can
training the model to rely less on these features improve ViT robustness and
out-of-distribution performance? We use the images transformed with our
patch-based operations as negatively augmented views and offer losses to
regularize the training away from using non-robust features. This is a
complementary view to existing research that mostly focuses on augmenting
inputs with semantic-preserving transformations to enforce models' invariance.
We show that patch-based negative augmentation consistently improves robustness
of ViTs across a wide set of ImageNet based robustness benchmarks. Furthermore,
we find our patch-based negative augmentation are complementary to traditional
(positive) data augmentation, and together boost the performance further. All
the code in this work will be open-sourced.


http://arxiv.org/abs/2110.08113
Hand Me Your PIN! Inferring ATM PINs of Users Typing with a Covered Hand. (1%)
Matteo Cardaioli; Stefano Cecconello; Mauro Conti; Simone Milani; Stjepan Picek; Eugen Saraci
  Automated Teller Machines (ATMs) represent the most used system for
withdrawing cash. The European Central Bank reported more than 11 billion cash
withdrawals and loading/unloading transactions on the European ATMs in 2019.
Although ATMs have undergone various technological evolutions, Personal
Identification Numbers (PINs) are still the most common authentication method
for these devices. Unfortunately, the PIN mechanism is vulnerable to
shoulder-surfing attacks performed via hidden cameras installed near the ATM to
catch the PIN pad. To overcome this problem, people get used to covering the
typing hand with the other hand. While such users probably believe this
behavior is safe enough to protect against mentioned attacks, there is no clear
assessment of this countermeasure in the scientific literature.
  This paper proposes a novel attack to reconstruct PINs entered by victims
covering the typing hand with the other hand. We consider the setting where the
attacker can access an ATM PIN pad of the same brand/model as the target one.
Afterward, the attacker uses that model to infer the digits pressed by the
victim while entering the PIN. Our attack owes its success to a carefully
selected deep learning architecture that can infer the PIN from the typing hand
position and movements. We run a detailed experimental analysis including 58
users. With our approach, we can guess 30% of the 5-digit PINs within three
attempts -- the ones usually allowed by ATM before blocking the card. We also
conducted a survey with 78 users that managed to reach an accuracy of only
7.92% on average for the same setting. Finally, we evaluate a shielding
countermeasure that proved to be rather inefficient unless the whole keypad is
shielded.


http://arxiv.org/abs/2110.07182
Adversarial examples by perturbing high-level features in intermediate decoder layers. (99%)
Vojtěch Čermák; Lukáš Adam
  We propose a novel method for creating adversarial examples. Instead of
perturbing pixels, we use an encoder-decoder representation of the input image
and perturb intermediate layers in the decoder. This changes the high-level
features provided by the generative model. Therefore, our perturbation
possesses semantic meaning, such as a longer beak or green tints. We formulate
this task as an optimization problem by minimizing the Wasserstein distance
between the adversarial and initial images under a misclassification
constraint. We employ the projected gradient method with a simple inexact
projection. Due to the projection, all iterations are feasible, and our method
always generates adversarial images. We perform numerical experiments on the
MNIST and ImageNet datasets in both targeted and untargeted settings. We
demonstrate that our adversarial images are much less vulnerable to
steganographic defence techniques than pixel-based attacks. Moreover, we show
that our method modifies key features such as edges and that defence techniques
based on adversarial training are vulnerable to our attacks.


http://arxiv.org/abs/2110.07305
DI-AA: An Interpretable White-box Attack for Fooling Deep Neural Networks. (99%)
Yixiang Wang; Jiqiang Liu; Xiaolin Chang; Jianhua Wang; Ricardo J. Rodríguez
  White-box Adversarial Example (AE) attacks towards Deep Neural Networks
(DNNs) have a more powerful destructive capacity than black-box AE attacks in
the fields of AE strategies. However, almost all the white-box approaches lack
interpretation from the point of view of DNNs. That is, adversaries did not
investigate the attacks from the perspective of interpretable features, and few
of these approaches considered what features the DNN actually learns. In this
paper, we propose an interpretable white-box AE attack approach, DI-AA, which
explores the application of the interpretable approach of the deep Taylor
decomposition in the selection of the most contributing features and adopts the
Lagrangian relaxation optimization of the logit output and L_p norm to further
decrease the perturbation. We compare DI-AA with six baseline attacks
(including the state-of-the-art attack AutoAttack) on three datasets.
Experimental results reveal that our proposed approach can 1) attack non-robust
models with comparatively low perturbation, where the perturbation is closer to
or lower than the AutoAttack approach; 2) break the TRADES adversarial training
models with the highest success rate; 3) the generated AE can reduce the robust
accuracy of the robust black-box models by 16% to 31% in the black-box transfer
attack.


http://arxiv.org/abs/2110.07801
Adversarial Purification through Representation Disentanglement. (99%)
Tao Bai; Jun Zhao; Lanqing Guo; Bihan Wen
  Deep learning models are vulnerable to adversarial examples and make
incomprehensible mistakes, which puts a threat on their real-world deployment.
Combined with the idea of adversarial training, preprocessing-based defenses
are popular and convenient to use because of their task independence and good
generalizability. Current defense methods, especially purification, tend to
remove ``noise" by learning and recovering the natural images. However,
different from random noise, the adversarial patterns are much easier to be
overfitted during model training due to their strong correlation to the images.
In this work, we propose a novel adversarial purification scheme by presenting
disentanglement of natural images and adversarial perturbations as a
preprocessing defense. With extensive experiments, our defense is shown to be
generalizable and make significant protection against unseen strong adversarial
attacks. It reduces the success rates of state-of-the-art \textbf{ensemble}
attacks from \textbf{61.7\%} to \textbf{14.9\%} on average, superior to a
number of existing methods. Notably, our defense restores the perturbed images
perfectly and does not hurt the clean accuracy of backbone models, which is
highly desirable in practice.


http://arxiv.org/abs/2110.07831
RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models. (93%)
Wenkai Yang; Yankai Lin; Peng Li; Jie Zhou; Xu Sun
  Backdoor attacks, which maliciously control a well-trained model's outputs of
the instances with specific triggers, are recently shown to be serious threats
to the safety of reusing deep neural networks (DNNs). In this work, we propose
an efficient online defense mechanism based on robustness-aware perturbations.
Specifically, by analyzing the backdoor training process, we point out that
there exists a big gap of robustness between poisoned and clean samples.
Motivated by this observation, we construct a word-based robustness-aware
perturbation to distinguish poisoned samples from clean samples to defend
against the backdoor attacks on natural language processing (NLP) models.
Moreover, we give a theoretical analysis about the feasibility of our
robustness-aware perturbation-based defense method. Experimental results on
sentiment analysis and toxic detection tasks show that our method achieves
better defending performance and much lower computational costs than existing
online defense methods. Our code is available at
https://github.com/lancopku/RAP.


http://arxiv.org/abs/2110.07683
An Optimization Perspective on Realizing Backdoor Injection Attacks on Deep Neural Networks in Hardware. (82%)
M. Caner Tol; Saad Islam; Berk Sunar; Ziming Zhang
  State-of-the-art deep neural networks (DNNs) have been proven to be
vulnerable to adversarial manipulation and backdoor attacks. Backdoored models
deviate from expected behavior on inputs with predefined triggers while
retaining performance on clean data. Recent works focus on software simulation
of backdoor injection during the inference phase by modifying network weights,
which we find often unrealistic in practice due to the hardware restriction
such as bit allocation in memory. In contrast, in this work, we investigate the
viability of backdoor injection attacks in real-life deployments of DNNs on
hardware and address such practical issues in hardware implementation from a
novel optimization perspective. We are motivated by the fact that the
vulnerable memory locations are very rare, device-specific, and sparsely
distributed. Consequently, we propose a novel network training algorithm based
on constrained optimization for realistic backdoor injection attack in
hardware. By modifying parameters uniformly across the convolutional and
fully-connected layers as well as optimizing the trigger pattern together, we
achieve the state-of-the-art attack performance with fewer bit flips. For
instance, our method on a hardware-deployed ResNet-20 model trained on CIFAR-10
can achieve over 91% test accuracy and 94% attack success rate by flipping only
10 bits out of 2.2 million bits.


http://arxiv.org/abs/2110.07667
Interactive Analysis of CNN Robustness. (80%)
Stefan Sietzen; Mathias Lechner; Judy Borowski; Ramin Hasani; Manuela Waldner
  While convolutional neural networks (CNNs) have found wide adoption as
state-of-the-art models for image-related tasks, their predictions are often
highly sensitive to small input perturbations, which the human vision is robust
against. This paper presents Perturber, a web-based application that allows
users to instantaneously explore how CNN activations and predictions evolve
when a 3D input scene is interactively perturbed. Perturber offers a large
variety of scene modifications, such as camera controls, lighting and shading
effects, background modifications, object morphing, as well as adversarial
attacks, to facilitate the discovery of potential vulnerabilities. Fine-tuned
model versions can be directly compared for qualitative evaluation of their
robustness. Case studies with machine learning experts have shown that
Perturber helps users to quickly generate hypotheses about model
vulnerabilities and to qualitatively compare model behavior. Using quantitative
analyses, we could replicate users' insights with other CNN architectures and
input images, yielding new insights about the vulnerability of adversarially
trained models.


http://arxiv.org/abs/2110.07462
On Adversarial Vulnerability of PHM algorithms: An Initial Study. (69%)
Weizhong Yan; Zhaoyuan Yang; Jianwei Qiu
  With proliferation of deep learning (DL) applications in diverse domains,
vulnerability of DL models to adversarial attacks has become an increasingly
interesting research topic in the domains of Computer Vision (CV) and Natural
Language Processing (NLP). DL has also been widely adopted to diverse PHM
applications, where data are primarily time-series sensor measurements. While
those advanced DL algorithms/models have resulted in an improved PHM
algorithms' performance, the vulnerability of those PHM algorithms to
adversarial attacks has not drawn much attention in the PHM community. In this
paper we attempt to explore the vulnerability of PHM algorithms. More
specifically, we investigate the strategies of attacking PHM algorithms by
considering several unique characteristics associated with time-series sensor
measurements data. We use two real-world PHM applications as examples to
validate our attack strategies and to demonstrate that PHM algorithms indeed
are vulnerable to adversarial attacks.


http://arxiv.org/abs/2110.07736
Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models. (61%)
Tianlu Wang; Diyi Yang; Xuezhi Wang
  Recently, NLP models have achieved remarkable progress across a variety of
tasks; however, they have also been criticized for being not robust. Many
robustness problems can be attributed to models exploiting spurious
correlations, or shortcuts between the training data and the task labels.
Models may fail to generalize to out-of-distribution data or be vulnerable to
adversarial attacks if spurious correlations are exploited through the training
process. In this paper, we aim to automatically identify such spurious
correlations in NLP models at scale. We first leverage existing
interpretability methods to extract tokens that significantly affect model's
decision process from the input text. We then distinguish "genuine" tokens and
"spurious" tokens by analyzing model predictions across multiple corpora and
further verify them through knowledge-aware perturbations. We show that our
proposed method can effectively and efficiently identify a scalable set of
"shortcuts", and mitigating these leads to more robust models in multiple
applications.


http://arxiv.org/abs/2110.07537
Toward Degradation-Robust Voice Conversion. (9%)
Chien-yu Huang; Kai-Wei Chang; Hung-yi Lee
  Any-to-any voice conversion technologies convert the vocal timbre of an
utterance to any speaker even unseen during training. Although there have been
several state-of-the-art any-to-any voice conversion models, they were all
based on clean utterances to convert successfully. However, in real-world
scenarios, it is difficult to collect clean utterances of a speaker, and they
are usually degraded by noises or reverberations. It thus becomes highly
desired to understand how these degradations affect voice conversion and build
a degradation-robust model. We report in this paper the first comprehensive
study on the degradation robustness of any-to-any voice conversion. We show
that the performance of state-of-the-art models nowadays was severely hampered
given degraded utterances. To this end, we then propose speech enhancement
concatenation and denoising training to improve the robustness. In addition to
common degradations, we also consider adversarial noises, which alter the model
output significantly yet are human-imperceptible. It was shown that both
concatenations with off-the-shelf speech enhancement models and denoising
training on voice conversion models could improve the robustness, while each of
them had pros and cons.


http://arxiv.org/abs/2110.07596
Retrieval-guided Counterfactual Generation for QA. (2%)
Bhargavi Paranjape; Matthew Lamm; Ian Tenney
  Deep NLP models have been shown to learn spurious correlations, leaving them
brittle to input perturbations. Recent work has shown that counterfactual or
contrastive data -- i.e. minimally perturbed inputs -- can reveal these
weaknesses, and that data augmentation using counterfactuals can help
ameliorate them. Proposed techniques for generating counterfactuals rely on
human annotations, perturbations based on simple heuristics, and meaning
representation frameworks. We focus on the task of creating counterfactuals for
question answering, which presents unique challenges related to world
knowledge, semantic diversity, and answerability. To address these challenges,
we develop a Retrieve-Generate-Filter(RGF) technique to create counterfactual
evaluation and training data with minimal human supervision. Using an
open-domain QA framework and question generation model trained on original task
data, we create counterfactuals that are fluent, semantically diverse, and
automatically labeled. Data augmentation with RGF counterfactuals improves
performance on out-of-domain and challenging evaluation sets over and above
existing methods, in both the reading comprehension and open-domain QA
settings. Moreover, we find that RGF data leads to significant improvements in
a model's robustness to local perturbations.


http://arxiv.org/abs/2110.08260
Effective Certification of Monotone Deep Equilibrium Models. (1%)
Mark Niklas Müller; Robin Staab; Marc Fischer; Martin Vechev
  Monotone Operator Equilibrium Models (monDEQs) represent a class of models
combining the powerful deep equilibrium paradigm with convergence guarantees.
Further, their inherent robustness to adversarial perturbations makes
investigating their certifiability a promising research direction.
Unfortunately, existing approaches are either imprecise or severely limited in
scalability. In this work, we propose the first scalable and precise monDEQ
verifier, based on two key ideas: (i) a novel convex relaxation enabling
efficient inclusion checks, and (ii) non-trivial mathematical insights
characterizing the fixpoint operations at the heart of monDEQs on sets rather
than concrete inputs. An extensive evaluation of our verifier on the
challenging $\ell_\infty$ perturbations demonstrates that it exceeds
state-of-the-art performance in terms of speed (two orders of magnitude) and
scalability (an order of magnitude) while yielding 25% higher certified
accuracies on the same networks.


http://arxiv.org/abs/2110.06816
A Framework for Verification of Wasserstein Adversarial Robustness. (99%)
Tobias Wegel; Felix Assion; David Mickisch; Florens Greßner
  Machine learning image classifiers are susceptible to adversarial and
corruption perturbations. Adding imperceptible noise to images can lead to
severe misclassifications of the machine learning model. Using $L_p$-norms for
measuring the size of the noise fails to capture human similarity perception,
which is why optimal transport based distance measures like the Wasserstein
metric are increasingly being used in the field of adversarial robustness.
Verifying the robustness of classifiers using the Wasserstein metric can be
achieved by proving the absence of adversarial examples (certification) or
proving their presence (attack). In this work we present a framework based on
the work by Levine and Feizi, which allows us to transfer existing
certification methods for convex polytopes or $L_1$-balls to the Wasserstein
threat model. The resulting certification can be complete or incomplete,
depending on whether convex polytopes or $L_1$-balls were chosen. Additionally,
we present a new Wasserstein adversarial attack that is projected gradient
descent based and which has a significantly reduced computational burden
compared to existing attack approaches.


http://arxiv.org/abs/2110.06802
Identification of Attack-Specific Signatures in Adversarial Examples. (99%)
Hossein Souri; Pirazh Khorramshahi; Chun Pong Lau; Micah Goldblum; Rama Chellappa
  The adversarial attack literature contains a myriad of algorithms for
crafting perturbations which yield pathological behavior in neural networks. In
many cases, multiple algorithms target the same tasks and even enforce the same
constraints. In this work, we show that different attack algorithms produce
adversarial examples which are distinct not only in their effectiveness but
also in how they qualitatively affect their victims. We begin by demonstrating
that one can determine the attack algorithm that crafted an adversarial
example. Then, we leverage recent advances in parameter-space saliency maps to
show, both visually and quantitatively, that adversarial attack algorithms
differ in which parts of the network and image they target. Our findings
suggest that prospective adversarial attacks should be compared not only via
their success rates at fooling models but also via deeper downstream effects
they have on victims.


http://arxiv.org/abs/2110.08256
Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness. (99%)
Xiao Yang; Yinpeng Dong; Wenzhao Xiang; Tianyu Pang; Hang Su; Jun Zhu
  The vulnerability of deep neural networks to adversarial examples has
motivated an increasing number of defense strategies for promoting model
robustness. However, the progress is usually hampered by insufficient
robustness evaluations. As the de facto standard to evaluate adversarial
robustness, adversarial attacks typically solve an optimization problem of
crafting adversarial examples with an iterative process. In this work, we
propose a Model-Agnostic Meta-Attack (MAMA) approach to discover stronger
attack algorithms automatically. Our method learns the optimizer in adversarial
attacks parameterized by a recurrent neural network, which is trained over a
class of data samples and defenses to produce effective update directions
during adversarial example generation. Furthermore, we develop a model-agnostic
training algorithm to improve the generalization ability of the learned
optimizer when attacking unseen defenses. Our approach can be flexibly
incorporated with various attacks and consistently improves the performance
with little extra computational cost. Extensive experiments demonstrate the
effectiveness of the learned attacks by MAMA compared to the state-of-the-art
attacks on different defenses, leading to a more reliable evaluation of
adversarial robustness.


http://arxiv.org/abs/2110.07139
Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. (98%)
Fanchao Qi; Yangyi Chen; Xurui Zhang; Mukai Li; Zhiyuan Liu; Maosong Sun
  Adversarial attacks and backdoor attacks are two common security threats that
hang over deep learning. Both of them harness task-irrelevant features of data
in their implementation. Text style is a feature that is naturally irrelevant
to most NLP tasks, and thus suitable for adversarial and backdoor attacks. In
this paper, we make the first attempt to conduct adversarial and backdoor
attacks based on text style transfer, which is aimed at altering the style of a
sentence while preserving its meaning. We design an adversarial attack method
and a backdoor attack method, and conduct extensive experiments to evaluate
them. Experimental results show that popular NLP models are vulnerable to both
adversarial and backdoor attacks based on text style transfer -- the attack
success rates can exceed 90% without much effort. It reflects the limited
ability of NLP models to handle the feature of text style that has not been
widely realized. In addition, the style transfer-based adversarial and backdoor
attack methods show superiority to baselines in many aspects. All the code and
data of this paper can be obtained at https://github.com/thunlp/StyleAttack.


http://arxiv.org/abs/2110.07120
Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack. (93%)
Davis Brown; Henry Kvinge
  Methods for model explainability have become increasingly critical for
testing the fairness and soundness of deep learning. A number of explainability
techniques have been developed which use a set of examples to represent a
human-interpretable concept in a model's activations. In this work we show that
these explainability methods can suffer the same vulnerability to adversarial
attacks as the models they are meant to analyze. We demonstrate this phenomenon
on two well-known concept-based approaches to the explainability of deep
learning models: TCAV and faceted feature visualization. We show that by
carefully perturbing the examples of the concept that is being investigated, we
can radically change the output of the interpretability method, e.g. showing
that stripes are not an important factor in identifying images of a zebra. Our
work highlights the fact that in safety-critical applications, there is need
for security around not only the machine learning pipeline but also the model
interpretation process.


http://arxiv.org/abs/2110.06904
Traceback of Data Poisoning Attacks in Neural Networks. (92%)
Shawn Shan; Arjun Nitin Bhagoji; Haitao Zheng; Ben Y. Zhao
  In adversarial machine learning, new defenses against attacks on deep
learning systems are routinely broken soon after their release by more powerful
attacks. In this context, forensic tools can offer a valuable complement to
existing defenses, by tracing back a successful attack to its root cause, and
offering a path forward for mitigation to prevent similar attacks in the
future.
  In this paper, we describe our efforts in developing a forensic traceback
tool for poison attacks on deep neural networks. We propose a novel iterative
clustering and pruning solution that trims "innocent" training samples, until
all that remains is the set of poisoned data responsible for the attack. Our
method clusters training samples based on their impact on model parameters,
then uses an efficient data unlearning method to prune innocent clusters. We
empirically demonstrate the efficacy of our system on three types of
dirty-label (backdoor) poison attacks and three types of clean-label poison
attacks, across domains of computer vision and malware classification. Our
system achieves over 98.4% precision and 96.8% recall across all attacks. We
also show that our system is robust against four anti-forensics measures
specifically designed to attack it.


http://arxiv.org/abs/2110.06513
Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions. (1%)
Chenyu Yi; SIYUAN YANG; Haoliang Li; Yap-peng Tan; Alex Kot
  The state-of-the-art deep neural networks are vulnerable to common
corruptions (e.g., input data degradations, distortions, and disturbances
caused by weather changes, system error, and processing). While much progress
has been made in analyzing and improving the robustness of models in image
understanding, the robustness in video understanding is largely unexplored. In
this paper, we establish a corruption robustness benchmark, Mini Kinetics-C and
Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in
images. We make the first attempt to conduct an exhaustive study on the
corruption robustness of established CNN-based and Transformer-based
spatial-temporal models. The study provides some guidance on robust model
design and training: Transformer-based model performs better than CNN-based
models on corruption robustness; the generalization ability of spatial-temporal
models implies robustness against temporal corruptions; model corruption
robustness (especially robustness in the temporal domain) enhances with
computational cost and model capacity, which may contradict the current trend
of improving the computational efficiency of models. Moreover, we find the
robustness intervention for image-related tasks (e.g., training models with
noise) may not work for spatial-temporal models.


http://arxiv.org/abs/2110.06468
Graph-Fraudster: Adversarial Attacks on Graph Neural Network Based Vertical Federated Learning. (99%)
Jinyin Chen; Guohan Huang; Shanqing Yu; Wenrong Jiang; Chen Cui
  Graph neural network (GNN) models have achieved great success on graph
representation learning. Challenged by large scale private data collection from
user-side, GNN models may not be able to reflect the excellent performance,
without rich features and complete adjacent relationships. Addressing to the
problem, vertical federated learning (VFL) is proposed to implement local data
protection through training a global model collaboratively. Consequently, for
graph-structured data, it is natural idea to construct VFL framework with GNN
models. However, GNN models are proven to be vulnerable to adversarial attacks.
Whether the vulnerability will be brought into the VFL has not been studied. In
this paper, we devote to study the security issues of GNN based VFL (GVFL),
i.e., robustness against adversarial attacks. Further, we propose an
adversarial attack method, named Graph-Fraudster. It generates adversarial
perturbations based on the noise-added global node embeddings via GVFL's
privacy leakage, and the gradient of pairwise node. First, it steals the global
node embeddings and sets up a shadow server model for attack generator. Second,
noises are added into node embeddings to confuse the shadow server model. At
last, the gradient of pairwise node is used to generate attacks with the
guidance of noise-added node embeddings. To the best of our knowledge, this is
the first study of adversarial attacks on GVFL. The extensive experiments on
five benchmark datasets demonstrate that Graph-Fraudster performs better than
three possible baselines in GVFL. Furthermore, Graph-Fraudster can remain a
threat to GVFL even if two possible defense mechanisms are applied. This paper
reveals that GVFL is vulnerable to adversarial attack similar to centralized
GNN models.


http://arxiv.org/abs/2110.07718
Adversarial Attack across Datasets. (99%)
Yunxiao Qin; Yuanhao Xiong; Jinfeng Yi; Cho-Jui Hsieh
  It has been observed that Deep Neural Networks (DNNs) are vulnerable to
transfer attacks in the query-free black-box setting. However, all the previous
studies on transfer attack assume that the white-box surrogate models possessed
by the attacker and the black-box victim models are trained on the same
dataset, which means the attacker implicitly knows the label set and the input
size of the victim model. However, this assumption is usually unrealistic as
the attacker may not know the dataset used by the victim model, and further,
the attacker needs to attack any randomly encountered images that may not come
from the same dataset. Therefore, in this paper we define a new Generalized
Transferable Attack (GTA) problem where we assume the attacker has a set of
surrogate models trained on different datasets (with different label sets and
image sizes), and none of them is equal to the dataset used by the victim
model. We then propose a novel method called Image Classification Eraser (ICE)
to erase classification information for any encountered images from arbitrary
dataset. Extensive experiments on Cifar-10, Cifar-100, and TieredImageNet
demonstrate the effectiveness of the proposed ICE on the GTA problem.
Furthermore, we show that existing transfer attack methods can be modified to
tackle the GTA problem, but with significantly worse performance compared with
ICE.


http://arxiv.org/abs/2110.05748
SEPP: Similarity Estimation of Predicted Probabilities for Defending and Detecting Adversarial Text. (92%)
Hoang-Quoc Nguyen-Son; Seira Hidano; Kazuhide Fukushima; Shinsaku Kiyomoto
  There are two cases describing how a classifier processes input text, namely,
misclassification and correct classification. In terms of misclassified texts,
a classifier handles the texts with both incorrect predictions and adversarial
texts, which are generated to fool the classifier, which is called a victim.
Both types are misunderstood by the victim, but they can still be recognized by
other classifiers. This induces large gaps in predicted probabilities between
the victim and the other classifiers. In contrast, text correctly classified by
the victim is often successfully predicted by the others and induces small
gaps. In this paper, we propose an ensemble model based on similarity
estimation of predicted probabilities (SEPP) to exploit the large gaps in the
misclassified predictions in contrast to small gaps in the correct
classification. SEPP then corrects the incorrect predictions of the
misclassified texts. We demonstrate the resilience of SEPP in defending and
detecting adversarial texts through different types of victim classifiers,
classification tasks, and adversarial attacks.


http://arxiv.org/abs/2110.06018
On the Security Risks of AutoML. (45%)
Ren Pang; Zhaohan Xi; Shouling Ji; Xiapu Luo; Ting Wang
  Neural Architecture Search (NAS) represents an emerging machine learning (ML)
paradigm that automatically searches for models tailored to given tasks, which
greatly simplifies the development of ML systems and propels the trend of ML
democratization. Yet, little is known about the potential security risks
incurred by NAS, which is concerning given the increasing use of NAS-generated
models in critical domains.
  This work represents a solid initial step towards bridging the gap. Through
an extensive empirical study of 10 popular NAS methods, we show that compared
with their manually designed counterparts, NAS-generated models tend to suffer
greater vulnerability to various malicious attacks (e.g., adversarial evasion,
model poisoning, and functionality stealing). Further, with both empirical and
analytical evidence, we provide possible explanations for such phenomena: given
the prohibitive search space and training cost, most NAS methods favor models
that converge fast at early training stages; this preference results in
architectural properties associated with attack vulnerability (e.g., high loss
smoothness and low gradient variance). Our findings not only reveal the
relationships between model characteristics and attack vulnerability but also
suggest the inherent connections underlying different attacks. Finally, we
discuss potential remedies to mitigate such drawbacks, including increasing
cell depth and suppressing skip connects, which lead to several promising
research directions.


http://arxiv.org/abs/2110.05797
Zero-bias Deep Neural Network for Quickest RF Signal Surveillance. (1%)
Yongxin Liu; Yingjie Chen; Jian Wang; Shuteng Niu; Dahai Liu; Houbing Song
  The Internet of Things (IoT) is reshaping modern society by allowing a decent
number of RF devices to connect and share information through RF channels.
However, such an open nature also brings obstacles to surveillance. For
alleviation, a surveillance oracle, or a cognitive communication entity needs
to identify and confirm the appearance of known or unknown signal sources in
real-time. In this paper, we provide a deep learning framework for RF signal
surveillance. Specifically, we jointly integrate the Deep Neural Networks
(DNNs) and Quickest Detection (QD) to form a sequential signal surveillance
scheme. We first analyze the latent space characteristic of neural network
classification models, and then we leverage the response characteristics of DNN
classifiers and propose a novel method to transform existing DNN classifiers
into performance-assured binary abnormality detectors. In this way, we
seamlessly integrate the DNNs with the parametric quickest detection. Finally,
we propose an enhanced Elastic Weight Consolidation (EWC) algorithm with better
numerical stability for DNNs in signal surveillance systems to evolve
incrementally, we demonstrate that the zero-bias DNN is superior to regular DNN
models considering incremental learning and decision fairness. We evaluated the
proposed framework using real signal datasets and we believe this framework is
helpful in developing a trustworthy IoT ecosystem.


http://arxiv.org/abs/2110.05007
Boosting Fast Adversarial Training with Learnable Adversarial Initialization. (99%)
Xiaojun Jia; Yong Zhang; Baoyuan Wu; Jue Wang; Xiaochun Cao
  Adversarial training (AT) has been demonstrated to be effective in improving
model robustness by leveraging adversarial examples for training. However, most
AT methods are in face of expensive time and computational cost for calculating
gradients at multiple steps in generating adversarial examples. To boost
training efficiency, fast gradient sign method (FGSM) is adopted in fast AT
methods by calculating gradient only once. Unfortunately, the robustness is far
from satisfactory. One reason may arise from the initialization fashion.
Existing fast AT generally uses a random sample-agnostic initialization, which
facilitates the efficiency yet hinders a further robustness improvement. Up to
now, the initialization in fast AT is still not extensively explored. In this
paper, we boost fast AT with a sample-dependent adversarial initialization,
i.e., an output from a generative network conditioned on a benign image and its
gradient information from the target network. As the generative network and the
target network are optimized jointly in the training phase, the former can
adaptively generate an effective initialization with respect to the latter,
which motivates gradually improved robustness. Experimental evaluations on four
benchmark databases demonstrate the superiority of our proposed method over
state-of-the-art fast AT methods, as well as comparable robustness to advanced
multi-step AT methods. The code is released at
https://github.com//jiaxiaojunQAQ//FGSM-SDI.


http://arxiv.org/abs/2110.05626
Parameterizing Activation Functions for Adversarial Robustness. (98%)
Sihui Dai; Saeed Mahloujifar; Prateek Mittal
  Deep neural networks are known to be vulnerable to adversarially perturbed
inputs. A commonly used defense is adversarial training, whose performance is
influenced by model capacity. While previous works have studied the impact of
varying model width and depth on robustness, the impact of increasing capacity
by using learnable parametric activation functions (PAFs) has not been studied.
We study how using learnable PAFs can improve robustness in conjunction with
adversarial training. We first ask the question: how should we incorporate
parameters into activation functions to improve robustness? To address this, we
analyze the direct impact of activation shape on robustness through PAFs and
observe that activation shapes with positive outputs on negative inputs and
with high finite curvature can increase robustness. We combine these properties
to create a new PAF, which we call Parametric Shifted Sigmoidal Linear Unit
(PSSiLU). We then combine PAFs (including PReLU, PSoftplus and PSSiLU) with
adversarial training and analyze robust performance. We find that PAFs optimize
towards activation shape properties found to directly affect robustness.
Additionally, we find that while introducing only 1-2 learnable parameters into
the network, smooth PAFs can significantly increase robustness over ReLU. For
instance, when trained on CIFAR-10 with additional synthetic data, PSSiLU
improves robust accuracy by 4.54% over ReLU on ResNet-18 and 2.69% over ReLU on
WRN-28-10 in the $\ell_{\infty}$ threat model while adding only 2 additional
parameters into the network architecture. The PSSiLU WRN-28-10 model achieves
61.96% AutoAttack accuracy, improving over the state-of-the-art robust accuracy
on RobustBench (Croce et al., 2020).


http://arxiv.org/abs/2110.05059
Amicable examples for informed source separation. (86%)
Naoya Takahashi; Yuki Mitsufuji
  This paper deals with the problem of informed source separation (ISS), where
the sources are accessible during the so-called \textit{encoding} stage.
Previous works computed side-information during the encoding stage and source
separation models were designed to utilize the side-information to improve the
separation performance. In contrast, in this work, we improve the performance
of a pretrained separation model that does not use any side-information. To
this end, we propose to adopt an adversarial attack for the opposite purpose,
i.e., rather than computing the perturbation to degrade the separation, we
compute an imperceptible perturbation called amicable noise to improve the
separation. Experimental results show that the proposed approach selectively
improves the performance of the targeted separation model by 2.23 dB on average
and is robust to signal compression. Moreover, we propose multi-model
multi-purpose learning that control the effect of the perturbation on different
models individually.


http://arxiv.org/abs/2110.05691
Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation. (12%)
Weiting Tan; Shuoyang Ding; Huda Khayrallah; Philipp Koehn
  Neural Machine Translation (NMT) models are known to suffer from noisy
inputs. To make models robust, we generate adversarial augmentation samples
that attack the model and preserve the source-side semantic meaning at the same
time. To generate such samples, we propose a doubly-trained architecture that
pairs two NMT models of opposite translation directions with a joint loss
function, which combines the target-side attack and the source-side semantic
similarity constraint. The results from our experiments across three different
language pairs and two evaluation metrics show that these adversarial samples
improve the model robustness.


http://arxiv.org/abs/2110.05679
Large Language Models Can Be Strong Differentially Private Learners. (1%)
Xuechen Li; Florian Tramèr; Percy Liang; Tatsunori Hashimoto
  Differentially Private (DP) learning has seen limited success for building
large deep learning models of text, and attempts at straightforwardly applying
Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have
resulted in large performance drops and high computational overhead. We show
that this performance drop can be mitigated with (1) the use of large
pretrained models; (2) hyperparameters that suit DP optimization; and (3)
fine-tuning objectives aligned with the pretraining procedure. With these
factors set right, we obtain private NLP models that outperform
state-of-the-art private training approaches and strong non-private baselines
-- by directly fine-tuning pretrained models with DP optimization on
moderately-sized corpora. To address the computational challenge of running
DP-SGD with large Transformers, we propose a memory saving technique that
allows clipping in DP-SGD to run without instantiating per-example gradients
for any layer in the model. The technique enables privately training
Transformers with almost the same memory cost as non-private training at a
modest run-time overhead. Contrary to conventional wisdom that DP optimization
fails at learning high-dimensional models (due to noise that scales with
dimension) empirical results reveal that private learning with pretrained
models tends to not suffer from dimension-dependent performance degradation.


http://arxiv.org/abs/2110.05290
Homogeneous Learning: Self-Attention Decentralized Deep Learning. (1%)
Yuwei Sun; Hideya Ochiai
  Federated learning (FL) has been facilitating privacy-preserving deep
learning in many walks of life such as medical image classification, network
intrusion detection, and so forth. Whereas it necessitates a central parameter
server for model aggregation, which brings about delayed model communication
and vulnerability to adversarial attacks. A fully decentralized architecture
like Swarm Learning allows peer-to-peer communication among distributed nodes,
without the central server. One of the most challenging issues in decentralized
deep learning is that data owned by each node are usually non-independent and
identically distributed (non-IID), causing time-consuming convergence of model
training. To this end, we propose a decentralized learning model called
Homogeneous Learning (HL) for tackling non-IID data with a self-attention
mechanism. In HL, training performs on each round's selected node, and the
trained model of a node is sent to the next selected node at the end of each
round. Notably, for the selection, the self-attention mechanism leverages
reinforcement learning to observe a node's inner state and its surrounding
environment's state, and find out which node should be selected to optimize the
training. We evaluate our method with various scenarios for an image
classification task. The result suggests that HL can produce a better
performance compared with standalone learning and greatly reduce both the total
training rounds by 50.8% and the communication cost by 74.6% compared with
random policy-based decentralized learning for training on non-IID data.


http://arxiv.org/abs/2110.05054
Source Mixing and Separation Robust Audio Steganography. (1%)
Naoya Takahashi; Mayank Kumar Singh; Yuki Mitsufuji
  Audio steganography aims at concealing secret information in carrier audio
with imperceptible modification on the carrier. Although previous works
addressed the robustness of concealed message recovery against distortions
introduced during transmission, they do not address the robustness against
aggressive editing such as mixing of other audio sources and source separation.
In this work, we propose for the first time a steganography method that can
embed information into individual sound sources in a mixture such as
instrumental tracks in music. To this end, we propose a time-domain model and
curriculum learning essential to learn to decode the concealed message from the
separated sources. Experimental results show that the proposed method
successfully conceals the information in an imperceptible perturbation and that
the information can be correctly recovered even after mixing of other sources
and separation by a source separation algorithm. Furthermore, we show that the
proposed method can be applied to multiple sources simultaneously without
interfering with the decoder for other sources even after the sources are mixed
and separated.


http://arxiv.org/abs/2110.05365
Intriguing Properties of Input-dependent Randomized Smoothing. (1%)
Peter Súkeník; Aleksei Kuvshinov; Stephan Günnemann
  Randomized smoothing is currently considered the state-of-the-art method to
obtain certifiably robust classifiers. Despite its remarkable performance, the
method is associated with various serious problems such as ``certified accuracy
waterfalls'', certification vs. accuracy trade-off, or even fairness issues.
Input-dependent smoothing approaches have been proposed to overcome these
flaws. However, we demonstrate that these methods lack formal guarantees and so
the resulting certificates are not justified. We show that the input-dependent
smoothing, in general, suffers from the curse of dimensionality, forcing the
variance function to have low semi-elasticity. On the other hand, we provide a
theoretical and practical framework that enables the usage of input-dependent
smoothing even in the presence of the curse of dimensionality, under strict
restrictions. We present one concrete design of the smoothing variance and test
it on CIFAR10 and MNIST. Our design solves some of the problems of classical
smoothing and is formally underlined, yet further improvement of the design is
still necessary.


http://arxiv.org/abs/2110.05689
Hiding Images into Images with Real-world Robustness. (1%)
Qichao Ying; Hang Zhou; Xianhan Zeng; Haisheng Xu; Zhenxing Qian; Xinpeng Zhang
  The existing image embedding networks are basically vulnerable to malicious
attacks such as JPEG compression and noise adding, not applicable for
real-world copyright protection tasks. To solve this problem, we introduce a
generative deep network based method for hiding images into images while
assuring high-quality extraction from the destructive synthesized images. An
embedding network is sequentially concatenated with an attack layer, a
decoupling network and an image extraction network. The addition of decoupling
network learns to extract the embedded watermark from the attacked image. We
also pinpoint the weaknesses of the adversarial training for robustness in
previous works and build our improved real-world attack simulator. Experimental
results demonstrate the superiority of the proposed method against typical
digital attacks by a large margin, as well as the performance boost of the
recovered images with the aid of progressive recovery strategy. Besides, we are
the first to robustly hide three secret images.


http://arxiv.org/abs/2110.07719
Certified Patch Robustness via Smoothed Vision Transformers. (1%)
Hadi Salman; Saachi Jain; Eric Wong; Aleksander Mądry
  Certified patch defenses can guarantee robustness of an image classifier to
arbitrary changes within a bounded contiguous region. But, currently, this
robustness comes at a cost of degraded standard accuracies and slower inference
times. We demonstrate how using vision transformers enables significantly
better certified patch robustness that is also more computationally efficient
and does not incur a substantial drop in standard accuracy. These improvements
stem from the inherent ability of the vision transformer to gracefully handle
largely masked images. Our code is available at
https://github.com/MadryLab/smoothed-vit.


http://arxiv.org/abs/2110.04887
Adversarial Attacks in a Multi-view Setting: An Empirical Study of the Adversarial Patches Inter-view Transferability. (98%)
Bilel Tarchoun; Ihsen Alouani; Anouar Ben Khalifa; Mohamed Ali Mahjoub
  While machine learning applications are getting mainstream owing to a
demonstrated efficiency in solving complex problems, they suffer from inherent
vulnerability to adversarial attacks. Adversarial attacks consist of additive
noise to an input which can fool a detector. Recently, successful real-world
printable adversarial patches were proven efficient against state-of-the-art
neural networks. In the transition from digital noise based attacks to
real-world physical attacks, the myriad of factors affecting object detection
will also affect adversarial patches. Among these factors, view angle is one of
the most influential, yet under-explored. In this paper, we study the effect of
view angle on the effectiveness of an adversarial patch. To this aim, we
propose the first approach that considers a multi-view context by combining
existing adversarial patches with a perspective geometric transformation in
order to simulate the effect of view angle changes. Our approach has been
evaluated on two datasets: the first dataset which contains most real world
constraints of a multi-view context, and the second dataset which empirically
isolates the effect of view angle. The experiments show that view angle
significantly affects the performance of adversarial patches, where in some
cases the patch loses most of its effectiveness. We believe that these results
motivate taking into account the effect of view angles in future adversarial
attacks, and open up new opportunities for adversarial defenses.


http://arxiv.org/abs/2110.04731
Universal Adversarial Attacks on Neural Networks for Power Allocation in a Massive MIMO System. (92%)
Pablo Millán Santos; B. R. Manoj; Meysam Sadeghi; Erik G. Larsson
  Deep learning (DL) architectures have been successfully used in many
applications including wireless systems. However, they have been shown to be
susceptible to adversarial attacks. We analyze DL-based models for a regression
problem in the context of downlink power allocation in massive
multiple-input-multiple-output systems and propose universal adversarial
perturbation (UAP)-crafting methods as white-box and black-box attacks. We
benchmark the UAP performance of white-box and black-box attacks for the
considered application and show that the adversarial success rate can achieve
up to 60% and 40%, respectively. The proposed UAP-based attacks make a more
practical and realistic approach as compared to classical white-box attacks.


http://arxiv.org/abs/2110.04488
Demystifying the Transferability of Adversarial Attacks in Computer Networks. (99%)
Ehsan Nowroozi; Mauro Conti; Yassine Mekdad; Mohammad Hajian Berenjestanaki; Abdeslam EL Fergougui
  Deep Convolutional Neural Networks (CNN) models are one of the most popular
networks in deep learning. With their large fields of application in different
areas, they are extensively used in both academia and industry. CNN-based
models include several exciting implementations such as early breast cancer
detection or detecting developmental delays in children (e.g., autism, speech
disorders, etc.). However, previous studies demonstrate that these models are
subject to various adversarial attacks. Interestingly, some adversarial
examples could potentially still be effective against different unknown models.
This particular property is known as adversarial transferability, and prior
works slightly analyzed this characteristic in a very limited application
domain. In this paper, we aim to demystify the transferability threats in
computer networks by studying the possibility of transferring adversarial
examples. In particular, we provide the first comprehensive study which
assesses the robustness of CNN-based models for computer networks against
adversarial transferability. In our experiments, we consider five different
attacks: (1) the Iterative Fast Gradient Method (I-FGSM), (2) the
Jacobian-based Saliency Map attack (JSMA), (3) the L-BFGS attack, (4) the
Projected Gradient Descent attack (PGD), and (5) the DeepFool attack. These
attacks are performed against two well-known datasets: the N-BaIoT dataset and
the Domain Generating Algorithms (DGA) dataset. Our results show that the
transferability happens in specific use cases where the adversary can easily
compromise the victim's network with very few knowledge of the targeted model.


http://arxiv.org/abs/2110.04471
Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning. (93%)
Guanlin Liu; Lifeng Lai
  Due to the broad range of applications of reinforcement learning (RL),
understanding the effects of adversarial attacks against RL model is essential
for the safe applications of this model. Prior works on adversarial attacks
against RL mainly focus on either observation poisoning attacks or environment
poisoning attacks. In this paper, we introduce a new class of attacks named
action poisoning attacks, where an adversary can change the action signal
selected by the agent. Compared with existing attack models, the attacker's
ability in the proposed action poisoning attack model is more restricted, and
hence the attack model is more practical. We study the action poisoning attack
in both white-box and black-box settings. We introduce an adaptive attack
scheme called LCB-H, which works for most RL agents in the black-box setting.
We prove that the LCB-H attack can force any efficient RL agent, whose dynamic
regret scales sublinearly with the total number of steps taken, to choose
actions according to a policy selected by the attacker very frequently, with
only sublinear cost. In addition, we apply LCB-H attack against a popular
model-free RL algorithm: UCB-H. We show that, even in the black-box setting, by
spending only logarithm cost, the proposed LCB-H attack scheme can force the
UCB-H agent to choose actions according to the policy selected by the attacker
very frequently.


http://arxiv.org/abs/2110.04571
Widen The Backdoor To Let More Attackers In. (13%)
Siddhartha Datta; Giulio Lovisotto; Ivan Martinovic; Nigel Shadbolt
  As collaborative learning and the outsourcing of data collection become more
common, malicious actors (or agents) which attempt to manipulate the learning
process face an additional obstacle as they compete with each other. In
backdoor attacks, where an adversary attempts to poison a model by introducing
malicious samples into the training data, adversaries have to consider that the
presence of additional backdoor attackers may hamper the success of their own
backdoor. In this paper, we investigate the scenario of a multi-agent backdoor
attack, where multiple non-colluding attackers craft and insert triggered
samples in a shared dataset which is used by a model (a defender) to learn a
task. We discover a clear backfiring phenomenon: increasing the number of
attackers shrinks each attacker's attack success rate (ASR). We then exploit
this phenomenon to minimize the collective ASR of attackers and maximize
defender's robustness accuracy by (i) artificially augmenting the number of
attackers, and (ii) indexing to remove the attacker's sub-dataset from the
model for inference, hence proposing 2 defenses.


http://arxiv.org/abs/2110.04158
Explainability-Aware One Point Attack for Point Cloud Neural Networks. (99%)
Hanxiao Tan; Helena Kotthaus
  With the proposition of neural networks for point clouds, deep learning has
started to shine in the field of 3D object recognition while researchers have
shown an increased interest to investigate the reliability of point cloud
networks by fooling them with perturbed instances. However, most studies focus
on the imperceptibility or surface consistency, with humans perceiving no
perturbations on the adversarial examples. This work proposes two new attack
methods: opa and cta, which go in the opposite direction: we restrict the
perturbation dimensions to a human cognizable range with the help of
explainability methods, which enables the working principle or decision
boundary of the models to be comprehensible through the observable perturbation
magnitude. Our results show that the popular point cloud networks can be
deceived with almost 100% success rate by shifting only one point from the
input instance. In addition, we attempt to provide a more persuasive viewpoint
of comparing the robustness of point cloud models against adversarial attacks.
We also show the interesting impact of different point attribution
distributions on the adversarial robustness of point cloud networks. Finally,
we discuss how our approaches facilitate the explainability study for point
cloud networks. To the best of our knowledge, this is the first
point-cloud-based adversarial approach concerning explainability. Our code is
available at https://github.com/Explain3D/Exp-One-Point-Atk-PC.


http://arxiv.org/abs/2110.06166
Game Theory for Adversarial Attacks and Defenses. (98%)
Shorya Sharma
  Adversarial attacks can generate adversarial inputs by applying small but
intentionally worst-case perturbations to samples from the dataset, which leads
to even state-of-the-art deep neural networks outputting incorrect answers with
high confidence. Hence, some adversarial defense techniques are developed to
improve the security and robustness of the models and avoid them being
attacked. Gradually, a game-like competition between attackers and defenders
formed, in which both players would attempt to play their best strategies
against each other while maximizing their own payoffs. To solve the game, each
player would choose an optimal strategy against the opponent based on the
prediction of the opponent's strategy choice. In this work, we are on the
defensive side to apply game-theoretic approaches on defending against attacks.
We use two randomization methods, random initialization and stochastic
activation pruning, to create diversity of networks. Furthermore, we use one
denoising technique, super resolution, to improve models' robustness by
preprocessing images before attacks. Our experimental results indicate that
those three methods can effectively improve the robustness of deep-learning
neural networks.


http://arxiv.org/abs/2110.03999
Graphs as Tools to Improve Deep Learning Methods. (10%)
Carlos Lassance; Myriam Bontonou; Mounia Hamidouche; Bastien Pasdeloup; Lucas Drumetz; Vincent Gripon
  In recent years, deep neural networks (DNNs) have known an important rise in
popularity. However, although they are state-of-the-art in many machine
learning challenges, they still suffer from several limitations. For example,
DNNs require a lot of training data, which might not be available in some
practical applications. In addition, when small perturbations are added to the
inputs, DNNs are prone to misclassification errors. DNNs are also viewed as
black-boxes and as such their decisions are often criticized for their lack of
interpretability.
  In this chapter, we review recent works that aim at using graphs as tools to
improve deep learning methods. These graphs are defined considering a specific
layer in a deep learning architecture. Their vertices represent distinct
samples, and their edges depend on the similarity of the corresponding
intermediate representations. These graphs can then be leveraged using various
methodologies, many of which built on top of graph signal processing.
  This chapter is composed of four main parts: tools for visualizing
intermediate layers in a DNN, denoising data representations, optimizing graph
objective functions and regularizing the learning process.


http://arxiv.org/abs/2110.04180
IHOP: Improved Statistical Query Recovery against Searchable Symmetric Encryption through Quadratic Optimization. (3%)
Simon Oya; Florian Kerschbaum
  Searchable Symmetric Encryption (SSE) schemes allow a client to perform
secure searches over encrypted databases on a remote server. These schemes leak
certain information that an honest-but-curious service provider can use to
recover the keywords of the client's queries. Effective query recovery attacks
typically rely on auxiliary ground-truth information about the queries or
dataset. Query recovery is also possible under the weaker statistical auxiliary
information assumption, although statistical-based attacks achieve lower
accuracy and are not considered a serious threat. In this work we present IHOP,
a statistical-based query recovery attack that formulates query recovery as a
quadratic optimization problem and reaches a solution by iterating over linear
assignment problems. We show that IHOP outperforms all other statistical-based
query recovery attacks on SSE schemes with typical access and search pattern
leakage, reaching query recovery accuracies around 80% and 90%. We adapt IHOP
against access-pattern obfuscation defenses and show that it still achieves
reasonable recovery rates, outperforming existing attacks in this scenario.
Finally, we use IHOP in a frequency-only leakage setting where the client's
queries are correlated, and show that our attack can exploit query dependencies
even when PANCAKE, a recent frequency-hiding defense by Grubbs et al., is
applied. Our findings indicate that statistical query recovery attacks pose a
severe threat to privacy-preserving SSE schemes.


http://arxiv.org/abs/2110.04259
A Wireless Intrusion Detection System for 802.11 WPA3 Networks. (1%)
Neil Dalal; Nadeem Akhtar; Anubhav Gupta; Nikhil Karamchandani; Gaurav S. Kasbekar; Jatin Parekh
  Wi-Fi (802.11) networks have become an essential part of our daily lives;
hence, their security is of utmost importance. However, Wi-Fi Protected Access
3 (WPA3), the latest security certification for 802.11 standards, has recently
been shown to be vulnerable to several attacks. In this paper, we first
describe the attacks on WPA3 networks that have been reported in prior work;
additionally, we show that a deauthentication attack and a beacon flood attack,
known to be possible on a WPA2 network, are still possible with WPA3. We launch
and test all the above (a total of nine) attacks using a testbed that contains
an enterprise Access Point (AP) and Intrusion Detection System (IDS). Our
experimental results show that the AP is vulnerable to eight out of the nine
attacks and the IDS is unable to detect any of them. We propose a design for a
signature-based IDS, which incorporates techniques to detect all the above
attacks. Also, we implement these techniques on our testbed and verify that our
IDS is able to successfully detect all the above attacks. We provide schemes
for mitigating the impact of the above attacks once they are detected. We make
the code to perform the above attacks as well as that of our IDS publicly
available, so that it can be used for future work by the research community at
large.


http://arxiv.org/abs/2110.03301
EvadeDroid: A Practical Evasion Attack on Machine Learning for Black-box Android Malware Detection. (99%)
Hamid Bostani; Veelasha Moonsamy
  Over the last decade, several studies have investigated the weaknesses of
Android malware detectors against adversarial examples by proposing novel
evasion attacks; however, the practicality of most studies in manipulating
real-world malware is arguable. The majority of studies have assumed attackers
know the details of the target classifiers used for malware detection, while in
real life, malicious actors have limited access to the target classifiers. This
paper presents a practical evasion attack, EvadeDroid, to circumvent black-box
Android malware detectors. In addition to generating real-world adversarial
malware, the proposed evasion attack can preserve the functionality of the
original malware samples. EvadeDroid applies a set of functionality-preserving
transformations to morph malware instances into benign ones using an iterative
and incremental manipulation strategy. The proposed manipulation technique is a
novel, query-efficient optimization algorithm with the aim of finding and
injecting optimal sequences of transformations into malware samples. Our
empirical evaluation demonstrates the efficacy of EvadeDroid under hard- and
soft-label attacks. Moreover, EvadeDroid is capable to generate practical
adversarial examples with only a small number of queries, with evasion rate of
81%, 73%, and 75% for DREBIN, Sec-SVM, and MaMaDroid, respectively. Finally, we
show that EvadeDroid is able to preserve its stealthiness against four popular
commercial antivirus, thus demonstrating its feasibility in the real world.


http://arxiv.org/abs/2110.03745
Adversarial Attack by Limited Point Cloud Surface Modifications. (98%)
Atrin Arya; Hanieh Naderi; Shohreh Kasaei
  Recent research has revealed that the security of deep neural networks that
directly process 3D point clouds to classify objects can be threatened by
adversarial samples. Although existing adversarial attack methods achieve high
success rates, they do not restrict the point modifications enough to preserve
the point cloud appearance. To overcome this shortcoming, two constraints are
proposed. These include applying hard boundary constraints on the number of
modified points and on the point perturbation norms. Due to the restrictive
nature of the problem, the search space contains many local maxima. The
proposed method addresses this issue by using a high step-size at the beginning
of the algorithm to search the main surface of the point cloud fast and
effectively. Then, in order to converge to the desired output, the step-size is
gradually decreased. To evaluate the performance of the proposed method, it is
run on the ModelNet40 and ScanObjectNN datasets by employing the
state-of-the-art point cloud classification models; including PointNet,
PointNet++, and DGCNN. The obtained results show that it can perform successful
attacks and achieve state-of-the-art results by only a limited number of point
modifications while preserving the appearance of the point cloud. Moreover, due
to the effective search algorithm, it can perform successful attacks in just a
few steps. Additionally, the proposed step-size scheduling algorithm shows an
improvement of up to $14.5\%$ when adopted by other methods as well. The
proposed method also performs effectively against popular defense methods.


http://arxiv.org/abs/2110.03825
Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks. (98%)
Hanxun Huang; Yisen Wang; Sarah Monazam Erfani; Quanquan Gu; James Bailey; Xingjun Ma
  Deep neural networks (DNNs) are known to be vulnerable to adversarial
attacks. A range of defense methods have been proposed to train adversarially
robust DNNs, among which adversarial training has demonstrated promising
results. However, despite preliminary understandings developed for adversarial
training, it is still not clear, from the architectural perspective, what
configurations can lead to more robust DNNs. In this paper, we address this gap
via a comprehensive investigation on the impact of network width and depth on
the robustness of adversarially trained DNNs. Specifically, we make the
following key observations: 1) more parameters (higher model capacity) does not
necessarily help adversarial robustness; 2) reducing capacity at the last stage
(the last group of blocks) of the network can actually improve adversarial
robustness; and 3) under the same parameter budget, there exists an optimal
architectural configuration for adversarial robustness. We also provide a
theoretical analysis explaning why such network configuration can help
robustness. These architectural insights can help design adversarially robust
DNNs. Code is available at \url{https://github.com/HanxunH/RobustWRN}.


http://arxiv.org/abs/2110.03605
One Thing to Fool them All: Generating Interpretable, Universal, and Physically-Realizable Adversarial Features. (92%)
Stephen Casper; Max Nadeau; Gabriel Kreiman
  It is well understood that modern deep networks are vulnerable to adversarial
attacks. However, conventional methods fail to produce adversarial
perturbations that are intelligible to humans, and they pose limited threats in
the physical world. To study feature-class associations in networks and better
understand the real-world threats they face, we develop feature-level
adversarial perturbations using deep image generators and a novel optimization
objective. We term these feature-fool attacks. We show that they are versatile
and use them to generate targeted feature-level attacks at the ImageNet scale
that are simultaneously interpretable, universal to any source image, and
physically-realizable. These attacks can also reveal spurious,
semantically-describable feature/class associations, and we use them to guide
the design of "copy/paste" adversaries in which one natural image is pasted
into another to cause a targeted misclassification.


http://arxiv.org/abs/2110.03875
Dyn-Backdoor: Backdoor Attack on Dynamic Link Prediction. (80%)
Jinyin Chen; Haiyang Xiong; Haibin Zheng; Jian Zhang; Guodong Jiang; Yi Liu
  Dynamic link prediction (DLP) makes graph prediction based on historical
information. Since most DLP methods are highly dependent on the training data
to achieve satisfying prediction performance, the quality of the training data
is crucial. Backdoor attacks induce the DLP methods to make wrong prediction by
the malicious training data, i.e., generating a subgraph sequence as the
trigger and embedding it to the training data. However, the vulnerability of
DLP toward backdoor attacks has not been studied yet. To address the issue, we
propose a novel backdoor attack framework on DLP, denoted as Dyn-Backdoor.
Specifically, Dyn-Backdoor generates diverse initial-triggers by a generative
adversarial network (GAN). Then partial links of the initial-triggers are
selected to form a trigger set, according to the gradient information of the
attack discriminator in the GAN, so as to reduce the size of triggers and
improve the concealment of the attack. Experimental results show that
Dyn-Backdoor launches successful backdoor attacks on the state-of-the-art DLP
models with success rate more than 90%. Additionally, we conduct a possible
defense against Dyn-Backdoor to testify its resistance in defensive settings,
highlighting the needs of defenses for backdoor attacks on DLP.


http://arxiv.org/abs/2110.03175
Fingerprinting Multi-exit Deep Neural Network Models via Inference Time. (62%)
Tian Dong; Han Qiu; Tianwei Zhang; Jiwei Li; Hewu Li; Jialiang Lu
  Transforming large deep neural network (DNN) models into the multi-exit
architectures can overcome the overthinking issue and distribute a large DNN
model on resource-constrained scenarios (e.g. IoT frontend devices and backend
servers) for inference and transmission efficiency. Nevertheless, intellectual
property (IP) protection for the multi-exit models in the wild is still an
unsolved challenge. Previous efforts to verify DNN model ownership mainly rely
on querying the model with specific samples and checking the responses, e.g.,
DNN watermarking and fingerprinting. However, they are vulnerable to
adversarial settings such as adversarial training and are not suitable for the
IP verification for multi-exit DNN models. In this paper, we propose a novel
approach to fingerprint multi-exit models via inference time rather than
inference predictions. Specifically, we design an effective method to generate
a set of fingerprint samples to craft the inference process with a unique and
robust inference time cost as the evidence for model ownership. We conduct
extensive experiments to prove the uniqueness and robustness of our method on
three structures (ResNet-56, VGG-16, and MobileNet) and three datasets
(CIFAR-10, CIFAR-100, and Tiny-ImageNet) under comprehensive adversarial
settings.


http://arxiv.org/abs/2110.03735
Adversarial Unlearning of Backdoors via Implicit Hypergradient. (56%)
Yi Zeng; Si Chen; Won Park; Z. Morley Mao; Ming Jin; Ruoxi Jia
  We propose a minimax formulation for removing backdoors from a given poisoned
model based on a small set of clean data. This formulation encompasses much of
prior work on backdoor removal. We propose the Implicit Bacdoor Adversarial
Unlearning (I-BAU) algorithm to solve the minimax. Unlike previous work, which
breaks down the minimax into separate inner and outer problems, our algorithm
utilizes the implicit hypergradient to account for the interdependence between
inner and outer optimization. We theoretically analyze its convergence and the
generalizability of the robustness gained by solving minimax on clean data to
unseen test data. In our evaluation, we compare I-BAU with six state-of-art
backdoor defenses on seven backdoor attacks over two datasets and various
attack settings, including the common setting where the attacker targets one
class as well as important but underexplored settings where multiple classes
are targeted. I-BAU's performance is comparable to and most often significantly
better than the best baseline. Particularly, its performance is more robust to
the variation on triggers, attack settings, poison ratio, and clean data size.
Moreover, I-BAU requires less computation to take effect; particularly, it is
more than $13\times$ faster than the most efficient baseline in the
single-target attack setting. Furthermore, it can remain effective in the
extreme case where the defender can only access 100 clean samples -- a setting
where all the baselines fail to produce acceptable results.


http://arxiv.org/abs/2110.11417
HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise. (99%)
Souvik Kundu; Massoud Pedram; Peter A. Beerel
  Low-latency deep spiking neural networks (SNNs) have become a promising
alternative to conventional artificial neural networks (ANNs) because of their
potential for increased energy efficiency on event-driven neuromorphic
hardware. Neural networks, including SNNs, however, are subject to various
adversarial attacks and must be trained to remain resilient against such
attacks for many applications. Nevertheless, due to prohibitively high training
costs associated with SNNs, analysis, and optimization of deep SNNs under
various adversarial attacks have been largely overlooked. In this paper, we
first present a detailed analysis of the inherent robustness of low-latency
SNNs against popular gradient-based attacks, namely fast gradient sign method
(FGSM) and projected gradient descent (PGD). Motivated by this analysis, to
harness the model robustness against these attacks we present an SNN training
algorithm that uses crafted input noise and incurs no additional training time.
To evaluate the merits of our algorithm, we conducted extensive experiments
with variants of VGG and ResNet on both CIFAR-10 and CIFAR-100 datasets.
Compared to standard trained direct input SNNs, our trained models yield
improved classification accuracy of up to 13.7% and 10.1% on FGSM and PGD
attack-generated images, respectively, with negligible loss in clean image
accuracy. Our models also outperform inherently robust SNNs trained on
rate-coded inputs with improved or similar classification performance on
attack-generated images while having up to 25x and 4.6x lower latency and
computation energy, respectively.


http://arxiv.org/abs/2110.02700
Reversible adversarial examples against local visual perturbation. (99%)
Zhaoxia Yin; Li Chen; Shaowei Zhu
  Recently, studies have indicated that adversarial attacks pose a threat to
deep learning systems. However, when there are only adversarial examples,
people cannot get the original images, so there is research on reversible
adversarial attacks. However, the existing strategies are aimed at invisible
adversarial perturbation, and do not consider the case of locally visible
adversarial perturbation. In this article, we generate reversible adversarial
examples for local visual adversarial perturbation, and use reversible data
embedding technology to embed the information needed to restore the original
image into the adversarial examples to generate examples that are both
adversarial and reversible. Experiments on ImageNet dataset show that our
method can restore the original image losslessly while ensuring the attack
capability.


http://arxiv.org/abs/2110.02516
Attack as the Best Defense: Nullifying Image-to-image Translation GANs via Limit-aware Adversarial Attack. (99%)
Chin-Yuan Yeh; Hsi-Wen Chen; Hong-Han Shuai; De-Nian Yang; Ming-Syan Chen
  With the successful creation of high-quality image-to-image (Img2Img)
translation GANs comes the non-ethical applications of DeepFake and DeepNude.
Such misuses of img2img techniques present a challenging problem for society.
In this work, we tackle the problem by introducing the Limit-Aware Self-Guiding
Gradient Sliding Attack (LaS-GSA). LaS-GSA follows the Nullifying Attack to
cancel the img2img translation process under a black-box setting. In other
words, by processing input images with the proposed LaS-GSA before publishing,
any targeted img2img GANs can be nullified, preventing the model from
maliciously manipulating the images. To improve efficiency, we introduce the
limit-aware random gradient-free estimation and the gradient sliding mechanism
to estimate the gradient that adheres to the adversarial limit, i.e., the pixel
value limitations of the adversarial example. Theoretical justifications
validate how the above techniques prevent inefficiency caused by the
adversarial limit in both the direction and the step length. Furthermore, an
effective self-guiding prior is extracted solely from the threat model and the
target image to efficiently leverage the prior information and guide the
gradient estimation process. Extensive experiments demonstrate that LaS-GSA
requires fewer queries to nullify the image translation process with higher
success rates than 4 state-of-the-art black-box methods.


http://arxiv.org/abs/2110.02797
Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs. (99%)
Philipp Benz; Soomin Ham; Chaoning Zhang; Adil Karjauv; In So Kweon
  Convolutional Neural Networks (CNNs) have become the de facto gold standard
in computer vision applications in the past years. Recently, however, new model
architectures have been proposed challenging the status quo. The Vision
Transformer (ViT) relies solely on attention modules, while the MLP-Mixer
architecture substitutes the self-attention modules with Multi-Layer
Perceptrons (MLPs). Despite their great success, CNNs have been widely known to
be vulnerable to adversarial attacks, causing serious concerns for
security-sensitive applications. Thus, it is critical for the community to know
whether the newly proposed ViT and MLP-Mixer are also vulnerable to adversarial
attacks. To this end, we empirically evaluate their adversarial robustness
under several adversarial attack setups and benchmark them against the widely
used CNNs. Overall, we find that the two architectures, especially ViT, are
more robust than their CNN models. Using a toy example, we also provide
empirical evidence that the lower adversarial robustness of CNNs can be
partially attributed to their shift-invariant property. Our frequency analysis
suggests that the most robust ViT architectures tend to rely more on
low-frequency features compared with CNNs. Additionally, we have an intriguing
finding that MLP-Mixer is extremely vulnerable to universal adversarial
perturbations.


http://arxiv.org/abs/2110.02498
Adversarial Attacks on Machinery Fault Diagnosis. (99%)
Jiahao Chen; Diqun Yan
  Despite the great progress of neural network-based (NN-based) machinery fault
diagnosis methods, their robustness has been largely neglected, for they can be
easily fooled through adding imperceptible perturbation to the input. For fault
diagnosis problems, in this paper, we reformulate various adversarial attacks
and intensively investigate them under untargeted and targeted conditions.
Experimental results on six typical NN-based models show that accuracies of the
models are greatly reduced by adding small perturbations. We further propose a
simple, efficient and universal scheme to protect the victim models. This work
provides an in-depth look at adversarial examples of machinery vibration
signals for developing protection methods against adversarial attack and
improving the robustness of NN-based models.


http://arxiv.org/abs/2110.02929
Adversarial Attacks on Spiking Convolutional Networks for Event-based Vision. (98%)
Julian Büchel; Gregor Lenz; Yalun Hu; Sadique Sheik; Martino Sorbaro
  Event-based sensing using dynamic vision sensors is gaining traction in
low-power vision applications. Spiking neural networks work well with the
sparse nature of event-based data and suit deployment on low-power neuromorphic
hardware. Being a nascent field, the sensitivity of spiking neural networks to
potentially malicious adversarial attacks has received very little attention so
far. In this work, we show how white-box adversarial attack algorithms can be
adapted to the discrete and sparse nature of event-based visual data, and to
the continuous-time setting of spiking neural networks. We test our methods on
the N-MNIST and IBM Gestures neuromorphic vision datasets and show adversarial
perturbations achieve a high success rate, by injecting a relatively small
number of appropriately placed events. We also verify, for the first time, the
effectiveness of these perturbations directly on neuromorphic hardware.
Finally, we discuss the properties of the resulting perturbations and possible
future directions.


http://arxiv.org/abs/2110.03092
A Uniform Framework for Anomaly Detection in Deep Neural Networks. (97%)
Fangzhen Zhao; Chenyi Zhang; Naipeng Dong; Zefeng You; Zhenxin Wu
  Deep neural networks (DNN) can achieve high performance when applied to
In-Distribution (ID) data which come from the same distribution as the training
set. When presented with anomaly inputs not from the ID, the outputs of a DNN
should be regarded as meaningless. However, modern DNN often predict anomaly
inputs as an ID class with high confidence, which is dangerous and misleading.
In this work, we consider three classes of anomaly inputs, (1) natural inputs
from a different distribution than the DNN is trained for, known as
Out-of-Distribution (OOD) samples, (2) crafted inputs generated from ID by
attackers, often known as adversarial (AD) samples, and (3) noise (NS) samples
generated from meaningless data. We propose a framework that aims to detect all
these anomalies for a pre-trained DNN. Unlike some of the existing works, our
method does not require preprocessing of input data, nor is it dependent to any
known OOD set or adversarial attack algorithm. Through extensive experiments
over a variety of DNN models for the detection of aforementioned anomalies, we
show that in most cases our method outperforms state-of-the-art anomaly
detection methods in identifying all three classes of anomalies.


http://arxiv.org/abs/2110.03135
Double Descent in Adversarial Training: An Implicit Label Noise Perspective. (88%)
Chengyu Dong; Liyuan Liu; Jingbo Shang
  Here, we show that the robust overfitting shall be viewed as the early part
of an epoch-wise double descent -- the robust test error will start to decrease
again after training the model for a considerable number of epochs. Inspired by
our observations, we further advance the analyses of double descent to
understand robust overfitting better. In standard training, double descent has
been shown to be a result of label flipping noise. However, this reasoning is
not applicable in our setting, since adversarial perturbations are believed not
to change the label. Going beyond label flipping noise, we propose to measure
the mismatch between the assigned and (unknown) true label distributions,
denoted as \emph{implicit label noise}. We show that the traditional labeling
of adversarial examples inherited from their clean counterparts will lead to
implicit label noise. Towards better labeling, we show that predicted
distribution from a classifier, after scaling and interpolation, can provably
reduce the implicit label noise under mild assumptions. In light of our
analyses, we tailored the training objective accordingly to effectively
mitigate the double descent and verified its effectiveness on three benchmark
datasets.


http://arxiv.org/abs/2110.03124
Improving Adversarial Robustness for Free with Snapshot Ensemble. (83%)
Yihao Wang
  Adversarial training, as one of the few certified defenses against
adversarial attacks, can be quite complicated and time-consuming, while the
results might not be robust enough. To address the issue of lack of robustness,
ensemble methods were proposed, aiming to get the final output by weighting the
selected results from repeatedly trained processes. It is proved to be very
useful in achieving robust and accurate results, but the computational and
memory costs are even higher. Snapshot ensemble, a new ensemble method that
combines several local minima in a single training process to make the final
prediction, was proposed recently, which reduces the time spent on training
multiple networks and the memory to store the results. Based on the snapshot
ensemble, we present a new method that is easier to implement: unlike original
snapshot ensemble that seeks for local minima, our snapshot ensemble focuses on
the last few iterations of a training and stores the sets of parameters from
them. Our algorithm is much simpler but the results are no less accurate than
the original ones: based on different hyperparameters and datasets, our
snapshot ensemble has shown a 5% to 30% increase in accuracy when compared to
the traditional adversarial training.


http://arxiv.org/abs/2110.03154
DoubleStar: Long-Range Attack Towards Depth Estimation based Obstacle Avoidance in Autonomous Systems. (45%)
Ce Michigan State University Zhou; Qiben Michigan State University Yan; Yan Michigan State University Shi; Lichao Lehigh University Sun
  Depth estimation-based obstacle avoidance has been widely adopted by
autonomous systems (drones and vehicles) for safety purpose. It normally relies
on a stereo camera to automatically detect obstacles and make flying/driving
decisions, e.g., stopping several meters ahead of the obstacle in the path or
moving away from the detected obstacle. In this paper, we explore new security
risks associated with the stereo vision-based depth estimation algorithms used
for obstacle avoidance. By exploiting the weaknesses of the stereo matching in
depth estimation algorithms and the lens flare effect in optical imaging, we
propose DoubleStar, a long-range attack that injects fake obstacle depth by
projecting pure light from two complementary light sources.
  DoubleStar includes two distinctive attack formats: beams attack and orbs
attack, which leverage projected light beams and lens flare orbs respectively
to cause false depth perception. We successfully attack two commercial stereo
cameras designed for autonomous systems (ZED and Intel RealSense). The
visualization of fake depth perceived by the stereo cameras illustrates the
false stereo matching induced by DoubleStar. We further use Ardupilot to
simulate the attack and demonstrate its impact on drones. To validate the
attack on real systems, we perform a real-world attack towards a commercial
drone equipped with state-of-the-art obstacle avoidance algorithms. Our attack
can continuously bring a flying drone to a sudden stop or drift it away across
a long distance under various lighting conditions, even bypassing sensor fusion
mechanisms. Specifically, our experimental results show that DoubleStar creates
fake depth up to 15 meters in distance at night and up to 8 meters during the
daytime. To mitigate this newly discovered threat, we provide discussions on
potential countermeasures to defend against DoubleStar.


http://arxiv.org/abs/2110.02631
Inference Attacks Against Graph Neural Networks. (2%)
Zhikun Zhang; Min Chen; Michael Backes; Yun Shen; Yang Zhang
  Graph is an important data representation ubiquitously existing in the real
world. However, analyzing the graph data is computationally difficult due to
its non-Euclidean nature. Graph embedding is a powerful tool to solve the graph
analytics problem by transforming the graph data into low-dimensional vectors.
These vectors could also be shared with third parties to gain additional
insights of what is behind the data. While sharing graph embedding is
intriguing, the associated privacy risks are unexplored. In this paper, we
systematically investigate the information leakage of the graph embedding by
mounting three inference attacks. First, we can successfully infer basic graph
properties, such as the number of nodes, the number of edges, and graph
density, of the target graph with up to 0.89 accuracy. Second, given a subgraph
of interest and the graph embedding, we can determine with high confidence that
whether the subgraph is contained in the target graph. For instance, we achieve
0.98 attack AUC on the DD dataset. Third, we propose a novel graph
reconstruction attack that can reconstruct a graph that has similar graph
structural statistics to the target graph. We further propose an effective
defense mechanism based on graph embedding perturbation to mitigate the
inference attacks without noticeable performance degradation for graph
classification tasks. Our code is available at
https://github.com/Zhangzhk0819/GNN-Embedding-Leaks.


http://arxiv.org/abs/2110.03141
Efficient Sharpness-aware Minimization for Improved Training of Neural Networks. (1%)
Jiawei Du; Hanshu Yan; Jiashi Feng; Joey Tianyi Zhou; Liangli Zhen; Rick Siow Mong Goh; Vincent Y. F. Tan
  Overparametrized Deep Neural Networks (DNNs) often achieve astounding
performances, but may potentially result in severe generalization error.
Recently, the relation between the sharpness of the loss landscape and the
generalization error has been established by Foret et al. (2020), in which the
Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the
generalization. Unfortunately, SAM s computational cost is roughly double that
of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus
proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts SAM s
efficiency at no cost to its generalization performance. ESAM includes two
novel and efficient training strategies-StochasticWeight Perturbation and
Sharpness-Sensitive Data Selection. In the former, the sharpness measure is
approximated by perturbing a stochastically chosen set of weights in each
iteration; in the latter, the SAM loss is optimized using only a judiciously
selected subset of data that is sensitive to the sharpness. We provide
theoretical explanations as to why these strategies perform well. We also show,
via extensive experiments on the CIFAR and ImageNet datasets, that ESAM
enhances the efficiency over SAM from requiring 100% extra computations to 40%
vis-a-vis base optimizers, while test accuracies are preserved or even
improved.


http://arxiv.org/abs/2110.03149
Data-driven behavioural biometrics for continuous and adaptive user verification using Smartphone and Smartwatch. (1%)
Akriti Verma; Valeh Moghaddam; Adnan Anwar
  Recent studies have shown how motion-based biometrics can be used as a form
of user authentication and identification without requiring any human
cooperation. This category of behavioural biometrics deals with the features we
learn in our life as a result of our interaction with the environment and
nature. This modality is related to change in human behaviour over time. The
developments in these methods aim to amplify continuous authentication such as
biometrics to protect their privacy on user devices. Various Continuous
Authentication (CA) systems have been proposed in the literature. They
represent a new generation of security mechanisms that continuously monitor
user behaviour and use this as the basis to re-authenticate them periodically
throughout a login session. However, these methods usually constitute a single
classification model which is used to identify or verify a user. This work
proposes an algorithm to blend behavioural biometrics with multi-factor
authentication (MFA) by introducing a two-step user verification algorithm that
verifies the user's identity using motion-based biometrics and complements the
multi-factor authentication, thus making it more secure and flexible. This
two-step user verification algorithm is also immune to adversarial attacks,
based on our experimental results which show how the rate of misclassification
drops while using this model with adversarial data.


http://arxiv.org/abs/2110.03054
On The Vulnerability of Recurrent Neural Networks to Membership Inference Attacks. (1%)
Yunhao Yang; Parham Gohari; Ufuk Topcu
  We study the privacy implications of deploying recurrent neural networks in
machine learning. We consider membership inference attacks (MIAs) in which an
attacker aims to infer whether a given data record has been used in the
training of a learning agent. Using existing MIAs that target feed-forward
neural networks, we empirically demonstrate that the attack accuracy wanes for
data records used earlier in the training history. Alternatively, recurrent
networks are specifically designed to better remember their past experience;
hence, they are likely to be more vulnerable to MIAs than their feed-forward
counterparts. We develop a pair of MIA layouts for two primary applications of
recurrent networks, namely, deep reinforcement learning and
sequence-to-sequence tasks. We use the first attack to provide empirical
evidence that recurrent networks are indeed more vulnerable to MIAs than
feed-forward networks with the same performance level. We use the second attack
to showcase the differences between the effects of overtraining recurrent and
feed-forward networks on the accuracy of their respective MIAs. Finally, we
deploy a differential privacy mechanism to resolve the privacy vulnerability
that the MIAs exploit. For both attack layouts, the privacy mechanism degrades
the attack accuracy from above 80% to 50%, which is equal to guessing the data
membership uniformly at random, while trading off less than 10% utility.


http://arxiv.org/abs/2110.02504
Stegomalware: A Systematic Survey of MalwareHiding and Detection in Images, Machine LearningModels and Research Challenges. (1%)
Rajasekhar Chaganti; Vinayakumar Ravi; Mamoun Alazab; Tuan D. Pham
  Malware distribution to the victim network is commonly performed through file
attachments in phishing email or from the internet, when the victim interacts
with the source of infection. To detect and prevent the malware distribution in
the victim machine, the existing end device security applications may leverage
techniques such as signature or anomaly-based, machine learning techniques. The
well-known file formats Portable Executable (PE) for Windows and Executable and
Linkable Format (ELF) for Linux based operating system are used for malware
analysis, and the malware detection capabilities of these files has been well
advanced for real-time detection. But the malware payload hiding in multimedia
using steganography detection has been a challenge for enterprises, as these
are rarely seen and usually act as a stager in sophisticated attacks. In this
article, to our knowledge, we are the first to try to address the knowledge gap
between the current progress in image steganography and steganalysis academic
research focusing on data hiding and the review of the stegomalware (malware
payload hiding in images) targeting enterprises with cyberattacks current
status. We present the stegomalware history, generation tools, file format
specification description. Based on our findings, we perform the detail review
of the image steganography techniques including the recent Generative
Adversarial Networks (GAN) based models and the image steganalysis methods
including the Deep Learning(DL) models for hiding data detection. Additionally,
the stegomalware detection framework for enterprise is proposed for anomaly
based stegomalware detection emphasizing the architecture details for different
network environments. Finally, the research opportunities and challenges in
stegomalware generation and detection are also presented.


http://arxiv.org/abs/2110.02863
Exploring the Common Principal Subspace of Deep Features in Neural Networks. (1%)
Haoran Liu; Haoyi Xiong; Yaqing Wang; Haozhe An; Dongrui Wu; Dejing Dou
  We find that different Deep Neural Networks (DNNs) trained with the same
dataset share a common principal subspace in latent spaces, no matter in which
architectures (e.g., Convolutional Neural Networks (CNNs), Multi-Layer
Preceptors (MLPs) and Autoencoders (AEs)) the DNNs were built or even whether
labels have been used in training (e.g., supervised, unsupervised, and
self-supervised learning). Specifically, we design a new metric
$\mathcal{P}$-vector to represent the principal subspace of deep features
learned in a DNN, and propose to measure angles between the principal subspaces
using $\mathcal{P}$-vectors. Small angles (with cosine close to $1.0$) have
been found in the comparisons between any two DNNs trained with different
algorithms/architectures. Furthermore, during the training procedure from
random scratch, the angle decrease from a larger one ($70^\circ-80^\circ$
usually) to the small one, which coincides the progress of feature space
learning from scratch to convergence. Then, we carry out case studies to
measure the angle between the $\mathcal{P}$-vector and the principal subspace
of training dataset, and connect such angle with generalization performance.
Extensive experiments with practically-used Multi-Layer Perceptron (MLPs), AEs
and CNNs for classification, image reconstruction, and self-supervised learning
tasks on MNIST, CIFAR-10 and CIFAR-100 datasets have been done to support our
claims with solid evidences.
  Interpretability of Deep Learning, Feature Learning, and Subspaces of Deep
Features


http://arxiv.org/abs/2110.02718
Generalizing Neural Networks by Reflecting Deviating Data in Production. (1%)
Yan Xiao; Yun Lin; Ivan Beschastnikh; Changsheng Sun; David S. Rosenblum; Jin Song Dong
  Trained with a sufficiently large training and testing dataset, Deep Neural
Networks (DNNs) are expected to generalize. However, inputs may deviate from
the training dataset distribution in real deployments. This is a fundamental
issue with using a finite dataset. Even worse, real inputs may change over time
from the expected distribution. Taken together, these issues may lead deployed
DNNs to mis-predict in production.
  In this work, we present a runtime approach that mitigates DNN
mis-predictions caused by the unexpected runtime inputs to the DNN. In contrast
to previous work that considers the structure and parameters of the DNN itself,
our approach treats the DNN as a blackbox and focuses on the inputs to the DNN.
Our approach has two steps. First, it recognizes and distinguishes "unseen"
semantically-preserving inputs. For this we use a distribution analyzer based
on the distance metric learned by a Siamese network. Second, our approach
transforms those unexpected inputs into inputs from the training set that are
identified as having similar semantics. We call this process input reflection
and formulate it as a search problem over the embedding space on the training
set. This embedding space is learned by a Quadruplet network as an auxiliary
model for the subject model to improve the generalization.
  We implemented a tool called InputReflector based on the above two-step
approach and evaluated it with experiments on three DNN models trained on
CIFAR-10, MNIST, and FMINST image datasets. The results show that
InputReflector can effectively distinguish inputs that retain semantics of the
distribution (e.g., blurred, brightened, contrasted, and zoomed images) and
out-of-distribution inputs from normal inputs.


http://arxiv.org/abs/2110.02125
Adversarial Robustness Verification and Attack Synthesis in Stochastic Systems. (99%)
Lisa Oakley; Alina Oprea; Stavros Tripakis
  Probabilistic model checking is a useful technique for specifying and
verifying properties of stochastic systems including randomized protocols and
the theoretical underpinnings of reinforcement learning models. However, these
methods rely on the assumed structure and probabilities of certain system
transitions. These assumptions may be incorrect, and may even be violated in
the event that an adversary gains control of some or all components in the
system.
  In this paper, motivated by research in adversarial machine learning on
adversarial examples, we develop a formal framework for adversarial robustness
in systems defined as discrete time Markov chains (DTMCs), and extend to
include deterministic, memoryless policies acting in Markov decision processes
(MDPs). Our framework includes a flexible approach for specifying several
adversarial models with different capabilities to manipulate the system. We
outline a class of threat models under which adversaries can perturb system
transitions, constrained by an $\varepsilon$ ball around the original
transition probabilities and define four specific instances of this threat
model.
  We define three main DTMC adversarial robustness problems and present two
optimization-based solutions, leveraging traditional and parametric
probabilistic model checking techniques. We then evaluate our solutions on two
stochastic protocols and a collection of GridWorld case studies, which model an
agent acting in an environment described as an MDP. We find that the parametric
solution results in fast computation for small parameter spaces. In the case of
less restrictive (stronger) adversaries, the number of parameters increases,
and directly computing property satisfaction probabilities is more scalable. We
demonstrate the usefulness of our definitions and solutions by comparing system
outcomes over various properties, threat models, and case studies.


http://arxiv.org/abs/2110.01823
Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations. (99%)
Shasha Li; Abhishek Aich; Shitong Zhu; M. Salman Asif; Chengyu Song; Amit K. Roy-Chowdhury; Srikanth Krishnamurthy
  When compared to the image classification models, black-box adversarial
attacks against video classification models have been largely understudied.
This could be possible because, with video, the temporal dimension poses
significant additional challenges in gradient estimation. Query-efficient
black-box attacks rely on effectively estimated gradients towards maximizing
the probability of misclassifying the target video. In this work, we
demonstrate that such effective gradients can be searched for by parameterizing
the temporal structure of the search space with geometric transformations.
Specifically, we design a novel iterative algorithm Geometric TRAnsformed
Perturbations (GEO-TRAP), for attacking video classification models. GEO-TRAP
employs standard geometric transformation operations to reduce the search space
for effective gradients into searching for a small group of parameters that
define these operations. This group of parameters describes the geometric
progression of gradients, resulting in a reduced and structured search space.
Our algorithm inherently leads to successful perturbations with surprisingly
few queries. For example, adversarial examples generated from GEO-TRAP have
better attack success rates with ~73.55% fewer queries compared to the
state-of-the-art method for video adversarial attacks on the widely used Jester
dataset. Overall, our algorithm exposes vulnerabilities of diverse video
classification models and achieves new state-of-the-art results under black-box
settings on two large datasets.


http://arxiv.org/abs/2110.02364
Adversarial defenses via a mixture of generators. (99%)
Maciej Żelaszczyk; Jacek Mańdziuk
  In spite of the enormous success of neural networks, adversarial examples
remain a relatively weakly understood feature of deep learning systems. There
is a considerable effort in both building more powerful adversarial attacks and
designing methods to counter the effects of adversarial examples. We propose a
method to transform the adversarial input data through a mixture of generators
in order to recover the correct class obfuscated by the adversarial attack. A
canonical set of images is used to generate adversarial examples through
potentially multiple attacks. Such transformed images are processed by a set of
generators, which are trained adversarially as a whole to compete in inverting
the initial transformations. To our knowledge, this is the first use of a
mixture-based adversarially trained system as a defense mechanism. We show that
it is possible to train such a system without supervision, simultaneously on
multiple adversarial attacks. Our system is able to recover class information
for previously-unseen examples with neither attack nor data labels on the MNIST
dataset. The results demonstrate that this multi-attack approach is competitive
with adversarial defenses tested in single-attack settings.


http://arxiv.org/abs/2110.01818
Neural Network Adversarial Attack Method Based on Improved Genetic Algorithm. (92%)
Dingming Yang; Yanrong Cui; Hongqiang Yuan
  Deep learning algorithms are widely used in fields such as computer vision
and natural language processing, but they are vulnerable to security threats
from adversarial attacks because of their internal presence of a large number
of nonlinear functions and parameters leading to their uninterpretability. In
this paper, we propose a neural network adversarial attack method based on an
improved genetic algorithm. The improved genetic algorithm improves the
variation and crossover links based on the original genetic optimization
algorithm, which greatly improves the iteration efficiency and shortens the
running time. The method does not need the internal structure and parameter
information of the neural network model, and it can obtain the adversarial
samples with high confidence in a short time by the classification and
confidence information of the neural network. The experimental results show
that the method in this paper has a wide range of applicability and high
efficiency for the model, and provides a new idea for the adversarial attack.


http://arxiv.org/abs/2110.02467
BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models. (33%)
Kangjie Chen; Yuxian Meng; Xiaofei Sun; Shangwei Guo; Tianwei Zhang; Jiwei Li; Chun Fan
  Pre-trained Natural Language Processing (NLP) models can be easily adapted to
a variety of downstream language tasks. This significantly accelerates the
development of language models. However, NLP models have been shown to be
vulnerable to backdoor attacks, where a pre-defined trigger word in the input
text causes model misprediction. Previous NLP backdoor attacks mainly focus on
some specific tasks. This makes those attacks less general and applicable to
other kinds of NLP models and tasks. In this work, we propose \Name, the first
task-agnostic backdoor attack against the pre-trained NLP models. The key
feature of our attack is that the adversary does not need prior information
about the downstream tasks when implanting the backdoor to the pre-trained
model. When this malicious model is released, any downstream models transferred
from it will also inherit the backdoor, even after the extensive transfer
learning process. We further design a simple yet effective strategy to bypass a
state-of-the-art defense. Experimental results indicate that our approach can
compromise a wide range of downstream NLP tasks in an effective and stealthy
way.


http://arxiv.org/abs/2110.02417
CADA: Multi-scale Collaborative Adversarial Domain Adaptation for Unsupervised Optic Disc and Cup Segmentation. (1%)
Peng Liu; Charlie T. Tran; Bin Kong; Ruogu Fang
  The diversity of retinal imaging devices poses a significant challenge:
domain shift, which leads to performance degradation when applying the deep
learning models trained on one domain to new testing domains. In this paper, we
propose a multi-scale input along with multiple domain adaptors applied
hierarchically in both feature and output spaces. The proposed training
strategy and novel unsupervised domain adaptation framework, called
Collaborative Adversarial Domain Adaptation (CADA), can effectively overcome
the challenge. Multi-scale inputs can reduce the information loss due to the
pooling layers used in the network for feature extraction, while our proposed
CADA is an interactive paradigm that presents an exquisite collaborative
adaptation through both adversarial learning and ensembling weights at
different network layers. In particular, to produce a better prediction for the
unlabeled target domain data, we simultaneously achieve domain invariance and
model generalizability via adversarial learning at multi-scale outputs from
different levels of network layers and maintaining an exponential moving
average (EMA) of the historical weights during training. Without annotating any
sample from the target domain, multiple adversarial losses in encoder and
decoder layers guide the extraction of domain-invariant features to confuse the
domain classifier. Meanwhile, the ensembling of weights via EMA reduces the
uncertainty of adapting multiple discriminator learning. Comprehensive
experimental results demonstrate that our CADA model incorporating multi-scale
input training can overcome performance degradation and outperform
state-of-the-art domain adaptation methods in segmenting retinal optic disc and
cup from fundus images stemming from the REFUGE, Drishti-GS, and Rim-One-r3
datasets.


http://arxiv.org/abs/2110.02180
Noisy Feature Mixup. (1%)
Soon Hoe Lim; N. Benjamin Erichson; Francisco Utrera; Winnie Xu; Michael W. Mahoney
  We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method
for data augmentation that combines the best of interpolation based training
and noise injection schemes. Rather than training with convex combinations of
pairs of examples and their labels, we use noise-perturbed convex combinations
of pairs of data points in both input and feature space. This method includes
mixup and manifold mixup as special cases, but it has additional advantages,
including better smoothing of decision boundaries and enabling improved model
robustness. We provide theory to understand this as well as the implicit
regularization effects of NFM. Our theory is supported by empirical results,
demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We
show that residual networks and vision transformers trained with NFM have
favorable trade-offs between predictive accuracy on clean data and robustness
with respect to various types of data perturbation across a range of computer
vision benchmark datasets.


http://arxiv.org/abs/2110.01232
Benchmarking Safety Monitors for Image Classifiers with Machine Learning. (1%)
Raul Sena LAAS Ferreira; Jean LAAS Arlat; Jeremie LAAS Guiochet; Hélène LAAS Waeselynck
  High-accurate machine learning (ML) image classifiers cannot guarantee that
they will not fail at operation. Thus, their deployment in safety-critical
applications such as autonomous vehicles is still an open issue. The use of
fault tolerance mechanisms such as safety monitors is a promising direction to
keep the system in a safe state despite errors of the ML classifier. As the
prediction from the ML is the core information directly impacting safety, many
works are focusing on monitoring the ML model itself. Checking the efficiency
of such monitors in the context of safety-critical applications is thus a
significant challenge. Therefore, this paper aims at establishing a baseline
framework for benchmarking monitors for ML image classifiers. Furthermore, we
propose a framework covering the entire pipeline, from data generation to
evaluation. Our approach measures monitor performance with a broader set of
metrics than usually proposed in the literature. Moreover, we benchmark three
different monitor approaches in 79 benchmark datasets containing five
categories of out-of-distribution data for image classifiers: class novelty,
noise, anomalies, distributional shifts, and adversarial attacks. Our results
indicate that these monitors are no more accurate than a random monitor. We
also release the code of all experiments for reproducibility.


http://arxiv.org/abs/2110.01094
Adversarial Examples Generation for Reducing Implicit Gender Bias in Pre-trained Models. (82%)
Wenqian Ye; Fei Xu; Yaojia Huang; Cassie Huang; Ji A
  Over the last few years, Contextualized Pre-trained Neural Language Models,
such as BERT, GPT, have shown significant gains in various NLP tasks. To
enhance the robustness of existing pre-trained models, one way is adversarial
examples generation and evaluation for conducting data augmentation or
adversarial learning. In the meanwhile, gender bias embedded in the models
seems to be a serious problem in practical applications. Many researches have
covered the gender bias produced by word-level information(e.g.
gender-stereotypical occupations), while few researchers have investigated the
sentence-level cases and implicit cases.
  In this paper, we proposed a method to automatically generate implicit gender
bias samples at sentence-level and a metric to measure gender bias. Samples
generated by our method will be evaluated in terms of accuracy. The metric will
be used to guide the generation of examples from Pre-trained models. Therefore,
those examples could be used to impose attacks on Pre-trained Models. Finally,
we discussed the evaluation efficacy of our generated examples on reducing
gender bias for future research.


http://arxiv.org/abs/2110.14597
Evaluating Deep Learning Models and Adversarial Attacks on Accelerometer-Based Gesture Authentication. (98%)
Elliu Huang; Troia Fabio Di; Mark Stamp
  Gesture-based authentication has emerged as a non-intrusive, effective means
of authenticating users on mobile devices. Typically, such authentication
techniques have relied on classical machine learning techniques, but recently,
deep learning techniques have been applied this problem. Although prior
research has shown that deep learning models are vulnerable to adversarial
attacks, relatively little research has been done in the adversarial domain for
behavioral biometrics. In this research, we collect tri-axial accelerometer
gesture data (TAGD) from 46 users and perform classification experiments with
both classical machine learning and deep learning models. Specifically, we
train and test support vector machines (SVM) and convolutional neural networks
(CNN). We then consider a realistic adversarial attack, where we assume the
attacker has access to real users' TAGD data, but not the authentication model.
We use a deep convolutional generative adversarial network (DC-GAN) to create
adversarial samples, and we show that our deep learning model is surprisingly
robust to such an attack scenario.


http://arxiv.org/abs/2110.00899
Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function. (13%)
Md Tahmid Hossain; Shyh Wei Teng; Ferdous Sohel; Guojun Lu
  Deep convolutional networks are vulnerable to image translation or shift,
partly due to common down-sampling layers, e.g., max-pooling and strided
convolution. These operations violate the Nyquist sampling rate and cause
aliasing. The textbook solution is low-pass filtering (blurring) before
down-sampling, which can benefit deep networks as well. Even so, non-linearity
units, such as ReLU, often re-introduce the problem, suggesting that blurring
alone may not suffice. In this work, first, we analyse deep features with
Fourier transform and show that Depth Adaptive Blurring is more effective, as
opposed to monotonic blurring. To this end, we outline how this can replace
existing down-sampling methods. Second, we introduce a novel activation
function -- with a built-in low pass filter, to keep the problem from
reappearing. From experiments, we observe generalisation on other forms of
transformations and corruptions as well, e.g., rotation, scale, and noise. We
evaluate our method under three challenging settings: (1) a variety of image
translations; (2) adversarial attacks -- both $\ell_{p}$ bounded and unbounded;
and (3) data corruptions and perturbations. In each setting, our method
achieves state-of-the-art results and improves clean accuracy on various
benchmark datasets.


http://arxiv.org/abs/2110.00623
Calibrated Adversarial Training. (98%)
Tianjin Huang; Vlado Menkovski; Yulong Pei; Mykola Pechenizkiy
  Adversarial training is an approach of increasing the robustness of models to
adversarial attacks by including adversarial examples in the training set. One
major challenge of producing adversarial examples is to contain sufficient
perturbation in the example to flip the model's output while not making severe
changes in the example's semantical content. Exuberant change in the semantical
content could also change the true label of the example. Adding such examples
to the training set results in adverse effects. In this paper, we present the
Calibrated Adversarial Training, a method that reduces the adverse effects of
semantic perturbations in adversarial training. The method produces pixel-level
adaptations to the perturbations based on novel calibrated robust error. We
provide theoretical analysis on the calibrated robust error and derive an upper
bound for it. Our empirical results show a superior performance of the
Calibrated Adversarial Training over a number of public datasets.


http://arxiv.org/abs/2110.00708
Universal Adversarial Spoofing Attacks against Face Recognition. (87%)
Takuma Amada; Seng Pei Liew; Kazuya Kakizaki; Toshinori Araki
  We assess the vulnerabilities of deep face recognition systems for images
that falsify/spoof multiple identities simultaneously. We demonstrate that, by
manipulating the deep feature representation extracted from a face image via
imperceptibly small perturbations added at the pixel level using our proposed
Universal Adversarial Spoofing Examples (UAXs), one can fool a face
verification system into recognizing that the face image belongs to multiple
different identities with a high success rate. One characteristic of the UAXs
crafted with our method is that they are universal (identity-agnostic); they
are successful even against identities not known in advance. For a certain deep
neural network, we show that we are able to spoof almost all tested identities
(99\%), including those not known beforehand (not included in training). Our
results indicate that a multiple-identity attack is a real threat and should be
taken into account when deploying face recognition systems.


http://arxiv.org/abs/2110.00473
Score-Based Generative Classifiers. (84%)
Roland S. Zimmermann; Lukas Schott; Yang Song; Benjamin A. Dunn; David A. Klindt
  The tremendous success of generative models in recent years raises the
question whether they can also be used to perform classification. Generative
models have been used as adversarially robust classifiers on simple datasets
such as MNIST, but this robustness has not been observed on more complex
datasets like CIFAR-10. Additionally, on natural image datasets, previous
results have suggested a trade-off between the likelihood of the data and
classification accuracy. In this work, we investigate score-based generative
models as classifiers for natural images. We show that these models not only
obtain competitive likelihood values but simultaneously achieve
state-of-the-art classification accuracy for generative classifiers on
CIFAR-10. Nevertheless, we find that these models are only slightly, if at all,
more robust than discriminative baseline models on out-of-distribution tasks
based on common image corruptions. Similarly and contrary to prior results, we
find that score-based are prone to worst-case distribution shifts in the form
of adversarial perturbations. Our work highlights that score-based generative
models are closing the gap in classification accuracy compared to standard
discriminative models. While they do not yet deliver on the promise of
adversarial and out-of-domain robustness, they provide a different approach to
classification that warrants further research.


http://arxiv.org/abs/2110.05929
One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency. (1%)
Sayeed Shafayet Chowdhury; Nitin Rathi; Kaushik Roy
  Spiking Neural Networks (SNNs) are energy efficient alternatives to commonly
used deep neural networks (DNNs). Through event-driven information processing,
SNNs can reduce the expensive compute requirements of DNNs considerably, while
achieving comparable performance. However, high inference latency is a
significant hindrance to the edge deployment of deep SNNs. Computation over
multiple timesteps not only increases latency as well as overall energy budget
due to higher number of operations, but also incurs memory access overhead of
fetching membrane potentials, both of which lessen the energy benefits of SNNs.
To overcome this bottleneck and leverage the full potential of SNNs, we propose
an Iterative Initialization and Retraining method for SNNs (IIR-SNN) to perform
single shot inference in the temporal axis. The method starts with an SNN
trained with T timesteps (T>1). Then at each stage of latency reduction, the
network trained at previous stage with higher timestep is utilized as
initialization for subsequent training with lower timestep. This acts as a
compression method, as the network is gradually shrunk in the temporal domain.
In this paper, we use direct input encoding and choose T=5, since as per
literature, it is the minimum required latency to achieve satisfactory
performance on ImageNet. The proposed scheme allows us to obtain SNNs with up
to unit latency, requiring a single forward pass during inference. We achieve
top-1 accuracy of 93.05%, 70.15% and 67.71% on CIFAR-10, CIFAR-100 and
ImageNet, respectively using VGG16, with just 1 timestep. In addition, IIR-SNNs
perform inference with 5-2500X reduced latency compared to other
state-of-the-art SNNs, maintaining comparable or even better accuracy.
Furthermore, in comparison with standard DNNs, the proposed IIR-SNNs
provide25-33X higher energy efficiency, while being comparable to them in
classification performance.


http://arxiv.org/abs/2109.15160
Mitigating Black-Box Adversarial Attacks via Output Noise Perturbation. (98%)
Manjushree B. Aithal; Xiaohua Li
  In black-box adversarial attacks, adversaries query the deep neural network
(DNN), use the output to reconstruct gradients, and then optimize the
adversarial inputs iteratively. In this paper, we study the method of adding
white noise to the DNN output to mitigate such attacks, with a unique focus on
the trade-off analysis of noise level and query cost. The attacker's query
count (QC) is derived mathematically as a function of noise standard deviation.
With this result, the defender can conveniently find the noise level needed to
mitigate attacks for the desired security level specified by QC and limited DNN
performance loss. Our analysis shows that the added noise is drastically
magnified by the small variation of DNN outputs, which makes the reconstructed
gradient have an extremely low signal-to-noise ratio (SNR). Adding slight white
noise with a standard deviation less than 0.01 is enough to increase QC by many
orders of magnitude without introducing any noticeable classification accuracy
reduction. Our experiments demonstrate that this method can effectively
mitigate both soft-label and hard-label black-box attacks under realistic QC
constraints. We also show that this method outperforms many other defense
methods and is robust to the attacker's countermeasures.


http://arxiv.org/abs/2109.15177
You Cannot Easily Catch Me: A Low-Detectable Adversarial Patch for Object Detectors. (95%)
Zijian Zhu; Hang Su; Chang Liu; Wenzhao Xiang; Shibao Zheng
  Blind spots or outright deceit can bedevil and deceive machine learning
models. Unidentified objects such as digital "stickers," also known as
adversarial patches, can fool facial recognition systems, surveillance systems
and self-driving cars. Fortunately, most existing adversarial patches can be
outwitted, disabled and rejected by a simple classification network called an
adversarial patch detector, which distinguishes adversarial patches from
original images. An object detector classifies and predicts the types of
objects within an image, such as by distinguishing a motorcyclist from the
motorcycle, while also localizing each object's placement within the image by
"drawing" so-called bounding boxes around each object, once again separating
the motorcyclist from the motorcycle. To train detectors even better, however,
we need to keep subjecting them to confusing or deceitful adversarial patches
as we probe for the models' blind spots. For such probes, we came up with a
novel approach, a Low-Detectable Adversarial Patch, which attacks an object
detector with small and texture-consistent adversarial patches, making these
adversaries less likely to be recognized. Concretely, we use several geometric
primitives to model the shapes and positions of the patches. To enhance our
attack performance, we also assign different weights to the bounding boxes in
terms of loss function. Our experiments on the common detection dataset COCO as
well as the driving-video dataset D2-City show that LDAP is an effective attack
method, and can resist the adversarial patch detector.


http://arxiv.org/abs/2109.15009
Adversarial Semantic Contour for Object Detection. (92%)
Yichi Zhang; Zijian Zhu; Xiao Yang; Jun Zhu
  Modern object detectors are vulnerable to adversarial examples, which brings
potential risks to numerous applications, e.g., self-driving car. Among attacks
regularized by $\ell_p$ norm, $\ell_0$-attack aims to modify as few pixels as
possible. Nevertheless, the problem is nontrivial since it generally requires
to optimize the shape along with the texture simultaneously, which is an
NP-hard problem. To address this issue, we propose a novel method of
Adversarial Semantic Contour (ASC) guided by object contour as prior. With this
prior, we reduce the searching space to accelerate the $\ell_0$ optimization,
and also introduce more semantic information which should affect the detectors
more. Based on the contour, we optimize the selection of modified pixels via
sampling and their colors with gradient descent alternately. Extensive
experiments demonstrate that our proposed ASC outperforms the most commonly
manually designed patterns (e.g., square patches and grids) on task of
disappearing. By modifying no more than 5\% and 3.5\% of the object area
respectively, our proposed ASC can successfully mislead the mainstream object
detectors including the SSD512, Yolov4, Mask RCNN, Faster RCNN, etc.


http://arxiv.org/abs/2109.14868
From Zero-Shot Machine Learning to Zero-Day Attack Detection. (10%)
Mohanad Sarhan; Siamak Layeghy; Marcus Gallagher; Marius Portmann
  The standard ML methodology assumes that the test samples are derived from a
set of pre-observed classes used in the training phase. Where the model
extracts and learns useful patterns to detect new data samples belonging to the
same data classes. However, in certain applications such as Network Intrusion
Detection Systems, it is challenging to obtain data samples for all attack
classes that the model will most likely observe in production. ML-based NIDSs
face new attack traffic known as zero-day attacks, that are not used in the
training of the learning models due to their non-existence at the time. In this
paper, a zero-shot learning methodology has been proposed to evaluate the ML
model performance in the detection of zero-day attack scenarios. In the
attribute learning stage, the ML models map the network data features to
distinguish semantic attributes from known attack (seen) classes. In the
inference stage, the models are evaluated in the detection of zero-day attack
(unseen) classes by constructing the relationships between known attacks and
zero-day attacks. A new metric is defined as Zero-day Detection Rate, which
measures the effectiveness of the learning model in the inference stage. The
results demonstrate that while the majority of the attack classes do not
represent significant risks to organisations adopting an ML-based NIDS in a
zero-day attack scenario. However, for certain attack groups identified in this
paper, such systems are not effective in applying the learnt attributes of
attack behaviour to detect them as malicious. Further Analysis was conducted
using the Wasserstein Distance technique to measure how different such attacks
are from other attack types used in the training of the ML model. The results
demonstrate that sophisticated attacks with a low zero-day detection rate have
a significantly distinct feature distribution compared to the other attack
classes.


http://arxiv.org/abs/2109.14205
On Brightness Agnostic Adversarial Examples Against Face Recognition Systems. (99%)
Inderjeet Singh; Satoru Momiyama; Kazuya Kakizaki; Toshinori Araki
  This paper introduces a novel adversarial example generation method against
face recognition systems (FRSs). An adversarial example (AX) is an image with
deliberately crafted noise to cause incorrect predictions by a target system.
The AXs generated from our method remain robust under real-world brightness
changes. Our method performs non-linear brightness transformations while
leveraging the concept of curriculum learning during the attack generation
procedure. We demonstrate that our method outperforms conventional techniques
from comprehensive experimental investigations in the digital and physical
world. Furthermore, this method enables practical risk assessment of FRSs
against brightness agnostic AXs.


http://arxiv.org/abs/2109.15031
Back in Black: A Comparative Evaluation of Recent State-Of-The-Art Black-Box Attacks. (70%)
Kaleel Mahmood; Rigel Mahmood; Ethan Rathbun; Dijk Marten van
  The field of adversarial machine learning has experienced a near exponential
growth in the amount of papers being produced since 2018. This massive
information output has yet to be properly processed and categorized. In this
paper, we seek to help alleviate this problem by systematizing the recent
advances in adversarial machine learning black-box attacks since 2019. Our
survey summarizes and categorizes 20 recent black-box attacks. We also present
a new analysis for understanding the attack success rate with respect to the
adversarial model used in each paper. Overall, our paper surveys a wide body of
literature to highlight recent attack developments and organizes them into four
attack categories: score based attacks, decision based attacks, transfer
attacks and non-traditional attacks. Further, we provide a new mathematical
framework to show exactly how attack results can fairly be compared.


http://arxiv.org/abs/2109.14707
BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining. (41%)
Weizhe Hua; Yichi Zhang; Chuan Guo; Zhiru Zhang; G. Edward Suh
  Neural network robustness has become a central topic in machine learning in
recent years. Most training algorithms that improve the model's robustness to
adversarial and common corruptions also introduce a large computational
overhead, requiring as many as ten times the number of forward and backward
passes in order to converge. To combat this inefficiency, we propose
BulletTrain $-$ a boundary example mining technique to drastically reduce the
computational cost of robust training. Our key observation is that only a small
fraction of examples are beneficial for improving robustness. BulletTrain
dynamically predicts these important examples and optimizes robust training
algorithms to focus on the important examples. We apply our technique to
several existing robust training algorithms and achieve a 2.1$\times$ speed-up
for TRADES and MART on CIFAR-10 and a 1.7$\times$ speed-up for AugMix on
CIFAR-10-C and CIFAR-100-C without any reduction in clean and robust accuracy.


http://arxiv.org/abs/2109.14678
Mitigation of Adversarial Policy Imitation via Constrained Randomization of Policy (CRoP). (10%)
Nancirose Piazza; Vahid Behzadan
  Deep reinforcement learning (DRL) policies are vulnerable to unauthorized
replication attacks, where an adversary exploits imitation learning to
reproduce target policies from observed behavior. In this paper, we propose
Constrained Randomization of Policy (CRoP) as a mitigation technique against
such attacks. CRoP induces the execution of sub-optimal actions at random under
performance loss constraints. We present a parametric analysis of CRoP, address
the optimality of CRoP, and establish theoretical bounds on the adversarial
budget and the expectation of loss. Furthermore, we report the experimental
evaluation of CRoP in Atari environments under adversarial imitation, which
demonstrate the efficacy and feasibility of our proposed method against policy
replication attacks.


http://arxiv.org/abs/2109.14002
slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networks. (1%)
Elizabeth Newman; Julianne Chung; Matthias Chung; Lars Ruthotto
  Deep neural networks (DNNs) have shown their success as high-dimensional
function approximators in many applications; however, training DNNs can be
challenging in general. DNN training is commonly phrased as a stochastic
optimization problem whose challenges include non-convexity, non-smoothness,
insufficient regularization, and complicated data distributions. Hence, the
performance of DNNs on a given task depends crucially on tuning
hyperparameters, especially learning rates and regularization parameters. In
the absence of theoretical guidelines or prior experience on similar tasks,
this requires solving many training problems, which can be time-consuming and
demanding on computational resources. This can limit the applicability of DNNs
to problems with non-standard, complex, and scarce datasets, e.g., those
arising in many scientific applications. To remedy the challenges of DNN
training, we propose slimTrain, a stochastic optimization method for training
DNNs with reduced sensitivity to the choice hyperparameters and fast initial
convergence. The central idea of slimTrain is to exploit the separability
inherent in many DNN architectures; that is, we separate the DNN into a
nonlinear feature extractor followed by a linear model. This separability
allows us to leverage recent advances made for solving large-scale, linear,
ill-posed inverse problems. Crucially, for the linear weights, slimTrain does
not require a learning rate and automatically adapts the regularization
parameter. Since our method operates on mini-batches, its computational
overhead per iteration is modest. In our numerical experiments, slimTrain
outperforms existing DNN training methods with the recommended hyperparameter
settings and reduces the sensitivity of DNN training to the remaining
hyperparameters.


http://arxiv.org/abs/2109.12838
MUTEN: Boosting Gradient-Based Adversarial Attacks via Mutant-Based Ensembles. (99%)
Yuejun Guo; Qiang Hu; Maxime Cordy; Michail Papadakis; Yves Le Traon
  Deep Neural Networks (DNNs) are vulnerable to adversarial examples, which
causes serious threats to security-critical applications. This motivated much
research on providing mechanisms to make models more robust against adversarial
attacks. Unfortunately, most of these defenses, such as gradient masking, are
easily overcome through different attack means. In this paper, we propose
MUTEN, a low-cost method to improve the success rate of well-known attacks
against gradient-masking models. Our idea is to apply the attacks on an
ensemble model which is built by mutating the original model elements after
training. As we found out that mutant diversity is a key factor in improving
success rate, we design a greedy algorithm for generating diverse mutants
efficiently. Experimental results on MNIST, SVHN, and CIFAR10 show that MUTEN
can increase the success rate of four attacks by up to 0.45.


http://arxiv.org/abs/2109.13069
Query-based Adversarial Attacks on Graph with Fake Nodes. (99%)
Zhengyi Wang; Zhongkai Hao; Hang Su; Jun Zhu
  While deep neural networks have achieved great success on the graph analysis,
recent works have shown that they are also vulnerable to adversarial attacks
where fraudulent users can fool the model with a limited number of queries.
Compared with adversarial attacks on image classification, performing
adversarial attack on graphs is challenging because of the discrete and
non-differential nature of a graph. To address these issues, we proposed
Cluster Attack, a novel adversarial attack by introducing a set of fake nodes
to the original graph which can mislead the classification on certain victim
nodes. Specifically, we query the victim model for each victim node to acquire
their most adversarial feature, which is related to how the fake node's feature
will affect the victim nodes. We further cluster the victim nodes into several
subgroups according to their most adversarial features such that we can reduce
the searching space. Moreover, our attack is performed in a practical and
unnoticeable manner: (1) We protect the predicted labels of nodes which we are
not aimed for from being changed during attack. (2) We attack by introducing
fake nodes into the original graph without changing existing links and
features. (3) We attack with only partial information about the attacked graph,
i.e., by leveraging the information of victim nodes along with their neighbors
within $k$-hop instead of the whole graph. (4) We perform attack with a limited
number of queries about the predicted scores of the model in a black-box
manner, i.e., without model architecture and parameters. Extensive experiments
demonstrate the effectiveness of our method in terms of the success rate of
attack.


http://arxiv.org/abs/2109.13215
Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective. (98%)
Adhyyan Narang; Vidya Muthukumar; Anant Sahai
  State-of-the-art deep learning classifiers are heavily overparameterized with
respect to the amount of training examples and observed to generalize well on
"clean" data, but be highly susceptible to infinitesmal adversarial
perturbations. In this paper, we identify an overparameterized linear ensemble,
that uses the "lifted" Fourier feature map, that demonstrates both of these
behaviors. The input is one-dimensional, and the adversary is only allowed to
perturb these inputs and not the non-linear features directly. We find that the
learned model is susceptible to adversaries in an intermediate regime where
classification generalizes but regression does not. Notably, the susceptibility
arises despite the absence of model mis-specification or label noise, which are
commonly cited reasons for adversarial-susceptibility. These results are
extended theoretically to a random-Fourier-sum setup that exhibits
double-descent behavior. In both feature-setups, the adversarial vulnerability
arises because of a phenomenon we term spatial localization: the predictions of
the learned model are markedly more sensitive in the vicinity of training
points than elsewhere. This sensitivity is a consequence of feature lifting and
is reminiscent of Gibb's and Runge's phenomena from signal processing and
functional analysis. Despite the adversarial susceptibility, we find that
classification with these features can be easier than the more commonly studied
"independent feature" models.


http://arxiv.org/abs/2109.13297
GANG-MAM: GAN based enGine for Modifying Android Malware. (64%)
Renjith G; Sonia Laudanna; Aji S; Corrado Aaron Visaggio; Vinod P
  Malware detectors based on machine learning are vulnerable to adversarial
attacks. Generative Adversarial Networks (GAN) are architectures based on
Neural Networks that could produce successful adversarial samples. The interest
towards this technology is quickly growing. In this paper, we propose a system
that produces a feature vector for making an Android malware strongly evasive
and then modify the malicious program accordingly. Such a system could have a
twofold contribution: it could be used to generate datasets to validate systems
for detecting GAN-based malware and to enlarge the training and testing dataset
for making more robust malware classifiers.


http://arxiv.org/abs/2109.12803
Distributionally Robust Multi-Output Regression Ranking. (3%)
Shahabeddin Sotudian; Ruidi Chen; Ioannis Paschalidis
  Despite their empirical success, most existing listwiselearning-to-rank (LTR)
models are not built to be robust to errors in labeling or annotation,
distributional data shift, or adversarial data perturbations. To fill this gap,
we introduce a new listwise LTR model called Distributionally Robust
Multi-output Regression Ranking (DRMRR). Different from existing methods, the
scoring function of DRMRR was designed as a multivariate mapping from a feature
vector to a vector of deviation scores, which captures local context
information and cross-document interactions. DRMRR uses a Distributionally
Robust Optimization (DRO) framework to minimize a multi-output loss function
under the most adverse distributions in the neighborhood of the empirical data
distribution defined by a Wasserstein ball. We show that this is equivalent to
a regularized regression problem with a matrix norm regularizer. Our
experiments were conducted on two real-world applications, medical document
retrieval, and drug response prediction, showing that DRMRR notably outperforms
state-of-the-art LTR models. We also conducted a comprehensive analysis to
assess the resilience of DRMRR against various types of noise: Gaussian noise,
adversarial perturbations, and label poisoning. We show that DRMRR is not only
able to achieve significantly better performance than other baselines, but it
can maintain a relatively stable performance as more noise is added to the
data.


http://arxiv.org/abs/2109.12851
Improving Uncertainty of Deep Learning-based Object Classification on Radar Spectra using Label Smoothing. (1%)
Kanil Patel; William Beluch; Kilian Rambach; Michael Pfeiffer; Bin Yang
  Object type classification for automotive radar has greatly improved with
recent deep learning (DL) solutions, however these developments have mostly
focused on the classification accuracy. Before employing DL solutions in
safety-critical applications, such as automated driving, an indispensable
prerequisite is the accurate quantification of the classifiers' reliability.
Unfortunately, DL classifiers are characterized as black-box systems which
output severely over-confident predictions, leading downstream decision-making
systems to false conclusions with possibly catastrophic consequences. We find
that deep radar classifiers maintain high-confidences for ambiguous, difficult
samples, e.g. small objects measured at large distances, under domain shift and
signal corruptions, regardless of the correctness of the predictions. The focus
of this article is to learn deep radar spectra classifiers which offer robust
real-time uncertainty estimates using label smoothing during training. Label
smoothing is a technique of refining, or softening, the hard labels typically
available in classification datasets. In this article, we exploit
radar-specific know-how to define soft labels which encourage the classifiers
to learn to output high-quality calibrated uncertainty estimates, thereby
partially resolving the problem of over-confidence. Our investigations show how
simple radar knowledge can easily be combined with complex data-driven learning
algorithms to yield safe automotive radar perception.


http://arxiv.org/abs/2109.13012
Federated Deep Learning with Bayesian Privacy. (1%)
Hanlin Gu; Lixin Fan; Bowen Li; Yan Kang; Yuan Yao; Qiang Yang
  Federated learning (FL) aims to protect data privacy by cooperatively
learning a model without sharing private data among users. For Federated
Learning of Deep Neural Network with billions of model parameters, existing
privacy-preserving solutions are unsatisfactory. Homomorphic encryption (HE)
based methods provide secure privacy protections but suffer from extremely high
computational and communication overheads rendering it almost useless in
practice . Deep learning with Differential Privacy (DP) was implemented as a
practical learning algorithm at a manageable cost in complexity. However, DP is
vulnerable to aggressive Bayesian restoration attacks as disclosed in the
literature and demonstrated in experimental results of this work. To address
the aforementioned perplexity, we propose a novel Bayesian Privacy (BP)
framework which enables Bayesian restoration attacks to be formulated as the
probability of reconstructing private data from observed public information.
Specifically, the proposed BP framework accurately quantifies privacy loss by
Kullback-Leibler (KL) Divergence between the prior distribution about the
privacy data and the posterior distribution of restoration private data
conditioning on exposed information}. To our best knowledge, this Bayesian
Privacy analysis is the first to provides theoretical justification of secure
privacy-preserving capabilities against Bayesian restoration attacks. As a
concrete use case, we demonstrate that a novel federated deep learning method
using private passport layers is able to simultaneously achieve high model
performance, privacy-preserving capability and low computational complexity.
Theoretical analysis is in accordance with empirical measurements of
information leakage extensively experimented with a variety of DNN networks on
image classification MNIST, CIFAR10, and CIFAR100 datasets.


http://arxiv.org/abs/2109.12772
Distributionally Robust Multiclass Classification and Applications in Deep CNN Image Classifiers. (11%)
Ruidi Chen; Boran Hao; Ioannis Paschalidis
  We develop a Distributionally Robust Optimization (DRO) formulation for
Multiclass Logistic Regression (MLR), which could tolerate data contaminated by
outliers. The DRO framework uses a probabilistic ambiguity set defined as a
ball of distributions that are close to the empirical distribution of the
training set in the sense of the Wasserstein metric. We relax the DRO
formulation into a regularized learning problem whose regularizer is a norm of
the coefficient matrix. We establish out-of-sample performance guarantees for
the solutions to our model, offering insights on the role of the regularizer in
controlling the prediction error. We apply the proposed method in rendering
deep CNN-based image classifiers robust to random and adversarial attacks.
Specifically, using the MNIST and CIFAR-10 datasets, we demonstrate reductions
in test error rate by up to 78.8% and loss by up to 90.8%. We also show that
with a limited number of perturbed images in the training set, our method can
improve the error rate by up to 49.49% and the loss by up to 68.93% compared to
Empirical Risk Minimization (ERM), converging faster to an ideal loss/error
rate as the number of perturbed images increases.


http://arxiv.org/abs/2109.12459
Two Souls in an Adversarial Image: Towards Universal Adversarial Example Detection using Multi-view Inconsistency. (99%)
Sohaib Kiani; Sana Awan; Chao Lan; Fengjun Li; Bo Luo
  In the evasion attacks against deep neural networks (DNN), the attacker
generates adversarial instances that are visually indistinguishable from benign
samples and sends them to the target DNN to trigger misclassifications. In this
paper, we propose a novel multi-view adversarial image detector, namely Argos,
based on a novel observation. That is, there exist two "souls" in an
adversarial instance, i.e., the visually unchanged content, which corresponds
to the true label, and the added invisible perturbation, which corresponds to
the misclassified label. Such inconsistencies could be further amplified
through an autoregressive generative approach that generates images with seed
pixels selected from the original image, a selected label, and pixel
distributions learned from the training data. The generated images (i.e., the
"views") will deviate significantly from the original one if the label is
adversarial, demonstrating inconsistencies that Argos expects to detect. To
this end, Argos first amplifies the discrepancies between the visual content of
an image and its misclassified label induced by the attack using a set of
regeneration mechanisms and then identifies an image as adversarial if the
reproduced views deviate to a preset degree. Our experimental results show that
Argos significantly outperforms two representative adversarial detectors in
both detection accuracy and robustness against six well-known adversarial
attacks. Code is available at:
https://github.com/sohaib730/Argos-Adversarial_Detection


http://arxiv.org/abs/2109.13232
Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning. (98%)
Víctor Gallego
  The rampant adoption of ML methodologies has revealed that models are usually
adopted to make decisions without taking into account the uncertainties in
their predictions. More critically, they can be vulnerable to adversarial
examples. Thus, we believe that developing ML systems that take into account
predictive uncertainties and are robust against adversarial examples is a must
for critical, real-world tasks. We start with a case study in retailing. We
propose a robust implementation of the Nerlove-Arrow model using a Bayesian
structural time series model. Its Bayesian nature facilitates incorporating
prior information reflecting the manager's views, which can be updated with
relevant data. However, this case adopted classical Bayesian techniques, such
as the Gibbs sampler. Nowadays, the ML landscape is pervaded with neural
networks and this chapter also surveys current developments in this sub-field.
Then, we tackle the problem of scaling Bayesian inference to complex models and
large data regimes. In the first part, we propose a unifying view of two
different Bayesian inference algorithms, Stochastic Gradient Markov Chain Monte
Carlo (SG-MCMC) and Stein Variational Gradient Descent (SVGD), leading to
improved and efficient novel sampling schemes. In the second part, we develop a
framework to boost the efficiency of Bayesian inference in probabilistic models
by embedding a Markov chain sampler within a variational posterior
approximation. After that, we present an alternative perspective on adversarial
classification based on adversarial risk analysis, and leveraging the scalable
Bayesian approaches from chapter 2. In chapter 4 we turn to reinforcement
learning, introducing Threatened Markov Decision Processes, showing the
benefits of accounting for adversaries in RL while the agent learns.


http://arxiv.org/abs/2109.12406
MINIMAL: Mining Models for Data Free Universal Adversarial Triggers. (93%)
Swapnil Parekh; Yaman Singla Kumar; Somesh Singh; Changyou Chen; Balaji Krishnamurthy; Rajiv Ratn Shah
  It is well known that natural language models are vulnerable to adversarial
attacks, which are mostly input-specific in nature. Recently, it has been shown
that there also exist input-agnostic attacks in NLP models, called universal
adversarial triggers. However, existing methods to craft universal triggers are
data intensive. They require large amounts of data samples to generate
adversarial triggers, which are typically inaccessible by attackers. For
instance, previous works take 3000 data samples per class for the SNLI dataset
to generate adversarial triggers. In this paper, we present a novel data-free
approach, MINIMAL, to mine input-agnostic adversarial triggers from models.
Using the triggers produced with our data-free algorithm, we reduce the
accuracy of Stanford Sentiment Treebank's positive class from 93.6% to 9.6%.
Similarly, for the Stanford Natural Language Inference (SNLI), our single-word
trigger reduces the accuracy of the entailment class from 90.95% to less than
0.6\%. Despite being completely data-free, we get equivalent accuracy drops as
data-dependent methods.


http://arxiv.org/abs/2109.11803
Local Intrinsic Dimensionality Signals Adversarial Perturbations. (98%)
Sandamal Weerasinghe; Tansu Alpcan; Sarah M. Erfani; Christopher Leckie; Benjamin I. P. Rubinstein
  The vulnerability of machine learning models to adversarial perturbations has
motivated a significant amount of research under the broad umbrella of
adversarial machine learning. Sophisticated attacks may cause learning
algorithms to learn decision functions or make decisions with poor predictive
performance. In this context, there is a growing body of literature that uses
local intrinsic dimensionality (LID), a local metric that describes the minimum
number of latent variables required to describe each data point, for detecting
adversarial samples and subsequently mitigating their effects. The research to
date has tended to focus on using LID as a practical defence method often
without fully explaining why LID can detect adversarial samples. In this paper,
we derive a lower-bound and an upper-bound for the LID value of a perturbed
data point and demonstrate that the bounds, in particular the lower-bound, has
a positive correlation with the magnitude of the perturbation. Hence, we
demonstrate that data points that are perturbed by a large amount would have
large LID values compared to unperturbed samples, thus justifying its use in
the prior literature. Furthermore, our empirical validation demonstrates the
validity of the bounds on benchmark datasets.


http://arxiv.org/abs/2109.11308
Breaking BERT: Understanding its Vulnerabilities for Biomedical Named Entity Recognition through Adversarial Attack. (98%)
Anne Dirkson; Suzan Verberne; Wessel Kraaij
  Biomedical named entity recognition (NER) is a key task in the extraction of
information from biomedical literature and electronic health records. For this
task, both generic and biomedical BERT models are widely used. Robustness of
these models is vital for medical applications, such as automated medical
decision making. In this paper we investigate the vulnerability of BERT models
to variation in input data for NER through adversarial attack. Since
adversarial attack methods for NER are sparse, we propose two black-box methods
for NER based on existing methods for classification tasks. Experimental
results show that the original as well as the biomedical BERT models are highly
vulnerable to entity replacement: They can be fooled in 89.2 to 99.4% of the
cases to mislabel previously correct entities. BERT models are also vulnerable
to variation in the entity context with 20.2 to 45.0% of entities predicted
completely wrong and another 29.3 to 53.3% of entities predicted wrong
partially. Often a single change is sufficient to fool the model. BERT models
seem most vulnerable to changes in the local context of entities. Of the
biomedical BERT models, the vulnerability of BioBERT is comparable to the
original BERT model whereas SciBERT is even more vulnerable. Our results chart
the vulnerabilities of BERT models for biomedical NER and emphasize the
importance of further research into uncovering and reducing these weaknesses.


http://arxiv.org/abs/2109.11249
FooBaR: Fault Fooling Backdoor Attack on Neural Network Training. (88%)
Jakub Breier; Xiaolu Hou; Martín Ochoa; Jesus Solano
  Neural network implementations are known to be vulnerable to physical attack
vectors such as fault injection attacks. As of now, these attacks were only
utilized during the inference phase with the intention to cause a
misclassification. In this work, we explore a novel attack paradigm by
injecting faults during the training phase of a neural network in a way that
the resulting network can be attacked during deployment without the necessity
of further faulting. In particular, we discuss attacks against ReLU activation
functions that make it possible to generate a family of malicious inputs, which
are called fooling inputs, to be used at inference time to induce controlled
misclassifications. Such malicious inputs are obtained by mathematically
solving a system of linear equations that would cause a particular behaviour on
the attacked activation functions, similar to the one induced in training
through faulting. We call such attacks fooling backdoors as the fault attacks
at the training phase inject backdoors into the network that allow an attacker
to produce fooling inputs. We evaluate our approach against multi-layer
perceptron networks and convolutional networks on a popular image
classification task obtaining high attack success rates (from 60% to 100%) and
high classification confidence when as little as 25 neurons are attacked while
preserving high accuracy on the originally intended classification task.


http://arxiv.org/abs/2109.11728
AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses. (68%)
Yaman Kumar Singla; Swapnil Parekh; Somesh Singh; Junyi Jessy Li; Rajiv Ratn Shah; Changyou Chen
  Deep-learning based Automatic Essay Scoring (AES) systems are being actively
used by states and language testing agencies alike to evaluate millions of
candidates for life-changing decisions ranging from college applications to
visa approvals. However, little research has been put to understand and
interpret the black-box nature of deep-learning based scoring algorithms.
Previous studies indicate that scoring models can be easily fooled. In this
paper, we explore the reason behind their surprising adversarial brittleness.
We utilize recent advances in interpretability to find the extent to which
features such as coherence, content, vocabulary, and relevance are important
for automated scoring mechanisms. We use this to investigate the
oversensitivity i.e., large change in output score with a little change in
input essay content) and overstability i.e., little change in output scores
with large changes in input essay content) of AES. Our results indicate that
autoscoring models, despite getting trained as "end-to-end" models with rich
contextual embeddings such as BERT, behave like bag-of-words models. A few
words determine the essay score without the requirement of any context making
the model largely overstable. This is in stark contrast to recent probing
studies on pre-trained representation learning models, which show that rich
linguistic features such as parts-of-speech and morphology are encoded by them.
Further, we also find that the models have learnt dataset biases, making them
oversensitive. To deal with these issues, we propose detection-based protection
models that can detect oversensitivity and overstability causing samples with
high accuracies. We find that our proposed models are able to detect unusual
attribution patterns and flag adversarial samples successfully.


http://arxiv.org/abs/2109.11495
DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications. (1%)
Dongqi Han; Zhiliang Wang; Wenqi Chen; Ying Zhong; Su Wang; Han Zhang; Jiahai Yang; Xingang Shi; Xia Yin
  Unsupervised Deep Learning (DL) techniques have been widely used in various
security-related anomaly detection applications, owing to the great promise of
being able to detect unforeseen threats and superior performance provided by
Deep Neural Networks (DNN). However, the lack of interpretability creates key
barriers to the adoption of DL models in practice. Unfortunately, existing
interpretation approaches are proposed for supervised learning models and/or
non-security domains, which are unadaptable for unsupervised DL models and fail
to satisfy special requirements in security domains.
  In this paper, we propose DeepAID, a general framework aiming to (1)
interpret DL-based anomaly detection systems in security domains, and (2)
improve the practicality of these systems based on the interpretations. We
first propose a novel interpretation method for unsupervised DNNs by
formulating and solving well-designed optimization problems with special
constraints for security domains. Then, we provide several applications based
on our Interpreter as well as a model-based extension Distiller to improve
security systems by solving domain-specific problems. We apply DeepAID over
three types of security-related anomaly detection systems and extensively
evaluate our Interpreter with representative prior works. Experimental results
show that DeepAID can provide high-quality interpretations for unsupervised DL
models while meeting the special requirements of security domains. We also
provide several use cases to show that DeepAID can help security operators to
understand model decisions, diagnose system mistakes, give feedback to models,
and reduce false positives.


http://arxiv.org/abs/2109.10770
Exploring Adversarial Examples for Efficient Active Learning in Machine Learning Classifiers. (99%)
Honggang Yu; Shihfeng Zeng; Teng Zhang; Ing-Chao Lin; Yier Jin
  Machine learning researchers have long noticed the phenomenon that the model
training process will be more effective and efficient when the training samples
are densely sampled around the underlying decision boundary. While this
observation has already been widely applied in a range of machine learning
security techniques, it lacks theoretical analyses of the correctness of the
observation. To address this challenge, we first add particular perturbation to
original training examples using adversarial attack methods so that the
generated examples could lie approximately on the decision boundary of the ML
classifiers. We then investigate the connections between active learning and
these particular training examples. Through analyzing various representative
classifiers such as k-NN classifiers, kernel methods as well as deep neural
networks, we establish a theoretical foundation for the observation. As a
result, our theoretical proofs provide support to more efficient active
learning methods with the help of adversarial examples, contrary to previous
works where adversarial examples are often used as destructive solutions.
Experimental results show that the established theoretical foundation will
guide better active learning strategies based on adversarial examples.


http://arxiv.org/abs/2109.10696
CC-Cert: A Probabilistic Approach to Certify General Robustness of Neural Networks. (81%)
Mikhail Pautov; Nurislam Tursynbek; Marina Munkhoeva; Nikita Muravev; Aleksandr Petiushko; Ivan Oseledets
  In safety-critical machine learning applications, it is crucial to defend
models against adversarial attacks -- small modifications of the input that
change the predictions. Besides rigorously studied $\ell_p$-bounded additive
perturbations, recently proposed semantic perturbations (e.g. rotation,
translation) raise a serious concern on deploying ML systems in real-world.
Therefore, it is important to provide provable guarantees for deep learning
models against semantically meaningful input transformations. In this paper, we
propose a new universal probabilistic certification approach based on
Chernoff-Cramer bounds that can be used in general attack settings. We estimate
the probability of a model to fail if the attack is sampled from a certain
distribution. Our theoretical findings are supported by experimental results on
different datasets.


http://arxiv.org/abs/2109.11041
Security Analysis of Capsule Network Inference using Horizontal Collaboration. (69%)
Adewale Adeyemo; Faiq Khalid; Tolulope A. Odetola; Syed Rafay Hasan
  The traditional convolution neural networks (CNN) have several drawbacks like
the Picasso effect and the loss of information by the pooling layer. The
Capsule network (CapsNet) was proposed to address these challenges because its
architecture can encode and preserve the spatial orientation of input images.
Similar to traditional CNNs, CapsNet is also vulnerable to several malicious
attacks, as studied by several researchers in the literature. However, most of
these studies focus on single-device-based inference, but horizontally
collaborative inference in state-of-the-art systems, like intelligent edge
services in self-driving cars, voice controllable systems, and drones, nullify
most of these analyses. Horizontal collaboration implies partitioning the
trained CNN models or CNN tasks to multiple end devices or edge nodes.
Therefore, it is imperative to examine the robustness of the CapsNet against
malicious attacks when deployed in horizontally collaborative environments.
Towards this, we examine the robustness of the CapsNet when subjected to
noise-based inference attacks in a horizontal collaborative environment. In
this analysis, we perturbed the feature maps of the different layers of four
DNN models, i.e., CapsNet, Mini-VGG, LeNet, and an in-house designed CNN
(ConvNet) with the same number of parameters as CapsNet, using two types of
noised-based attacks, i.e., Gaussian Noise Attack and FGSM noise attack. The
experimental results show that similar to the traditional CNNs, depending upon
the access of the attacker to the DNN layer, the classification accuracy of the
CapsNet drops significantly. For example, when Gaussian Noise Attack
classification is performed at the DigitCap layer of the CapsNet, the maximum
classification accuracy drop is approximately 97%.


http://arxiv.org/abs/2109.11125
Adversarial Transfer Attacks With Unknown Data and Class Overlap. (62%)
Luke E. Richards; André Nguyen; Ryan Capps; Steven Forsythe; Cynthia Matuszek; Edward Raff
  The ability to transfer adversarial attacks from one model (the surrogate) to
another model (the victim) has been an issue of concern within the machine
learning (ML) community. The ability to successfully evade unseen models
represents an uncomfortable level of ease toward implementing attacks. In this
work we note that as studied, current transfer attack research has an
unrealistic advantage for the attacker: the attacker has the exact same
training data as the victim. We present the first study of transferring
adversarial attacks focusing on the data available to attacker and victim under
imperfect settings without querying the victim, where there is some variable
level of overlap in the exact data used or in the classes learned by each
model. This threat model is relevant to applications in medicine, malware, and
others. Under this new threat model attack success rate is not correlated with
data or class overlap in the way one would expect, and varies with dataset.
This makes it difficult for attacker and defender to reason about each other
and contributes to the broader study of model robustness and security. We
remedy this by developing a masked version of Projected Gradient Descent that
simulates class disparity, which enables the attacker to reliably estimate a
lower-bound on their attack's success.


http://arxiv.org/abs/2109.10859
Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation. (1%)
Diptesh Kanojia; Marina Fomicheva; Tharindu Ranasinghe; Frédéric Blain; Constantin Orăsan; Lucia Specia
  Current Machine Translation (MT) systems achieve very good results on a
growing variety of language pairs and datasets. However, they are known to
produce fluent translation outputs that can contain important meaning errors,
thus undermining their reliability in practice. Quality Estimation (QE) is the
task of automatically assessing the performance of MT systems at test time.
Thus, in order to be useful, QE systems should be able to detect such errors.
However, this ability is yet to be tested in the current evaluation practices,
where QE systems are assessed only in terms of their correlation with human
judgements. In this work, we bridge this gap by proposing a general methodology
for adversarial testing of QE for MT. First, we show that despite a high
correlation with human judgements achieved by the recent SOTA, certain types of
meaning errors are still problematic for QE to detect. Second, we show that on
average, the ability of a given model to discriminate between
meaning-preserving and meaning-altering perturbations is predictive of its
overall performance, thus potentially allowing for comparing QE systems without
relying on manual quality annotation.


http://arxiv.org/abs/2109.10512
Backdoor Attacks on Federated Learning with Lottery Ticket Hypothesis. (1%)
Zeyuan Yin; Ye Yuan; Panfeng Guo; Pan Zhou
  Edge devices in federated learning usually have much more limited computation
and communication resources compared to servers in a data center. Recently,
advanced model compression methods, like the Lottery Ticket Hypothesis, have
already been implemented on federated learning to reduce the model size and
communication cost. However, Backdoor Attack can compromise its implementation
in the federated learning scenario. The malicious edge device trains the client
model with poisoned private data and uploads parameters to the center,
embedding a backdoor to the global shared model after unwitting aggregative
optimization. During the inference phase, the model with backdoors classifies
samples with a certain trigger as one target category, while shows a slight
decrease in inference accuracy to clean samples. In this work, we empirically
demonstrate that Lottery Ticket models are equally vulnerable to backdoor
attacks as the original dense models, and backdoor attacks can influence the
structure of extracted tickets. Based on tickets' similarities between each
other, we provide a feasible defense for federated learning against backdoor
attacks on various datasets.


http://arxiv.org/abs/2109.10417
Attacks on Visualization-Based Malware Detection: Balancing Effectiveness and Executability. (99%)
Hadjer Benkraouda; Jingyu Qian; Hung Quoc Tran; Berkay Kaplan
  With the rapid development of machine learning for image classification,
researchers have found new applications of visualization techniques in malware
detection. By converting binary code into images, researchers have shown
satisfactory results in applying machine learning to extract features that are
difficult to discover manually. Such visualization-based malware detection
methods can capture malware patterns from many different malware families and
improve malware detection speed. On the other hand, recent research has also
shown adversarial attacks against such visualization-based malware detection.
Attackers can generate adversarial examples by perturbing the malware binary in
non-reachable regions, such as padding at the end of the binary. Alternatively,
attackers can perturb the malware image embedding and then verify the
executability of the malware post-transformation. One major limitation of the
first attack scenario is that a simple pre-processing step can remove the
perturbations before classification. For the second attack scenario, it is hard
to maintain the original malware's executability and functionality. In this
work, we provide literature review on existing malware visualization techniques
and attacks against them. We summarize the limitation of the previous work, and
design a new adversarial example attack against visualization-based malware
detection that can evade pre-processing filtering and maintain the original
malware functionality. We test our attack on a public malware dataset and
achieve a 98% success rate.


http://arxiv.org/abs/2109.10161
3D Point Cloud Completion with Geometric-Aware Adversarial Augmentation. (93%)
Mengxi Wu; Hao Huang; Yi Fang
  With the popularity of 3D sensors in self-driving and other robotics
applications, extensive research has focused on designing novel neural network
architectures for accurate 3D point cloud completion. However, unlike in point
cloud classification and reconstruction, the role of adversarial samples in3D
point cloud completion has seldom been explored. In this work, we show that
training with adversarial samples can improve the performance of neural
networks on 3D point cloud completion tasks. We propose a novel approach to
generate adversarial samples that benefit both the performance of clean and
adversarial samples. In contrast to the PGD-k attack, our method generates
adversarial samples that keep the geometric features in clean samples and
contain few outliers. In particular, we use principal directions to constrain
the adversarial perturbations for each input point. The gradient components in
the mean direction of principal directions are taken as adversarial
perturbations. In addition, we also investigate the effect of using the minimum
curvature direction. Besides, we adopt attack strength accumulation and
auxiliary Batch Normalization layers method to speed up the training process
and alleviate the distribution mismatch between clean and adversarial samples.
Experimental results show that training with the adversarial samples crafted by
our method effectively enhances the performance of PCN on the ShapeNet dataset.


http://arxiv.org/abs/2109.09955
DeSMP: Differential Privacy-exploited Stealthy Model Poisoning Attacks in Federated Learning. (76%)
Md Tamjid Hossain; Shafkat Islam; Shahriar Badsha; Haoting Shen
  Federated learning (FL) has become an emerging machine learning technique
lately due to its efficacy in safeguarding the client's confidential
information. Nevertheless, despite the inherent and additional
privacy-preserving mechanisms (e.g., differential privacy, secure multi-party
computation, etc.), the FL models are still vulnerable to various
privacy-violating and security-compromising attacks (e.g., data or model
poisoning) due to their numerous attack vectors which in turn, make the models
either ineffective or sub-optimal. Existing adversarial models focusing on
untargeted model poisoning attacks are not enough stealthy and persistent at
the same time because of their conflicting nature (large scale attacks are
easier to detect and vice versa) and thus, remain an unsolved research problem
in this adversarial learning paradigm. Considering this, in this paper, we
analyze this adversarial learning process in an FL setting and show that a
stealthy and persistent model poisoning attack can be conducted exploiting the
differential noise. More specifically, we develop an unprecedented DP-exploited
stealthy model poisoning (DeSMP) attack for FL models. Our empirical analysis
on both the classification and regression tasks using two popular datasets
reflects the effectiveness of the proposed DeSMP attack. Moreover, we develop a
novel reinforcement learning (RL)-based defense strategy against such model
poisoning attacks which can intelligently and dynamically select the privacy
level of the FL models to minimize the DeSMP attack surface and facilitate the
attack detection.


http://arxiv.org/abs/2109.09963
Privacy, Security, and Utility Analysis of Differentially Private CPES Data. (13%)
Md Tamjid Hossain; Shahriar Badsha; Haoting Shen
  Differential privacy (DP) has been widely used to protect the privacy of
confidential cyber physical energy systems (CPES) data. However, applying DP
without analyzing the utility, privacy, and security requirements can affect
the data utility as well as help the attacker to conduct integrity attacks
(e.g., False Data Injection(FDI)) leveraging the differentially private data.
Existing anomaly-detection-based defense strategies against data integrity
attacks in DP-based smart grids fail to minimize the attack impact while
maximizing data privacy and utility. To address this challenge, it is
nontrivial to apply a defensive approach during the design process. In this
paper, we formulate and develop the defense strategy as a part of the design
process to investigate data privacy, security, and utility in a DP-based smart
grid network. We have proposed a provable relationship among the DP-parameters
that enables the defender to design a fault-tolerant system against FDI
attacks. To experimentally evaluate and prove the effectiveness of our proposed
design approach, we have simulated the FDI attack in a DP-based grid. The
evaluation indicates that the attack impact can be minimized if the designer
calibrates the privacy level according to the proposed correlation of the
DP-parameters to design the grid network. Moreover, we analyze the feasibility
of the DP mechanism and QoS of the smart grid network in an adversarial
setting. Our analysis suggests that the DP mechanism is feasible over existing
privacy-preserving mechanisms in the smart grid domain. Also, the QoS of the
differentially private grid applications is found satisfactory in adversarial
presence.


http://arxiv.org/abs/2109.09320
Robust Physical-World Attacks on Face Recognition. (99%)
Xin Zheng; Yanbo Fan; Baoyuan Wu; Yong Zhang; Jue Wang; Shirui Pan
  Face recognition has been greatly facilitated by the development of deep
neural networks (DNNs) and has been widely applied to many safety-critical
applications. However, recent studies have shown that DNNs are very vulnerable
to adversarial examples, raising serious concerns on the security of real-world
face recognition. In this work, we study sticker-based physical attacks on face
recognition for better understanding its adversarial robustness. To this end,
we first analyze in-depth the complicated physical-world conditions confronted
by attacking face recognition, including the different variations of stickers,
faces, and environmental conditions. Then, we propose a novel robust physical
attack framework, dubbed PadvFace, to model these challenging variations
specifically. Furthermore, considering the difference in attack complexity, we
propose an efficient Curriculum Adversarial Attack (CAA) algorithm that
gradually adapts adversarial stickers to environmental variations from easy to
complex. Finally, we construct a standardized testing protocol to facilitate
the fair evaluation of physical attacks on face recognition, and extensive
experiments on both dodging and impersonation attacks demonstrate the superior
performance of the proposed method.


http://arxiv.org/abs/2109.09901
Modeling Adversarial Noise for Adversarial Defense. (99%)
Dawei Zhou; Nannan Wang; Bo Han; Tongliang Liu
  Deep neural networks have been demonstrated to be vulnerable to adversarial
noise, promoting the development of defense against adversarial attacks.
Motivated by the fact that adversarial noise contains well-generalizing
features and that the relationship between adversarial data and natural data
can help infer natural data and make reliable predictions, in this paper, we
study to model adversarial noise by learning the transition relationship
between adversarial labels (i.e. the flipped labels used to generate
adversarial data) and natural labels (i.e. the ground truth labels of the
natural data). Specifically, we introduce an instance-dependent transition
matrix to relate adversarial labels and natural labels, which can be seamlessly
embedded with the target model (enabling us to model stronger adaptive
adversarial noise). Empirical evaluations demonstrate that our method could
effectively improve adversarial accuracy.


http://arxiv.org/abs/2109.09654
Can We Leverage Predictive Uncertainty to Detect Dataset Shift and Adversarial Examples in Android Malware Detection? (99%)
Deqiang Li; Tian Qiu; Shuo Chen; Qianmu Li; Shouhuai Xu
  The deep learning approach to detecting malicious software (malware) is
promising but has yet to tackle the problem of dataset shift, namely that the
joint distribution of examples and their labels associated with the test set is
different from that of the training set. This problem causes the degradation of
deep learning models without users' notice. In order to alleviate the problem,
one approach is to let a classifier not only predict the label on a given
example but also present its uncertainty (or confidence) on the predicted
label, whereby a defender can decide whether to use the predicted label or not.
While intuitive and clearly important, the capabilities and limitations of this
approach have not been well understood. In this paper, we conduct an empirical
study to evaluate the quality of predictive uncertainties of malware detectors.
Specifically, we re-design and build 24 Android malware detectors (by
transforming four off-the-shelf detectors with six calibration methods) and
quantify their uncertainties with nine metrics, including three metrics dealing
with data imbalance. Our main findings are: (i) predictive uncertainty indeed
helps achieve reliable malware detection in the presence of dataset shift, but
cannot cope with adversarial evasion attacks; (ii) approximate Bayesian methods
are promising to calibrate and generalize malware detectors to deal with
dataset shift, but cannot cope with adversarial evasion attacks; (iii)
adversarial evasion attacks can render calibration methods useless, and it is
an open problem to quantify the uncertainty associated with the predicted
labels of adversarial examples (i.e., it is not effective to use predictive
uncertainty to detect adversarial examples).


http://arxiv.org/abs/2109.09869
Robustness Analysis of Deep Learning Frameworks on Mobile Platforms. (10%)
Amin Eslami Abyane; Hadi Hemmati
  With the recent increase in the computational power of modern mobile devices,
machine learning-based heavy tasks such as face detection and speech
recognition are now integral parts of such devices. This requires frameworks to
execute machine learning models (e.g., Deep Neural Networks) on mobile devices.
Although there exist studies on the accuracy and performance of these
frameworks, the quality of on-device deep learning frameworks, in terms of
their robustness, has not been systematically studied yet. In this paper, we
empirically compare two on-device deep learning frameworks with three
adversarial attacks on three different model architectures. We also use both
the quantized and unquantized variants for each architecture. The results show
that, in general, neither of the deep learning frameworks is better than the
other in terms of robustness, and there is not a significant difference between
the PC and mobile frameworks either. However, in cases like Boundary attack,
mobile version is more robust than PC. In addition, quantization improves
robustness in all cases when moving from PC to mobile.


http://arxiv.org/abs/2109.09598
"Hello, It's Me": Deep Learning-based Speech Synthesis Attacks in the Real World. (2%)
Emily Wenger; Max Bronckers; Christian Cianfarani; Jenna Cryan; Angela Sha; Haitao Zheng; Ben Y. Zhao
  Advances in deep learning have introduced a new wave of voice synthesis
tools, capable of producing audio that sounds as if spoken by a target speaker.
If successful, such tools in the wrong hands will enable a range of powerful
attacks against both humans and software systems (aka machines). This paper
documents efforts and findings from a comprehensive experimental study on the
impact of deep-learning based speech synthesis attacks on both human listeners
and machines such as speaker recognition and voice-signin systems. We find that
both humans and machines can be reliably fooled by synthetic speech and that
existing defenses against synthesized speech fall short. These findings
highlight the need to raise awareness and develop new protections against
synthetic speech for both humans and machines.


http://arxiv.org/abs/2109.09829
Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework. (1%)
Muhammad Shafique; Alberto Marchisio; Rachmad Vidya Wicaksana Putra; Muhammad Abdullah Hanif
  The security and privacy concerns along with the amount of data that is
required to be processed on regular basis has pushed processing to the edge of
the computing systems. Deploying advanced Neural Networks (NN), such as deep
neural networks (DNNs) and spiking neural networks (SNNs), that offer
state-of-the-art results on resource-constrained edge devices is challenging
due to the stringent memory and power/energy constraints. Moreover, these
systems are required to maintain correct functionality under diverse security
and reliability threats. This paper first discusses existing approaches to
address energy efficiency, reliability, and security issues at different system
layers, i.e., hardware (HW) and software (SW). Afterward, we discuss how to
further improve the performance (latency) and the energy efficiency of Edge AI
systems through HW/SW-level optimizations, such as pruning, quantization, and
approximation. To address reliability threats (like permanent and transient
faults), we highlight cost-effective mitigation techniques, like fault-aware
training and mapping. Moreover, we briefly discuss effective detection and
protection techniques to address security threats (like model and data
corruption). Towards the end, we discuss how these techniques can be combined
in an integrated cross-layer framework for realizing robust and
energy-efficient Edge AI systems.


http://arxiv.org/abs/2109.09060
On the Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbars. (99%)
Deboleena Roy; Chun Tao; Indranil Chakraborty; Kaushik Roy
  Applications based on Deep Neural Networks (DNNs) have grown exponentially in
the past decade. To match their increasing computational needs, several
Non-Volatile Memory (NVM) crossbar-based accelerators have been proposed. Apart
from improved energy efficiency and performance, these approximate hardware
also possess intrinsic robustness for defense against Adversarial Attacks,
which is an important security concern for DNNs. Prior works have focused on
quantifying this intrinsic robustness for vanilla networks, that is DNNs
trained on unperturbed inputs. However, adversarial training of DNNs is the
benchmark technique for robustness, and sole reliance on intrinsic robustness
of the hardware may not be sufficient. In this work, we explore the design of
robust DNNs through the amalgamation of adversarial training and the intrinsic
robustness offered by NVM crossbar-based analog hardware. First, we study the
noise stability of such networks on unperturbed inputs and observe that
internal activations of adversarially trained networks have lower
Signal-to-Noise Ratio (SNR), and are sensitive to noise than vanilla networks.
As a result, they suffer significantly higher performance degradation due to
the non-ideal computations; on an average 2x accuracy drop. On the other hand,
for adversarial images generated using Projected-Gradient-Descent (PGD)
White-Box attacks, ResNet-10/20 adversarially trained on CIFAR-10/100 display a
5-10% gain in robust accuracy due to the underlying NVM crossbar when the
attack epsilon ($\epsilon_{attack}$, the degree of input perturbations) is
greater than the epsilon of the adversarial training ($\epsilon_{train}$). Our
results indicate that implementing adversarially trained networks on analog
hardware requires careful calibration between hardware non-idealities and
$\epsilon_{train}$ to achieve optimum robustness and performance.


http://arxiv.org/abs/2109.09075
Adversarial Training with Contrastive Learning in NLP. (16%)
Daniela N. Rim; DongNyeong Heo; Heeyoul Choi
  For years, adversarial training has been extensively studied in natural
language processing (NLP) settings. The main goal is to make models robust so
that similar inputs derive in semantically similar outcomes, which is not a
trivial problem since there is no objective measure of semantic similarity in
language. Previous works use an external pre-trained NLP model to tackle this
challenge, introducing an extra training stage with huge memory consumption
during training. However, the recent popular approach of contrastive learning
in language processing hints a convenient way of obtaining such similarity
restrictions. The main advantage of the contrastive learning approach is that
it aims for similar data points to be mapped close to each other and further
from different ones in the representation space. In this work, we propose
adversarial training with contrastive learning (ATCL) to adversarially train a
language processing task using the benefits of contrastive learning. The core
idea is to make linear perturbations in the embedding space of the input via
fast gradient methods (FGM) and train the model to keep the original and
perturbed representations close via contrastive learning. In NLP experiments,
we applied ATCL to language modeling and neural machine translation tasks. The
results show not only an improvement in the quantitative (perplexity and BLEU)
scores when compared to the baselines, but ATCL also achieves good qualitative
results in the semantic level for both tasks without using a pre-trained model.


http://arxiv.org/abs/2109.08868
Clean-label Backdoor Attack against Deep Hashing based Retrieval. (98%)
Kuofeng Gao; Jiawang Bai; Bin Chen; Dongxian Wu; Shu-Tao Xia
  Deep hashing has become a popular method in large-scale image retrieval due
to its computational and storage efficiency. However, recent works raise the
security concerns of deep hashing. Although existing works focus on the
vulnerability of deep hashing in terms of adversarial perturbations, we
identify a more pressing threat, backdoor attack, when the attacker has access
to the training data. A backdoored deep hashing model behaves normally on
original query images, while returning the images with the target label when
the trigger presents, which makes the attack hard to be detected. In this
paper, we uncover this security concern by utilizing clean-label data
poisoning. To the best of our knowledge, this is the first attempt at the
backdoor attack against deep hashing models. To craft the poisoned images, we
first generate the targeted adversarial patch as the backdoor trigger.
Furthermore, we propose the confusing perturbations to disturb the hashing code
learning, such that the hashing model can learn more about the trigger. The
confusing perturbations are imperceptible and generated by dispersing the
images with the target label in the Hamming space. We have conducted extensive
experiments to verify the efficacy of our backdoor attack under various
settings. For instance, it can achieve 63% targeted mean average precision on
ImageNet under 48 bits code length with only 40 poisoned images.


http://arxiv.org/abs/2109.08465
Messing Up 3D Virtual Environments: Transferable Adversarial 3D Objects. (98%)
Enrico Meloni; Matteo Tiezzi; Luca Pasqualini; Marco Gori; Stefano Melacci
  In the last few years, the scientific community showed a remarkable and
increasing interest towards 3D Virtual Environments, training and testing
Machine Learning-based models in realistic virtual worlds. On one hand, these
environments could also become a mean to study the weaknesses of Machine
Learning algorithms, or to simulate training settings that allow Machine
Learning models to gain robustness to 3D adversarial attacks. On the other
hand, their growing popularity might also attract those that aim at creating
adversarial conditions to invalidate the benchmarking process, especially in
the case of public environments that allow the contribution from a large
community of people. Most of the existing Adversarial Machine Learning
approaches are focused on static images, and little work has been done in
studying how to deal with 3D environments and how a 3D object should be altered
to fool a classifier that observes it. In this paper, we study how to craft
adversarial 3D objects by altering their textures, using a tool chain composed
of easily accessible elements. We show that it is possible, and indeed simple,
to create adversarial objects using off-the-shelf limited surrogate renderers
that can compute gradients with respect to the parameters of the rendering
process, and, to a certain extent, to transfer the attacks to more advanced 3D
engines. We propose a saliency-based attack that intersects the two classes of
renderers in order to focus the alteration to those texture elements that are
estimated to be effective in the target engine, evaluating its impact in
popular neural classifiers.


http://arxiv.org/abs/2109.07986
Harnessing Perceptual Adversarial Patches for Crowd Counting. (99%)
Shunchang Liu; Jiakai Wang; Aishan Liu; Yingwei Li; Yijie Gao; Xianglong Liu; Dacheng Tao
  Crowd counting, which is significantly important for estimating the number of
people in safety-critical scenes, has been shown to be vulnerable to
adversarial examples in the physical world (e.g., adversarial patches). Though
harmful, adversarial examples are also valuable for assessing and better
understanding model robustness. However, existing adversarial example
generation methods in crowd counting scenarios lack strong transferability
among different black-box models. Motivated by the fact that transferability is
positively correlated to the model-invariant characteristics, this paper
proposes the Perceptual Adversarial Patch (PAP) generation framework to learn
the shared perceptual features between models by exploiting both the model
scale perception and position perception. Specifically, PAP exploits
differentiable interpolation and density attention to help learn the invariance
between models during training, leading to better transferability. In addition,
we surprisingly found that our adversarial patches could also be utilized to
benefit the performance of vanilla models for alleviating several challenges
including cross datasets and complex backgrounds. Extensive experiments under
both digital and physical world scenarios demonstrate the effectiveness of our
PAP.


http://arxiv.org/abs/2109.08191
KATANA: Simple Post-Training Robustness Using Test Time Augmentations. (98%)
Gilad Cohen; Raja Giryes
  Although Deep Neural Networks (DNNs) achieve excellent performance on many
real-world tasks, they are highly vulnerable to adversarial attacks. A leading
defense against such attacks is adversarial training, a technique in which a
DNN is trained to be robust to adversarial attacks by introducing adversarial
noise to its input. This procedure is effective but must be done during the
training phase. In this work, we propose a new simple and easy-to-use
technique, KATANA, for robustifying an existing pretrained DNN without
modifying its weights. For every image, we generate N randomized Test Time
Augmentations (TTAs) by applying diverse color, blur, noise, and geometric
transforms. Next, we utilize the DNN's logits output to train a simple random
forest classifier to predict the real class label. Our strategy achieves
state-of-the-art adversarial robustness on diverse attacks with minimal
compromise on the natural images' classification. We test KATANA also against
two adaptive white-box attacks and it shows excellent results when combined
with adversarial training. Code is available in
https://github.com/giladcohen/KATANA.


http://arxiv.org/abs/2109.08139
Adversarial Attacks against Deep Learning Based Power Control in Wireless Communications. (95%)
Brian Kim; Yi Shi; Yalin E. Sagduyu; Tugba Erpek; Sennur Ulukus
  We consider adversarial machine learning based attacks on power allocation
where the base station (BS) allocates its transmit power to multiple orthogonal
subcarriers by using a deep neural network (DNN) to serve multiple user
equipments (UEs). The DNN that corresponds to a regression model is trained
with channel gains as the input and allocated transmit powers as the output.
While the BS allocates the transmit power to the UEs to maximize rates for all
UEs, there is an adversary that aims to minimize these rates. The adversary may
be an external transmitter that aims to manipulate the inputs to the DNN by
interfering with the pilot signals that are transmitted to measure the channel
gain. Alternatively, the adversary may be a rogue UE that transmits fabricated
channel estimates to the BS. In both cases, the adversary carefully crafts
adversarial perturbations to manipulate the inputs to the DNN of the BS subject
to an upper bound on the strengths of these perturbations. We consider the
attacks targeted on a single UE or all UEs. We compare these attacks with a
benchmark, where the adversary scales down the input to the DNN. We show that
adversarial attacks are much more effective than the benchmark attack in terms
of reducing the rate of communications. We also show that adversarial attacks
are robust to the uncertainty at the adversary including the erroneous
knowledge of channel gains and the potential errors in exercising the attacks
exactly as specified.


http://arxiv.org/abs/2109.07723
Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patterns. (92%)
Prasanth Buddareddygari; Travis Zhang; Yezhou Yang; Yi Ren
  Recent studies demonstrated the vulnerability of control policies learned
through deep reinforcement learning against adversarial attacks, raising
concerns about the application of such models to risk-sensitive tasks such as
autonomous driving. Threat models for these demonstrations are limited to (1)
targeted attacks through real-time manipulation of the agent's observation, and
(2) untargeted attacks through manipulation of the physical environment. The
former assumes full access to the agent's states/observations at all times,
while the latter has no control over attack outcomes. This paper investigates
the feasibility of targeted attacks through visually learned patterns placed on
physical object in the environment, a threat model that combines the
practicality and effectiveness of the existing ones. Through analysis, we
demonstrate that a pre-trained policy can be hijacked within a time window,
e.g., performing an unintended self-parking, when an adversarial object is
present. To enable the attack, we adopt an assumption that the dynamics of both
the environment and the agent can be learned by the attacker. Lastly, we
empirically show the effectiveness of the proposed attack on different driving
scenarios, perform a location robustness test, and study the tradeoff between
the attack strength and its effectiveness.


http://arxiv.org/abs/2109.07926
Don't Search for a Search Method -- Simple Heuristics Suffice for Adversarial Text Attacks. (68%)
Nathaniel Berger; Stefan Riezler; Artem Sokolov; Sebastian Ebert
  Recently more attention has been given to adversarial attacks on neural
networks for natural language processing (NLP). A central research topic has
been the investigation of search algorithms and search constraints, accompanied
by benchmark algorithms and tasks. We implement an algorithm inspired by zeroth
order optimization-based attacks and compare with the benchmark results in the
TextAttack framework. Surprisingly, we find that optimization-based methods do
not yield any improvement in a constrained setup and slightly benefit from
approximate gradient information only in unconstrained setups where search
spaces are larger. In contrast, simple heuristics exploiting nearest neighbors
without querying the target function yield substantial success rates in
constrained setups, and nearly full success rate in unconstrained setups, at an
order of magnitude fewer queries. We conclude from these results that current
TextAttack benchmark tasks are too easy and constraints are too strict,
preventing meaningful research on black-box adversarial text attacks.


http://arxiv.org/abs/2109.08045
Membership Inference Attacks Against Recommender Systems. (3%)
Minxing Zhang; Zhaochun Ren; Zihan Wang; Pengjie Ren; Zhumin Chen; Pengfei Hu; Yang Zhang
  Recently, recommender systems have achieved promising performances and become
one of the most widely used web applications. However, recommender systems are
often trained on highly sensitive user data, thus potential data leakage from
recommender systems may lead to severe privacy problems.
  In this paper, we make the first attempt on quantifying the privacy leakage
of recommender systems through the lens of membership inference. In contrast
with traditional membership inference against machine learning classifiers, our
attack faces two main differences. First, our attack is on the user-level but
not on the data sample-level. Second, the adversary can only observe the
ordered recommended items from a recommender system instead of prediction
results in the form of posterior probabilities. To address the above
challenges, we propose a novel method by representing users from relevant
items. Moreover, a shadow recommender is established to derive the labeled
training data for training the attack model. Extensive experimental results
show that our attack framework achieves a strong performance. In addition, we
design a defense mechanism to effectively mitigate the membership inference
threat of recommender systems.


http://arxiv.org/abs/2109.07142
Universal Adversarial Attack on Deep Learning Based Prognostics. (99%)
Arghya Basak; Pradeep Rathore; Sri Harsha Nistala; Sagar Srinivas; Venkataramana Runkana
  Deep learning-based time series models are being extensively utilized in
engineering and manufacturing industries for process control and optimization,
asset monitoring, diagnostic and predictive maintenance. These models have
shown great improvement in the prediction of the remaining useful life (RUL) of
industrial equipment but suffer from inherent vulnerability to adversarial
attacks. These attacks can be easily exploited and can lead to catastrophic
failure of critical industrial equipment. In general, different adversarial
perturbations are computed for each instance of the input data. This is,
however, difficult for the attacker to achieve in real time due to higher
computational requirement and lack of uninterrupted access to the input data.
Hence, we present the concept of universal adversarial perturbation, a special
imperceptible noise to fool regression based RUL prediction models. Attackers
can easily utilize universal adversarial perturbations for real-time attack
since continuous access to input data and repetitive computation of adversarial
perturbations are not a prerequisite for the same. We evaluate the effect of
universal adversarial attacks using NASA turbofan engine dataset. We show that
addition of universal adversarial perturbation to any instance of the input
data increases error in the output predicted by the model. To the best of our
knowledge, we are the first to study the effect of the universal adversarial
perturbation on time series regression models. We further demonstrate the
effect of varying the strength of perturbations on RUL prediction models and
found that model accuracy decreases with the increase in perturbation strength
of the universal adversarial attack. We also showcase that universal
adversarial perturbation can be transferred across different models.


http://arxiv.org/abs/2109.07171
Balancing detectability and performance of attacks on the control channel of Markov Decision Processes. (98%)
Alessio Russo; Alexandre Proutiere
  We investigate the problem of designing optimal stealthy poisoning attacks on
the control channel of Markov decision processes (MDPs). This research is
motivated by the recent interest of the research community for adversarial and
poisoning attacks applied to MDPs, and reinforcement learning (RL) methods. The
policies resulting from these methods have been shown to be vulnerable to
attacks perturbing the observations of the decision-maker. In such an attack,
drawing inspiration from adversarial examples used in supervised learning, the
amplitude of the adversarial perturbation is limited according to some norm,
with the hope that this constraint will make the attack imperceptible. However,
such constraints do not grant any level of undetectability and do not take into
account the dynamic nature of the underlying Markov process. In this paper, we
propose a new attack formulation, based on information-theoretical quantities,
that considers the objective of minimizing the detectability of the attack as
well as the performance of the controlled process. We analyze the trade-off
between the efficiency of the attack and its detectability. We conclude with
examples and numerical simulations illustrating this trade-off.


http://arxiv.org/abs/2109.07193
FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack. (93%)
DonghuaWang; Tingsong Jiang; Jialiang Sun; Weien Zhou; Xiaoya Zhang; Zhiqiang Gong; Wen Yao; Xiaoqian Chen
  Physical adversarial attacks in object detection have attracted increasing
attention. However, most previous works focus on hiding the objects from the
detector by generating an individual adversarial patch, which only covers the
planar part of the vehicle's surface and fails to attack the detector in
physical scenarios for multi-view, long-distance and partially occluded
objects. To bridge the gap between digital attacks and physical attacks, we
exploit the full 3D vehicle surface to propose a robust Full-coverage
Camouflage Attack (FCA) to fool detectors. Specifically, we first try rendering
the non-planar camouflage texture over the full vehicle surface. To mimic the
real-world environment conditions, we then introduce a transformation function
to transfer the rendered camouflaged vehicle into a photo-realistic scenario.
Finally, we design an efficient loss function to optimize the camouflage
texture. Experiments show that the full-coverage camouflage attack can not only
outperform state-of-the-art methods under various test cases but also
generalize to different environments, vehicles, and object detectors.


http://arxiv.org/abs/2109.07403
BERT is Robust! A Case Against Synonym-Based Adversarial Examples in Text Classification. (92%)
Jens Hauser; Zhao Meng; Damián Pascual; Roger Wattenhofer
  Deep Neural Networks have taken Natural Language Processing by storm. While
this led to incredible improvements across many tasks, it also initiated a new
research field, questioning the robustness of these neural networks by
attacking them. In this paper, we investigate four word substitution-based
attacks on BERT. We combine a human evaluation of individual word substitutions
and a probabilistic analysis to show that between 96% and 99% of the analyzed
attacks do not preserve semantics, indicating that their success is mainly
based on feeding poor data to the model. To further confirm that, we introduce
an efficient data augmentation procedure and show that many adversarial
examples can be prevented by including data similar to the attacks during
training. An additional post-processing step reduces the success rates of
state-of-the-art attacks below 5%. Finally, by looking at more reasonable
thresholds on constraints for word substitutions, we conclude that BERT is a
lot more robust than research on attacks suggests.


http://arxiv.org/abs/2109.07177
Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup. (13%)
Guang Liu; Yuzhao Mao; Hailong Huang; Weiguo Gao; Xuan Li
  Mixup is a recent regularizer for current deep classification networks.
Through training a neural network on convex combinations of pairs of examples
and their labels, it imposes locally linear constraints on the model's input
space. However, such strict linear constraints often lead to under-fitting
which degrades the effects of regularization. Noticeably, this issue is getting
more serious when the resource is extremely limited. To address these issues,
we propose the Adversarial Mixing Policy (AMP), organized in a min-max-rand
formulation, to relax the Locally Linear Constraints in Mixup. Specifically,
AMP adds a small adversarial perturbation to the mixing coefficients rather
than the examples. Thus, slight non-linearity is injected in-between the
synthetic examples and synthetic labels. By training on these data, the deep
networks are further regularized, and thus achieve a lower predictive error
rate. Experiments on five text classification benchmarks and five backbone
models have empirically shown that our methods reduce the error rate over Mixup
variants in a significant margin (up to 31.3%), especially in low-resource
conditions (up to 17.5%).


http://arxiv.org/abs/2109.07395
Can one hear the shape of a neural network?: Snooping the GPU via Magnetic Side Channel. (10%)
Henrique Teles Maia; Chang Xiao; Dingzeyu Li; Eitan Grinspun; Changxi Zheng
  Neural network applications have become popular in both enterprise and
personal settings. Network solutions are tuned meticulously for each task, and
designs that can robustly resolve queries end up in high demand. As the
commercial value of accurate and performant machine learning models increases,
so too does the demand to protect neural architectures as confidential
investments. We explore the vulnerability of neural networks deployed as black
boxes across accelerated hardware through electromagnetic side channels. We
examine the magnetic flux emanating from a graphics processing unit's power
cable, as acquired by a cheap $3 induction sensor, and find that this signal
betrays the detailed topology and hyperparameters of a black-box neural network
model. The attack acquires the magnetic signal for one query with unknown input
values, but known input dimensions. The network reconstruction is possible due
to the modular layer sequence in which deep neural networks are evaluated. We
find that each layer component's evaluation produces an identifiable magnetic
signal signature, from which layer topology, width, function type, and sequence
order can be inferred using a suitably trained classifier and a joint
consistency optimization based on integer programming. We study the extent to
which network specifications can be recovered, and consider metrics for
comparing network similarity. We demonstrate the potential accuracy of this
side channel attack in recovering the details for a broad range of network
architectures, including random designs. We consider applications that may
exploit this novel side channel exposure, such as adversarial transfer attacks.
In response, we discuss countermeasures to protect against our method and other
similar snooping techniques.


http://arxiv.org/abs/2109.06634
A Novel Data Encryption Method Inspired by Adversarial Attacks. (99%)
Praveen Fernando; Jin Wei-Kocsis
  Due to the advances of sensing and storage technologies, a tremendous amount
of data becomes available and, it supports the phenomenal growth of artificial
intelligence (AI) techniques especially, deep learning (DL), in various
application domains. While the data sources become valuable assets for enabling
the success of autonomous decision-making, they also lead to critical
vulnerabilities in privacy and security. For example, data leakage can be
exploited via querying and eavesdropping in the exploratory phase for black-box
attacks against DL-based autonomous decision-making systems. To address this
issue, in this work, we propose a novel data encryption method, called
AdvEncryption, by exploiting the principle of adversarial attacks. Different
from existing encryption technologies, the AdvEncryption method is not
developed to prevent attackers from exploiting the dataset. Instead, our
proposed method aims to trap the attackers in a misleading feature distillation
of the data. To achieve this goal, our AdvEncryption method consists of two
essential components: 1) an adversarial attack-inspired encryption mechanism to
encrypt the data with stealthy adversarial perturbation, and 2) a decryption
mechanism that minimizes the impact of the perturbations on the effectiveness
of autonomous decision making. In the performance evaluation section, we
evaluate the performance of our proposed AdvEncryption method through case
studies considering different scenarios.


http://arxiv.org/abs/2109.06536
Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder. (99%)
Yao Qiu; Jinchao Zhang; Jie Zhou
  Recent work has proposed several efficient approaches for generating
gradient-based adversarial perturbations on embeddings and proved that the
model's performance and robustness can be improved when they are trained with
these contaminated embeddings. While they paid little attention to how to help
the model to learn these adversarial samples more efficiently. In this work, we
focus on enhancing the model's ability to defend gradient-based adversarial
attack during the model's training process and propose two novel adversarial
training approaches: (1) CARL narrows the original sample and its adversarial
sample in the representation space while enlarging their distance from
different labeled samples. (2) RAR forces the model to reconstruct the original
sample from its adversarial representation. Experiments show that the proposed
two approaches outperform strong baselines on various text classification
datasets. Analysis experiments find that when using our approaches, the
semantic representation of the input sentence won't be significantly affected
by adversarial perturbations, and the model's performance drops less under
adversarial attack. That is to say, our approaches can effectively improve the
robustness of the model. Besides, RAR can also be used to generate text-form
adversarial samples.


http://arxiv.org/abs/2109.06777
PETGEN: Personalized Text Generation Attack on Deep Sequence Embedding-based Classification Models. (99%)
Bing He; Mustaque Ahamad; Srijan Kumar
  \textit{What should a malicious user write next to fool a detection model?}
Identifying malicious users is critical to ensure the safety and integrity of
internet platforms. Several deep learning based detection models have been
created. However, malicious users can evade deep detection models by
manipulating their behavior, rendering these models of little use. The
vulnerability of such deep detection models against adversarial attacks is
unknown. Here we create a novel adversarial attack model against deep user
sequence embedding-based classification models, which use the sequence of user
posts to generate user embeddings and detect malicious users. In the attack,
the adversary generates a new post to fool the classifier. We propose a novel
end-to-end Personalized Text Generation Attack model, called \texttt{PETGEN},
that simultaneously reduces the efficacy of the detection model and generates
posts that have several key desirable properties. Specifically, \texttt{PETGEN}
generates posts that are personalized to the user's writing style, have
knowledge about a given target context, are aware of the user's historical
posts on the target context, and encapsulate the user's recent topical
interests. We conduct extensive experiments on two real-world datasets (Yelp
and Wikipedia, both with ground-truth of malicious users) to show that
\texttt{PETGEN} significantly reduces the performance of popular deep user
sequence embedding-based classification models. \texttt{PETGEN} outperforms
five attack baselines in terms of text quality and attack efficacy in both
white-box and black-box classifier settings. Overall, this work paves the path
towards the next generation of adversary-aware sequence classification models.


http://arxiv.org/abs/2109.08026
EVAGAN: Evasion Generative Adversarial Network for Low Data Regimes. (76%)
Rizwan Hamid Randhawa; Nauman Aslam; Muhammad Alauthman; Husnain Rafiq; Muhammad Khalid
  Many recent literary works have leveraged generative adversarial networks
(GANs) to spawn unseen evasion samples. The purpose is to annex the generated
data with the original train set for adversarial training to improve the
detection performance of machine learning (ML) classifiers. The quality
generation of adversarial samples relies on the adequacy of training data
samples. Sadly in low data regimes like medical anomaly detection, drug
discovery and cybersecurity, the attack samples are scarce in number. This
paper proposes a novel GAN design called Evasion Generative Adversarial Network
(EVAGAN), more suitable for low data regime problems that use oversampling for
detection improvement of ML classifiers. EVAGAN not only can generate evasion
samples, but its discriminator can act as an evasion aware classifier. We have
considered Auxiliary Classifier GAN (ACGAN) as a benchmark to evaluate the
performance of EVAGAN on cybersecurity (ISCX-2014, CIC-2017 and CIC2018) botnet
and CV (MNIST) datasets. We demonstrate that EVAGAN outperforms ACGAN for
imbalance datasets regarding detection performance, training stability, time
complexity. EVAGAN's generator quickly learns to generate the low sample class
and hardens its discriminator simultaneously. In contrast to ML classifiers
that require security hardening after being adversarially trained by GAN
generated data, EVAGAN renders it needless. The experimental analysis proves
EVAGAN to be an efficient evasion hardened model for low data regimes in
cybersecurity and CV. Code will be available at
https://github.com/rhr407/EVAGAN.


http://arxiv.org/abs/2109.06467
Dodging Attack Using Carefully Crafted Natural Makeup. (47%)
Nitzan Guetta; Asaf Shabtai; Inderjeet Singh; Satoru Momiyama; Yuval Elovici
  Deep learning face recognition models are used by state-of-the-art
surveillance systems to identify individuals passing through public areas
(e.g., airports). Previous studies have demonstrated the use of adversarial
machine learning (AML) attacks to successfully evade identification by such
systems, both in the digital and physical domains. Attacks in the physical
domain, however, require significant manipulation to the human participant's
face, which can raise suspicion by human observers (e.g. airport security
officers). In this study, we present a novel black-box AML attack which
carefully crafts natural makeup, which, when applied on a human participant,
prevents the participant from being identified by facial recognition models. We
evaluated our proposed attack against the ArcFace face recognition model, with
20 participants in a real-world setup that includes two cameras, different
shooting angles, and different lighting conditions. The evaluation results show
that in the digital domain, the face recognition system was unable to identify
all of the participants, while in the physical domain, the face recognition
system was able to identify the participants in only 1.22% of the frames
(compared to 47.57% without makeup and 33.73% with random natural makeup),
which is below a reasonable threshold of a realistic operational environment.


http://arxiv.org/abs/2109.07028
Avengers Ensemble! Improving Transferability of Authorship Obfuscation. (12%)
Muhammad Haroon; Muhammad Fareed Zaffar; Padmini Srinivasan; Zubair Shafiq
  Stylometric approaches have been shown to be quite effective for real-world
authorship attribution. To mitigate the privacy threat posed by authorship
attribution, researchers have proposed automated authorship obfuscation
approaches that aim to conceal the stylometric artefacts that give away the
identity of an anonymous document's author. Recent work has focused on
authorship obfuscation approaches that rely on black-box access to an
attribution classifier to evade attribution while preserving semantics.
However, to be useful under a realistic threat model, it is important that
these obfuscation approaches work well even when the adversary's attribution
classifier is different from the one used internally by the obfuscator.
Unfortunately, existing authorship obfuscation approaches do not transfer well
to unseen attribution classifiers. In this paper, we propose an ensemble-based
approach for transferable authorship obfuscation. Our experiments show that if
an obfuscator can evade an ensemble attribution classifier, which is based on
multiple base attribution classifiers, it is more likely to transfer to
different attribution classifiers. Our analysis shows that ensemble-based
authorship obfuscation achieves better transferability because it combines the
knowledge from each of the base attribution classifiers by essentially
averaging their decision boundaries.


http://arxiv.org/abs/2109.07048
ARCH: Efficient Adversarial Regularized Training with Caching. (2%)
Simiao Zuo; Chen Liang; Haoming Jiang; Pengcheng He; Xiaodong Liu; Jianfeng Gao; Weizhu Chen; Tuo Zhao
  Adversarial regularization can improve model generalization in many natural
language processing tasks. However, conventional approaches are computationally
expensive since they need to generate a perturbation for each sample in each
epoch. We propose a new adversarial regularization method ARCH (adversarial
regularization with caching), where perturbations are generated and cached once
every several epochs. As caching all the perturbations imposes memory usage
concerns, we adopt a K-nearest neighbors-based strategy to tackle this issue.
The strategy only requires caching a small amount of perturbations, without
introducing additional training time. We evaluate our proposed method on a set
of neural machine translation and natural language understanding tasks. We
observe that ARCH significantly eases the computational burden (saves up to
70\% of computational time in comparison with conventional approaches). More
surprisingly, by reducing the variance of stochastic gradients, ARCH produces a
notably better (in most of the tasks) or comparable model generalization. Our
code is publicly available.


http://arxiv.org/abs/2109.05830
Adversarial Bone Length Attack on Action Recognition. (99%)
Nariki Tanaka; Hiroshi Kera; Kazuhiko Kawamoto
  Skeleton-based action recognition models have recently been shown to be
vulnerable to adversarial attacks. Compared to adversarial attacks on images,
perturbations to skeletons are typically bounded to a lower dimension of
approximately 100 per frame. This lower-dimensional setting makes it more
difficult to generate imperceptible perturbations. Existing attacks resolve
this by exploiting the temporal structure of the skeleton motion so that the
perturbation dimension increases to thousands. In this paper, we show that
adversarial attacks can be performed on skeleton-based action recognition
models, even in a significantly low-dimensional setting without any temporal
manipulation. Specifically, we restrict the perturbations to the lengths of the
skeleton's bones, which allows an adversary to manipulate only approximately 30
effective dimensions. We conducted experiments on the NTU RGB+D and HDM05
datasets and demonstrate that the proposed attack successfully deceived models
with sometimes greater than 90\% success rate by small perturbations.
Furthermore, we discovered an interesting phenomenon: in our low-dimensional
setting, the adversarial training with the bone length attack shares a similar
property with data augmentation, and it not only improves the adversarial
robustness but also improves the classification accuracy on the original
original data. This is an interesting counterexample of the trade-off between
adversarial robustness and clean accuracy, which has been widely observed in
studies on adversarial training in the high-dimensional regime.


http://arxiv.org/abs/2109.05698
Randomized Substitution and Vote for Textual Adversarial Example Detection. (99%)
Xiaosen Wang; Yifeng Xiong; Kun He
  A line of work has shown that natural text processing models are vulnerable
to adversarial examples. Correspondingly, various defense methods are proposed
to mitigate the threat of textual adversarial examples, e.g. adversarial
training, certified defense, input pre-processing, detection, etc. In this
work, we treat the optimization process for synonym substitution based textual
adversarial attacks as a specific sequence of word replacement, in which each
word mutually influences other words. We identify that we could destroy such
mutual interaction and eliminate the adversarial perturbation by randomly
substituting a word with its synonyms. Based on this observation, we propose a
novel textual adversarial example detection method, termed Randomized
Substitution and Vote (RS&V), which votes the prediction label by accumulating
the logits of k samples generated by randomly substituting the words in the
input text with synonyms. The proposed RS&V is generally applicable to any
existing neural networks without modification on the architecture or extra
training, and it is orthogonal to prior work on making the classification
network itself more robust. Empirical evaluations on three benchmark datasets
demonstrate that RS&V could detect the textual adversarial examples more
successfully than the existing detection methods while maintaining the high
classification accuracy on benign samples.


http://arxiv.org/abs/2109.05820
Improving Robustness of Adversarial Attacks Using an Affine-Invariant Gradient Estimator. (99%)
Wenzhao Xiang; Hang Su; Chang Liu; Yandong Guo; Shibao Zheng
  Adversarial examples can deceive a deep neural network (DNN) by significantly
altering its response with imperceptible perturbations, which poses new
potential vulnerabilities as the growing ubiquity of DNNs. However, most of the
existing adversarial examples cannot maintain the malicious functionality if we
apply an affine transformation on the resultant examples, which is an important
measurement to the robustness of adversarial attacks for the practical risks.
To address this issue, we propose an affine-invariant adversarial attack which
can consistently construct adversarial examples robust over a distribution of
affine transformation. To further improve the efficiency, we propose to
disentangle the affine transformation into rotations, translations,
magnifications, and reformulate the transformation in polar space. Afterwards,
we construct an affine-invariant gradient estimator by convolving the gradient
at the original image with derived kernels, which can be integrated with any
gradient-based attack methods. Extensive experiments on the ImageNet
demonstrate that our method can consistently produce more robust adversarial
examples under significant affine transformations, and as a byproduct, improve
the transferability of adversarial examples compared with the alternative
state-of-the-art methods.


http://arxiv.org/abs/2109.05919
Evolving Architectures with Gradient Misalignment toward Low Adversarial Transferability. (98%)
Kevin Richard G. Operiano; Wanchalerm Pora; Hitoshi Iba; Hiroshi Kera
  Deep neural network image classifiers are known to be susceptible not only to
adversarial examples created for them but even those created for others. This
phenomenon poses a potential security risk in various black-box systems relying
on image classifiers. The reason behind such transferability of adversarial
examples is not yet fully understood and many studies have proposed training
methods to obtain classifiers with low transferability. In this study, we
address this problem from a novel perspective through investigating the
contribution of the network architecture to transferability. Specifically, we
propose an architecture searching framework that employs neuroevolution to
evolve network architectures and the gradient misalignment loss to encourage
networks to converge into dissimilar functions after training. Our experiments
show that the proposed framework successfully discovers architectures that
reduce transferability from four standard networks including ResNet and VGG,
while maintaining a good accuracy on unperturbed images. In addition, the
evolved networks trained with gradient misalignment exhibit significantly lower
transferability compared to standard networks trained with gradient
misalignment, which indicates that the network architecture plays an important
role in reducing transferability. This study demonstrates that designing or
exploring proper network architectures is a promising approach to tackle the
transferability issue and train adversarially robust image classifiers.


http://arxiv.org/abs/2109.06358
A Practical Adversarial Attack on Contingency Detection of Smart Energy Systems. (98%)
Moein Sabounchi; Jin Wei-Kocsis
  Due to the advances in computing and sensing, deep learning (DL) has widely
been applied in smart energy systems (SESs). These DL-based solutions have
proved their potentials in improving the effectiveness and adaptiveness of the
control systems. However, in recent years, increasing evidence shows that DL
techniques can be manipulated by adversarial attacks with carefully-crafted
perturbations. Adversarial attacks have been studied in computer vision and
natural language processing. However, there is very limited work focusing on
the adversarial attack deployment and mitigation in energy systems. In this
regard, to better prepare the SESs against potential adversarial attacks, we
propose an innovative adversarial attack model that can practically compromise
dynamical controls of energy system. We also optimize the deployment of the
proposed adversarial attack model by employing deep reinforcement learning (RL)
techniques. In this paper, we present our first-stage work in this direction.
In simulation section, we evaluate the performance of our proposed adversarial
attack model using standard IEEE 9-bus system.


http://arxiv.org/abs/2109.05925
Adversarial Examples for Evaluating Math Word Problem Solvers. (96%)
Vivek Kumar; Rishabh Maheshwary; Vikram Pudi
  Standard accuracy metrics have shown that Math Word Problem (MWP) solvers
have achieved high performance on benchmark datasets. However, the extent to
which existing MWP solvers truly understand language and its relation with
numbers is still unclear. In this paper, we generate adversarial attacks to
evaluate the robustness of state-of-the-art MWP solvers. We propose two methods
Question Reordering and Sentence Paraphrasing to generate adversarial attacks.
We conduct experiments across three neural MWP solvers over two benchmark
datasets. On average, our attack method is able to reduce the accuracy of MWP
solvers by over 40 percentage points on these datasets. Our results demonstrate
that existing MWP solvers are sensitive to linguistic variations in the problem
text. We verify the validity and quality of generated adversarial examples
through human evaluation.


http://arxiv.org/abs/2109.05695
PAT: Pseudo-Adversarial Training For Detecting Adversarial Videos. (86%)
Nupur Thakur; Baoxin Li
  Extensive research has demonstrated that deep neural networks (DNNs) are
prone to adversarial attacks. Although various defense mechanisms have been
proposed for image classification networks, fewer approaches exist for
video-based models that are used in security-sensitive applications like
surveillance. In this paper, we propose a novel yet simple algorithm called
Pseudo-Adversarial Training (PAT), to detect the adversarial frames in a video
without requiring knowledge of the attack. Our approach generates `transition
frames' that capture critical deviation from the original frames and eliminate
the components insignificant to the detection task. To avoid the necessity of
knowing the attack model, we produce `pseudo perturbations' to train our
detection network. Adversarial detection is then achieved through the use of
the detected frames. Experimental results on UCF-101 and 20BN-Jester datasets
show that PAT can detect the adversarial video frames and videos with a high
detection rate. We also unveil the potential reasons for the effectiveness of
the transition frames and pseudo perturbations through extensive experiments.


http://arxiv.org/abs/2109.05872
SignGuard: Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering. (81%)
Jian Xu; Shao-Lun Huang; Linqi Song; Tian Lan
  Gradient-based training in federated learning is known to be vulnerable to
faulty/malicious worker nodes, which are often modeled as Byzantine clients.
Previous work either makes use of auxiliary data at parameter server to verify
the received gradients or leverages statistic-based methods to identify and
remove malicious gradients from Byzantine clients. In this paper, we
acknowledge that auxiliary data may not always be available in practice and
focus on the statistic-based approach. However, recent work on model poisoning
attacks have shown that well-crafted attacks can circumvent most of existing
median- and distance-based statistical defense methods, making malicious
gradients indistinguishable from honest ones. To tackle this challenge, we show
that the element-wise sign of gradient vector can provide valuable insight in
detecting model poisoning attacks. Based on our theoretical analysis of
state-of-the-art attack, we propose a novel approach, \textit{SignGuard}, to
enable Byzantine-robust federated learning through collaborative malicious
gradient filtering. More precisely, the received gradients are first processed
to generate relevant magnitude, sign, and similarity statistics, which are then
collaboratively utilized by multiple, parallel filters to eliminate malicious
gradients before final aggregation. We further provide theoretical analysis of
SignGuard by quantifying its convergence with appropriate choice of learning
rate and under non-IID training data. Finally, extensive experiments of image
and text classification tasks - including MNIST, Fashion-MNIST, CIFAR-10, and
AG-News - are conducted together with recently proposed attacks and defense
strategies. The numerical results demonstrate the effectiveness and superiority
of our proposed approach.


http://arxiv.org/abs/2109.05793
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models. (50%)
Kun Zhou; Wayne Xin Zhao; Sirui Wang; Fuzheng Zhang; Wei Wu; Ji-Rong Wen
  Recent works have shown that powerful pre-trained language models (PLM) can
be fooled by small perturbations or intentional attacks. To solve this issue,
various data augmentation techniques are proposed to improve the robustness of
PLMs. However, it is still challenging to augment semantically relevant
examples with sufficient diversity. In this work, we present Virtual Data
Augmentation (VDA), a general framework for robustly fine-tuning PLMs. Based on
the original token embeddings, we construct a multinomial mixture for
augmenting virtual data embeddings, where a masked language model guarantees
the semantic relevance and the Gaussian noise provides the augmentation
diversity. Furthermore, a regularized training strategy is proposed to balance
the two aspects. Extensive experiments on six datasets show that our approach
is able to improve the robustness of PLMs and alleviate the performance
degradation under adversarial attacks. Our codes and data are publicly
available at \textcolor{blue}{\url{https://github.com/RUCAIBox/VDA}}.


http://arxiv.org/abs/2109.06024
Formalizing and Estimating Distribution Inference Risks. (50%)
Anshuman Suri; David Evans
  Property inference attacks reveal statistical properties about a training set
but are difficult to distinguish from the intrinsic purpose of statistical
machine learning, namely to produce models that capture statistical properties
about a distribution. Motivated by Yeom et al.'s membership inference
framework, we propose a formal and general definition of property inference
attacks. The proposed notion describes attacks that can distinguish between
possible training distributions, extending beyond previous property inference
attacks that infer the ratio of a particular type of data in the training data
set such as the proportion of females. We show how our definition captures
previous property inference attacks as well as a new attack that can reveal the
average node degree or clustering coefficient of a training graph. Our
definition also enables a theorem that connects the maximum possible accuracy
of inference attacks distinguishing between distributions to the effective size
of dataset leaked by the model. To quantify and understand property inference
risks, we conduct a series of experiments across a range of different
distributions using both black-box and white-box attacks. Our results show that
inexpensive attacks are often as effective as expensive meta-classifier
attacks, and that there are surprising asymmetries in the effectiveness of
attacks. We also extend the state-of-the-art property inference attack to work
on convolutional neural networks, and propose techniques to help identify
parameters in a model that leak the most information, thus significantly
lowering resource requirements for meta-classifier attacks.


http://arxiv.org/abs/2109.06363
Sensor Adversarial Traits: Analyzing Robustness of 3D Object Detection Sensor Fusion Models. (16%)
Won Park; Nan Li; Qi Alfred Chen; Z. Morley Mao
  A critical aspect of autonomous vehicles (AVs) is the object detection stage,
which is increasingly being performed with sensor fusion models: multimodal 3D
object detection models which utilize both 2D RGB image data and 3D data from a
LIDAR sensor as inputs. In this work, we perform the first study to analyze the
robustness of a high-performance, open source sensor fusion model architecture
towards adversarial attacks and challenge the popular belief that the use of
additional sensors automatically mitigate the risk of adversarial attacks. We
find that despite the use of a LIDAR sensor, the model is vulnerable to our
purposefully crafted image-based adversarial attacks including disappearance,
universal patch, and spoofing. After identifying the underlying reason, we
explore some potential defenses and provide some recommendations for improved
sensor fusion models.


http://arxiv.org/abs/2109.05751
Adversarially Trained Object Detector for Unsupervised Domain Adaptation. (2%)
Kazuma Fujii; Hiroshi Kera; Kazuhiko Kawamoto
  Unsupervised domain adaptation, which involves transferring knowledge from a
label-rich source domain to an unlabeled target domain, can be used to
substantially reduce annotation costs in the field of object detection. In this
study, we demonstrate that adversarial training in the source domain can be
employed as a new approach for unsupervised domain adaptation. Specifically, we
establish that adversarially trained detectors achieve improved detection
performance in target domains that are significantly shifted from source
domains. This phenomenon is attributed to the fact that adversarially trained
detectors can be used to extract robust features that are in alignment with
human perception and worth transferring across domains while discarding
domain-specific non-robust features. In addition, we propose a method that
combines adversarial training and feature alignment to ensure the improved
alignment of robust features with the target domain. We conduct experiments on
four benchmark datasets and confirm the effectiveness of our proposed approach
on large domain shifts from real to artistic images. Compared to the baseline
models, the adversarially trained detectors improve the mean average precision
by up to 7.7\%, and further by up to 11.8\% when feature alignments are
incorporated.


http://arxiv.org/abs/2109.05771
Perturbation CheckLists for Evaluating NLG Evaluation Metrics. (1%)
Ananya B. Sai; Tanay Dixit; Dev Yashpal Sheth; Sreyas Mohan; Mitesh M. Khapra
  Natural Language Generation (NLG) evaluation is a multifaceted task requiring
assessment of multiple desirable criteria, e.g., fluency, coherency, coverage,
relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG
tasks, we observe that the human evaluation scores on these multiple criteria
are often not correlated. For example, there is a very low correlation between
human scores on fluency and data coverage for the task of structured data to
text generation. This suggests that the current recipe of proposing new
automatic evaluation metrics for NLG by showing that they correlate well with
scores assigned by humans for a single criteria (overall quality) alone is
inadequate. Indeed, our extensive study involving 25 automatic evaluation
metrics across 6 different tasks and 18 different evaluation criteria shows
that there is no single metric which correlates well with human scores on all
desirable criteria, for most NLG tasks. Given this situation, we propose
CheckLists for better design and evaluation of automatic metrics. We design
templates which target a specific criteria (e.g., coverage) and perturb the
output such that the quality gets affected only along this specific criteria
(e.g., the coverage drops). We show that existing evaluation metrics are not
robust against even such simple perturbations and disagree with scores assigned
by humans to the perturbed output. The proposed templates thus allow for a
fine-grained assessment of automatic evaluation metrics exposing their
limitations and will facilitate better design, analysis and evaluation of such
metrics.


http://arxiv.org/abs/2109.05696
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding. (1%)
Tianda Li; Ahmad Rashid; Aref Jafari; Pranav Sharma; Ali Ghodsi; Mehdi Rezagholizadeh
  Knowledge Distillation (KD) is a model compression algorithm that helps
transfer the knowledge of a large neural network into a smaller one. Even
though KD has shown promise on a wide range of Natural Language Processing
(NLP) applications, little is understood about how one KD algorithm compares to
another and whether these approaches can be complimentary to each other. In
this work, we evaluate various KD algorithms on in-domain, out-of-domain and
adversarial testing. We propose a framework to assess the adversarial
robustness of multiple KD algorithms. Moreover, we introduce a new KD
algorithm, Combined-KD, which takes advantage of two promising approaches
(better training scheme and more efficient data augmentation). Our extensive
experimental results show that Combined-KD achieves state-of-the-art results on
the GLUE benchmark, out-of-domain generalization, and adversarial robustness
compared to competitive methods.


http://arxiv.org/abs/2109.06404
Detecting Safety Problems of Multi-Sensor Fusion in Autonomous Driving. (1%)
Ziyuan Zhong; Zhisheng Hu; Shengjian Guo; Xinyang Zhang; Zhenyu Zhong; Baishakhi Ray
  Autonomous driving (AD) systems have been thriving in recent years. In
general, they receive sensor data, compute driving decisions, and output
control signals to the vehicles. To smooth out the uncertainties brought by
sensor inputs, AD systems usually leverage multi-sensor fusion (MSF) to fuse
the sensor inputs and produce a more reliable understanding of the
surroundings. However, MSF cannot completely eliminate the uncertainties since
it lacks the knowledge about which sensor provides the most accurate data. As a
result, critical consequences might happen unexpectedly. In this work, we
observed that the popular MSF methods in an industry-grade Advanced
Driver-Assistance System (ADAS) can mislead the car control and result in
serious safety hazards. Misbehavior can happen regardless of the used fusion
methods and the accurate data from at least one sensor. To attribute the safety
hazards to a MSF method, we formally define the fusion errors and propose a way
to distinguish safety violations causally induced by such errors. Further, we
develop a novel evolutionary-based domain-specific search framework,
FusionFuzz, for the efficient detection of fusion errors. We evaluate our
framework on two widely used MSF methods. %in two driving environments.
Experimental results show that FusionFuzz identifies more than 150 fusion
errors. Finally, we provide several suggestions to improve the MSF methods
under study.


http://arxiv.org/abs/2109.06176
TREATED:Towards Universal Defense against Textual Adversarial Attacks. (99%)
Bin Zhu; Zhaoquan Gu; Le Wang; Zhihong Tian
  Recent work shows that deep neural networks are vulnerable to adversarial
examples. Much work studies adversarial example generation, while very little
work focuses on more critical adversarial defense. Existing adversarial
detection methods usually make assumptions about the adversarial example and
attack method (e.g., the word frequency of the adversarial example, the
perturbation level of the attack method). However, this limits the
applicability of the detection method. To this end, we propose TREATED, a
universal adversarial detection method that can defend against attacks of
various perturbation levels without making any assumptions. TREATED identifies
adversarial examples through a set of well-designed reference models. Extensive
experiments on three competitive neural networks and two widely used datasets
show that our method achieves better detection performance than baselines. We
finally conduct ablation studies to verify the effectiveness of our method.


http://arxiv.org/abs/2109.05558
CoG: a Two-View Co-training Framework for Defending Adversarial Attacks on Graph. (98%)
Xugang Wu; Huijun Wu; Xu Zhou; Kai Lu
  Graph neural networks exhibit remarkable performance in graph data analysis.
However, the robustness of GNN models remains a challenge. As a result, they
are not reliable enough to be deployed in critical applications. Recent studies
demonstrate that GNNs could be easily fooled with adversarial perturbations,
especially structural perturbations. Such vulnerability is attributed to the
excessive dependence on the structure information to make predictions. To
achieve better robustness, it is desirable to build the prediction of GNNs with
more comprehensive features. Graph data, in most cases, has two views of
information, namely structure information and feature information. In this
paper, we propose CoG, a simple yet effective co-training framework to combine
these two views for the purpose of robustness. CoG trains sub-models from the
feature view and the structure view independently and allows them to distill
knowledge from each other by adding their most confident unlabeled data into
the training set. The orthogonality of these two views diversifies the
sub-models, thus enhancing the robustness of their ensemble. We evaluate our
framework on three popular datasets, and results show that CoG significantly
improves the robustness of graph models against adversarial attacks without
sacrificing their performance on clean data. We also show that CoG still
achieves good robustness when both node features and graph structures are
perturbed.


http://arxiv.org/abs/2109.05620
RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models. (84%)
Bill Yuchen Lin; Wenyang Gao; Jun Yan; Ryan Moreno; Xiang Ren
  To audit the robustness of named entity recognition (NER) models, we propose
RockNER, a simple yet effective method to create natural adversarial examples.
Specifically, at the entity level, we replace target entities with other
entities of the same semantic class in Wikidata; at the context level, we use
pre-trained language models (e.g., BERT) to generate word substitutions.
Together, the two levels of attack produce natural adversarial examples that
result in a shifted distribution from the training data on which our target
models have been trained. We apply the proposed method to the OntoNotes dataset
and create a new benchmark named OntoRock for evaluating the robustness of
existing NER models via a systematic evaluation protocol. Our experiments and
analysis reveal that even the best model has a significant performance drop,
and these models seem to memorize in-domain entity patterns instead of
reasoning from the context. Our work also studies the effects of a few simple
data augmentation methods to improve the robustness of NER models.


http://arxiv.org/abs/2109.05507
Check Your Other Door! Establishing Backdoor Attacks in the Frequency Domain. (41%)
Hasan Abed Al Kader Hammoud; Bernard Ghanem
  Deep Neural Networks (DNNs) have been utilized in various applications
ranging from image classification and facial recognition to medical imagery
analysis and real-time object detection. As our models become more
sophisticated and complex, the computational cost of training such models
becomes a burden for small companies and individuals; for this reason,
outsourcing the training process has been the go-to option for such users.
Unfortunately, outsourcing the training process comes at the cost of
vulnerability to backdoor attacks. These attacks aim at establishing hidden
backdoors in the DNN such that the model performs well on benign samples but
outputs a particular target label when a trigger is applied to the input.
Current backdoor attacks rely on generating triggers in the image/pixel domain;
however, as we show in this paper, it is not the only domain to exploit and one
should always "check the other doors". In this work, we propose a complete
pipeline for generating a dynamic, efficient, and invisible backdoor attack in
the frequency domain. We show the advantages of utilizing the frequency domain
for establishing undetectable and powerful backdoor attacks through extensive
experiments on various datasets and network architectures. The backdoored
models are shown to break various state-of-the-art defences. We also show two
possible defences that succeed against frequency-based backdoor attacks and
possible ways for the attacker to bypass them. We conclude the work with some
remarks regarding a network's learning capacity and the capability of embedding
a backdoor attack in the model.


http://arxiv.org/abs/2109.05671
Shape-Biased Domain Generalization via Shock Graph Embeddings. (2%)
Maruthi Narayanan; Vickram Rajendran; Benjamin Kimia
  There is an emerging sense that the vulnerability of Image Convolutional
Neural Networks (CNN), i.e., sensitivity to image corruptions, perturbations,
and adversarial attacks, is connected with Texture Bias. This relative lack of
Shape Bias is also responsible for poor performance in Domain Generalization
(DG). The inclusion of a role of shape alleviates these vulnerabilities and
some approaches have achieved this by training on negative images, images
endowed with edge maps, or images with conflicting shape and texture
information. This paper advocates an explicit and complete representation of
shape using a classical computer vision approach, namely, representing the
shape content of an image with the shock graph of its contour map. The
resulting graph and its descriptor is a complete representation of contour
content and is classified using recent Graph Neural Network (GNN) methods. The
experimental results on three domain shift datasets, Colored MNIST, PACS, and
VLCS demonstrate that even without using appearance the shape-based approach
exceeds classical Image CNN based methods in domain generalization.


http://arxiv.org/abs/2109.05659
Source Inference Attacks in Federated Learning. (1%)
Hongsheng Hu; Zoran Salcic; Lichao Sun; Gillian Dobbie; Xuyun Zhang
  Federated learning (FL) has emerged as a promising privacy-aware paradigm
that allows multiple clients to jointly train a model without sharing their
private data. Recently, many studies have shown that FL is vulnerable to
membership inference attacks (MIAs) that can distinguish the training members
of the given model from the non-members. However, existing MIAs ignore the
source of a training member, i.e., the information of which client owns the
training member, while it is essential to explore source privacy in FL beyond
membership privacy of examples from all clients. The leakage of source
information can lead to severe privacy issues. For example, identification of
the hospital contributing to the training of an FL model for COVID-19 pandemic
can render the owner of a data record from this hospital more prone to
discrimination if the hospital is in a high risk region. In this paper, we
propose a new inference attack called source inference attack (SIA), which can
derive an optimal estimation of the source of a training member. Specifically,
we innovatively adopt the Bayesian perspective to demonstrate that an
honest-but-curious server can launch an SIA to steal non-trivial source
information of the training members without violating the FL protocol. The
server leverages the prediction loss of local models on the training members to
achieve the attack effectively and non-intrusively. We conduct extensive
experiments on one synthetic and five real datasets to evaluate the key factors
in an SIA, and the results show the efficacy of the proposed source inference
attack.


http://arxiv.org/abs/2109.05211
RobustART: Benchmarking Robustness on Architecture Design and Training Techniques. (98%)
Shiyu Tang; Ruihao Gong; Yan Wang; Aishan Liu; Jiakai Wang; Xinyun Chen; Fengwei Yu; Xianglong Liu; Dawn Song; Alan Yuille; Philip H. S. Torr; Dacheng Tao
  Deep neural networks (DNNs) are vulnerable to adversarial noises, which
motivates the benchmark of model robustness. Existing benchmarks mainly focus
on evaluating the defenses, but there are no comprehensive studies of how
architecture design and general training techniques affect robustness.
Comprehensively benchmarking their relationships will be highly beneficial for
better understanding and developing robust DNNs. Thus, we propose RobustART,
the first comprehensive Robustness investigation benchmark on ImageNet
(including open-source toolkit, pre-trained model zoo, datasets, and analyses)
regarding ARchitecture design (44 human-designed off-the-shelf architectures
and 1200+ networks from neural architecture search) and Training techniques
(10+ general techniques, e.g., data augmentation) towards diverse noises
(adversarial, natural, and system noises). Extensive experiments revealed and
substantiated several insights for the first time, for example: (1) adversarial
training largely improves the clean accuracy and all types of robustness for
Transformers and MLP-Mixers; (2) with comparable sizes, CNNs > Transformers >
MLP-Mixers on robustness against natural and system noises; Transformers >
MLP-Mixers > CNNs on adversarial robustness; (3) for some light-weight
architectures (e.g., EfficientNet, MobileNetV2, and MobileNetV3), increasing
model sizes or using extra training data cannot improve robustness. Our
benchmark http://robust.art/ : (1) presents an open-source platform for
conducting comprehensive evaluation on diverse robustness types; (2) provides a
variety of pre-trained models with different training techniques to facilitate
robustness evaluation; (3) proposes a new view to better understand the
mechanism towards designing robust DNN architectures, backed up by the
analysis. We will continuously contribute to building this ecosystem for the
community.


http://arxiv.org/abs/2109.05223
2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency. (81%)
Yonggan Fu; Yang Zhao; Qixuan Yu; Chaojian Li; Yingyan Lin
  The recent breakthroughs of deep neural networks (DNNs) and the advent of
billions of Internet of Things (IoT) devices have excited an explosive demand
for intelligent IoT devices equipped with domain-specific DNN accelerators.
However, the deployment of DNN accelerator enabled intelligent functionality
into real-world IoT devices still remains particularly challenging. First,
powerful DNNs often come at prohibitive complexities, whereas IoT devices often
suffer from stringent resource constraints. Second, while DNNs are vulnerable
to adversarial attacks especially on IoT devices exposed to complex real-world
environments, many IoT applications require strict security. Existing DNN
accelerators mostly tackle only one of the two aforementioned challenges (i.e.,
efficiency or adversarial robustness) while neglecting or even sacrificing the
other. To this end, we propose a 2-in-1 Accelerator, an integrated
algorithm-accelerator co-design framework aiming at winning both the
adversarial robustness and efficiency of DNN accelerators. Specifically, we
first propose a Random Precision Switch (RPS) algorithm that can effectively
defend DNNs against adversarial attacks by enabling random DNN quantization as
an in-situ model switch. Furthermore, we propose a new precision-scalable
accelerator featuring (1) a new precision-scalable MAC unit architecture which
spatially tiles the temporal MAC units to boost both the achievable efficiency
and flexibility and (2) a systematically optimized dataflow that is searched by
our generic accelerator optimizer. Extensive experiments and ablation studies
validate that our 2-in-1 Accelerator can not only aggressively boost both the
adversarial robustness and efficiency of DNN accelerators under various
attacks, but also naturally support instantaneous robustness-efficiency
trade-offs adapting to varied resources without the necessity of DNN
retraining.


http://arxiv.org/abs/2109.04775
A Strong Baseline for Query Efficient Attacks in a Black Box Setting. (99%)
Rishabh Maheshwary; Saket Maheshwary; Vikram Pudi
  Existing black box search methods have achieved high success rate in
generating adversarial attacks against NLP models. However, such search methods
are inefficient as they do not consider the amount of queries required to
generate adversarial attacks. Also, prior attacks do not maintain a consistent
search space while comparing different search methods. In this paper, we
propose a query efficient attack strategy to generate plausible adversarial
examples on text classification and entailment tasks. Our attack jointly
leverages attention mechanism and locality sensitive hashing (LSH) to reduce
the query count. We demonstrate the efficacy of our approach by comparing our
attack with four baselines across three different search spaces. Further, we
benchmark our results across the same search space used in prior attacks. In
comparison to attacks proposed, on an average, we are able to reduce the query
count by 75% across all datasets and target models. We also demonstrate that
our attack achieves a higher success rate when compared to prior attacks in a
limited query setting.


http://arxiv.org/abs/2109.04385
Contrasting Human- and Machine-Generated Word-Level Adversarial Examples for Text Classification. (99%)
Maximilian Mozes; Max Bartolo; Pontus Stenetorp; Bennett Kleinberg; Lewis D. Griffin
  Research shows that natural language processing models are generally
considered to be vulnerable to adversarial attacks; but recent work has drawn
attention to the issue of validating these adversarial inputs against certain
criteria (e.g., the preservation of semantics and grammaticality). Enforcing
constraints to uphold such criteria may render attacks unsuccessful, raising
the question of whether valid attacks are actually feasible. In this work, we
investigate this through the lens of human language ability. We report on
crowdsourcing studies in which we task humans with iteratively modifying words
in an input text, while receiving immediate model feedback, with the aim of
causing a sentiment classification model to misclassify the example. Our
findings suggest that humans are capable of generating a substantial amount of
adversarial examples using semantics-preserving word substitutions. We analyze
how human-generated adversarial examples compare to the recently proposed
TextFooler, Genetic, BAE and SememePSO attack algorithms on the dimensions
naturalness, preservation of sentiment, grammaticality and substitution rate.
Our findings suggest that human-generated adversarial examples are not more
able than the best algorithms to generate natural-reading, sentiment-preserving
examples, though they do so by being much more computationally efficient.


http://arxiv.org/abs/2109.04300
Energy Attack: On Transferring Adversarial Examples. (99%)
Ruoxi Shi; Borui Yang; Yangzhou Jiang; Chenglong Zhao; Bingbing Ni
  In this work we propose Energy Attack, a transfer-based black-box
$L_\infty$-adversarial attack. The attack is parameter-free and does not
require gradient approximation. In particular, we first obtain white-box
adversarial perturbations of a surrogate model and divide these perturbations
into small patches. Then we extract the unit component vectors and eigenvalues
of these patches with principal component analysis (PCA). Base on the
eigenvalues, we can model the energy distribution of adversarial perturbations.
We then perform black-box attacks by sampling from the perturbation patches
according to their energy distribution, and tiling the sampled patches to form
a full-size adversarial perturbation. This can be done without the available
access to victim models. Extensive experiments well demonstrate that the
proposed Energy Attack achieves state-of-the-art performance in black-box
attacks on various models and several datasets. Moreover, the extracted
distribution is able to transfer among different model architectures and
different datasets, and is therefore intrinsic to vision architectures.


http://arxiv.org/abs/2109.04460
Protein Folding Neural Networks Are Not Robust. (99%)
Sumit Kumar Jha; Arvind Ramanathan; Rickard Ewetz; Alvaro Velasquez; Susmit Jha
  Deep neural networks such as AlphaFold and RoseTTAFold predict remarkably
accurate structures of proteins compared to other algorithmic approaches. It is
known that biologically small perturbations in the protein sequence do not lead
to drastic changes in the protein structure. In this paper, we demonstrate that
RoseTTAFold does not exhibit such a robustness despite its high accuracy, and
biologically small perturbations for some input sequences result in radically
different predicted protein structures. This raises the challenge of detecting
when these predicted protein structures cannot be trusted. We define the
robustness measure for the predicted structure of a protein sequence to be the
inverse of the root-mean-square distance (RMSD) in the predicted structure and
the structure of its adversarially perturbed sequence. We use adversarial
attack methods to create adversarial protein sequences, and show that the RMSD
in the predicted protein structure ranges from 0.119\r{A} to 34.162\r{A} when
the adversarial perturbations are bounded by 20 units in the BLOSUM62 distance.
This demonstrates very high variance in the robustness measure of the predicted
structures. We show that the magnitude of the correlation (0.917) between our
robustness measure and the RMSD between the predicted structure and the ground
truth is high, that is, the predictions with low robustness measure cannot be
trusted. This is the first paper demonstrating the susceptibility of
RoseTTAFold to adversarial attacks.


http://arxiv.org/abs/2109.04176
Towards Transferable Adversarial Attacks on Vision Transformers. (99%)
Zhipeng Wei; Jingjing Chen; Micah Goldblum; Zuxuan Wu; Tom Goldstein; Yu-Gang Jiang
  Vision transformers (ViTs) have demonstrated impressive performance on a
series of computer vision tasks, yet they still suffer from adversarial
examples. In this paper, we posit that adversarial attacks on transformers
should be specially tailored for their architecture, jointly considering both
patches and self-attention, in order to achieve high transferability. More
specifically, we introduce a dual attack framework, which contains a Pay No
Attention (PNA) attack and a PatchOut attack, to improve the transferability of
adversarial samples across different ViTs. We show that skipping the gradients
of attention during backpropagation can generate adversarial examples with high
transferability. In addition, adversarial perturbations generated by optimizing
randomly sampled subsets of patches at each iteration achieve higher attack
success rates than attacks using all patches. We evaluate the transferability
of attacks on state-of-the-art ViTs, CNNs and robustly trained CNNs. The
results of these experiments demonstrate that the proposed dual attack can
greatly boost transferability between ViTs and from ViTs to CNNs. In addition,
the proposed method can easily be combined with existing transfer methods to
boost performance.


http://arxiv.org/abs/2109.04367
Multi-granularity Textual Adversarial Attack with Behavior Cloning. (98%)
Yangyi Chen; Jin Su; Wei Wei
  Recently, the textual adversarial attack models become increasingly popular
due to their successful in estimating the robustness of NLP models. However,
existing works have obvious deficiencies. (1) They usually consider only a
single granularity of modification strategies (e.g. word-level or
sentence-level), which is insufficient to explore the holistic textual space
for generation; (2) They need to query victim models hundreds of times to make
a successful attack, which is highly inefficient in practice. To address such
problems, in this paper we propose MAYA, a Multi-grAnularitY Attack model to
effectively generate high-quality adversarial samples with fewer queries to
victim models. Furthermore, we propose a reinforcement-learning based method to
train a multi-granularity attack agent through behavior cloning with the expert
knowledge from our MAYA algorithm to further reduce the query times.
Additionally, we also adapt the agent to attack black-box models that only
output labels without confidence scores. We conduct comprehensive experiments
to evaluate our attack models by attacking BiLSTM, BERT and RoBERTa in two
different black-box attack settings and three benchmark datasets. Experimental
results show that our models achieve overall better attacking performance and
produce more fluent and grammatical adversarial samples compared to baseline
models. Besides, our adversarial attack agent significantly reduces the query
times in both attack settings. Our codes are released at
https://github.com/Yangyi-Chen/MAYA.


http://arxiv.org/abs/2109.04608
Spatially Focused Attack against Spatiotemporal Graph Neural Networks. (81%)
Fuqiang Liu; Luis Miranda-Moreno; Lijun Sun
  Spatiotemporal forecasting plays an essential role in various applications in
intelligent transportation systems (ITS), such as route planning, navigation,
and traffic control and management. Deep Spatiotemporal graph neural networks
(GNNs), which capture both spatial and temporal patterns, have achieved great
success in traffic forecasting applications. Understanding how GNNs-based
forecasting work and the vulnerability and robustness of these models becomes
critical to real-world applications. For example, if spatiotemporal GNNs are
vulnerable in real-world traffic prediction applications, a hacker can easily
manipulate the results and cause serious traffic congestion and even a
city-scale breakdown. However, despite that recent studies have demonstrated
that deep neural networks (DNNs) are vulnerable to carefully designed
perturbations in multiple domains like objection classification and graph
representation, current adversarial works cannot be directly applied to
spatiotemporal forecasting due to the causal nature and spatiotemporal
mechanisms in forecasting models. To fill this gap, in this paper we design
Spatially Focused Attack (SFA) to break spatiotemporal GNNs by attacking a
single vertex. To achieve this, we first propose the inverse estimation to
address the causality issue; then, we apply genetic algorithms with a universal
attack method as the evaluation function to locate the weakest vertex; finally,
perturbations are generated by solving an inverse estimation-based optimization
problem. We conduct experiments on real-world traffic data and our results show
that perturbations in one vertex designed by SA can be diffused into a large
part of the graph.


http://arxiv.org/abs/2109.04615
Differential Privacy in Personalized Pricing with Nonparametric Demand Models. (26%)
Xi Chen; Sentao Miao; Yining Wang
  In the recent decades, the advance of information technology and abundant
personal data facilitate the application of algorithmic personalized pricing.
However, this leads to the growing concern of potential violation of privacy
due to adversarial attack. To address the privacy issue, this paper studies a
dynamic personalized pricing problem with \textit{unknown} nonparametric demand
models under data privacy protection. Two concepts of data privacy, which have
been widely applied in practices, are introduced: \textit{central differential
privacy (CDP)} and \textit{local differential privacy (LDP)}, which is proved
to be stronger than CDP in many cases. We develop two algorithms which make
pricing decisions and learn the unknown demand on the fly, while satisfying the
CDP and LDP gurantees respectively. In particular, for the algorithm with CDP
guarantee, the regret is proved to be at most $\tilde
O(T^{(d+2)/(d+4)}+\varepsilon^{-1}T^{d/(d+4)})$. Here, the parameter $T$
denotes the length of the time horizon, $d$ is the dimension of the
personalized information vector, and the key parameter $\varepsilon>0$ measures
the strength of privacy (smaller $\varepsilon$ indicates a stronger privacy
protection). On the other hand, for the algorithm with LDP guarantee, its
regret is proved to be at most $\tilde
O(\varepsilon^{-2/(d+2)}T^{(d+1)/(d+2)})$, which is near-optimal as we prove a
lower bound of $\Omega(\varepsilon^{-2/(d+2)}T^{(d+1)/(d+2)})$ for any
algorithm with LDP guarantee.


http://arxiv.org/abs/2109.03975
Where Did You Learn That From? Surprising Effectiveness of Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning. (89%)
Maziar Gomrokchi; Susan Amin; Hossein Aboutalebi; Alexander Wong; Doina Precup
  While significant research advances have been made in the field of deep
reinforcement learning, a major challenge to widespread industrial adoption of
deep reinforcement learning that has recently surfaced but little explored is
the potential vulnerability to privacy breaches. In particular, there have been
no concrete adversarial attack strategies in literature tailored for studying
the vulnerability of deep reinforcement learning algorithms to membership
inference attacks. To address this gap, we propose an adversarial attack
framework tailored for testing the vulnerability of deep reinforcement learning
algorithms to membership inference attacks. More specifically, we design a
series of experiments to investigate the impact of temporal correlation, which
naturally exists in reinforcement learning training data, on the probability of
information leakage. Furthermore, we study the differences in the performance
of \emph{collective} and \emph{individual} membership attacks against deep
reinforcement learning algorithms. Experimental results show that the proposed
adversarial attack framework is surprisingly effective at inferring the data
used during deep reinforcement training with an accuracy exceeding $84\%$ in
individual and $97\%$ in collective mode on two different control tasks in
OpenAI Gym, which raises serious privacy concerns in the deployment of models
resulting from deep reinforcement learning. Moreover, we show that the learning
state of a reinforcement learning algorithm significantly influences the level
of the privacy breach.


http://arxiv.org/abs/2109.03857
Robust Optimal Classification Trees Against Adversarial Examples. (80%)
Daniël Vos; Sicco Verwer
  Decision trees are a popular choice of explainable model, but just like
neural networks, they suffer from adversarial examples. Existing algorithms for
fitting decision trees robust against adversarial examples are greedy
heuristics and lack approximation guarantees. In this paper we propose ROCT, a
collection of methods to train decision trees that are optimally robust against
user-specified attack models. We show that the min-max optimization problem
that arises in adversarial learning can be solved using a single minimization
formulation for decision trees with 0-1 loss. We propose such formulations in
Mixed-Integer Linear Programming and Maximum Satisfiability, which widely
available solvers can optimize. We also present a method that determines the
upper bound on adversarial accuracy for any model using bipartite matching. Our
experimental results demonstrate that the existing heuristics achieve close to
optimal scores while ROCT achieves state-of-the-art scores.


http://arxiv.org/abs/2109.02889
Adversarial Parameter Defense by Multi-Step Risk Minimization. (98%)
Zhiyuan Zhang; Ruixuan Luo; Xuancheng Ren; Qi Su; Liangyou Li; Xu Sun
  Previous studies demonstrate DNNs' vulnerability to adversarial examples and
adversarial training can establish a defense to adversarial examples. In
addition, recent studies show that deep neural networks also exhibit
vulnerability to parameter corruptions. The vulnerability of model parameters
is of crucial value to the study of model robustness and generalization. In
this work, we introduce the concept of parameter corruption and propose to
leverage the loss change indicators for measuring the flatness of the loss
basin and the parameter robustness of neural network parameters. On such basis,
we analyze parameter corruptions and propose the multi-step adversarial
corruption algorithm. To enhance neural networks, we propose the adversarial
parameter defense algorithm that minimizes the average risk of multiple
adversarial parameter corruptions. Experimental results show that the proposed
algorithm can improve both the parameter robustness and accuracy of neural
networks.


http://arxiv.org/abs/2109.02979
POW-HOW: An enduring timing side-channel to evade online malware sandboxes. (12%)
Antonio Nappa; Panagiotis Papadopoulos; Matteo Varvello; Daniel Aceituno Gomez; Juan Tapiador; Andrea Lanzi
  Online malware scanners are one of the best weapons in the arsenal of
cybersecurity companies and researchers. A fundamental part of such systems is
the sandbox that provides an instrumented and isolated environment (virtualized
or emulated) for any user to upload and run unknown artifacts and identify
potentially malicious behaviors. The provided API and the wealth of information
inthe reports produced by these services have also helped attackers test the
efficacy of numerous techniques to make malware hard to detect.The most common
technique used by malware for evading the analysis system is to monitor the
execution environment, detect the presence of any debugging artifacts, and hide
its malicious behavior if needed. This is usually achieved by looking for
signals suggesting that the execution environment does not belong to a the
native machine, such as specific memory patterns or behavioral traits of
certain CPU instructions.
  In this paper, we show how an attacker can evade detection on such online
services by incorporating a Proof-of-Work (PoW) algorithm into a malware
sample. Specifically, we leverage the asymptotic behavior of the computational
cost of PoW algorithms when they run on some classes of hardware platforms to
effectively detect a non bare-metal environment of the malware sandbox
analyzer. To prove the validity of this intuition, we design and implement the
POW-HOW framework, a tool to automatically implement sandbox detection
strategies and embed a test evasion program into an arbitrary malware sample.
Our empirical evaluation shows that the proposed evasion technique is durable,
hard to fingerprint, and reduces existing malware detection rate by a factor of
10. Moreover, we show how bare-metal environments cannot scale with actual
malware submissions rates for consumer services.


http://arxiv.org/abs/2109.02973
Unpaired Adversarial Learning for Single Image Deraining with Rain-Space Contrastive Constraints. (1%)
Xiang Chen; Jinshan Pan; Kui Jiang; Yufeng Huang; Caihua Kong; Longgang Dai; Yufeng Li
  Deep learning-based single image deraining (SID) with unpaired information is
of immense importance, as relying on paired synthetic data often limits their
generality and scalability in real-world applications. However, we noticed that
direct employ of unpaired adversarial learning and cycle-consistency
constraints in the SID task is insufficient to learn the underlying
relationship from rainy input to clean outputs, since the domain knowledge
between rainy and rain-free images is asymmetrical. To address such limitation,
we develop an effective unpaired SID method which explores mutual properties of
the unpaired exemplars by a contrastive learning manner in a GAN framework,
named as CDR-GAN. The proposed method mainly consists of two cooperative
branches: Bidirectional Translation Branch (BTB) and Contrastive Guidance
Branch (CGB). Specifically, BTB takes full advantage of the circulatory
architecture of adversarial consistency to exploit latent feature distributions
and guide transfer ability between two domains by equipping it with
bidirectional mapping. Simultaneously, CGB implicitly constrains the embeddings
of different exemplars in rain space by encouraging the similar feature
distributions closer while pushing the dissimilar further away, in order to
better help rain removal and image restoration. During training, we explore
several loss functions to further constrain the proposed CDR-GAN. Extensive
experiments show that our method performs favorably against existing unpaired
deraining approaches on both synthetic and real-world datasets, even
outperforms several fully-supervised or semi-supervised models.


http://arxiv.org/abs/2109.02765
Robustness and Generalization via Generative Adversarial Training. (82%)
Omid Poursaeed; Tianxing Jiang; Harry Yang; Serge Belongie; SerNam Lim
  While deep neural networks have achieved remarkable success in various
computer vision tasks, they often fail to generalize to new domains and subtle
variations of input images. Several defenses have been proposed to improve the
robustness against these variations. However, current defenses can only
withstand the specific attack used in training, and the models often remain
vulnerable to other input variations. Moreover, these methods often degrade
performance of the model on clean images and do not generalize to out-of-domain
samples. In this paper we present Generative Adversarial Training, an approach
to simultaneously improve the model's generalization to the test set and
out-of-domain samples as well as its robustness to unseen adversarial attacks.
Instead of altering a low-level pre-defined aspect of images, we generate a
spectrum of low-level, mid-level and high-level changes using generative models
with a disentangled latent space. Adversarial training with these examples
enable the model to withstand a wide range of attacks by observing a variety of
input alterations during training. We show that our approach not only improves
performance of the model on clean images and out-of-domain samples but also
makes it robust against unforeseen attacks and outperforms prior work. We
validate effectiveness of our method by demonstrating results on various tasks
such as classification, segmentation and object detection.


http://arxiv.org/abs/2109.02836
Trojan Signatures in DNN Weights. (33%)
Greg Fields; Mohammad Samragh; Mojan Javaheripi; Farinaz Koushanfar; Tara Javidi
  Deep neural networks have been shown to be vulnerable to backdoor, or trojan,
attacks where an adversary has embedded a trigger in the network at training
time such that the model correctly classifies all standard inputs, but
generates a targeted, incorrect classification on any input which contains the
trigger. In this paper, we present the first ultra light-weight and highly
effective trojan detection method that does not require access to the
training/test data, does not involve any expensive computations, and makes no
assumptions on the nature of the trojan trigger. Our approach focuses on
analysis of the weights of the final, linear layer of the network. We
empirically demonstrate several characteristics of these weights that occur
frequently in trojaned networks, but not in benign networks. In particular, we
show that the distribution of the weights associated with the trojan target
class is clearly distinguishable from the weights associated with other
classes. Using this, we demonstrate the effectiveness of our proposed detection
method against state-of-the-art attacks across a variety of architectures,
datasets, and trigger types.


http://arxiv.org/abs/2109.02532
Automated Robustness with Adversarial Training as a Post-Processing Step. (4%)
Ambrish Rawat; Mathieu Sinn; Beat Buesser
  Adversarial training is a computationally expensive task and hence searching
for neural network architectures with robustness as the criterion can be
challenging. As a step towards practical automation, this work explores the
efficacy of a simple post processing step in yielding robust deep learning
model. To achieve this, we adopt adversarial training as a post-processing step
for optimised network architectures obtained from a neural architecture search
algorithm. Specific policies are adopted for tuning the hyperparameters of the
different steps, resulting in a fully automated pipeline for generating
adversarially robust deep learning models. We evidence the usefulness of the
proposed pipeline with extensive experimentation across 11 image classification
and 9 text classification tasks.


http://arxiv.org/abs/2109.02431
Exposing Length Divergence Bias of Textual Matching Models. (2%)
Lan Jiang; Tianshu Lyu; Chong Meng; Xiaoyong Lyu; Dawei Yin
  Despite the remarkable success deep models have achieved in Textual Matching
(TM), their robustness issue is still a topic of concern. In this work, we
propose a new perspective to study this issue -- via the length divergence bias
of TM models. We conclude that this bias stems from two parts: the label bias
of existing TM datasets and the sensitivity of TM models to superficial
information. We critically examine widely used TM datasets, and find that all
of them follow specific length divergence distributions by labels, providing
direct cues for predictions. As for the TM models, we conduct adversarial
evaluation and show that all models' performances drop on the
out-of-distribution adversarial test sets we construct, which demonstrates that
they are all misled by biased training sets. This is also confirmed by the
\textit{SentLen} probing task that all models capture rich length information
during training to facilitate their performances. Finally, to alleviate the
length divergence bias in TM models, we propose a practical adversarial
training method using bias-free training data. Our experiments indicate that we
successfully improve the robustness and generalization ability of models at the
same time.


http://arxiv.org/abs/2109.02229
Efficient Combinatorial Optimization for Word-level Adversarial Textual Attack. (98%)
Shengcai Liu; Ning Lu; Cheng Chen; Ke Tang
  Over the past few years, various word-level textual attack approaches have
been proposed to reveal the vulnerability of deep neural networks used in
natural language processing. Typically, these approaches involve an important
optimization step to determine which substitute to be used for each word in the
original input. However, current research on this step is still rather limited,
from the perspectives of both problem-understanding and problem-solving. In
this paper, we address these issues by uncovering the theoretical properties of
the problem and proposing an efficient local search algorithm (LS) to solve it.
We establish the first provable approximation guarantee on solving the problem
in general cases. Notably, for adversarial textual attack, it is even better
than the previous bound which only holds in special case. Extensive experiments
involving five NLP tasks, six datasets and eleven NLP models show that LS can
largely reduce the number of queries usually by an order of magnitude to
achieve high attack success rates. Further experiments show that the
adversarial examples crafted by LS usually have higher quality, exhibit better
transferability, and can bring more robustness improvement to victim models by
adversarial training.


http://arxiv.org/abs/2109.02018
Tolerating Adversarial Attacks and Byzantine Faults in Distributed Machine Learning. (2%)
Yusen Wu; Hao Chen; Xin Wang; Chao Liu; Phuong Nguyen; Yelena Yesha
  Adversarial attacks attempt to disrupt the training, retraining and utilizing
of artificial intelligence and machine learning models in large-scale
distributed machine learning systems. This causes security risks on its
prediction outcome. For example, attackers attempt to poison the model by
either presenting inaccurate misrepresentative data or altering the models'
parameters. In addition, Byzantine faults including software, hardware, network
issues occur in distributed systems which also lead to a negative impact on the
prediction outcome. In this paper, we propose a novel distributed training
algorithm, partial synchronous stochastic gradient descent (ParSGD), which
defends adversarial attacks and/or tolerates Byzantine faults. We demonstrate
the effectiveness of our algorithm under three common adversarial attacks again
the ML models and a Byzantine fault during the training phase. Our results show
that using ParSGD, ML models can still produce accurate predictions as if it is
not being attacked nor having failures at all when almost half of the nodes are
being compromised or failed. We will report the experimental evaluations of
ParSGD in comparison with other algorithms.


http://arxiv.org/abs/2109.03326
DexRay: A Simple, yet Effective Deep Learning Approach to Android Malware Detection based on Image Representation of Bytecode. (1%)
Nadia Daoudi; Jordan Samhi; Abdoul Kader Kabore; Kevin Allix; Tegawendé F. Bissyandé; Jacques Klein
  Computer vision has witnessed several advances in recent years, with
unprecedented performance provided by deep representation learning research.
Image formats thus appear attractive to other fields such as malware detection,
where deep learning on images alleviates the need for comprehensively
hand-crafted features generalising to different malware variants. We postulate
that this research direction could become the next frontier in Android malware
detection, and therefore requires a clear roadmap to ensure that new approaches
indeed bring novel contributions. We contribute with a first building block by
developing and assessing a baseline pipeline for image-based malware detection
with straightforward steps. We propose DexRay, which converts the bytecode of
the app DEX files into grey-scale "vector" images and feeds them to a
1-dimensional Convolutional Neural Network model. We view DexRay as
foundational due to the exceedingly basic nature of the design choices,
allowing to infer what could be a minimal performance that can be obtained with
image-based learning in malware detection. The performance of DexRay evaluated
on over 158k apps demonstrates that, while simple, our approach is effective
with a high detection rate(F1-score= 0.96). Finally, we investigate the impact
of time decay and image-resizing on the performance of DexRay and assess its
resilience to obfuscation. This work-in-progress paper contributes to the
domain of Deep Learning based Malware detection by providing a sound, simple,
yet effective approach (with available artefacts) that can be the basis to
scope the many profound questions that will need to be investigated to fully
develop this domain.


http://arxiv.org/abs/2109.03329
Real-World Adversarial Examples involving Makeup Application. (99%)
Chang-Sheng Lin; Chia-Yi Hsu; Pin-Yu Chen; Chia-Mu Yu
  Deep neural networks have developed rapidly and have achieved outstanding
performance in several tasks, such as image classification and natural language
processing. However, recent studies have indicated that both digital and
physical adversarial examples can fool neural networks. Face-recognition
systems are used in various applications that involve security threats from
physical adversarial examples. Herein, we propose a physical adversarial attack
with the use of full-face makeup. The presence of makeup on the human face is a
reasonable possibility, which possibly increases the imperceptibility of
attacks. In our attack framework, we combine the cycle-adversarial generative
network (cycle-GAN) and a victimized classifier. The Cycle-GAN is used to
generate adversarial makeup, and the architecture of the victimized classifier
is VGG 16. Our experimental results show that our attack can effectively
overcome manual errors in makeup application, such as color and
position-related errors. We also demonstrate that the approaches used to train
the models can influence physical attacks; the adversarial perturbations
crafted from the pre-trained model are affected by the corresponding training
data.


http://arxiv.org/abs/2109.01945
Utilizing Adversarial Targeted Attacks to Boost Adversarial Robustness. (99%)
Uriya Pesso; Koby Bibas; Meir Feder
  Adversarial attacks have been shown to be highly effective at degrading the
performance of deep neural networks (DNNs). The most prominent defense is
adversarial training, a method for learning a robust model. Nevertheless,
adversarial training does not make DNNs immune to adversarial perturbations. We
propose a novel solution by adopting the recently suggested Predictive
Normalized Maximum Likelihood. Specifically, our defense performs adversarial
targeted attacks according to different hypotheses, where each hypothesis
assumes a specific label for the test sample. Then, by comparing the hypothesis
probabilities, we predict the label. Our refinement process corresponds to
recent findings of the adversarial subspace properties. We extensively evaluate
our approach on 16 adversarial attack benchmarks using ResNet-50,
WideResNet-28, and a2-layer ConvNet trained with ImageNet, CIFAR10, and MNIST,
showing a significant improvement of up to 5.7%, 3.7%, and 0.6% respectively.


http://arxiv.org/abs/2109.01983
Training Meta-Surrogate Model for Transferable Adversarial Attack. (99%)
Yunxiao Qin; Yuanhao Xiong; Jinfeng Yi; Cho-Jui Hsieh
  We consider adversarial attacks to a black-box model when no queries are
allowed. In this setting, many methods directly attack surrogate models and
transfer the obtained adversarial examples to fool the target model. Plenty of
previous works investigated what kind of attacks to the surrogate model can
generate more transferable adversarial examples, but their performances are
still limited due to the mismatches between surrogate models and the target
model. In this paper, we tackle this problem from a novel angle -- instead of
using the original surrogate models, can we obtain a Meta-Surrogate Model (MSM)
such that attacks to this model can be easier transferred to other models? We
show that this goal can be mathematically formulated as a well-posed
(bi-level-like) optimization problem and design a differentiable attacker to
make training feasible. Given one or a set of surrogate models, our method can
thus obtain an MSM such that adversarial examples generated on MSM enjoy
eximious transferability. Comprehensive experiments on Cifar-10 and ImageNet
demonstrate that by attacking the MSM, we can obtain stronger transferable
adversarial examples to fool black-box models including adversarially trained
ones, with much higher success rates than existing methods. The proposed method
reveals significant security challenges of deep models and is promising to be
served as a state-of-the-art benchmark for evaluating the robustness of deep
models in the black-box setting.


http://arxiv.org/abs/2109.01766
SEC4SR: A Security Analysis Platform for Speaker Recognition. (99%)
Guangke Chen; Zhe Zhao; Fu Song; Sen Chen; Lingling Fan; Yang Liu
  Adversarial attacks have been expanded to speaker recognition (SR). However,
existing attacks are often assessed using different SR models, recognition
tasks and datasets, and only few adversarial defenses borrowed from computer
vision are considered. Yet,these defenses have not been thoroughly evaluated
against adaptive attacks. Thus, there is still a lack of quantitative
understanding about the strengths and limitations of adversarial attacks and
defenses. More effective defenses are also required for securing SR systems. To
bridge this gap, we present SEC4SR, the first platform enabling researchers to
systematically and comprehensively evaluate adversarial attacks and defenses in
SR. SEC4SR incorporates 4 white-box and 2 black-box attacks, 24 defenses
including our novel feature-level transformations. It also contains techniques
for mounting adaptive attacks. Using SEC4SR, we conduct thus far the
largest-scale empirical study on adversarial attacks and defenses in SR,
involving 23 defenses, 15 attacks and 4 attack settings. Our study provides
lots of useful findings that may advance future research: such as (1) all the
transformations slightly degrade accuracy on benign examples and their
effectiveness vary with attacks; (2) most transformations become less effective
under adaptive attacks, but some transformations become more effective; (3) few
transformations combined with adversarial training yield stronger defenses over
some but not all attacks, while our feature-level transformation combined with
adversarial training yields the strongest defense over all the attacks.
Extensive experiments demonstrate capabilities and advantages of SEC4SR which
can benefit future research in SR.


http://arxiv.org/abs/2109.01553
Risk Assessment for Connected Vehicles under Stealthy Attacks on Vehicle-to-Vehicle Networks. (1%)
Tianci Yang; Carlos Murguia; Chen Lv
  Cooperative Adaptive Cruise Control (CACC) is an autonomous vehicle-following
technology that allows groups of vehicles on the highway to form in
tightly-coupled platoons. This is accomplished by exchanging inter-vehicle data
through Vehicle-to-Vehicle (V2V) wireless communication networks. CACC
increases traffic throughput and safety, and decreases fuel consumption.
However, the surge of vehicle connectivity has brought new security challenges
as vehicular networks increasingly serve as new access points for adversaries
trying to deteriorate the platooning performance or even cause collisions. In
this manuscript, we propose a novel attack detection scheme that leverage
real-time sensor/network data and physics-based mathematical models of vehicles
in the platoon. Nevertheless, even the best detection scheme could lead to
conservative detection results because of unavoidable modelling uncertainties,
network effects (delays, quantization, communication dropouts), and noise. It
is hard (often impossible) for any detector to distinguish between these
different perturbation sources and actual attack signals. This enables
adversaries to launch a range of attack strategies that can surpass the
detection scheme by hiding within the system uncertainty. Here, we provide risk
assessment tools (in terms of semidefinite programs) for Connected and
Automated Vehicles (CAVs) to quantify the potential effect of attacks that
remain hidden from the detector (referred here as \emph{stealthy attacks}). A
numerical case-study is presented to illustrate the effectiveness of our
methods.


http://arxiv.org/abs/2109.01275
A Synergetic Attack against Neural Network Classifiers combining Backdoor and Adversarial Examples. (99%)
Guanxiong Liu; Issa Khalil; Abdallah Khreishah; NhatHai Phan
  In this work, we show how to jointly exploit adversarial perturbation and
model poisoning vulnerabilities to practically launch a new stealthy attack,
dubbed AdvTrojan. AdvTrojan is stealthy because it can be activated only when:
1) a carefully crafted adversarial perturbation is injected into the input
examples during inference, and 2) a Trojan backdoor is implanted during the
training process of the model. We leverage adversarial noise in the input space
to move Trojan-infected examples across the model decision boundary, making it
difficult to detect. The stealthiness behavior of AdvTrojan fools the users
into accidentally trust the infected model as a robust classifier against
adversarial examples. AdvTrojan can be implemented by only poisoning the
training data similar to conventional Trojan backdoor attacks. Our thorough
analysis and extensive experiments on several benchmark datasets show that
AdvTrojan can bypass existing defenses with a success rate close to 100% in
most of our experimental scenarios and can be extended to attack federated
learning tasks as well.


http://arxiv.org/abs/2109.00936
Impact of Attention on Adversarial Robustness of Image Classification Models. (99%)
Prachi Agrawal; Narinder Singh Punn; Sanjay Kumar Sonbhadra; Sonali Agarwal
  Adversarial attacks against deep learning models have gained significant
attention and recent works have proposed explanations for the existence of
adversarial examples and techniques to defend the models against these attacks.
Attention in computer vision has been used to incorporate focused learning of
important features and has led to improved accuracy. Recently, models with
attention mechanisms have been proposed to enhance adversarial robustness.
Following this context, this work aims at a general understanding of the impact
of attention on adversarial robustness. This work presents a comparative study
of adversarial robustness of non-attention and attention based image
classification models trained on CIFAR-10, CIFAR-100 and Fashion MNIST datasets
under the popular white box and black box attacks. The experimental results
show that the robustness of attention based models may be dependent on the
datasets used i.e. the number of classes involved in the classification. In
contrast to the datasets with less number of classes, attention based models
are observed to show better robustness towards classification.


http://arxiv.org/abs/2109.00946
Adversarial Robustness for Unsupervised Domain Adaptation. (98%)
Muhammad Awais; Fengwei Zhou; Hang Xu; Lanqing Hong; Ping Luo; Sung-Ho Bae; Zhenguo Li
  Extensive Unsupervised Domain Adaptation (UDA) studies have shown great
success in practice by learning transferable representations across a labeled
source domain and an unlabeled target domain with deep models. However,
previous works focus on improving the generalization ability of UDA models on
clean examples without considering the adversarial robustness, which is crucial
in real-world applications. Conventional adversarial training methods are not
suitable for the adversarial robustness on the unlabeled target domain of UDA
since they train models with adversarial examples generated by the supervised
loss function. In this work, we leverage intermediate representations learned
by multiple robust ImageNet models to improve the robustness of UDA models. Our
method works by aligning the features of the UDA model with the robust features
learned by ImageNet pre-trained models along with domain adaptation training.
It utilizes both labeled and unlabeled domains and instills robustness without
any adversarial intervention or label requirement during domain adaptation
training. Experimental results show that our method significantly improves
adversarial robustness compared to the baseline while keeping clean accuracy on
various UDA benchmarks.


http://arxiv.org/abs/2109.00864
Real World Robustness from Systematic Noise. (91%)
Yan Wang; Yuhang Li; Ruihao Gong
  Systematic error, which is not determined by chance, often refers to the
inaccuracy (involving either the observation or measurement process) inherent
to a system. In this paper, we exhibit some long-neglected but
frequent-happening adversarial examples caused by systematic error. More
specifically, we find the trained neural network classifier can be fooled by
inconsistent implementations of image decoding and resize. This tiny difference
between these implementations often causes an accuracy drop from training to
deployment. To benchmark these real-world adversarial examples, we propose
ImageNet-S dataset, which enables researchers to measure a classifier's
robustness to systematic error. For example, we find a normal ResNet-50 trained
on ImageNet can have 1%-5% accuracy difference due to the systematic error.
Together our evaluation and dataset may aid future work toward real-world
robustness and practical generalization.


http://arxiv.org/abs/2109.00959
Building Compact and Robust Deep Neural Networks with Toeplitz Matrices. (61%)
Alexandre Araujo
  Deep neural networks are state-of-the-art in a wide variety of tasks,
however, they exhibit important limitations which hinder their use and
deployment in real-world applications. When developing and training neural
networks, the accuracy should not be the only concern, neural networks must
also be cost-effective and reliable. Although accurate, large neural networks
often lack these properties. This thesis focuses on the problem of training
neural networks which are not only accurate but also compact, easy to train,
reliable and robust to adversarial examples. To tackle these problems, we
leverage the properties of structured matrices from the Toeplitz family to
build compact and secure neural networks.


http://arxiv.org/abs/2109.00544
Towards Improving Adversarial Training of NLP Models. (98%)
Jin Yong Yoo; Yanjun Qi
  Adversarial training, a method for learning robust deep neural networks,
constructs adversarial examples during training. However, recent methods for
generating NLP adversarial examples involve combinatorial search and expensive
sentence encoders for constraining the generated instances. As a result, it
remains challenging to use vanilla adversarial training to improve NLP models'
performance, and the benefits are mainly uninvestigated. This paper proposes a
simple and improved vanilla adversarial training process for NLP, which we name
Attacking to Training ($\texttt{A2T}$). The core part of $\texttt{A2T}$ is a
new and cheaper word substitution attack optimized for vanilla adversarial
training. We use $\texttt{A2T}$ to train BERT and RoBERTa models on IMDB,
Rotten Tomatoes, Yelp, and SNLI datasets. Our results show that it is possible
to train empirically robust NLP models using a much cheaper adversary. We
demonstrate that vanilla adversarial training with $\texttt{A2T}$ can improve
an NLP model's robustness to the attack it was originally trained with and also
defend the model against other types of attacks. Furthermore, we show that
$\texttt{A2T}$ can improve NLP models' standard accuracy, cross-domain
generalization, and interpretability. Code is available at
http://github.com/jinyongyoo/A2T .


http://arxiv.org/abs/2109.00685
Excess Capacity and Backdoor Poisoning. (97%)
Naren Sarayu Manoj; Avrim Blum
  A backdoor data poisoning attack is an adversarial attack wherein the
attacker injects several watermarked, mislabeled training examples into a
training set. The watermark does not impact the test-time performance of the
model on typical data; however, the model reliably errs on watermarked
examples.
  To gain a better foundational understanding of backdoor data poisoning
attacks, we present a formal theoretical framework within which one can discuss
backdoor data poisoning attacks for classification problems. We then use this
to analyze important statistical and computational issues surrounding these
attacks.
  On the statistical front, we identify a parameter we call the memorization
capacity that captures the intrinsic vulnerability of a learning problem to a
backdoor attack. This allows us to argue about the robustness of several
natural learning problems to backdoor attacks. Our results favoring the
attacker involve presenting explicit constructions of backdoor attacks, and our
robustness results show that some natural problem settings cannot yield
successful backdoor attacks.
  From a computational standpoint, we show that under certain assumptions,
adversarial training can detect the presence of backdoors in a training set. We
then show that under similar assumptions, two closely related problems we call
backdoor filtering and robust generalization are nearly equivalent. This
implies that it is both asymptotically necessary and sufficient to design
algorithms that can identify watermarked examples in the training set in order
to obtain a learning algorithm that both generalizes well to unseen data and is
robust to backdoors.


http://arxiv.org/abs/2109.00678
Regional Adversarial Training for Better Robust Generalization. (96%)
Chuanbiao Song; Yanbo Fan; Yicheng Yang; Baoyuan Wu; Yiming Li; Zhifeng Li; Kun He
  Adversarial training (AT) has been demonstrated as one of the most promising
defense methods against various adversarial attacks. To our knowledge, existing
AT-based methods usually train with the locally most adversarial perturbed
points and treat all the perturbed points equally, which may lead to
considerably weaker adversarial robust generalization on test data. In this
work, we introduce a new adversarial training framework that considers the
diversity as well as characteristics of the perturbed points in the vicinity of
benign samples. To realize the framework, we propose a Regional Adversarial
Training (RAT) defense method that first utilizes the attack path generated by
the typical iterative attack method of projected gradient descent (PGD), and
constructs an adversarial region based on the attack path. Then, RAT samples
diverse perturbed training points efficiently inside this region, and utilizes
a distance-aware label smoothing mechanism to capture our intuition that
perturbed points at different locations should have different impact on the
model performance. Extensive experiments on several benchmark datasets show
that RAT consistently makes significant improvement on standard adversarial
training (SAT), and exhibits better robust generalization.


http://arxiv.org/abs/2109.00533
R-SNN: An Analysis and Design Methodology for Robustifying Spiking Neural Networks against Adversarial Attacks through Noise Filters for Dynamic Vision Sensors. (86%)
Alberto Marchisio; Giacomo Pira; Maurizio Martina; Guido Masera; Muhammad Shafique
  Spiking Neural Networks (SNNs) aim at providing energy-efficient learning
capabilities when implemented on neuromorphic chips with event-based Dynamic
Vision Sensors (DVS). This paper studies the robustness of SNNs against
adversarial attacks on such DVS-based systems, and proposes R-SNN, a novel
methodology for robustifying SNNs through efficient DVS-noise filtering. We are
the first to generate adversarial attacks on DVS signals (i.e., frames of
events in the spatio-temporal domain) and to apply noise filters for DVS
sensors in the quest for defending against adversarial attacks. Our results
show that the noise filters effectively prevent the SNNs from being fooled. The
SNNs in our experiments provide more than 90% accuracy on the DVS-Gesture and
NMNIST datasets under different adversarial threat models.


http://arxiv.org/abs/2109.00542
Proof Transfer for Neural Network Verification. (9%)
Christian Sprecher; Marc Fischer; Dimitar I. Dimitrov; Gagandeep Singh; Martin Vechev
  We introduce the novel concept of proof transfer for neural network
verification. We show that by generating proof templates that capture and
generalize existing proofs, we can speed up subsequent proofs. In particular we
create these templates from previous proofs on the same neural network and
consider two cases: (i) where the proofs are created online when verifying
other properties and (ii) where the templates are created offline using a
dataset. We base our methods on three key hypotheses of neural network
robustness proofs. Our evaluation shows the potential of proof transfer for
benefitting robustness verification of neural networks against adversarial
patches, geometric, and $\ell_{\infty}$-perturbations.


http://arxiv.org/abs/2109.00187
Guarding Machine Learning Hardware Against Physical Side-Channel Attacks. (2%)
Anuj Dubey; Rosario Cammarota; Vikram Suresh; Aydin Aysu
  Machine learning (ML) models can be trade secrets due to their development
cost. Hence, they need protection against malicious forms of reverse
engineering (e.g., in IP piracy). With a growing shift of ML to the edge
devices, in part for performance and in part for privacy benefits, the models
have become susceptible to the so-called physical side-channel attacks.
  ML being a relatively new target compared to cryptography poses the problem
of side-channel analysis in a context that lacks published literature. The gap
between the burgeoning edge-based ML devices and the research on adequate
defenses to provide side-channel security for them thus motivates our study.
Our work develops and combines different flavors of side-channel defenses for
ML models in the hardware blocks. We propose and optimize the first defense
based on Boolean masking. We first implement all the masked hardware blocks. We
then present an adder optimization to reduce the area and latency overheads.
Finally, we couple it with a shuffle-based defense.
  We quantify that the area-delay overhead of masking ranges from 5.4$\times$
to 4.7$\times$ depending on the adder topology used and demonstrate first-order
side-channel security of millions of power traces. Additionally, the shuffle
countermeasure impedes a straightforward second-order attack on our first-order
masked implementation.


http://arxiv.org/abs/2108.13930
EG-Booster: Explanation-Guided Booster of ML Evasion Attacks. (99%)
Abderrahmen Amich; Birhanu Eshete
  The widespread usage of machine learning (ML) in a myriad of domains has
raised questions about its trustworthiness in security-critical environments.
Part of the quest for trustworthy ML is robustness evaluation of ML models to
test-time adversarial examples. Inline with the trustworthy ML goal, a useful
input to potentially aid robustness evaluation is feature-based explanations of
model predictions. In this paper, we present a novel approach called EG-Booster
that leverages techniques from explainable ML to guide adversarial example
crafting for improved robustness evaluation of ML models before deploying them
in security-critical settings. The key insight in EG-Booster is the use of
feature-based explanations of model predictions to guide adversarial example
crafting by adding consequential perturbations likely to result in model
evasion and avoiding non-consequential ones unlikely to contribute to evasion.
EG-Booster is agnostic to model architecture, threat model, and supports
diverse distance metrics used previously in the literature. We evaluate
EG-Booster using image classification benchmark datasets, MNIST and CIFAR10.
Our findings suggest that EG-Booster significantly improves evasion rate of
state-of-the-art attacks while performing less number of perturbations. Through
extensive experiments that covers four white-box and three black-box attacks,
we demonstrate the effectiveness of EG-Booster against two undefended neural
networks trained on MNIST and CIFAR10, and another adversarially-trained ResNet
model trained on CIFAR10. Furthermore, we introduce a stability assessment
metric and evaluate the reliability of our explanation-based approach by
observing the similarity between the model's classification outputs across
multiple runs of EG-Booster.


http://arxiv.org/abs/2108.13952
Morphence: Moving Target Defense Against Adversarial Examples. (99%)
Abderrahmen Amich; Birhanu Eshete
  Robustness to adversarial examples of machine learning models remains an open
topic of research. Attacks often succeed by repeatedly probing a fixed target
model with adversarial examples purposely crafted to fool it. In this paper, we
introduce Morphence, an approach that shifts the defense landscape by making a
model a moving target against adversarial examples. By regularly moving the
decision function of a model, Morphence makes it significantly challenging for
repeated or correlated attacks to succeed. Morphence deploys a pool of models
generated from a base model in a manner that introduces sufficient randomness
when it responds to prediction queries. To ensure repeated or correlated
attacks fail, the deployed pool of models automatically expires after a query
budget is reached and the model pool is seamlessly replaced by a new model pool
generated in advance. We evaluate Morphence on two benchmark image
classification datasets (MNIST and CIFAR10) against five reference attacks (2
white-box and 3 black-box). In all cases, Morphence consistently outperforms
the thus-far effective defense, adversarial training, even in the face of
strong white-box attacks, while preserving accuracy on clean data.


http://arxiv.org/abs/2109.00124
DPA: Learning Robust Physical Adversarial Camouflages for Object Detectors. (86%)
Yexin Duan; Jialin Chen; Xingyu Zhou; Junhua Zou; Zhengyun He; Wu Zhang; Zhisong Pan
  Adversarial attacks are feasible in the real world for object detection.
However, most of the previous works have tried to learn "patches" applied to an
object to fool detectors, which become less effective or even ineffective in
squint view angles. To address this issue, we propose the Dense Proposals
Attack (DPA) to learn robust, physical and targeted adversarial camouflages for
detectors. The camouflages are robust because they remain adversarial when
filmed under arbitrary viewpoint and different illumination conditions,
physical because they function well both in the 3D virtual scene and the real
world, and targeted because they can cause detectors to misidentify an object
as a specific target class. In order to make the generated camouflages robust
in the physical world, we introduce a combination of viewpoint shifts, lighting
and other natural transformations to model the physical phenomena. In addition,
to improve the attacks, DPA substantially attacks all the classifications in
the fixed region proposals. Moreover, we build a virtual 3D scene using the
Unity simulation engine to fairly and reproducibly evaluate different physical
attacks. Extensive experiments demonstrate that DPA outperforms the
state-of-the-art methods significantly, and generalizes well to the real world,
posing a potential threat to the security-critical computer vision systems.


http://arxiv.org/abs/2109.01165
Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction. (83%)
Zhenrui Yue; Zhankui He; Huimin Zeng; Julian McAuley
  We investigate whether model extraction can be used to "steal" the weights of
sequential recommender systems, and the potential threats posed to victims of
such attacks. This type of risk has attracted attention in image and text
classification, but to our knowledge not in recommender systems. We argue that
sequential recommender systems are subject to unique vulnerabilities due to the
specific autoregressive regimes used to train them. Unlike many existing
recommender attackers, which assume the dataset used to train the victim model
is exposed to attackers, we consider a data-free setting, where training data
are not accessible. Under this setting, we propose an API-based model
extraction method via limited-budget synthetic data generation and knowledge
distillation. We investigate state-of-the-art models for sequential
recommendation and show their vulnerability under model extraction and
downstream attacks. We perform attacks in two stages. (1) Model extraction:
given different types of synthetic data and their labels retrieved from a
black-box recommender, we extract the black-box model to a white-box model via
distillation. (2) Downstream attacks: we attack the black-box model with
adversarial samples generated by the white-box recommender. Experiments show
the effectiveness of our data-free model extraction and downstream attacks on
sequential recommenders in both profile pollution and data poisoning settings.


http://arxiv.org/abs/2108.13617
Segmentation Fault: A Cheap Defense Against Adversarial Machine Learning. (75%)
Doha Al Bared; Mohamed Nassar
  Recently published attacks against deep neural networks (DNNs) have stressed
the importance of methodologies and tools to assess the security risks of using
this technology in critical systems. Efficient techniques for detecting
adversarial machine learning helps establishing trust and boost the adoption of
deep learning in sensitive and security systems. In this paper, we propose a
new technique for defending deep neural network classifiers, and convolutional
ones in particular. Our defense is cheap in the sense that it requires less
computation power despite a small cost to pay in terms of detection accuracy.
The work refers to a recently published technique called ML-LOO. We replace the
costly pixel by pixel leave-one-out approach of ML-LOO by adopting
coarse-grained leave-one-out. We evaluate and compare the efficiency of
different segmentation algorithms for this task. Our results show that a large
gain in efficiency is possible, even though penalized by a marginal decrease in
detection accuracy.


http://arxiv.org/abs/2108.13888
Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning. (4%)
Linyang Li; Demin Song; Xiaonan Li; Jiehang Zeng; Ruotian Ma; Xipeng Qiu
  \textbf{P}re-\textbf{T}rained \textbf{M}odel\textbf{s} have been widely
applied and recently proved vulnerable under backdoor attacks: the released
pre-trained weights can be maliciously poisoned with certain triggers. When the
triggers are activated, even the fine-tuned model will predict pre-defined
labels, causing a security threat. These backdoors generated by the poisoning
methods can be erased by changing hyper-parameters during fine-tuning or
detected by finding the triggers. In this paper, we propose a stronger
weight-poisoning attack method that introduces a layerwise weight poisoning
strategy to plant deeper backdoors; we also introduce a combinatorial trigger
that cannot be easily detected. The experiments on text classification tasks
show that previous defense methods cannot resist our weight-poisoning method,
which indicates that our method can be widely applied and may provide hints for
future model robustness studies.


http://arxiv.org/abs/2108.13797
Sample Efficient Detection and Classification of Adversarial Attacks via Self-Supervised Embeddings. (99%)
Mazda Moayeri; Soheil Feizi
  Adversarial robustness of deep models is pivotal in ensuring safe deployment
in real world settings, but most modern defenses have narrow scope and
expensive costs. In this paper, we propose a self-supervised method to detect
adversarial attacks and classify them to their respective threat models, based
on a linear model operating on the embeddings from a pre-trained
self-supervised encoder. We use a SimCLR encoder in our experiments, since we
show the SimCLR embedding distance is a good proxy for human perceptibility,
enabling it to encapsulate many threat models at once. We call our method
SimCat since it uses SimCLR encoder to catch and categorize various types of
adversarial attacks, including L_p and non-L_p evasion attacks, as well as data
poisonings. The simple nature of a linear classifier makes our method efficient
in both time and sample complexity. For example, on SVHN, using only five pairs
of clean and adversarial examples computed with a PGD-L_inf attack, SimCat's
detection accuracy is over 85%. Moreover, on ImageNet, using only 25 examples
from each threat model, SimCat can classify eight different attack types such
as PGD-L_2, PGD-L_inf, CW-L_2, PPGD, LPA, StAdv, ReColor, and JPEG-L_inf, with
over 40% accuracy. On STL10 data, we apply SimCat as a defense against
poisoning attacks, such as BP, CP, FC, CLBD, HTBD, halving the success rate
while using only twenty total poisons for training. We find that the detectors
generalize well to unseen threat models. Lastly, we investigate the performance
of our detection method under adaptive attacks and further boost its robustness
against such attacks via adversarial training.


http://arxiv.org/abs/2108.13093
Investigating Vulnerabilities of Deep Neural Policies. (99%)
Ezgi Korkmaz
  Reinforcement learning policies based on deep neural networks are vulnerable
to imperceptible adversarial perturbations to their inputs, in much the same
way as neural network image classifiers. Recent work has proposed several
methods to improve the robustness of deep reinforcement learning agents to
adversarial perturbations based on training in the presence of these
imperceptible perturbations (i.e. adversarial training). In this paper, we
study the effects of adversarial training on the neural policy learned by the
agent. In particular, we follow two distinct parallel approaches to investigate
the outcomes of adversarial training on deep neural policies based on
worst-case distributional shift and feature sensitivity. For the first
approach, we compare the Fourier spectrum of minimal perturbations computed for
both adversarially trained and vanilla trained neural policies. Via experiments
in the OpenAI Atari environments we show that minimal perturbations computed
for adversarially trained policies are more focused on lower frequencies in the
Fourier domain, indicating a higher sensitivity of these policies to low
frequency perturbations. For the second approach, we propose a novel method to
measure the feature sensitivities of deep neural policies and we compare these
feature sensitivity differences in state-of-the-art adversarially trained deep
neural policies and vanilla trained deep neural policies. We believe our
results can be an initial step towards understanding the relationship between
adversarial training and different notions of robustness for neural policies.


http://arxiv.org/abs/2108.13562
Adversarial Example Devastation and Detection on Speech Recognition System by Adding Random Noise. (99%)
Mingyu Dong; Diqun Yan; Yongkang Gong; Rangding Wang
  An automatic speech recognition (ASR) system based on a deep neural network
is vulnerable to attack by an adversarial example, especially if the
command-dependent ASR fails. A defense method against adversarial examples is
proposed to improve the robustness and security of the ASR system. We propose
an algorithm of devastation and detection on adversarial examples that can
attack current advanced ASR systems. We choose an advanced text- and
command-dependent ASR system as our target, generating adversarial examples by
an optimization-based attack on text-dependent ASR and the GA-based algorithm
on command-dependent ASR. The method is based on input transformation of
adversarial examples. Different random intensities and kinds of noise are added
to adversarial examples to devastate the perturbation previously added to
normal examples. Experimental results show that the method performs well. For
the devastation of examples, the original speech similarity after adding noise
can reach 99.68%, the similarity of adversarial examples can reach zero, and
the detection rate of adversarial examples can reach 94%.


http://arxiv.org/abs/2108.13049
Single Node Injection Attack against Graph Neural Networks. (68%)
Shuchang Tao; Qi Cao; Huawei Shen; Junjie Huang; Yunfan Wu; Xueqi Cheng
  Node injection attack on Graph Neural Networks (GNNs) is an emerging and
practical attack scenario that the attacker injects malicious nodes rather than
modifying original nodes or edges to affect the performance of GNNs. However,
existing node injection attacks ignore extremely limited scenarios, namely the
injected nodes might be excessive such that they may be perceptible to the
target GNN. In this paper, we focus on an extremely limited scenario of single
node injection evasion attack, i.e., the attacker is only allowed to inject one
single node during the test phase to hurt GNN's performance. The discreteness
of network structure and the coupling effect between network structure and node
features bring great challenges to this extremely limited scenario. We first
propose an optimization-based method to explore the performance upper bound of
single node injection evasion attack. Experimental results show that 100%,
98.60%, and 94.98% nodes on three public datasets are successfully attacked
even when only injecting one node with one edge, confirming the feasibility of
single node injection evasion attack. However, such an optimization-based
method needs to be re-optimized for each attack, which is computationally
unbearable. To solve the dilemma, we further propose a Generalizable Node
Injection Attack model, namely G-NIA, to improve the attack efficiency while
ensuring the attack performance. Experiments are conducted across three
well-known GNNs. Our proposed G-NIA significantly outperforms state-of-the-art
baselines and is 500 times faster than the optimization-based method when
inferring.


http://arxiv.org/abs/2108.13446
Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms. (41%)
Albert Jiménez Sanfiz; Mohamed Akrout
  Backpropagation is the default algorithm for training deep neural networks
due to its simplicity, efficiency and high convergence rate. However, its
requirements make it impossible to be implemented in a human brain. In recent
years, more biologically plausible learning methods have been proposed. Some of
these methods can match backpropagation accuracy, and simultaneously provide
other extra benefits such as faster training on specialized hardware (e.g.,
ASICs) or higher robustness against adversarial attacks. While the interest in
the field is growing, there is a necessity for open-source libraries and
toolkits to foster research and benchmark algorithms. In this paper, we present
BioTorch, a software framework to create, train, and benchmark biologically
motivated neural networks. In addition, we investigate the performance of
several feedback alignment methods proposed in the literature, thereby
unveiling the importance of the forward and backward weight initialization and
optimizer choice. Finally, we provide a novel robustness study of these methods
against state-of-the-art white and black-box adversarial attacks.


http://arxiv.org/abs/2108.13239
Adaptive perturbation adversarial training: based on reinforcement learning. (41%)
Zhishen Nie; Ying Lin; Sp Ren; Lan Zhang
  Adversarial training has become the primary method to defend against
adversarial samples. However, it is hard to practically apply due to many
shortcomings. One of the shortcomings of adversarial training is that it will
reduce the recognition accuracy of normal samples. Adaptive perturbation
adversarial training is proposed to alleviate this problem. It uses marginal
adversarial samples that are close to the decision boundary but does not cross
the decision boundary for adversarial training, which improves the accuracy of
model recognition while maintaining the robustness of the model. However,
searching for marginal adversarial samples brings additional computational
costs. This paper proposes a method for finding marginal adversarial samples
based on reinforcement learning, and combines it with the latest fast
adversarial training technology, which effectively speeds up training process
and reduces training costs.


http://arxiv.org/abs/2108.13602
How Does Adversarial Fine-Tuning Benefit BERT? (33%)
Javid Ebrahimi; Hao Yang; Wei Zhang
  Adversarial training (AT) is one of the most reliable methods for defending
against adversarial attacks in machine learning. Variants of this method have
been used as regularization mechanisms to achieve SOTA results on NLP
benchmarks, and they have been found to be useful for transfer learning and
continual learning. We search for the reasons for the effectiveness of AT by
contrasting vanilla and adversarially fine-tuned BERT models. We identify
partial preservation of BERT's syntactic abilities during fine-tuning as the
key to the success of AT. We observe that adversarially fine-tuned models
remain more faithful to BERT's language modeling behavior and are more
sensitive to the word order. As concrete examples of syntactic abilities, an
adversarially fine-tuned model could have an advantage of up to 38% on anaphora
agreement and up to 11% on dependency parsing. Our analysis demonstrates that
vanilla fine-tuning oversimplifies the sentence representation by focusing
heavily on a small subset of words. AT, however, moderates the effect of these
influential words and encourages representational diversity. This allows for a
more hierarchical representation of a sentence and leads to the mitigation of
BERT's loss of syntactic abilities.


http://arxiv.org/abs/2108.13373
ML-based IoT Malware Detection Under Adversarial Settings: A Systematic Evaluation. (26%)
Ahmed Abusnaina; Afsah Anwar; Sultan Alshamrani; Abdulrahman Alabduljabbar; RhongHo Jang; Daehun Nyang; David Mohaisen
  The rapid growth of the Internet of Things (IoT) devices is paralleled by
them being on the front-line of malicious attacks. This has led to an explosion
in the number of IoT malware, with continued mutations, evolution, and
sophistication. These malicious software are detected using machine learning
(ML) algorithms alongside the traditional signature-based methods. Although
ML-based detectors improve the detection performance, they are susceptible to
malware evolution and sophistication, making them limited to the patterns that
they have been trained upon. This continuous trend motivates the large body of
literature on malware analysis and detection research, with many systems
emerging constantly, and outperforming their predecessors. In this work, we
systematically examine the state-of-the-art malware detection approaches, that
utilize various representation and learning techniques, under a range of
adversarial settings. Our analyses highlight the instability of the proposed
detectors in learning patterns that distinguish the benign from the malicious
software. The results exhibit that software mutations with
functionality-preserving operations, such as stripping and padding,
significantly deteriorate the accuracy of such detectors. Additionally, our
analysis of the industry-standard malware detectors shows their instability to
the malware mutations.


http://arxiv.org/abs/2108.13140
DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation. (1%)
Lijie Wang; Hao Liu; Shuyuan Peng; Hongxuan Tang; Xinyan Xiao; Ying Chen; Hua Wu; Haifeng Wang
  While deep learning models have greatly improved the performance of most
artificial intelligence tasks, they are often criticized to be untrustworthy
due to the black-box problem. Consequently, many works have been proposed to
study the trustworthiness of deep learning. However, as most open datasets are
designed for evaluating the accuracy of model outputs, there is still a lack of
appropriate datasets for evaluating the inner workings of neural networks. The
lack of datasets obviously hinders the development of trustworthiness research.
Therefore, in order to systematically evaluate the factors for building
trustworthy systems, we propose a novel and well-annotated sentiment analysis
dataset to evaluate robustness and interpretability. To evaluate these factors,
our dataset contains diverse annotations about the challenging distribution of
instances, manual adversarial instances and sentiment explanations. Several
evaluation metrics are further proposed for interpretability and robustness.
Based on the dataset and metrics, we conduct comprehensive comparisons for the
trustworthiness of three typical models, and also study the relations between
accuracy, robustness and interpretability. We release this trustworthiness
evaluation dataset at \url{https://github/xyz} and hope our work can facilitate
the progress on building more trustworthy systems for real-world applications.


http://arxiv.org/abs/2108.12777
Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution. (99%)
Zongyi Li; Jianhan Xu; Jiehang Zeng; Linyang Li; Xiaoqing Zheng; Qi Zhang; Kai-Wei Chang; Cho-Jui Hsieh
  Recent studies have shown that deep neural networks are vulnerable to
intentionally crafted adversarial examples, and various methods have been
proposed to defend against adversarial word-substitution attacks for neural NLP
models. However, there is a lack of systematic study on comparing different
defense approaches under the same attacking setting. In this paper, we seek to
fill the gap of systematic studies through comprehensive researches on
understanding the behavior of neural text classifiers trained by various
defense methods under representative adversarial attacks. In addition, we
propose an effective method to further improve the robustness of neural text
classifiers against such attacks and achieved the highest accuracy on both
clean and adversarial examples on AGNEWS and IMDB datasets by a significant
margin.


http://arxiv.org/abs/2108.13872
Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models. (98%)
Zeyuan Wang; Chaofeng Sha; Su Yang
  We explore the black-box adversarial attack on video recognition models.
Attacks are only performed on selected key regions and key frames to reduce the
high computation cost of searching adversarial perturbations on a video due to
its high dimensionality. To select key frames, one way is to use heuristic
algorithms to evaluate the importance of each frame and choose the essential
ones. However, it is time inefficient on sorting and searching. In order to
speed up the attack process, we propose a reinforcement learning based frame
selection strategy. Specifically, the agent explores the difference between the
original class and the target class of videos to make selection decisions. It
receives rewards from threat models which indicate the quality of the
decisions. Besides, we also use saliency detection to select key regions and
only estimate the sign of gradient instead of the gradient itself in zeroth
order optimization to further boost the attack process. We can use the trained
model directly in the untargeted attack or with little fine-tune in the
targeted attack, which saves computation time. A range of empirical results on
real datasets demonstrate the effectiveness and efficiency of the proposed
method.


http://arxiv.org/abs/2108.12805
DropAttack: A Masked Weight Adversarial Training Method to Improve Generalization of Neural Networks. (82%)
Shiwen Ni; Jiawen Li; Hung-Yu Kao
  Adversarial training has been proven to be a powerful regularization method
to improve the generalization of models. However, current adversarial training
methods only attack the original input sample or the embedding vectors, and
their attacks lack coverage and diversity. To further enhance the breadth and
depth of attack, we propose a novel masked weight adversarial training method
called DropAttack, which enhances generalization of model by adding
intentionally worst-case adversarial perturbations to both the input and hidden
layers in different dimensions and minimize the adversarial risks generated by
each layer. DropAttack is a general technique and can be adopt to a wide
variety of neural networks with different architectures. To validate the
effectiveness of the proposed method, we used five public datasets in the
fields of natural language processing (NLP) and computer vision (CV) for
experimental evaluating. We compare the proposed method with other adversarial
training methods and regularization methods, and our method achieves
state-of-the-art on all datasets. In addition, Dropattack can achieve the same
performance when it use only a half training data compared to other standard
training method. Theoretical analysis reveals that DropAttack can perform
gradient regularization at random on some of the input and wight parameters of
the model. Further visualization experiments show that DropAttack can push the
minimum risk of the model to a lower and flatter loss landscapes. Our source
code is publicly available on https://github.com/nishiwen1214/DropAttack.


http://arxiv.org/abs/2110.00425
Rumor Detection on Social Media with Hierarchical Adversarial Training. (47%)
Shiwen Ni; Jiawen Li; Hung-Yu Kao
  The proliferation of rumors on social media has a huge impact on society.
However, natural language text is high-dimensional and sparse, and the same
rumor may be expressed in hundreds of ways on social media. As such, the
robustness and generalization of the current rumor detection model are put into
question. We propose a new hierarchical model called HAT-RD, which is divided
into two categories: post-level modules and event-level modules. HAT-RD adopts
a novel hierarchical adversarial training method based on gradient ascent by
adding adversarial perturbations to the embedding layers both of post-level
modules and event-level modules to deceive the detector. At the same time, the
detector uses stochastic gradient descent to minimize the adversarial risk to
learn a more robust model. In this way, the post-level and event-level sample
spaces are enhanced, and experiments indicate that the model drift into an area
with a flat loss landscape that leads to better generalization. Experiments on
two real-world datasets demonstrate that our model achieves better results than
state-of-the-art methods.


http://arxiv.org/abs/2108.12473
Mal2GCN: A Robust Malware Detection Approach Using Deep Graph Convolutional Networks With Non-Negative Weights. (99%)
Omid Kargarnovin; Amir Mahdi Sadeghzadeh; Rasool Jalili
  With the growing pace of using machine learning to solve various problems,
securing these models against adversaries has become one of the main concerns
of researchers. Recent studies have shown that in an adversarial environment,
machine learning models are vulnerable to adversarial examples, and adversaries
can create carefully crafted inputs to fool the models. With the advent of deep
neural networks, many researchers have used deep neural networks for various
tasks, and have achieved impressive results. These models must become robust
against attacks before being deployed safely, especially in security-related
fields such as malware detection. In this paper, we first present a black-box
source code-based adversarial malware generation approach that can be used to
evaluate the robustness of malware detection models against real-world
adversaries. The proposed approach injects adversarial codes into the various
locations of malware source codes to evade malware detection models. We then
propose Mal2GCN, a robust malware detection model. Mal2GCN uses the
representation power of graph convolutional networks combined with the
non-negative weights training method to create a malware detection model with
high detection accuracy, which is also robust against adversarial attacks that
add benign features to the input.


http://arxiv.org/abs/2108.12492
Disrupting Adversarial Transferability in Deep Neural Networks. (98%)
Christopher Wiedeman; Ge Wang
  Adversarial attack transferability is a well-recognized phenomenon in deep
learning. Prior work has partially explained transferability by recognizing
common adversarial subspaces and correlations between decision boundaries, but
we have found little explanation in the literature beyond this. In this paper,
we propose that transferability between seemingly different models is due to a
high linear correlation between features that different deep neural networks
extract. In other words, two models trained on the same task that are seemingly
distant in the parameter space likely extract features in the same fashion,
just with trivial shifts and rotations between the latent spaces. Furthermore,
we show how applying a feature correlation loss, which decorrelates the
extracted features in a latent space, can drastically reduce the
transferability of adversarial attacks between models, suggesting that the
models complete tasks in semantically different ways. Finally, we propose a
Dual Neck Autoencoder (DNA), which leverages this feature correlation loss to
create two meaningfully different encodings of input information with reduced
transferability.


http://arxiv.org/abs/2108.12237
Evaluating the Robustness of Neural Language Models to Input Perturbations. (16%)
Milad Moradi; Matthias Samwald
  High-performance neural language models have obtained state-of-the-art
results on a wide range of Natural Language Processing (NLP) tasks. However,
results for common benchmark datasets often do not reflect model reliability
and robustness when applied to noisy, real-world data. In this study, we design
and implement various types of character-level and word-level perturbation
methods to simulate realistic scenarios in which input texts may be slightly
noisy or different from the data distribution on which NLP systems were
trained. Conducting comprehensive experiments on different NLP tasks, we
investigate the ability of high-performance language models such as BERT,
XLNet, RoBERTa, and ELMo in handling different types of input perturbations.
The results suggest that language models are sensitive to input perturbations
and their performance can decrease even when small changes are introduced. We
highlight that models need to be further improved and that current benchmarks
are not reflecting model robustness well. We argue that evaluations on
perturbed inputs should routinely complement widely-used benchmarks in order to
yield a more realistic understanding of NLP systems robustness.


http://arxiv.org/abs/2108.12242
Deep learning models are not robust against noise in clinical text. (1%)
Milad Moradi; Kathrin Blagec; Matthias Samwald
  Artificial Intelligence (AI) systems are attracting increasing interest in
the medical domain due to their ability to learn complicated tasks that require
human intelligence and expert knowledge. AI systems that utilize
high-performance Natural Language Processing (NLP) models have achieved
state-of-the-art results on a wide variety of clinical text processing
benchmarks. They have even outperformed human accuracy on some tasks. However,
performance evaluation of such AI systems have been limited to accuracy
measures on curated and clean benchmark datasets that may not properly reflect
how robustly these systems can operate in real-world situations. In order to
address this challenge, we introduce and implement a wide variety of
perturbation methods that simulate different types of noise and variability in
clinical text data. While noisy samples produced by these perturbation methods
can often be understood by humans, they may cause AI systems to make erroneous
decisions. Conducting extensive experiments on several clinical text processing
tasks, we evaluated the robustness of high-performance NLP models against
various types of character-level and word-level noise. The results revealed
that the NLP models performance degrades when the input contains small amounts
of noise. This study is a significant step towards exposing vulnerabilities of
AI models utilized in clinical text processing systems. The proposed
perturbation methods can be used in performance evaluation tests to assess how
robustly clinical NLP models can operate on noisy data, in real-world settings.


http://arxiv.org/abs/2108.12001
Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks. (99%)
Landan Seguin; Anthony Ndirango; Neeli Mishra; SueYeon Chung; Tyler Lee
  Adversarial defenses train deep neural networks to be invariant to the input
perturbations from adversarial attacks. Almost all defense strategies achieve
this invariance through adversarial training i.e. training on inputs with
adversarial perturbations. Although adversarial training is successful at
mitigating adversarial attacks, the behavioral differences between
adversarially-trained (AT) models and standard models are still poorly
understood. Motivated by a recent study on learning robustness without input
perturbations by distilling an AT model, we explore what is learned during
adversarial training by analyzing the distribution of logits in AT models. We
identify three logit characteristics essential to learning adversarial
robustness. First, we provide a theoretical justification for the finding that
adversarial training shrinks two important characteristics of the logit
distribution: the max logit values and the "logit gaps" (difference between the
logit max and next largest values) are on average lower for AT models. Second,
we show that AT and standard models differ significantly on which samples are
high or low confidence, then illustrate clear qualitative differences by
visualizing samples with the largest confidence difference. Finally, we find
learning information about incorrect classes to be essential to learning
robustness by manipulating the non-max logit information during distillation
and measuring the impact on the student's robustness. Our results indicate that
learning some adversarial robustness without input perturbations requires a
model to learn specific sample-wise confidences and incorrect class orderings
that follow complex distributions.


http://arxiv.org/abs/2108.11785
A Hierarchical Assessment of Adversarial Severity. (98%)
Guillaume Jeanneret; Juan C Perez; Pablo Arbelaez
  Adversarial Robustness is a growing field that evidences the brittleness of
neural networks. Although the literature on adversarial robustness is vast, a
dimension is missing in these studies: assessing how severe the mistakes are.
We call this notion "Adversarial Severity" since it quantifies the downstream
impact of adversarial corruptions by computing the semantic error between the
misclassification and the proper label. We propose to study the effects of
adversarial noise by measuring the Robustness and Severity into a large-scale
dataset: iNaturalist-H. Our contributions are: (i) we introduce novel
Hierarchical Attacks that harness the rich structured space of labels to create
adversarial examples. (ii) These attacks allow us to benchmark the Adversarial
Robustness and Severity of classification models. (iii) We enhance the
traditional adversarial training with a simple yet effective Hierarchical
Curriculum Training to learn these nodes gradually within the hierarchical
tree. We perform extensive experiments showing that hierarchical defenses allow
deep models to boost the adversarial Robustness by 1.85% and reduce the
severity of all attacks by 0.17, on average.


http://arxiv.org/abs/2108.11765
Physical Adversarial Attacks on an Aerial Imagery Object Detector. (96%)
Andrew Du; Bo Chen; Tat-Jun Chin; Yee Wei Law; Michele Sasdelli; Ramesh Rajasegaran; Dillon Campbell
  Deep neural networks (DNNs) have become essential for processing the vast
amounts of aerial imagery collected using earth-observing satellite platforms.
However, DNNs are vulnerable towards adversarial examples, and it is expected
that this weakness also plagues DNNs for aerial imagery. In this work, we
demonstrate one of the first efforts on physical adversarial attacks on aerial
imagery, whereby adversarial patches were optimised, fabricated and installed
on or near target objects (cars) to significantly reduce the efficacy of an
object detector applied on overhead images. Physical adversarial attacks on
aerial images, particularly those captured from satellite platforms, are
challenged by atmospheric factors (lighting, weather, seasons) and the distance
between the observer and target. To investigate the effects of these
challenges, we devised novel experiments and metrics to evaluate the efficacy
of physical adversarial attacks against object detectors in aerial scenes. Our
results indicate the palpable threat posed by physical adversarial attacks
towards DNNs for processing satellite imagery.


http://arxiv.org/abs/2108.11673
Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference. (80%)
Yang Zheng; Xiaoyi Feng; Zhaoqiang Xia; Xiaoyue Jiang; Ambra Demontis; Maura Pintor; Battista Biggio; Fabio Roli
  Adversarial reprogramming allows repurposing a machine-learning model to
perform a different task. For example, a model trained to recognize animals can
be reprogrammed to recognize digits by embedding an adversarial program in the
digit images provided as input. Recent work has shown that adversarial
reprogramming may not only be used to abuse machine-learning models provided as
a service, but also beneficially, to improve transfer learning when training
data is scarce. However, the factors affecting its success are still largely
unexplained. In this work, we develop a first-order linear model of adversarial
reprogramming to show that its success inherently depends on the size of the
average input gradient, which grows when input gradients are more aligned, and
when inputs have higher dimensionality. The results of our experimental
analysis, involving fourteen distinct reprogramming tasks, show that the above
factors are correlated with the success and the failure of adversarial
reprogramming.


http://arxiv.org/abs/2108.12081
Detection and Continual Learning of Novel Face Presentation Attacks. (2%)
Mohammad Rostami; Leonidas Spinoulas; Mohamed Hussein; Joe Mathai; Wael Abd-Almageed
  Advances in deep learning, combined with availability of large datasets, have
led to impressive improvements in face presentation attack detection research.
However, state-of-the-art face antispoofing systems are still vulnerable to
novel types of attacks that are never seen during training. Moreover, even if
such attacks are correctly detected, these systems lack the ability to adapt to
newly encountered attacks. The post-training ability of continually detecting
new types of attacks and self-adaptation to identify these attack types, after
the initial detection phase, is highly appealing. In this paper, we enable a
deep neural network to detect anomalies in the observed input data points as
potential new types of attacks by suppressing the confidence-level of the
network outside the training samples' distribution. We then use experience
replay to update the model to incorporate knowledge about new types of attacks
without forgetting the past learned attack types. Experimental results are
provided to demonstrate the effectiveness of the proposed method on two
benchmark datasets as well as a newly introduced dataset which exhibits a large
variety of attack types.


http://arxiv.org/abs/2108.11168
Adversarially Robust One-class Novelty Detection. (99%)
Shao-Yuan Lo; Poojan Oza; Vishal M. Patel
  One-class novelty detectors are trained with examples of a particular class
and are tasked with identifying whether a query example belongs to the same
known class. Most recent advances adopt a deep auto-encoder style architecture
to compute novelty scores for detecting novel class data. Deep networks have
shown to be vulnerable to adversarial attacks, yet little focus is devoted to
studying the adversarial robustness of deep novelty detectors. In this paper,
we first show that existing novelty detectors are susceptible to adversarial
examples. We further demonstrate that commonly-used defense approaches for
classification tasks have limited effectiveness in one-class novelty detection.
Hence, we need a defense specifically designed for novelty detection. To this
end, we propose a defense strategy that manipulates the latent space of novelty
detectors to improve the robustness against adversarial examples. The proposed
method, referred to as Principal Latent Space (PLS), learns the
incrementally-trained cascade principal components in the latent space to
robustify novelty detectors. PLS can purify latent space against adversarial
examples and constrain latent space to exclusively model the known class
distribution. We conduct extensive experiments on multiple attacks, datasets
and novelty detectors, showing that PLS consistently enhances the adversarial
robustness of novelty detection models.


http://arxiv.org/abs/2108.11299
Backdoor Attacks on Network Certification via Data Poisoning. (98%)
Tobias Lorenz; Marta Kwiatkowska; Mario Fritz
  Certifiers for neural networks have made great progress towards provable
robustness guarantees against evasion attacks using adversarial examples.
However, introducing certifiers into deep learning systems also opens up new
attack vectors, which need to be considered before deployment. In this work, we
conduct the first systematic analysis of training time attacks against
certifiers in practical application pipelines, identifying new threat vectors
that can be exploited to degrade the overall system. Using these insights, we
design two backdoor attacks against network certifiers, which can drastically
reduce certified robustness when the backdoor is activated. For example, adding
1% poisoned data points during training is sufficient to reduce certified
robustness by up to 95 percentage points, effectively rendering the certifier
useless. We analyze how such novel attacks can compromise the overall system's
integrity or availability. Our extensive experiments across multiple datasets,
model architectures, and certifiers demonstrate the wide applicability of these
attacks. A first investigation into potential defenses shows that current
approaches only partially mitigate the issue, highlighting the need for new,
more specific solutions.


http://arxiv.org/abs/2108.11135
Bridged Adversarial Training. (93%)
Hoki Kim; Woojin Lee; Sungyoon Lee; Jaewook Lee
  Adversarial robustness is considered as a required property of deep neural
networks. In this study, we discover that adversarially trained models might
have significantly different characteristics in terms of margin and smoothness,
even they show similar robustness. Inspired by the observation, we investigate
the effect of different regularizers and discover the negative effect of the
smoothness regularizer on maximizing the margin. Based on the analyses, we
propose a new method called bridged adversarial training that mitigates the
negative effect by bridging the gap between clean and adversarial examples. We
provide theoretical and empirical evidence that the proposed method provides
stable and better robustness, especially for large perturbations.


http://arxiv.org/abs/2108.11505
Generalized Real-World Super-Resolution through Adversarial Robustness. (93%)
Angela Castillo; María Escobar; Juan C. Pérez; Andrés Romero; Radu Timofte; Gool Luc Van; Pablo Arbeláez
  Real-world Super-Resolution (SR) has been traditionally tackled by first
learning a specific degradation model that resembles the noise and corruption
artifacts in low-resolution imagery. Thus, current methods lack generalization
and lose their accuracy when tested on unseen types of corruption. In contrast
to the traditional proposal, we present Robust Super-Resolution (RSR), a method
that leverages the generalization capability of adversarial attacks to tackle
real-world SR. Our novel framework poses a paradigm shift in the development of
real-world SR methods. Instead of learning a dataset-specific degradation, we
employ adversarial attacks to create difficult examples that target the model's
weaknesses. Afterward, we use these adversarial examples during training to
improve our model's capacity to process noisy inputs. We perform extensive
experimentation on synthetic and real-world images and empirically demonstrate
that our RSR method generalizes well across datasets without re-training for
specific noise priors. By using a single robust model, we outperform
state-of-the-art specialized methods on real-world benchmarks.


http://arxiv.org/abs/2108.11032
Improving Visual Quality of Unrestricted Adversarial Examples with Wavelet-VAE. (99%)
Wenzhao Xiang; Chang Liu; Shibao Zheng
  Traditional adversarial examples are typically generated by adding
perturbation noise to the input image within a small matrix norm. In practice,
un-restricted adversarial attack has raised great concern and presented a new
threat to the AI safety. In this paper, we propose a wavelet-VAE structure to
reconstruct an input image and generate adversarial examples by modifying the
latent code. Different from perturbation-based attack, the modifications of the
proposed method are not limited but imperceptible to human eyes. Experiments
show that our method can generate high quality adversarial examples on ImageNet
dataset.


http://arxiv.org/abs/2108.10879
Are socially-aware trajectory prediction models really socially-aware? (92%)
Saeed Saadatnejad; Mohammadhossein Bahari; Pedram Khorsandi; Mohammad Saneian; Seyed-Mohsen Moosavi-Dezfooli; Alexandre Alahi
  Our field has recently witnessed an arms race of neural network-based
trajectory predictors. While these predictors are at the core of many
applications such as autonomous navigation or pedestrian flow simulations,
their adversarial robustness has not been carefully studied. In this paper, we
introduce a socially-attended attack to assess the social understanding of
prediction models in terms of collision avoidance. An attack is a small yet
carefully-crafted perturbations to fail predictors. Technically, we define
collision as a failure mode of the output, and propose hard- and soft-attention
mechanisms to guide our attack. Thanks to our attack, we shed light on the
limitations of the current models in terms of their social understanding. We
demonstrate the strengths of our method on the recent trajectory prediction
models. Finally, we show that our attack can be employed to increase the social
understanding of state-of-the-art models. The code is available online:
https://s-attack.github.io/


http://arxiv.org/abs/2108.10992
OOWL500: Overcoming Dataset Collection Bias in the Wild. (76%)
Brandon Leung; Chih-Hui Ho; Amir Persekian; David Orozco; Yen Chang; Erik Sandstrom; Bo Liu; Nuno Vasconcelos
  The hypothesis that image datasets gathered online "in the wild" can produce
biased object recognizers, e.g. preferring professional photography or certain
viewing angles, is studied. A new "in the lab" data collection infrastructure
is proposed consisting of a drone which captures images as it circles around
objects. Crucially, the control provided by this setup and the natural camera
shake inherent to flight mitigate many biases. It's inexpensive and easily
replicable nature may also potentially lead to a scalable data collection
effort by the vision community. The procedure's usefulness is demonstrated by
creating a dataset of Objects Obtained With fLight (OOWL). Denoted as OOWL500,
it contains 120,000 images of 500 objects and is the largest "in the lab" image
dataset available when both number of classes and objects per class are
considered. Furthermore, it has enabled several of new insights on object
recognition. First, a novel adversarial attack strategy is proposed, where
image perturbations are defined in terms of semantic properties such as camera
shake and pose. Indeed, experiments have shown that ImageNet has considerable
amounts of pose and professional photography bias. Second, it is used to show
that the augmentation of in the wild datasets, such as ImageNet, with in the
lab data, such as OOWL500, can significantly decrease these biases, leading to
object recognizers of improved generalization. Third, the dataset is used to
study questions on "best procedures" for dataset collection. It is revealed
that data augmentation with synthetic images does not suffice to eliminate in
the wild datasets biases, and that camera shake and pose diversity play a more
important role in object recognition robustness than previously thought.


http://arxiv.org/abs/2108.10549
StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures. (1%)
Sanghyuk Chun; Song Park
  Recent powerful vision classifiers are biased towards textures, while shape
information is overlooked by the models. A simple attempt by augmenting
training images using the artistic style transfer method, called Stylized
ImageNet, can reduce the texture bias. However, Stylized ImageNet approach has
two drawbacks in fidelity and diversity. First, the generated images show low
image quality due to the significant semantic gap betweeen natural images and
artistic paintings. Also, Stylized ImageNet training samples are pre-computed
before training, resulting in showing the lack of diversity for each sample. We
propose a StyleAugment by augmenting styles from the mini-batch. StyleAugment
does not rely on the pre-defined style references, but generates augmented
images on-the-fly by natural images in the mini-batch for the references.
Hence, StyleAugment let the model observe abundant confounding cues for each
image by on-the-fly the augmentation strategy, while the augmented images are
more realistic than artistic style transferred images. We validate the
effectiveness of StyleAugment in the ImageNet dataset with robustness
benchmarks, such as texture de-biased accuracy, corruption robustness, natural
adversarial samples, and occlusion robustness. StyleAugment shows better
generalization performances than previous unsupervised de-biasing methods and
state-of-the-art data augmentation methods in our experiments.


http://arxiv.org/abs/2108.10451
Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications. (99%)
Wenjie Ruan; Xinping Yi; Xiaowei Huang
  This tutorial aims to introduce the fundamentals of adversarial robustness of
deep learning, presenting a well-structured review of up-to-date techniques to
assess the vulnerability of various types of deep learning models to
adversarial examples. This tutorial will particularly highlight
state-of-the-art techniques in adversarial attacks and robustness verification
of deep neural networks (DNNs). We will also introduce some effective
countermeasures to improve the robustness of deep learning models, with a
particular focus on adversarial training. We aim to provide a comprehensive
overall picture about this emerging direction and enable the community to be
aware of the urgency and importance of designing robust deep learning models in
safety-critical data analytical applications, ultimately enabling the end-users
to trust deep learning classifiers. We will also summarize potential research
directions concerning the adversarial robustness of deep learning, and its
potential benefits to enable accountable and trustworthy deep learning-based
data analytical systems and applications.


http://arxiv.org/abs/2108.10015
Semantic-Preserving Adversarial Text Attacks. (99%)
Xinghao Yang; Weifeng Liu; James Bailey; Tianqing Zhu; Dacheng Tao; Wei Liu
  Deep neural networks (DNNs) are known to be vulnerable to adversarial images,
while their robustness in text classification is rarely studied. Several lines
of text attack methods have been proposed in the literature, including
character-level, word-level, and sentence-level attacks. However, it is still a
challenge to minimize the number of word changes necessary to induce
misclassification, while simultaneously ensuring lexical correctness, syntactic
soundness, and semantic similarity. In this paper, we propose a Bigram and
Unigram based adaptive Semantic Preservation Optimization (BU-SPO) method to
examine the vulnerability of deep models. Our method has four major merits.
Firstly, we propose to attack text documents not only at the unigram word level
but also at the bigram level which better keeps semantics and avoids producing
meaningless outputs. Secondly, we propose a hybrid method to replace the input
words with options among both their synonyms candidates and sememe candidates,
which greatly enriches the potential substitutions compared to only using
synonyms. Thirdly, we design an optimization algorithm, i.e., Semantic
Preservation Optimization (SPO), to determine the priority of word
replacements, aiming to reduce the modification cost. Finally, we further
improve the SPO with a semantic Filter (named SPOF) to find the adversarial
example with the highest semantic similarity. We evaluate the effectiveness of
our BU-SPO and BU-SPOF on IMDB, AG's News, and Yahoo! Answers text datasets by
attacking four popular DNNs models. Results show that our methods achieve the
highest attack success rates and semantics rates by changing the smallest
number of words compared with existing methods.


http://arxiv.org/abs/2108.10217
Deep Bayesian Image Set Classification: A Defence Approach against Adversarial Attacks. (99%)
Nima Mirnateghi; Syed Afaq Ali Shah; Mohammed Bennamoun
  Deep learning has become an integral part of various computer vision systems
in recent years due to its outstanding achievements for object recognition,
facial recognition, and scene understanding. However, deep neural networks
(DNNs) are susceptible to be fooled with nearly high confidence by an
adversary. In practice, the vulnerability of deep learning systems against
carefully perturbed images, known as adversarial examples, poses a dire
security threat in the physical world applications. To address this phenomenon,
we present, what to our knowledge, is the first ever image set based
adversarial defence approach. Image set classification has shown an exceptional
performance for object and face recognition, owing to its intrinsic property of
handling appearance variability. We propose a robust deep Bayesian image set
classification as a defence framework against a broad range of adversarial
attacks. We extensively experiment the performance of the proposed technique
with several voting strategies. We further analyse the effects of image size,
perturbation magnitude, along with the ratio of perturbed images in each image
set. We also evaluate our technique with the recent state-of-the-art defence
methods, and single-shot recognition task. The empirical results demonstrate
superior performance on CIFAR-10, MNIST, ETH-80, and Tiny ImageNet datasets.


http://arxiv.org/abs/2108.10251
Kryptonite: An Adversarial Attack Using Regional Focus. (99%)
Yogesh Kulkarni; Krisha Bhambani
  With the Rise of Adversarial Machine Learning and increasingly robust
adversarial attacks, the security of applications utilizing the power of
Machine Learning has been questioned. Over the past few years, applications of
Deep Learning using Deep Neural Networks(DNN) in several fields including
Medical Diagnosis, Security Systems, Virtual Assistants, etc. have become
extremely commonplace, and hence become more exposed and susceptible to attack.
In this paper, we present a novel study analyzing the weaknesses in the
security of deep learning systems. We propose 'Kryptonite', an adversarial
attack on images. We explicitly extract the Region of Interest (RoI) for the
images and use it to add imperceptible adversarial perturbations to images to
fool the DNN. We test our attack on several DNN's and compare our results with
state of the art adversarial attacks like Fast Gradient Sign Method (FGSM),
DeepFool (DF), Momentum Iterative Fast Gradient Sign Method (MIFGSM), and
Projected Gradient Descent (PGD). The results obtained by us cause a maximum
drop in network accuracy while yielding minimum possible perturbation and in
considerably less amount of time per sample. We thoroughly evaluate our attack
against three adversarial defence techniques and the promising results showcase
the efficacy of our attack.


http://arxiv.org/abs/2108.10241
Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Federated Learning. (54%)
Virat Shejwalkar; Amir Houmansadr; Peter Kairouz; Daniel Ramage
  While recent works have indicated that federated learning (FL) is vulnerable
to poisoning attacks by compromised clients, we show that these works make a
number of unrealistic assumptions and arrive at somewhat misleading
conclusions. For instance, they often use impractically high percentages of
compromised clients or assume unrealistic capabilities for the adversary. We
perform the first critical analysis of poisoning attacks under practical
production FL environments by carefully characterizing the set of realistic
threat models and adversarial capabilities. Our findings are rather surprising:
contrary to the established belief, we show that FL, even without any defenses,
is highly robust in practice. In fact, we go even further and propose novel,
state-of-the-art poisoning attacks under two realistic threat models, and show
via an extensive set of experiments across three benchmark datasets how
(in)effective poisoning attacks are, especially when simple defense mechanisms
are used. We correct previous misconceptions and give concrete guidelines that
we hope will encourage our community to conduct more accurate research in this
space and build stronger (and more realistic) attacks and defenses.


http://arxiv.org/abs/2108.09929
SegMix: Co-occurrence Driven Mixup for Semantic Segmentation and Adversarial Robustness. (4%)
Md Amirul Islam; Matthew Kowal; Konstantinos G. Derpanis; Neil D. B. Bruce
  In this paper, we present a strategy for training convolutional neural
networks to effectively resolve interference arising from competing hypotheses
relating to inter-categorical information throughout the network. The premise
is based on the notion of feature binding, which is defined as the process by
which activations spread across space and layers in the network are
successfully integrated to arrive at a correct inference decision. In our work,
this is accomplished for the task of dense image labelling by blending images
based on (i) categorical clustering or (ii) the co-occurrence likelihood of
categories. We then train a feature binding network which simultaneously
segments and separates the blended images. Subsequent feature denoising to
suppress noisy activations reveals additional desirable properties and high
degrees of successful predictions. Through this process, we reveal a general
mechanism, distinct from any prior methods, for boosting the performance of the
base segmentation and saliency network while simultaneously increasing
robustness to adversarial attacks.


http://arxiv.org/abs/2108.09713
Robustness-via-Synthesis: Robust Training with Generative Adversarial Perturbations. (99%)
Inci M. Baytas; Debayan Deb
  Upon the discovery of adversarial attacks, robust models have become
obligatory for deep learning-based systems. Adversarial training with
first-order attacks has been one of the most effective defenses against
adversarial perturbations to this day. The majority of the adversarial training
approaches focus on iteratively perturbing each pixel with the gradient of the
loss function with respect to the input image. However, the adversarial
training with gradient-based attacks lacks diversity and does not generalize
well to natural images and various attacks. This study presents a robust
training algorithm where the adversarial perturbations are automatically
synthesized from a random vector using a generator network. The classifier is
trained with cross-entropy loss regularized with the optimal transport distance
between the representations of the natural and synthesized adversarial samples.
Unlike prevailing generative defenses, the proposed one-step attack generation
framework synthesizes diverse perturbations without utilizing gradient of the
classifier's loss. Experimental results show that the proposed approach attains
comparable robustness with various gradient-based and generative robust
training techniques on CIFAR10, CIFAR100, and SVHN datasets. In addition,
compared to the baselines, the proposed robust training framework generalizes
well to the natural samples. Code and trained models will be made publicly
available.


http://arxiv.org/abs/2108.09891
Multi-Expert Adversarial Attack Detection in Person Re-identification Using Context Inconsistency. (92%)
Xueping Wang; Shasha Li; Min Liu; Yaonan Wang; Amit K. Roy-Chowdhury
  The success of deep neural networks (DNNs) haspromoted the widespread
applications of person re-identification (ReID). However, ReID systems inherit
thevulnerability of DNNs to malicious attacks of visually in-conspicuous
adversarial perturbations. Detection of adver-sarial attacks is, therefore, a
fundamental requirement forrobust ReID systems. In this work, we propose a
Multi-Expert Adversarial Attack Detection (MEAAD) approach toachieve this goal
by checking context inconsistency, whichis suitable for any DNN-based ReID
systems. Specifically,three kinds of context inconsistencies caused by
adversar-ial attacks are employed to learn a detector for distinguish-ing the
perturbed examples, i.e., a) the embedding distancesbetween a perturbed query
person image and its top-K re-trievals are generally larger than those between
a benignquery image and its top-K retrievals, b) the embedding dis-tances among
the top-K retrievals of a perturbed query im-age are larger than those of a
benign query image, c) thetop-K retrievals of a benign query image obtained
with mul-tiple expert ReID models tend to be consistent, which isnot preserved
when attacks are present. Extensive exper-iments on the Market1501 and
DukeMTMC-ReID datasetsshow that, as the first adversarial attack detection
approachfor ReID,MEAADeffectively detects various adversarial at-tacks and
achieves high ROC-AUC (over 97.5%).


http://arxiv.org/abs/2108.09768
Relating CNNs with brain: Challenges and findings. (10%)
Reem Abdel-Salam
  Conventional neural network models (CNN), loosely inspired by the primate
visual system, have been shown to predict neural responses in the visual
cortex. However, the relationship between CNNs and the visual system is
incomplete due to many reasons. On one hand state of the art CNN architecture
is very complex, yet can be fooled by imperceptibly small, explicitly crafted
perturbations which makes it hard difficult to map layers of the network with
the visual system and to understand what they are doing. On the other hand, we
don't know the exact mapping between feature space of the CNNs and the space
domain of the visual cortex, which makes it hard to accurately predict neural
responses. In this paper we review the challenges and the methods that have
been used to predict neural responses in the visual cortex and whole brain as
part of The Algonauts Project 2021 Challenge: "How the Human Brain Makes Sense
of a World in Motion".


http://arxiv.org/abs/2108.09513
A Hard Label Black-box Adversarial Attack Against Graph Neural Networks. (99%)
Jiaming Mu; Binghui Wang; Qi Li; Kun Sun; Mingwei Xu; Zhuotao Liu
  Graph Neural Networks (GNNs) have achieved state-of-the-art performance in
various graph structure related tasks such as node classification and graph
classification. However, GNNs are vulnerable to adversarial attacks. Existing
works mainly focus on attacking GNNs for node classification; nevertheless, the
attacks against GNNs for graph classification have not been well explored.
  In this work, we conduct a systematic study on adversarial attacks against
GNNs for graph classification via perturbing the graph structure. In
particular, we focus on the most challenging attack, i.e., hard label black-box
attack, where an attacker has no knowledge about the target GNN model and can
only obtain predicted labels through querying the target model.To achieve this
goal, we formulate our attack as an optimization problem, whose objective is to
minimize the number of edges to be perturbed in a graph while maintaining the
high attack success rate. The original optimization problem is intractable to
solve, and we relax the optimization problem to be a tractable one, which is
solved with theoretical convergence guarantee. We also design a coarse-grained
searching algorithm and a query-efficient gradient computation algorithm to
decrease the number of queries to the target GNN model. Our experimental
results on three real-world datasets demonstrate that our attack can
effectively attack representative GNNs for graph classification with less
queries and perturbations. We also evaluate the effectiveness of our attack
under two defenses: one is well-designed adversarial graph detector and the
other is that the target GNN model itself is equipped with a defense to prevent
adversarial graph generation. Our experimental results show that such defenses
are not effective enough, which highlights more advanced defenses.


http://arxiv.org/abs/2108.09454
"Adversarial Examples" for Proof-of-Learning. (98%)
Rui Zhang; Jian Liu; Yuan Ding; Qingbiao Wu; Kui Ren
  In S&P '21, Jia et al. proposed a new concept/mechanism named
proof-of-learning (PoL), which allows a prover to demonstrate ownership of a
machine learning model by proving integrity of the training procedure. It
guarantees that an adversary cannot construct a valid proof with less cost (in
both computation and storage) than that made by the prover in generating the
proof. A PoL proof includes a set of intermediate models recorded during
training, together with the corresponding data points used to obtain each
recorded model. Jia et al. claimed that an adversary merely knowing the final
model and training dataset cannot efficiently find a set of intermediate models
with correct data points. In this paper, however, we show that PoL is
vulnerable to "adversarial examples"! Specifically, in a similar way as
optimizing an adversarial example, we could make an arbitrarily-chosen data
point "generate" a given model, hence efficiently generating intermediate
models with correct data points. We demonstrate, both theoretically and
empirically, that we are able to generate a valid proof with significantly less
cost than generating a proof by the prover, thereby we successfully break PoL.


http://arxiv.org/abs/2108.13551
Regularizing (Stabilizing) Deep Learning Based Reconstruction Algorithms. (2%)
Abinash Nayak
  It's well-known that inverse problems are ill-posed and to solve them
meaningfully one has to employ regularization methods. Traditionally, popular
regularization methods have been the penalized Variational approaches. In
recent years, the classical regularized-reconstruction approaches have been
outclassed by the (deep-learning-based) learned reconstruction algorithms.
However, unlike the traditional regularization methods, the theoretical
underpinnings, such as stability and regularization, have been insufficient for
such learned reconstruction algorithms. Hence, the results obtained from such
algorithms, though empirically outstanding, can't always be completely trusted,
as they may contain certain instabilities or (hallucinated) features arising
from the learned process. In fact, it has been shown that such learning
algorithms are very susceptible to small (adversarial) noises in the data and
can lead to severe instabilities in the recovered solution, which can be quite
different than the inherent instabilities of the ill-posed (inverse) problem.
Whereas, the classical regularization methods can handle such (adversarial)
noises very well and can produce stable recovery. Here, we try to present
certain regularization methods to stabilize such (unstable) learned
reconstruction methods and recover a regularized solution, even in the presence
of adversarial noises. For this, we need to extend the classical notion of
regularization and incorporate it in the learned reconstruction algorithms. We
also present some regularization techniques to regularize two of the most
popular learning reconstruction algorithms, the Learned Post-Processing
Reconstruction and the Learned Unrolling Reconstruction.


http://arxiv.org/abs/2108.09135
PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifier. (99%)
Chong Xiang; Saeed Mahloujifar; Prateek Mittal
  The adversarial patch attack against image classification models aims to
inject adversarially crafted pixels within a localized restricted image region
(i.e., a patch) for inducing model misclassification. This attack can be
realized in the physical world by printing and attaching the patch to the
victim object and thus imposes a real-world threat to computer vision systems.
To counter this threat, we propose PatchCleanser as a certifiably robust
defense against adversarial patches that is compatible with any image
classifier. In PatchCleanser, we perform two rounds of pixel masking on the
input image to neutralize the effect of the adversarial patch. In the first
round of masking, we apply a set of carefully generated masks to the input
image and evaluate the model prediction on every masked image. If model
predictions on all one-masked images reach a unanimous agreement, we output the
agreed prediction label. Otherwise, we perform a second round of masking to
settle the disagreement, in which we evaluate model predictions on two-masked
images to robustly recover the correct prediction label. Notably, we can prove
that our defense will always make correct predictions on certain images against
any adaptive white-box attacker within our threat model, achieving certified
robustness. We extensively evaluate our defense on the ImageNet, ImageNette,
CIFAR-10, CIFAR-100, SVHN, and Flowers-102 datasets and demonstrate that our
defense achieves similar clean accuracy as state-of-the-art classification
models and also significantly improves certified robustness from prior works.
Notably, our defense can achieve 83.8% top-1 clean accuracy and 60.4% top-1
certified robust accuracy against a 2%-pixel square patch anywhere on the
1000-class ImageNet dataset.


http://arxiv.org/abs/2108.09034
AdvDrop: Adversarial Attack to DNNs by Dropping Information. (99%)
Ranjie Duan; Yuefeng Chen; Dantong Niu; Yun Yang; A. K. Qin; Yuan He
  Human can easily recognize visual objects with lost information: even losing
most details with only contour reserved, e.g. cartoon. However, in terms of
visual perception of Deep Neural Networks (DNNs), the ability for recognizing
abstract objects (visual objects with lost information) is still a challenge.
In this work, we investigate this issue from an adversarial viewpoint: will the
performance of DNNs decrease even for the images only losing a little
information? Towards this end, we propose a novel adversarial attack, named
\textit{AdvDrop}, which crafts adversarial examples by dropping existing
information of images. Previously, most adversarial attacks add extra
disturbing information on clean images explicitly. Opposite to previous works,
our proposed work explores the adversarial robustness of DNN models in a novel
perspective by dropping imperceptible details to craft adversarial examples. We
demonstrate the effectiveness of \textit{AdvDrop} by extensive experiments, and
show that this new type of adversarial examples is more difficult to be
defended by current defense systems.


http://arxiv.org/abs/2108.09413
Integer-arithmetic-only Certified Robustness for Quantized Neural Networks. (98%)
Haowen Lin; Jian Lou; Li Xiong; Cyrus Shahabi
  Adversarial data examples have drawn significant attention from the machine
learning and security communities. A line of work on tackling adversarial
examples is certified robustness via randomized smoothing that can provide a
theoretical robustness guarantee. However, such a mechanism usually uses
floating-point arithmetic for calculations in inference and requires large
memory footprints and daunting computational costs. These defensive models
cannot run efficiently on edge devices nor be deployed on integer-only logical
units such as Turing Tensor Cores or integer-only ARM processors. To overcome
these challenges, we propose an integer randomized smoothing approach with
quantization to convert any classifier into a new smoothed classifier, which
uses integer-only arithmetic for certified robustness against adversarial
perturbations. We prove a tight robustness guarantee under L2-norm for the
proposed approach. We show our approach can obtain a comparable accuracy and
4x~5x speedup over floating-point arithmetic certified robust methods on
general-purpose CPUs and mobile devices on two distinct datasets (CIFAR-10 and
Caltech-101).


http://arxiv.org/abs/2108.09093
Towards Understanding the Generative Capability of Adversarially Robust Classifiers. (98%)
Yao Zhu; Jiacheng Ma; Jiacheng Sun; Zewei Chen; Rongxin Jiang; Zhenguo Li
  Recently, some works found an interesting phenomenon that adversarially
robust classifiers can generate good images comparable to generative models. We
investigate this phenomenon from an energy perspective and provide a novel
explanation. We reformulate adversarial example generation, adversarial
training, and image generation in terms of an energy function. We find that
adversarial training contributes to obtaining an energy function that is flat
and has low energy around the real data, which is the key for generative
capability. Based on our new understanding, we further propose a better
adversarial training method, Joint Energy Adversarial Training (JEAT), which
can generate high-quality images and achieve new state-of-the-art robustness
under a wide range of attacks. The Inception Score of the images (CIFAR-10)
generated by JEAT is 8.80, much better than original robust classifiers (7.50).


http://arxiv.org/abs/2108.09383
Detecting and Segmenting Adversarial Graphics Patterns from Images. (93%)
Xiangyu Purdue University Qu; Stanley H. Purdue University Chan
  Adversarial attacks pose a substantial threat to computer vision system
security, but the social media industry constantly faces another form of
"adversarial attack" in which the hackers attempt to upload inappropriate
images and fool the automated screening systems by adding artificial graphics
patterns. In this paper, we formulate the defense against such attacks as an
artificial graphics pattern segmentation problem. We evaluate the efficacy of
several segmentation algorithms and, based on observation of their performance,
propose a new method tailored to this specific problem. Extensive experiments
show that the proposed method outperforms the baselines and has a promising
generalization capability, which is the most crucial aspect in segmenting
artificial graphics patterns.


http://arxiv.org/abs/2108.09033
UnSplit: Data-Oblivious Model Inversion, Model Stealing, and Label Inference Attacks Against Split Learning. (1%)
Ege Erdogan; Alptekin Kupcu; A. Ercument Cicek
  Training deep neural networks requires large scale data, which often forces
users to work in a distributed or outsourced setting, accompanied with privacy
concerns. Split learning framework aims to address this concern by splitting up
the model among the client and the server. The idea is that since the server
does not have access to client's part of the model, the scheme supposedly
provides privacy. We show that this is not true via two novel attacks. (1) We
show that an honest-but-curious split learning server, equipped only with the
knowledge of the client neural network architecture, can recover the input
samples and also obtain a functionally similar model to the client model,
without the client being able to detect the attack. (2) Furthermore, we show
that if split learning is used naively to protect the training labels, the
honest-but-curious server can infer the labels with perfect accuracy. We test
our attacks using three benchmark datasets and investigate various properties
of the overall system that affect the attacks' effectiveness. Our results show
that plaintext split learning paradigm can pose serious security risks and
provide no more than a false sense of security.


http://arxiv.org/abs/2108.09343
Early-exit deep neural networks for distorted images: providing an efficient edge offloading. (1%)
Roberto G. Pacheco; Fernanda D. V. R. Oliveira; Rodrigo S. Couto
  Edge offloading for deep neural networks (DNNs) can be adaptive to the
input's complexity by using early-exit DNNs. These DNNs have side branches
throughout their architecture, allowing the inference to end earlier in the
edge. The branches estimate the accuracy for a given input. If this estimated
accuracy reaches a threshold, the inference ends on the edge. Otherwise, the
edge offloads the inference to the cloud to process the remaining DNN layers.
However, DNNs for image classification deals with distorted images, which
negatively impact the branches' estimated accuracy. Consequently, the edge
offloads more inferences to the cloud. This work introduces expert side
branches trained on a particular distortion type to improve robustness against
image distortion. The edge detects the distortion type and selects appropriate
expert branches to perform the inference. This approach increases the estimated
accuracy on the edge, improving the offloading decisions. We validate our
proposal in a realistic scenario, in which the edge offloads DNN inference to
Amazon EC2 instances.


http://arxiv.org/abs/2108.08972
Application of Adversarial Examples to Physical ECG Signals. (99%)
Taiga Waseda University Ono; Takeshi The University of Electro-Communications Sugawara; Jun University of Tsukuba Sakuma; Tatsuya Waseda University RIKEN AIP Mori
  This work aims to assess the reality and feasibility of the adversarial
attack against cardiac diagnosis system powered by machine learning algorithms.
To this end, we introduce adversarial beats, which are adversarial
perturbations tailored specifically against electrocardiograms (ECGs)
beat-by-beat classification system. We first formulate an algorithm to generate
adversarial examples for the ECG classification neural network model, and study
its attack success rate. Next, to evaluate its feasibility in a physical
environment, we mount a hardware attack by designing a malicious signal
generator which injects adversarial beats into ECG sensor readings. To the best
of our knowledge, our work is the first in evaluating the proficiency of
adversarial examples for ECGs in a physical setup. Our real-world experiments
demonstrate that adversarial beats successfully manipulated the diagnosis
results 3-5 times out of 40 attempts throughout the course of 2 minutes.
Finally, we discuss the overall feasibility and impact of the attack, by
clearly defining motives and constraints of expected attackers along with our
experimental results.


http://arxiv.org/abs/2108.08560
Pruning in the Face of Adversaries. (99%)
Florian Merkle; Maximilian Samsinger; Pascal Schöttle
  The vulnerability of deep neural networks against adversarial examples -
inputs with small imperceptible perturbations - has gained a lot of attention
in the research community recently. Simultaneously, the number of parameters of
state-of-the-art deep learning models has been growing massively, with
implications on the memory and computational resources required to train and
deploy such models. One approach to control the size of neural networks is
retrospectively reducing the number of parameters, so-called neural network
pruning. Available research on the impact of neural network pruning on the
adversarial robustness is fragmentary and often does not adhere to established
principles of robustness evaluation. We close this gap by evaluating the
robustness of pruned models against L-0, L-2 and L-infinity attacks for a wide
range of attack strengths, several architectures, data sets, pruning methods,
and compression rates. Our results confirm that neural network pruning and
adversarial robustness are not mutually exclusive. Instead, sweet spots can be
found that are favorable in terms of model size and adversarial robustness.
Furthermore, we extend our analysis to situations that incorporate additional
assumptions on the adversarial scenario and show that depending on the
situation, different strategies are optimal.


http://arxiv.org/abs/2108.08976
ASAT: Adaptively Scaled Adversarial Training in Time Series. (98%)
Zhiyuan Zhang; Wei Li; Ruihan Bao; Keiko Harimoto; Yunfang Wu; Xu Sun
  Adversarial training is a method for enhancing neural networks to improve the
robustness against adversarial examples. Besides the security concerns of
potential adversarial examples, adversarial training can also improve the
performance of the neural networks, train robust neural networks, and provide
interpretability for neural networks. In this work, we take the first step to
introduce adversarial training in time series analysis by taking the finance
field as an example. Rethinking existing researches of adversarial training, we
propose the adaptively scaled adversarial training (ASAT) in time series
analysis, by treating data at different time slots with time-dependent
importance weights. Experimental results show that the proposed ASAT can
improve both the accuracy and the adversarial robustness of neural networks.
Besides enhancing neural networks, we also propose the dimension-wise
adversarial sensitivity indicator to probe the sensitivities and importance of
input dimensions. With the proposed indicator, we can explain the decision
bases of black box neural networks.


http://arxiv.org/abs/2108.08487
Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain. (80%)
Guangyao Chen; Peixi Peng; Li Ma; Jia Li; Lin Du; Yonghong Tian
  Recently, the generalization behavior of Convolutional Neural Networks (CNN)
is gradually transparent through explanation techniques with the frequency
components decomposition. However, the importance of the phase spectrum of the
image for a robust vision system is still ignored. In this paper, we notice
that the CNN tends to converge at the local optimum which is closely related to
the high-frequency components of the training images, while the amplitude
spectrum is easily disturbed such as noises or common corruptions. In contrast,
more empirical studies found that humans rely on more phase components to
achieve robust recognition. This observation leads to more explanations of the
CNN's generalization behaviors in both robustness to common perturbations and
out-of-distribution detection, and motivates a new perspective on data
augmentation designed by re-combing the phase spectrum of the current image and
the amplitude spectrum of the distracter image. That is, the generated samples
force the CNN to pay more attention to the structured information from phase
components and keep robust to the variation of the amplitude. Experiments on
several image datasets indicate that the proposed method achieves
state-of-the-art performances on multiple generalizations and calibration
tasks, including adaptability for common corruptions and surface variations,
out-of-distribution detection, and adversarial attack.


http://arxiv.org/abs/2108.07969
Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better. (99%)
Bojia Zi; Shihao Zhao; Xingjun Ma; Yu-Gang Jiang
  Adversarial training is one effective approach for training robust deep
neural networks against adversarial attacks. While being able to bring reliable
robustness, adversarial training (AT) methods in general favor high capacity
models, i.e., the larger the model the better the robustness. This tends to
limit their effectiveness on small models, which are more preferable in
scenarios where storage or computing resources are very limited (e.g., mobile
devices). In this paper, we leverage the concept of knowledge distillation to
improve the robustness of small models by distilling from adversarially trained
large models. We first revisit several state-of-the-art AT methods from a
distillation perspective and identify one common technique that can lead to
improved robustness: the use of robust soft labels -- predictions of a robust
model. Following this observation, we propose a novel adversarial robustness
distillation method called Robust Soft Label Adversarial Distillation (RSLAD)
to train robust small student models. RSLAD fully exploits the robust soft
labels produced by a robust (adversarially-trained) large teacher model to
guide the student's learning on both natural and adversarial examples in all
loss terms. We empirically demonstrate the effectiveness of our RSLAD approach
over existing adversarial training and distillation methods in improving the
robustness of small models against state-of-the-art attacks including the
AutoAttack. We also provide a set of understandings on our RSLAD and the
importance of robust soft labels for adversarial robustness distillation.


http://arxiv.org/abs/2108.08421
Exploiting Multi-Object Relationships for Detecting Adversarial Attacks in Complex Scenes. (98%)
Mingjun Yin; Shasha Li; Zikui Cai; Chengyu Song; M. Salman Asif; Amit K. Roy-Chowdhury; Srikanth V. Krishnamurthy
  Vision systems that deploy Deep Neural Networks (DNNs) are known to be
vulnerable to adversarial examples. Recent research has shown that checking the
intrinsic consistencies in the input data is a promising way to detect
adversarial attacks (e.g., by checking the object co-occurrence relationships
in complex scenes). However, existing approaches are tied to specific models
and do not offer generalizability. Motivated by the observation that language
descriptions of natural scene images have already captured the object
co-occurrence relationships that can be learned by a language model, we develop
a novel approach to perform context consistency checks using such language
models. The distinguishing aspect of our approach is that it is independent of
the deployed object detector and yet offers very high accuracy in terms of
detecting adversarial examples in practical scenes with multiple objects.


http://arxiv.org/abs/2108.08211
MBRS : Enhancing Robustness of DNN-based Watermarking by Mini-Batch of Real and Simulated JPEG Compression. (45%)
Zhaoyang Jia; Han Fang; Weiming Zhang
  Based on the powerful feature extraction ability of deep learning
architecture, recently, deep-learning based watermarking algorithms have been
widely studied. The basic framework of such algorithm is the auto-encoder like
end-to-end architecture with an encoder, a noise layer and a decoder. The key
to guarantee robustness is the adversarial training with the differential noise
layer. However, we found that none of the existing framework can well ensure
the robustness against JPEG compression, which is non-differential but is an
essential and important image processing operation. To address such
limitations, we proposed a novel end-to-end training architecture, which
utilizes Mini-Batch of Real and Simulated JPEG compression (MBRS) to enhance
the JPEG robustness. Precisely, for different mini-batches, we randomly choose
one of real JPEG, simulated JPEG and noise-free layer as the noise layer.
Besides, we suggest to utilize the Squeeze-and-Excitation blocks which can
learn better feature in embedding and extracting stage, and propose a "message
processor" to expand the message in a more appreciate way. Meanwhile, to
improve the robustness against crop attack, we propose an additive diffusion
block into the network. The extensive experimental results have demonstrated
the superior performance of the proposed scheme compared with the
state-of-the-art algorithms. Under the JPEG compression with quality factor
Q=50, our models achieve a bit error rate less than 0.01% for extracted
messages, with PSNR larger than 36 for the encoded images, which shows the
well-enhanced robustness against JPEG attack. Besides, under many other
distortions such as Gaussian filter, crop, cropout and dropout, the proposed
framework also obtains strong robustness. The code implemented by PyTorch
\cite{2011torch7} is avaiable in https://github.com/jzyustc/MBRS.


http://arxiv.org/abs/2108.08476
Proceedings of the 1st International Workshop on Adaptive Cyber Defense. (1%)
Damian Marriott; Kimberly Ferguson-Walter; Sunny Fugate; Marco Carvalho
  The 1st International Workshop on Adaptive Cyber Defense was held as part of
the 2021 International Joint Conference on Artificial Intelligence. This
workshop was organized to share research that explores unique applications of
Artificial Intelligence (AI) and Machine Learning (ML) as foundational
capabilities for the pursuit of adaptive cyber defense. The cyber domain cannot
currently be reliably and effectively defended without extensive reliance on
human experts. Skilled cyber defenders are in short supply and often cannot
respond fast enough to cyber threats.
  Building on recent advances in AI and ML the Cyber defense research community
has been motivated to develop new dynamic and sustainable defenses through the
adoption of AI and ML techniques to both cyber and non-cyber settings. Bridging
critical gaps between AI and Cyber researchers and practitioners can accelerate
efforts to create semi-autonomous cyber defenses that can learn to recognize
and respond to cyber attacks or discover and mitigate weaknesses in cooperation
with other cyber operation systems and human experts. Furthermore, these
defenses are expected to be adaptive and able to evolve over time to thwart
changes in attacker behavior, changes in the system health and readiness, and
natural shifts in user behavior over time.
  The Workshop (held on August 19th and 20th 2021 in Montreal-themed virtual
reality) was comprised of technical presentations and a panel discussion
focused on open problems and potential research solutions. Workshop submissions
were peer reviewed by a panel of domain experts with a proceedings consisting
of 10 technical articles exploring challenging problems of critical importance
to national and global security. Participation in this workshop offered new
opportunities to stimulate research and innovation in the emerging domain of
adaptive and autonomous cyber defense.


http://arxiv.org/abs/2108.07602
When Should You Defend Your Classifier -- A Game-theoretical Analysis of Countermeasures against Adversarial Examples. (98%)
Maximilian Samsinger; Florian Merkle; Pascal Schöttle; Tomas Pevny
  Adversarial machine learning, i.e., increasing the robustness of machine
learning algorithms against so-called adversarial examples, is now an
established field. Yet, newly proposed methods are evaluated and compared under
unrealistic scenarios where costs for adversary and defender are not considered
and either all samples are attacked or no sample is attacked. We scrutinize
these assumptions and propose the advanced adversarial classification game,
which incorporates all relevant parameters of an adversary and a defender in
adversarial classification. Especially, we take into account economic factors
on both sides and the fact that all so far proposed countermeasures against
adversarial examples reduce accuracy on benign samples. Analyzing the scenario
in detail, where both players have two pure strategies, we identify all best
responses and conclude that in practical settings, the most influential factor
might be the maximum amount of adversarial examples.


http://arxiv.org/abs/2108.07920
Adversarial Relighting against Face Recognition. (98%)
Ruijun Gao; Qing Guo; Qian Zhang; Felix Juefei-Xu; Hongkai Yu; Wei Feng
  Deep face recognition (FR) has achieved significantly high accuracy on
several challenging datasets and fosters successful real-world applications,
even showing high robustness to the illumination variation that is usually
regarded as a main threat to the FR system. However, in the real world,
illumination variation caused by diverse lighting conditions cannot be fully
covered by the limited face dataset. In this paper, we study the threat of
lighting against FR from a new angle, i.e., adversarial attack, and identify a
new task, i.e., adversarial relighting. Given a face image, adversarial
relighting aims to produce a naturally relighted counterpart while fooling the
state-of-the-art deep FR methods. To this end, we first propose the physical
model-based adversarial relighting attack (ARA) denoted as
albedo-quotient-based adversarial relighting attack (AQ-ARA). It generates
natural adversarial light under the physical lighting model and guidance of FR
systems and synthesizes adversarially relighted face images. Moreover, we
propose the auto-predictive adversarial relighting attack (AP-ARA) by training
an adversarial relighting network (ARNet) to automatically predict the
adversarial light in a one-step manner according to different input faces,
allowing efficiency-sensitive applications. More importantly, we propose to
transfer the above digital attacks to physical ARA (Phy-ARA) through a precise
relighting device, making the estimated adversarial lighting condition
reproducible in the real world. We validate our methods on three
state-of-the-art deep FR methods, i.e., FaceNet, ArcFace, and CosFace, on two
public datasets. The extensive and insightful results demonstrate our work can
generate realistic adversarial relighted face images fooling FR easily,
revealing the threat of specific light directions and strengths.


http://arxiv.org/abs/2108.07958
Semantic Perturbations with Normalizing Flows for Improved Generalization. (13%)
Oguz Kaan Yuksel; Sebastian U. Stich; Martin Jaggi; Tatjana Chavdarova
  Data augmentation is a widely adopted technique for avoiding overfitting when
training deep neural networks. However, this approach requires domain-specific
knowledge and is often limited to a fixed set of hard-coded transformations.
Recently, several works proposed to use generative models for generating
semantically meaningful perturbations to train a classifier. However, because
accurate encoding and decoding are critical, these methods, which use
architectures that approximate the latent-variable inference, remained limited
to pilot studies on small datasets.
  Exploiting the exactly reversible encoder-decoder structure of normalizing
flows, we perform on-manifold perturbations in the latent space to define fully
unsupervised data augmentations. We demonstrate that such perturbations match
the performance of advanced data augmentation techniques -- reaching 96.6% test
accuracy for CIFAR-10 using ResNet-18 and outperform existing methods,
particularly in low data regimes -- yielding 10--25% relative improvement of
test accuracy from classical training. We find that our latent adversarial
perturbations adaptive to the classifier throughout its training are most
effective, yielding the first test accuracy improvement results on real-world
datasets -- CIFAR-10/100 -- via latent-space perturbations.


http://arxiv.org/abs/2108.07594
Coalesced Multi-Output Tsetlin Machines with Clause Sharing. (1%)
Sondre Glimsdal; Ole-Christoffer Granmo
  Using finite-state machines to learn patterns, Tsetlin machines (TMs) have
obtained competitive accuracy and learning speed across several benchmarks,
with frugal memory- and energy footprint. A TM represents patterns as
conjunctive clauses in propositional logic (AND-rules), each clause voting for
or against a particular output. While efficient for single-output problems, one
needs a separate TM per output for multi-output problems. Employing multiple
TMs hinders pattern reuse because each TM then operates in a silo. In this
paper, we introduce clause sharing, merging multiple TMs into a single one.
Each clause is related to each output by using a weight. A positive weight
makes the clause vote for output $1$, while a negative weight makes the clause
vote for output $0$. The clauses thus coalesce to produce multiple outputs. The
resulting coalesced Tsetlin Machine (CoTM) simultaneously learns both the
weights and the composition of each clause by employing interacting Stochastic
Searching on the Line (SSL) and Tsetlin Automata (TA) teams. Our empirical
results on MNIST, Fashion-MNIST, and Kuzushiji-MNIST show that CoTM obtains
significantly higher accuracy than TM on $50$- to $1$K-clause configurations,
indicating an ability to repurpose clauses. E.g., accuracy goes from $71.99$%
to $89.66$% on Fashion-MNIST when employing $50$ clauses per class (22 Kb
memory). While TM and CoTM accuracy is similar when using more than $1$K
clauses per class, CoTM reaches peak accuracy $3\times$ faster on MNIST with
$8$K clauses. We further investigate robustness towards imbalanced training
data. Our evaluations on imbalanced versions of IMDb- and CIFAR10 data show
that CoTM is robust towards high degrees of class imbalance. Being able to
share clauses, we believe CoTM will enable new TM application domains that
involve multiple outputs, such as learning language models and auto-encoding.


http://arxiv.org/abs/2108.07779
Appearance Based Deep Domain Adaptation for the Classification of Aerial Images. (1%)
Dennis Wittich; Franz Rottensteiner
  This paper addresses domain adaptation for the pixel-wise classification of
remotely sensed data using deep neural networks (DNN) as a strategy to reduce
the requirements of DNN with respect to the availability of training data. We
focus on the setting in which labelled data are only available in a source
domain DS, but not in a target domain DT. Our method is based on adversarial
training of an appearance adaptation network (AAN) that transforms images from
DS such that they look like images from DT. Together with the original label
maps from DS, the transformed images are used to adapt a DNN to DT. We propose
a joint training strategy of the AAN and the classifier, which constrains the
AAN to transform the images such that they are correctly classified. In this
way, objects of a certain class are changed such that they resemble objects of
the same class in DT. To further improve the adaptation performance, we propose
a new regularization loss for the discriminator network used in domain
adversarial training. We also address the problem of finding the optimal values
of the trained network parameters, proposing an unsupervised entropy based
parameter selection criterion which compensates for the fact that there is no
validation set in DT that could be monitored. As a minor contribution, we
present a new weighting strategy for the cross-entropy loss, addressing the
problem of imbalanced class distributions. Our method is evaluated in 42
adaptation scenarios using datasets from 7 cities, all consisting of
high-resolution digital orthophotos and height data. It achieves a positive
transfer in all cases, and on average it improves the performance in the target
domain by 4.3% in overall accuracy. In adaptation scenarios between datasets
from the ISPRS semantic labelling benchmark our method outperforms those from
recent publications by 10-20% with respect to the mean intersection over union.


http://arxiv.org/abs/2108.07033
Exploring Transferable and Robust Adversarial Perturbation Generation from the Perspective of Network Hierarchy. (99%)
Ruikui Wang; Yuanfang Guo; Ruijie Yang; Yunhong Wang
  The transferability and robustness of adversarial examples are two practical
yet important properties for black-box adversarial attacks. In this paper, we
explore effective mechanisms to boost both of them from the perspective of
network hierarchy, where a typical network can be hierarchically divided into
output stage, intermediate stage and input stage. Since over-specialization of
source model, we can hardly improve the transferability and robustness of the
adversarial perturbations in the output stage. Therefore, we focus on the
intermediate and input stages in this paper and propose a transferable and
robust adversarial perturbation generation (TRAP) method. Specifically, we
propose the dynamically guided mechanism to continuously calculate accurate
directional guidances for perturbation generation in the intermediate stage. In
the input stage, instead of the single-form transformation augmentations
adopted in the existing methods, we leverage multiform affine transformation
augmentations to further enrich the input diversity and boost the robustness
and transferability of the adversarial perturbations. Extensive experiments
demonstrate that our TRAP achieves impressive transferability and high
robustness against certain interferences.


http://arxiv.org/abs/2108.06895
Interpreting Attributions and Interactions of Adversarial Attacks. (83%)
Xin Wang; Shuyun Lin; Hao Zhang; Yufei Zhu; Quanshi Zhang
  This paper aims to explain adversarial attacks in terms of how adversarial
perturbations contribute to the attacking task. We estimate attributions of
different image regions to the decrease of the attacking cost based on the
Shapley value. We define and quantify interactions among adversarial
perturbation pixels, and decompose the entire perturbation map into relatively
independent perturbation components. The decomposition of the perturbation map
shows that adversarially-trained DNNs have more perturbation components in the
foreground than normally-trained DNNs. Moreover, compared to the
normally-trained DNN, the adversarially-trained DNN have more components which
mainly decrease the score of the true category. Above analyses provide new
insights into the understanding of adversarial attacks.


http://arxiv.org/abs/2108.07229
Patch Attack Invariance: How Sensitive are Patch Attacks to 3D Pose? (62%)
Max Lennon; Nathan Drenkow; Philippe Burlina
  Perturbation-based attacks, while not physically realizable, have been the
main emphasis of adversarial machine learning (ML) research. Patch-based
attacks by contrast are physically realizable, yet most work has focused on 2D
domain with recent forays into 3D. Characterizing the robustness properties of
patch attacks and their invariance to 3D pose is important, yet not fully
elucidated, and is the focus of this paper. To this end, several contributions
are made here: A) we develop a new metric called mean Attack Success over
Transformations (mAST) to evaluate patch attack robustness and invariance; and
B), we systematically assess robustness of patch attacks to 3D position and
orientation for various conditions; in particular, we conduct a sensitivity
analysis which provides important qualitative insights into attack
effectiveness as a function of the 3D pose of a patch relative to the camera
(rotation, translation) and sets forth some properties for patch attack 3D
invariance; and C), we draw novel qualitative conclusions including: 1) we
demonstrate that for some 3D transformations, namely rotation and loom,
increasing the training distribution support yields an increase in patch
success over the full range at test time. 2) We provide new insights into the
existence of a fundamental cutoff limit in patch attack effectiveness that
depends on the extent of out-of-plane rotation angles. These findings should
collectively guide future design of 3D patch attacks and defenses.


http://arxiv.org/abs/2108.07256
NeuraCrypt is not private. (10%)
Nicholas Carlini; Sanjam Garg; Somesh Jha; Saeed Mahloujifar; Mohammad Mahmoody; Florian Tramer
  NeuraCrypt (Yara et al. arXiv 2021) is an algorithm that converts a sensitive
dataset to an encoded dataset so that (1) it is still possible to train machine
learning models on the encoded data, but (2) an adversary who has access only
to the encoded dataset can not learn much about the original sensitive dataset.
We break NeuraCrypt privacy claims, by perfectly solving the authors' public
challenge, and by showing that NeuraCrypt does not satisfy the formal privacy
definitions posed in the original paper. Our attack consists of a series of
boosting steps that, coupled with various design flaws, turns a 1% attack
advantage into a 100% complete break of the scheme.


http://arxiv.org/abs/2108.07083
Identifying and Exploiting Structures for Reliable Deep Learning. (2%)
Amartya Sanyal
  Deep learning research has recently witnessed an impressively fast-paced
progress in a wide range of tasks including computer vision, natural language
processing, and reinforcement learning. The extraordinary performance of these
systems often gives the impression that they can be used to revolutionise our
lives for the better. However, as recent works point out, these systems suffer
from several issues that make them unreliable for use in the real world,
including vulnerability to adversarial attacks (Szegedy et al. [248]), tendency
to memorise noise (Zhang et al. [292]), being over-confident on incorrect
predictions (miscalibration) (Guo et al. [99]), and unsuitability for handling
private data (Gilad-Bachrach et al. [88]). In this thesis, we look at each of
these issues in detail, investigate their causes, and propose computationally
cheap algorithms for mitigating them in practice. To do this, we identify
structures in deep neural networks that can be exploited to mitigate the above
causes of unreliability of deep learning algorithms.


http://arxiv.org/abs/2108.07258
On the Opportunities and Risks of Foundation Models. (2%)
Rishi Bommasani; Drew A. Hudson; Ehsan Adeli; Russ Altman; Simran Arora; Arx Sydney von; Michael S. Bernstein; Jeannette Bohg; Antoine Bosselut; Emma Brunskill; Erik Brynjolfsson; Shyamal Buch; Dallas Card; Rodrigo Castellon; Niladri Chatterji; Annie Chen; Kathleen Creel; Jared Quincy Davis; Dora Demszky; Chris Donahue; Moussa Doumbouya; Esin Durmus; Stefano Ermon; John Etchemendy; Kawin Ethayarajh; Li Fei-Fei; Chelsea Finn; Trevor Gale; Lauren Gillespie; Karan Goel; Noah Goodman; Shelby Grossman; Neel Guha; Tatsunori Hashimoto; Peter Henderson; John Hewitt; Daniel E. Ho; Jenny Hong; Kyle Hsu; Jing Huang; Thomas Icard; Saahil Jain; Dan Jurafsky; Pratyusha Kalluri; Siddharth Karamcheti; Geoff Keeling; Fereshte Khani; Omar Khattab; Pang Wei Koh; Mark Krass; Ranjay Krishna; Rohith Kuditipudi; Ananya Kumar; Faisal Ladhak; Mina Lee; Tony Lee; Jure Leskovec; Isabelle Levent; Xiang Lisa Li; Xuechen Li; Tengyu Ma; Ali Malik; Christopher D. Manning; Suvir Mirchandani; Eric Mitchell; Zanele Munyikwa; Suraj Nair; Avanika Narayan; Deepak Narayanan; Ben Newman; Allen Nie; Juan Carlos Niebles; Hamed Nilforoshan; Julian Nyarko; Giray Ogut; Laurel Orr; Isabel Papadimitriou; Joon Sung Park; Chris Piech; Eva Portelance; Christopher Potts; Aditi Raghunathan; Rob Reich; Hongyu Ren; Frieda Rong; Yusuf Roohani; Camilo Ruiz; Jack Ryan; Christopher Ré; Dorsa Sadigh; Shiori Sagawa; Keshav Santhanam; Andy Shih; Krishnan Srinivasan; Alex Tamkin; Rohan Taori; Armin W. Thomas; Florian Tramèr; Rose E. Wang; William Wang; Bohan Wu; Jiajun Wu; Yuhuai Wu; Sang Michael Xie; Michihiro Yasunaga; Jiaxuan You; Matei Zaharia; Michael Zhang; Tianyi Zhang; Xikun Zhang; Yuhui Zhang; Lucia Zheng; Kaitlyn Zhou; Percy Liang
  AI is undergoing a paradigm shift with the rise of models (e.g., BERT,
DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a
wide range of downstream tasks. We call these models foundation models to
underscore their critically central yet incomplete character. This report
provides a thorough account of the opportunities and risks of foundation
models, ranging from their capabilities (e.g., language, vision, robotics,
reasoning, human interaction) and technical principles(e.g., model
architectures, training procedures, data, systems, security, evaluation,
theory) to their applications (e.g., law, healthcare, education) and societal
impact (e.g., inequity, misuse, economic and environmental impact, legal and
ethical considerations). Though foundation models are based on standard deep
learning and transfer learning, their scale results in new emergent
capabilities,and their effectiveness across so many tasks incentivizes
homogenization. Homogenization provides powerful leverage but demands caution,
as the defects of the foundation model are inherited by all the adapted models
downstream. Despite the impending widespread deployment of foundation models,
we currently lack a clear understanding of how they work, when they fail, and
what they are even capable of due to their emergent properties. To tackle these
questions, we believe much of the critical research on foundation models will
require deep interdisciplinary collaboration commensurate with their
fundamentally sociotechnical nature.


http://arxiv.org/abs/2108.06885
Neural Architecture Dilation for Adversarial Robustness. (81%)
Yanxi Li; Zhaohui Yang; Yunhe Wang; Chang Xu
  With the tremendous advances in the architecture and scale of convolutional
neural networks (CNNs) over the past few decades, they can easily reach or even
exceed the performance of humans in certain tasks. However, a recently
discovered shortcoming of CNNs is that they are vulnerable to adversarial
attacks. Although the adversarial robustness of CNNs can be improved by
adversarial training, there is a trade-off between standard accuracy and
adversarial robustness. From the neural architecture perspective, this paper
aims to improve the adversarial robustness of the backbone CNNs that have a
satisfactory accuracy. Under a minimal computational overhead, the introduction
of a dilation architecture is expected to be friendly with the standard
performance of the backbone CNN while pursuing adversarial robustness.
Theoretical analyses on the standard and adversarial error bounds naturally
motivate the proposed neural architecture dilation algorithm. Experimental
results on real-world datasets and benchmark neural networks demonstrate the
effectiveness of the proposed algorithm to balance the accuracy and adversarial
robustness.


http://arxiv.org/abs/2108.06797
Deep Adversarially-Enhanced k-Nearest Neighbors. (74%)
Ren Wang; Tianqi Chen
  Recent works have theoretically and empirically shown that deep neural
networks (DNNs) have an inherent vulnerability to small perturbations. Applying
the Deep k-Nearest Neighbors (DkNN) classifier, we observe a dramatically
increasing robustness-accuracy trade-off as the layer goes deeper. In this
work, we propose a Deep Adversarially-Enhanced k-Nearest Neighbors (DAEkNN)
method which achieves higher robustness than DkNN and mitigates the
robustness-accuracy trade-off in deep layers through two key elements. First,
DAEkNN is based on an adversarially trained model. Second, DAEkNN makes
predictions by leveraging a weighted combination of benign and adversarial
training data. Empirically, we find that DAEkNN improves both the robustness
and the robustness-accuracy trade-off on MNIST and CIFAR-10 datasets.


http://arxiv.org/abs/2108.06871
IADA: Iterative Adversarial Data Augmentation Using Formal Verification and Expert Guidance. (1%)
Ruixuan Liu; Changliu Liu
  Neural networks (NNs) are widely used for classification tasks for their
remarkable performance. However, the robustness and accuracy of NNs heavily
depend on the training data. In many applications, massive training data is
usually not available. To address the challenge, this paper proposes an
iterative adversarial data augmentation (IADA) framework to learn neural
network models from an insufficient amount of training data. The method uses
formal verification to identify the most "confusing" input samples, and
leverages human guidance to safely and iteratively augment the training data
with these samples. The proposed framework is applied to an artificial 2D
dataset, the MNIST dataset, and a human motion dataset. By applying IADA to
fully-connected NN classifiers, we show that our training method can improve
the robustness and accuracy of the learned model. By comparing to regular
supervised training, on the MNIST dataset, the average perturbation bound
improved 107.4%. The classification accuracy improved 1.77%, 3.76%, 10.85% on
the 2D dataset, the MNIST dataset, and the human motion dataset respectively.


http://arxiv.org/abs/2108.06504
LinkTeller: Recovering Private Edges from Graph Neural Networks via Influence Analysis. (1%)
Fan Wu; Yunhui Long; Ce Zhang; Bo Li
  Graph structured data have enabled several successful applications such as
recommendation systems and traffic prediction, given the rich node features and
edges information. However, these high-dimensional features and high-order
adjacency information are usually heterogeneous and held by different data
holders in practice. Given such vertical data partition (e.g., one data holder
will only own either the node features or edge information), different data
holders have to develop efficient joint training protocols rather than directly
transfer data to each other due to privacy concerns. In this paper, we focus on
the edge privacy, and consider a training scenario where Bob with node features
will first send training node features to Alice who owns the adjacency
information. Alice will then train a graph neural network (GNN) with the joint
information and release an inference API. During inference, Bob is able to
provide test node features and query the API to obtain the predictions for test
nodes. Under this setting, we first propose a privacy attack LinkTeller via
influence analysis to infer the private edge information held by Alice via
designing adversarial queries for Bob. We then empirically show that LinkTeller
is able to recover a significant amount of private edges, outperforming
existing baselines. To further evaluate the privacy leakage, we adapt an
existing algorithm for differentially private graph convolutional network (DP
GCN) training and propose a new DP GCN mechanism LapGraph. We show that these
DP GCN mechanisms are not always resilient against LinkTeller empirically under
mild privacy guarantees ($\varepsilon>5$). Our studies will shed light on
future research towards designing more resilient privacy-preserving GCN models;
in the meantime, provide an in-depth understanding of the tradeoff between GCN
model utility and robustness against potential privacy attacks.


http://arxiv.org/abs/2108.06179
Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks. (99%)
Federico Nesti; Giulio Rossolini; Saasha Nair; Alessandro Biondi; Giorgio Buttazzo
  Deep learning and convolutional neural networks allow achieving impressive
performance in computer vision tasks, such as object detection and semantic
segmentation (SS). However, recent studies have shown evident weaknesses of
such models against adversarial perturbations. In a real-world scenario
instead, like autonomous driving, more attention should be devoted to
real-world adversarial examples (RWAEs), which are physical objects (e.g.,
billboards and printable patches) optimized to be adversarial to the entire
perception pipeline. This paper presents an in-depth evaluation of the
robustness of popular SS models by testing the effects of both digital and
real-world adversarial patches. These patches are crafted with powerful attacks
enriched with a novel loss function. Firstly, an investigation on the
Cityscapes dataset is conducted by extending the Expectation Over
Transformation (EOT) paradigm to cope with SS. Then, a novel attack
optimization, called scene-specific attack, is proposed. Such an attack
leverages the CARLA driving simulator to improve the transferability of the
proposed EOT-based attack to a real 3D environment. Finally, a printed physical
billboard containing an adversarial patch was tested in an outdoor driving
scenario to assess the feasibility of the studied attacks in the real world.
Exhaustive experiments revealed that the proposed attack formulations
outperform previous work to craft both digital and real-world adversarial
patches for SS. At the same time, the experimental results showed how these
attacks are notably less effective in the real world, hence questioning the
practical relevance of adversarial attacks to SS models for autonomous/assisted
driving.


http://arxiv.org/abs/2108.06247
Optical Adversarial Attack. (98%)
Abhiram Gnanasambandam; Alex M. Sherman; Stanley H. Chan
  We introduce OPtical ADversarial attack (OPAD). OPAD is an adversarial attack
in the physical space aiming to fool image classifiers without physically
touching the objects (e.g., moving or painting the objects). The principle of
OPAD is to use structured illumination to alter the appearance of the target
objects. The system consists of a low-cost projector, a camera, and a computer.
The challenge of the problem is the non-linearity of the radiometric response
of the projector and the spatially varying spectral response of the scene.
Attacks generated in a conventional approach do not work in this setting unless
they are calibrated to compensate for such a projector-camera model. The
proposed solution incorporates the projector-camera model into the adversarial
attack optimization, where a new attack formulation is derived. Experimental
results prove the validity of the solution. It is demonstrated that OPAD can
optically attack a real 3D object in the presence of background lighting for
white-box, black-box, targeted, and untargeted attacks. Theoretical analysis is
presented to quantify the fundamental performance limit of the system.


http://arxiv.org/abs/2108.06280
Understanding Structural Vulnerability in Graph Convolutional Networks. (96%)
Liang Chen; Jintang Li; Qibiao Peng; Yang Liu; Zibin Zheng; Carl Yang
  Recent studies have shown that Graph Convolutional Networks (GCNs) are
vulnerable to adversarial attacks on the graph structure. Although multiple
works have been proposed to improve their robustness against such structural
adversarial attacks, the reasons for the success of the attacks remain unclear.
In this work, we theoretically and empirically demonstrate that structural
adversarial examples can be attributed to the non-robust aggregation scheme
(i.e., the weighted mean) of GCNs. Specifically, our analysis takes advantage
of the breakdown point which can quantitatively measure the robustness of
aggregation schemes. The key insight is that weighted mean, as the basic design
of GCNs, has a low breakdown point and its output can be dramatically changed
by injecting a single edge. We show that adopting the aggregation scheme with a
high breakdown point (e.g., median or trimmed mean) could significantly enhance
the robustness of GCNs against structural attacks. Extensive experiments on
four real-world datasets demonstrate that such a simple but effective method
achieves the best robustness performance compared to state-of-the-art models.


http://arxiv.org/abs/2108.06131
The Forgotten Threat of Voltage Glitching: A Case Study on Nvidia Tegra X2 SoCs. (1%)
Otto Bittner; Thilo Krachenfels; Andreas Galauner; Jean-Pierre Seifert
  Voltage fault injection (FI) is a well-known attack technique that can be
used to force faulty behavior in processors during their operation. Glitching
the supply voltage can cause data value corruption, skip security checks, or
enable protected code paths. At the same time, modern systems on a chip (SoCs)
are used in security-critical applications, such as self-driving cars and
autonomous machines. Since these embedded devices are often physically
accessible by attackers, vendors must consider device tampering in their threat
models. However, while the threat of voltage FI is known since the early 2000s,
it seems as if vendors still forget to integrate countermeasures. This work
shows how the entire boot security of an Nvidia SoC, used in Tesla's autopilot
and Mercedes-Benz's infotainment system, can be circumvented using voltage FI.
We uncover a hidden bootloader that is only available to the manufacturer for
testing purposes and disabled by fuses in shipped products. We demonstrate how
to re-enable this bootloader using FI to gain code execution with the highest
privileges, enabling us to extract the bootloader's firmware and decryption
keys used in later boot stages. Using a hardware implant, an adversary might
misuse the hidden bootloader to bypass trusted code execution even during the
system's regular operation.


http://arxiv.org/abs/2108.06017
AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric Learning. (99%)
Hong Wang; Yuefan Deng; Shinjae Yoo; Haibin Ling; Yuewei Lin
  While deep neural networks have shown impressive performance in many tasks,
they are fragile to carefully designed adversarial attacks. We propose a novel
adversarial training-based model by Attention Guided Knowledge Distillation and
Bi-directional Metric Learning (AGKD-BML). The attention knowledge is obtained
from a weight-fixed model trained on a clean dataset, referred to as a teacher
model, and transferred to a model that is under training on adversarial
examples (AEs), referred to as a student model. In this way, the student model
is able to focus on the correct region, as well as correcting the intermediate
features corrupted by AEs to eventually improve the model accuracy. Moreover,
to efficiently regularize the representation in feature space, we propose a
bidirectional metric learning. Specifically, given a clean image, it is first
attacked to its most confusing class to get the forward AE. A clean image in
the most confusing class is then randomly picked and attacked back to the
original class to get the backward AE. A triplet loss is then used to shorten
the representation distance between original image and its AE, while enlarge
that between the forward and backward AEs. We conduct extensive adversarial
robustness experiments on two widely used datasets with different attacks. Our
proposed AGKD-BML model consistently outperforms the state-of-the-art
approaches. The code of AGKD-BML will be available at:
https://github.com/hongw579/AGKD-BML.


http://arxiv.org/abs/2108.05948
Deep adversarial attack on target detection systems. (99%)
Uche M. Osahor; Nasser M. Nasrabadi
  Target detection systems identify targets by localizing their coordinates on
the input image of interest. This is ideally achieved by labeling each pixel in
an image as a background or a potential target pixel. Deep Convolutional Neural
Network (DCNN) classifiers have proven to be successful tools for computer
vision applications. However,prior research confirms that even state of the art
classifier models are susceptible to adversarial attacks. In this paper, we
show how to generate adversarial infrared images by adding small perturbations
to the targets region to deceive a DCNN-based target detector at remarkable
levels. We demonstrate significant progress in developing visually
imperceptible adversarial infrared images where the targets are visually
recognizable by an expert but a DCNN-based target detector cannot detect the
targets in the image.


http://arxiv.org/abs/2108.05921
Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate. (68%)
Hannah Rose Kirk; Bertram Vidgen; Paul Röttger; Tristan Thrush; Scott A. Hale
  Detecting online hate is a complex task, and low-performing models have
harmful consequences when used for sensitive applications such as content
moderation. Emoji-based hate is a key emerging challenge for automated
detection. We present HatemojiCheck, a test suite of 3,930 short-form
statements that allows us to evaluate performance on hateful language expressed
with emoji. Using the test suite, we expose weaknesses in existing hate
detection models. To address these weaknesses, we create the HatemojiTrain
dataset using a human-and-model-in-the-loop approach. Models trained on these
5,912 adversarial examples perform substantially better at detecting
emoji-based hate, while retaining strong performance on text-only hate. Both
HatemojiCheck and HatemojiTrain are made publicly available.


http://arxiv.org/abs/2108.05075
Turning Your Strength against You: Detecting and Mitigating Robust and Universal Adversarial Patch Attack. (98%)
Zitao Chen; Pritam Dash; Karthik Pattabiraman
  Adversarial patch attack against image classification deep neural networks
(DNNs), in which the attacker can inject arbitrary distortions within a bounded
region of an image, is able to generate adversarial perturbations that are
robust (i.e., remain adversarial in physical world) and universal (i.e., remain
adversarial on any input). It is thus important to detect and mitigate such
attack to ensure the security of DNNs.
  This work proposes Jujutsu, a technique to detect and mitigate robust and
universal adversarial patch attack. Jujutsu leverages the universal property of
the patch attack for detection. It uses explainable AI technique to identify
suspicious features that are potentially malicious, and verify their
maliciousness by transplanting the suspicious features to new images. An
adversarial patch continues to exhibit the malicious behavior on the new images
and thus can be detected based on prediction consistency. Jujutsu leverages the
localized nature of the patch attack for mitigation, by randomly masking the
suspicious features to "remove" adversarial perturbations. However, the network
might fail to classify the images as some of the contents are removed (masked).
Therefore, Jujutsu uses image inpainting for synthesizing alternative contents
from the pixels that are masked, which can reconstruct the "clean" image for
correct prediction. We evaluate Jujutsu on five DNNs on two datasets, and show
that Jujutsu achieves superior performance and significantly outperforms
existing techniques. Jujutsu can further defend against various variants of the
basic attack, including 1) physical-world attack; 2) attacks that target
diverse classes; 3) attacks that use patches in different shapes and 4)
adaptive attacks.


http://arxiv.org/abs/2108.05490
Attacks against Ranking Algorithms with Text Embeddings: a Case Study on Recruitment Algorithms. (78%)
Anahita Samadi; Debapriya Banerjee; Shirin Nilizadeh
  Recently, some studies have shown that text classification tasks are
vulnerable to poisoning and evasion attacks. However, little work has
investigated attacks against decision making algorithms that use text
embeddings, and their output is a ranking. In this paper, we focus on ranking
algorithms for recruitment process, that employ text embeddings for ranking
applicants resumes when compared to a job description. We demonstrate both
white box and black box attacks that identify text items, that based on their
location in embedding space, have significant contribution in increasing the
similarity score between a resume and a job description. The adversary then
uses these text items to improve the ranking of their resume among others. We
tested recruitment algorithms that use the similarity scores obtained from
Universal Sentence Encoder (USE) and Term Frequency Inverse Document Frequency
(TF IDF) vectors. Our results show that in both adversarial settings, on
average the attacker is successful. We also found that attacks against TF IDF
is more successful compared to USE.


http://arxiv.org/abs/2108.05018
Are Neural Ranking Models Robust? (4%)
Chen Wu; Ruqing Zhang; Jiafeng Guo; Yixing Fan; Xueqi Cheng
  Recently, we have witnessed the bloom of neural ranking models in the
information retrieval (IR) field. So far, much effort has been devoted to
developing effective neural ranking models that can generalize well on new
data. There has been less attention paid to the robustness perspective. Unlike
the effectiveness which is about the average performance of a system under
normal purpose, robustness cares more about the system performance in the worst
case or under malicious operations instead. When a new technique enters into
the real-world application, it is critical to know not only how it works in
average, but also how would it behave in abnormal situations. So we raise the
question in this work: Are neural ranking models robust? To answer this
question, firstly, we need to clarify what we refer to when we talk about the
robustness of ranking models in IR. We show that robustness is actually a
multi-dimensional concept and there are three ways to define it in IR: 1) The
performance variance under the independent and identically distributed (I.I.D.)
setting; 2) The out-of-distribution (OOD) generalizability; and 3) The
defensive ability against adversarial operations. The latter two definitions
can be further specified into two different perspectives respectively, leading
to 5 robustness tasks in total. Based on this taxonomy, we build corresponding
benchmark datasets, design empirical experiments, and systematically analyze
the robustness of several representative neural ranking models against
traditional probabilistic ranking models and learning-to-rank (LTR) models. The
empirical results show that there is no simple answer to our question. While
neural ranking models are less robust against other IR models in most cases,
some of them can still win 1 out of 5 tasks. This is the first comprehensive
study on the robustness of neural ranking models.


http://arxiv.org/abs/2108.05149
Logic Explained Networks. (1%)
Gabriele Ciravegna; Pietro Barbiero; Francesco Giannini; Marco Gori; Pietro Lió; Marco Maggini; Stefano Melacci
  The large and still increasing popularity of deep learning clashes with a
major limit of neural network architectures, that consists in their lack of
capability in providing human-understandable motivations of their decisions. In
situations in which the machine is expected to support the decision of human
experts, providing a comprehensible explanation is a feature of crucial
importance. The language used to communicate the explanations must be formal
enough to be implementable in a machine and friendly enough to be
understandable by a wide audience. In this paper, we propose a general approach
to Explainable Artificial Intelligence in the case of neural architectures,
showing how a mindful design of the networks leads to a family of interpretable
deep learning models called Logic Explained Networks (LENs). LENs only require
their inputs to be human-understandable predicates, and they provide
explanations in terms of simple First-Order Logic (FOL) formulas involving such
predicates. LENs are general enough to cover a large number of scenarios.
Amongst them, we consider the case in which LENs are directly used as special
classifiers with the capability of being explainable, or when they act as
additional networks with the role of creating the conditions for making a
black-box classifier explainable by FOL formulas. Despite supervised learning
problems are mostly emphasized, we also show that LENs can learn and provide
explanations in unsupervised learning settings. Experimental results on several
datasets and tasks show that LENs may yield better classifications than
established white-box models, such as decision trees and Bayesian rule lists,
while providing more compact and meaningful explanations.


http://arxiv.org/abs/2108.04979
Simple black-box universal adversarial attacks on medical image classification based on deep neural networks. (99%)
Kazuki Koga; Kazuhiro Takemoto
  Universal adversarial attacks, which hinder most deep neural network (DNN)
tasks using only a small single perturbation called a universal adversarial
perturbation (UAP), is a realistic security threat to the practical application
of a DNN. In particular, such attacks cause serious problems in medical
imaging. Given that computer-based systems are generally operated under a
black-box condition in which only queries on inputs are allowed and outputs are
accessible, the impact of UAPs seems to be limited because well-used algorithms
for generating UAPs are limited to a white-box condition in which adversaries
can access the model weights and loss gradients. Nevertheless, we demonstrate
that UAPs are easily generatable using a relatively small dataset under
black-box conditions. In particular, we propose a method for generating UAPs
using a simple hill-climbing search based only on DNN outputs and demonstrate
the validity of the proposed method using representative DNN-based medical
image classifications. Black-box UAPs can be used to conduct both non-targeted
and targeted attacks. Overall, the black-box UAPs showed high attack success
rates (40% to 90%), although some of them had relatively low success rates
because the method only utilizes limited information to generate UAPs. The
vulnerability of black-box UAPs was observed in several model architectures.
The results indicate that adversaries can also generate UAPs through a simple
procedure under the black-box condition to foil or control DNN-based medical
image diagnoses, and that UAPs are a more realistic security threat.


http://arxiv.org/abs/2108.04890
On the Effect of Pruning on Adversarial Robustness. (81%)
Artur Jordao; Helio Pedrini
  Pruning is a well-known mechanism for reducing the computational cost of deep
convolutional networks. However, studies have shown the potential of pruning as
a form of regularization, which reduces overfitting and improves
generalization. We demonstrate that this family of strategies provides
additional benefits beyond computational performance and generalization. Our
analyses reveal that pruning structures (filters and/or layers) from
convolutional networks increase not only generalization but also robustness to
adversarial images (natural images with content modified). Such achievements
are possible since pruning reduces network capacity and provides
regularization, which have been proven effective tools against adversarial
images. In contrast to promising defense mechanisms that require training with
adversarial images and careful regularization, we show that pruning obtains
competitive results considering only natural images (e.g., the standard and
low-cost training). We confirm these findings on several adversarial attacks
and architectures; thus suggesting the potential of pruning as a novel defense
mechanism against adversarial images.


http://arxiv.org/abs/2108.04974
SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version). (68%)
Nils Lukas; Edward Jiang; Xinda Li; Florian Kerschbaum
  Deep Neural Network (DNN) watermarking is a method for provenance
verification of DNN models. Watermarking should be robust against watermark
removal attacks that derive a surrogate model that evades provenance
verification. Many watermarking schemes that claim robustness have been
proposed, but their robustness is only validated in isolation against a
relatively small set of attacks. There is no systematic, empirical evaluation
of these claims against a common, comprehensive set of removal attacks. This
uncertainty about a watermarking scheme's robustness causes difficulty to trust
their deployment in practice. In this paper, we evaluate whether recently
proposed watermarking schemes that claim robustness are robust against a large
set of removal attacks. We survey methods from the literature that (i) are
known removal attacks, (ii) derive surrogate models but have not been evaluated
as removal attacks, and (iii) novel removal attacks. Weight shifting and smooth
retraining are novel removal attacks adapted to the DNN watermarking schemes
surveyed in this paper. We propose taxonomies for watermarking schemes and
removal attacks. Our empirical evaluation includes an ablation study over sets
of parameters for each attack and watermarking scheme on the CIFAR-10 and
ImageNet datasets. Surprisingly, none of the surveyed watermarking schemes is
robust in practice. We find that schemes fail to withstand adaptive attacks and
known methods for deriving surrogate models that have not been evaluated as
removal attacks. This points to intrinsic flaws in how robustness is currently
evaluated. We show that watermarking schemes need to be evaluated against a
more extensive set of removal attacks with a more realistic adversary model.
Our source code and a complete dataset of evaluation results are publicly
available, which allows to independently verify our conclusions.


http://arxiv.org/abs/2108.04990
Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing. (64%)
Sanchit Sinha; Hanjie Chen; Arshdeep Sekhon; Yangfeng Ji; Yanjun Qi
  Interpretability methods like Integrated Gradient and LIME are popular
choices for explaining natural language model predictions with relative word
importance scores. These interpretations need to be robust for trustworthy NLP
applications in high-stake areas like medicine or finance. Our paper
demonstrates how interpretations can be manipulated by making simple word
perturbations on an input text. Via a small portion of word-level swaps, these
adversarial perturbations aim to make the resulting text semantically and
spatially similar to its seed input (therefore sharing similar
interpretations). Simultaneously, the generated examples achieve the same
prediction label as the seed yet are given a substantially different
explanation by the interpretation methods. Our experiments generate fragile
interpretations to attack two SOTA interpretation methods, across three popular
Transformer models and on two different NLP datasets. We observe that the rank
order correlation drops by over 20% when less than 10% of words are perturbed
on average. Further, rank-order correlation keeps decreasing as more words get
perturbed. Furthermore, we demonstrate that candidates generated from our
method have good quality metrics.


http://arxiv.org/abs/2108.04584
UniNet: A Unified Scene Understanding Network and Exploring Multi-Task Relationships through the Lens of Adversarial Attacks. (2%)
NareshKumar Gurulingan; Elahe Arani; Bahram Zonooz
  Scene understanding is crucial for autonomous systems which intend to operate
in the real world. Single task vision networks extract information only based
on some aspects of the scene. In multi-task learning (MTL), on the other hand,
these single tasks are jointly learned, thereby providing an opportunity for
tasks to share information and obtain a more comprehensive understanding. To
this end, we develop UniNet, a unified scene understanding network that
accurately and efficiently infers vital vision tasks including object
detection, semantic segmentation, instance segmentation, monocular depth
estimation, and monocular instance depth prediction. As these tasks look at
different semantic and geometric information, they can either complement or
conflict with each other. Therefore, understanding inter-task relationships can
provide useful cues to enable complementary information sharing. We evaluate
the task relationships in UniNet through the lens of adversarial attacks based
on the notion that they can exploit learned biases and task interactions in the
neural network. Extensive experiments on the Cityscapes dataset, using
untargeted and targeted attacks reveal that semantic tasks strongly interact
amongst themselves, and the same holds for geometric tasks. Additionally, we
show that the relationship between semantic and geometric tasks is asymmetric
and their interaction becomes weaker as we move towards higher-level
representations.


http://arxiv.org/abs/2108.04547
Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation. (1%)
Weilun Wang; Wengang Zhou; Jianmin Bao; Dong Chen; Houqiang Li
  Contrastive learning shows great potential in unpaired image-to-image
translation, but sometimes the translated results are in poor quality and the
contents are not preserved consistently. In this paper, we uncover that the
negative examples play a critical role in the performance of contrastive
learning for image translation. The negative examples in previous methods are
randomly sampled from the patches of different positions in the source image,
which are not effective to push the positive examples close to the query
examples. To address this issue, we present instance-wise hard Negative Example
Generation for Contrastive learning in Unpaired image-to-image
Translation~(NEGCUT). Specifically, we train a generator to produce negative
examples online. The generator is novel from two perspectives: 1) it is
instance-wise which means that the generated examples are based on the input
image, and 2) it can generate hard negative examples since it is trained with
an adversarial loss. With the generator, the performance of unpaired
image-to-image translation is significantly improved. Experiments on three
benchmark datasets demonstrate that the proposed NEGCUT framework achieves
state-of-the-art performance compared to previous methods.


http://arxiv.org/abs/2108.04204
Meta Gradient Adversarial Attack. (99%)
Zheng Yuan; Jie Zhang; Yunpei Jia; Chuanqi Tan; Tao Xue; Shiguang Shan
  In recent years, research on adversarial attacks has become a hot spot.
Although current literature on the transfer-based adversarial attack has
achieved promising results for improving the transferability to unseen
black-box models, it still leaves a long way to go. Inspired by the idea of
meta-learning, this paper proposes a novel architecture called Meta Gradient
Adversarial Attack (MGAA), which is plug-and-play and can be integrated with
any existing gradient-based attack method for improving the cross-model
transferability. Specifically, we randomly sample multiple models from a model
zoo to compose different tasks and iteratively simulate a white-box attack and
a black-box attack in each task. By narrowing the gap between the gradient
directions in white-box and black-box attacks, the transferability of
adversarial examples on the black-box setting can be improved. Extensive
experiments on the CIFAR10 and ImageNet datasets show that our architecture
outperforms the state-of-the-art methods for both black-box and white-box
attack settings.


http://arxiv.org/abs/2108.04409
On Procedural Adversarial Noise Attack And Defense. (99%)
Jun Yan; Xiaoyang Deng; Huilin Yin; Wancheng Ge
  Deep Neural Networks (DNNs) are vulnerable to adversarial examples which
would inveigle neural networks to make prediction errors with small
perturbations on the input images. Researchers have been devoted to promoting
the research on the universal adversarial perturbations (UAPs) which are
gradient-free and have little prior knowledge on data distributions. Procedural
adversarial noise attack is a data-free universal perturbation generation
method. In this paper, we propose two universal adversarial perturbation (UAP)
generation methods based on procedural noise functions: Simplex noise and
Worley noise. In our framework, the shading which disturbs visual
classification is generated with rendering technology. Without changing the
semantic representations, the adversarial examples generated via our methods
show superior performance on the attack.


http://arxiv.org/abs/2108.04430
Enhancing Knowledge Tracing via Adversarial Training. (98%)
Xiaopeng Guo; Zhijie Huang; Jie Gao; Mingyu Shang; Maojing Shu; Jun Sun
  We study the problem of knowledge tracing (KT) where the goal is to trace the
students' knowledge mastery over time so as to make predictions on their future
performance. Owing to the good representation capacity of deep neural networks
(DNNs), recent advances on KT have increasingly concentrated on exploring DNNs
to improve the performance of KT. However, we empirically reveal that the DNNs
based KT models may run the risk of overfitting, especially on small datasets,
leading to limited generalization. In this paper, by leveraging the current
advances in adversarial training (AT), we propose an efficient AT based KT
method (ATKT) to enhance KT model's generalization and thus push the limit of
KT. Specifically, we first construct adversarial perturbations and add them on
the original interaction embeddings as adversarial examples. The original and
adversarial examples are further used to jointly train the KT model, forcing it
is not only to be robust to the adversarial examples, but also to enhance the
generalization over the original ones. To better implement AT, we then present
an efficient attentive-LSTM model as KT backbone, where the key is a proposed
knowledge hidden state attention module that adaptively aggregates information
from previous knowledge hidden states while simultaneously highlighting the
importance of current knowledge hidden state to make a more accurate
prediction. Extensive experiments on four public benchmark datasets demonstrate
that our ATKT achieves new state-of-the-art performance. Code is available at:
\color{blue} {\url{https://github.com/xiaopengguo/ATKT}}.


http://arxiv.org/abs/2108.04214
Neural Network Repair with Reachability Analysis. (96%)
Xiaodong Yang; Tom Yamaguchi; Hoang-Dung Tran; Bardh Hoxha; Taylor T Johnson; Danil Prokhorov
  Safety is a critical concern for the next generation of autonomy that is
likely to rely heavily on deep neural networks for perception and control.
Formally verifying the safety and robustness of well-trained DNNs and
learning-enabled systems under attacks, model uncertainties, and sensing errors
is essential for safe autonomy. This research proposes a framework to repair
unsafe DNNs in safety-critical systems with reachability analysis. The repair
process is inspired by adversarial training which has demonstrated high
effectiveness in improving the safety and robustness of DNNs. Different from
traditional adversarial training approaches where adversarial examples are
utilized from random attacks and may not be representative of all unsafe
behaviors, our repair process uses reachability analysis to compute the exact
unsafe regions and identify sufficiently representative examples to enhance the
efficacy and efficiency of the adversarial training.
  The performance of our framework is evaluated on two types of benchmarks
without safe models as references. One is a DNN controller for aircraft
collision avoidance with access to training data. The other is a rocket lander
where our framework can be seamlessly integrated with the well-known deep
deterministic policy gradient (DDPG) reinforcement learning algorithm. The
experimental results show that our framework can successfully repair all
instances on multiple safety specifications with negligible performance
degradation. In addition, to increase the computational and memory efficiency
of the reachability analysis algorithm, we propose a depth-first-search
algorithm that combines an existing exact analysis method with an
over-approximation approach based on a new set representation. Experimental
results show that our method achieves a five-fold improvement in runtime and a
two-fold improvement in memory usage compared to exact analysis.


http://arxiv.org/abs/2108.04206
Classification Auto-Encoder based Detector against Diverse Data Poisoning Attacks. (92%)
Fereshteh Razmi; Li Xiong
  Poisoning attacks are a category of adversarial machine learning threats in
which an adversary attempts to subvert the outcome of the machine learning
systems by injecting crafted data into training data set, thus increasing the
machine learning model's test error. The adversary can tamper with the data
feature space, data labels, or both, each leading to a different attack
strategy with different strengths. Various detection approaches have recently
emerged, each focusing on one attack strategy. The Achilles heel of many of
these detection approaches is their dependence on having access to a clean,
untampered data set. In this paper, we propose CAE, a Classification
Auto-Encoder based detector against diverse poisoned data. CAE can detect all
forms of poisoning attacks using a combination of reconstruction and
classification errors without having any prior knowledge of the attack
strategy. We show that an enhanced version of CAE (called CAE+) does not have
to employ a clean data set to train the defense model. Our experimental results
on three real datasets MNIST, Fashion-MNIST and CIFAR demonstrate that our
proposed method can maintain its functionality under up to 30% contaminated
data and help the defended SVM classifier to regain its best accuracy.


http://arxiv.org/abs/2108.03803
Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning. (82%)
Wanqi Xue; Wei Qiu; Bo An; Zinovi Rabinovich; Svetlana Obraztsova; Chai Kiat Yeo
  Recent studies in multi-agent communicative reinforcement learning (MACRL)
demonstrate that multi-agent coordination can be significantly improved when
communication between agents is allowed. Meanwhile, advances in adversarial
machine learning (ML) have shown that ML and reinforcement learning (RL) models
are vulnerable to a variety of attacks that significantly degrade the
performance of learned behaviours. However, despite the obvious and growing
importance, the combination of adversarial ML and MACRL remains largely
uninvestigated. In this paper, we make the first step towards conducting
message attacks on MACRL methods. In our formulation, one agent in the
cooperating group is taken over by an adversary and can send malicious messages
to disrupt a deployed MACRL-based coordinated strategy during the deployment
phase. We further our study by developing a defence method via message
reconstruction. Finally, we address the resulting arms race, i.e., we consider
the ability of the malicious agent to adapt to the changing and improving
defensive communicative policies of the benign agents. Specifically, we model
the adversarial MACRL problem as a two-player zero-sum game and then utilize
Policy-Space Response Oracle to achieve communication robustness. Empirically,
we demonstrate that MACRL methods are vulnerable to message attacks while our
defence method the game-theoretic framework can effectively improve the
robustness of MACRL.


http://arxiv.org/abs/2108.04417
Privacy-Preserving Machine Learning: Methods, Challenges and Directions. (16%)
Runhua Xu; Nathalie Baracaldo; James Joshi
  Machine learning (ML) is increasingly being adopted in a wide variety of
application domains. Usually, a well-performing ML model, especially, emerging
deep neural network model, relies on a large volume of training data and
high-powered computational resources. The need for a vast volume of available
data raises serious privacy concerns because of the risk of leakage of highly
privacy-sensitive information and the evolving regulatory environments that
increasingly restrict access to and use of privacy-sensitive data. Furthermore,
a trained ML model may also be vulnerable to adversarial attacks such as
membership/property inference attacks and model inversion attacks. Hence,
well-designed privacy-preserving ML (PPML) solutions are crucial and have
attracted increasing research interest from academia and industry. More and
more efforts of PPML are proposed via integrating privacy-preserving techniques
into ML algorithms, fusing privacy-preserving approaches into ML pipeline, or
designing various privacy-preserving architectures for existing ML systems. In
particular, existing PPML arts cross-cut ML, system, security, and privacy;
hence, there is a critical need to understand state-of-art studies, related
challenges, and a roadmap for future research. This paper systematically
reviews and summarizes existing privacy-preserving approaches and proposes a
PGU model to guide evaluation for various PPML solutions through elaborately
decomposing their privacy-preserving functionalities. The PGU model is designed
as the triad of Phase, Guarantee, and technical Utility. Furthermore, we also
discuss the unique characteristics and challenges of PPML and outline possible
directions of future work that benefit a wide range of research communities
among ML, distributed systems, security, and privacy areas.


http://arxiv.org/abs/2108.04345
Explainable AI and susceptibility to adversarial attacks: a case study in classification of breast ultrasound images. (15%)
Hamza Rasaee; Hassan Rivaz
  Ultrasound is a non-invasive imaging modality that can be conveniently used
to classify suspicious breast nodules and potentially detect the onset of
breast cancer. Recently, Convolutional Neural Networks (CNN) techniques have
shown promising results in classifying ultrasound images of the breast into
benign or malignant. However, CNN inference acts as a black-box model, and as
such, its decision-making is not interpretable. Therefore, increasing effort
has been dedicated to explaining this process, most notably through GRAD-CAM
and other techniques that provide visual explanations into inner workings of
CNNs. In addition to interpretation, these methods provide clinically important
information, such as identifying the location for biopsy or treatment. In this
work, we analyze how adversarial assaults that are practically undetectable may
be devised to alter these importance maps dramatically. Furthermore, we will
show that this change in the importance maps can come with or without altering
the classification result, rendering them even harder to detect. As such, care
must be taken when using these importance maps to shed light on the inner
workings of deep learning. Finally, we utilize Multi-Task Learning (MTL) and
propose a new network based on ResNet-50 to improve the classification
accuracies. Our sensitivity and specificity is comparable to the state of the
art results.


http://arxiv.org/abs/2108.03388
Jointly Attacking Graph Neural Network and its Explanations. (96%)
Wenqi Fan; Wei Jin; Xiaorui Liu; Han Xu; Xianfeng Tang; Suhang Wang; Qing Li; Jiliang Tang; Jianping Wang; Charu Aggarwal
  Graph Neural Networks (GNNs) have boosted the performance for many
graph-related tasks. Despite the great success, recent studies have shown that
GNNs are highly vulnerable to adversarial attacks, where adversaries can
mislead the GNNs' prediction by modifying graphs. On the other hand, the
explanation of GNNs (GNNExplainer) provides a better understanding of a trained
GNN model by generating a small subgraph and features that are most influential
for its prediction. In this paper, we first perform empirical studies to
validate that GNNExplainer can act as an inspection tool and have the potential
to detect the adversarial perturbations for graphs. This finding motivates us
to further initiate a new problem investigation: Whether a graph neural network
and its explanations can be jointly attacked by modifying graphs with malicious
desires? It is challenging to answer this question since the goals of
adversarial attacks and bypassing the GNNExplainer essentially contradict each
other. In this work, we give a confirmative answer to this question by
proposing a novel attack framework (GEAttack), which can attack both a GNN
model and its explanations by simultaneously exploiting their vulnerabilities.
Extensive experiments on two explainers (GNNExplainer and PGExplainer) under
various real-world datasets demonstrate the effectiveness of the proposed
method.


http://arxiv.org/abs/2108.03506
Membership Inference Attacks on Lottery Ticket Networks. (33%)
Aadesh Bagmar; Shishira R Maiya; Shruti Bidwalka; Amol Deshpande
  The vulnerability of the Lottery Ticket Hypothesis has not been studied from
the purview of Membership Inference Attacks. Through this work, we are the
first to empirically show that the lottery ticket networks are equally
vulnerable to membership inference attacks. A Membership Inference Attack (MIA)
is the process of determining whether a data sample belongs to a training set
of a trained model or not. Membership Inference Attacks could leak critical
information about the training data that can be used for targeted attacks.
Recent deep learning models often have very large memory footprints and a high
computational cost associated with training and drawing inferences. Lottery
Ticket Hypothesis is used to prune the networks to find smaller sub-networks
that at least match the performance of the original model in terms of test
accuracy in a similar number of iterations. We used CIFAR-10, CIFAR-100, and
ImageNet datasets to perform image classification tasks and observe that the
attack accuracies are similar. We also see that the attack accuracy varies
directly according to the number of classes in the dataset and the sparsity of
the network. We demonstrate that these attacks are transferable across models
with high accuracy.


http://arxiv.org/abs/2108.03418
Information Bottleneck Approach to Spatial Attention Learning. (1%)
Qiuxia Lai; Yu Li; Ailing Zeng; Minhao Liu; Hanqiu Sun; Qiang Xu
  The selective visual attention mechanism in the human visual system (HVS)
restricts the amount of information to reach visual awareness for perceiving
natural scenes, allowing near real-time information processing with limited
computational capacity [Koch and Ullman, 1987]. This kind of selectivity acts
as an 'Information Bottleneck (IB)', which seeks a trade-off between
information compression and predictive accuracy. However, such information
constraints are rarely explored in the attention mechanism for deep neural
networks (DNNs). In this paper, we propose an IB-inspired spatial attention
module for DNN structures built for visual recognition. The module takes as
input an intermediate representation of the input image, and outputs a
variational 2D attention map that minimizes the mutual information (MI) between
the attention-modulated representation and the input, while maximizing the MI
between the attention-modulated representation and the task label. To further
restrict the information bypassed by the attention map, we quantize the
continuous attention scores to a set of learnable anchor values during
training. Extensive experiments show that the proposed IB-inspired spatial
attention mechanism can yield attention maps that neatly highlight the regions
of interest while suppressing backgrounds, and bootstrap standard DNN
structures for visual recognition tasks (e.g., image classification,
fine-grained recognition, cross-domain classification). The attention maps are
interpretable for the decision making of the DNNs as verified in the
experiments. Our code is available at https://github.com/ashleylqx/AIB.git.


http://arxiv.org/abs/2108.02940
Evaluating Adversarial Attacks on Driving Safety in Vision-Based Autonomous Vehicles. (80%)
Jindi Zhang; Yang Lou; Jianping Wang; Kui Wu; Kejie Lu; Xiaohua Jia
  In recent years, many deep learning models have been adopted in autonomous
driving. At the same time, these models introduce new vulnerabilities that may
compromise the safety of autonomous vehicles. Specifically, recent studies have
demonstrated that adversarial attacks can cause a significant decline in
detection precision of deep learning-based 3D object detection models. Although
driving safety is the ultimate concern for autonomous driving, there is no
comprehensive study on the linkage between the performance of deep learning
models and the driving safety of autonomous vehicles under adversarial attacks.
In this paper, we investigate the impact of two primary types of adversarial
attacks, perturbation attacks and patch attacks, on the driving safety of
vision-based autonomous vehicles rather than the detection precision of deep
learning models. In particular, we consider two state-of-the-art models in
vision-based 3D object detection, Stereo R-CNN and DSGN. To evaluate driving
safety, we propose an end-to-end evaluation framework with a set of driving
safety performance metrics. By analyzing the results of our extensive
evaluation experiments, we find that (1) the attack's impact on the driving
safety of autonomous vehicles and the attack's impact on the precision of 3D
object detectors are decoupled, and (2) the DSGN model demonstrates stronger
robustness to adversarial attacks than the Stereo R-CNN model. In addition, we
further investigate the causes behind the two findings with an ablation study.
The findings of this paper provide a new perspective to evaluate adversarial
attacks and guide the selection of deep learning models in autonomous driving.


http://arxiv.org/abs/2108.03288
Ensemble Augmentation for Deep Neural Networks Using 1-D Time Series Vibration Data. (2%)
Atik Faysal; Ngui Wai Keng; M. H. Lim
  Time-series data are one of the fundamental types of raw data representation
used in data-driven techniques. In machine condition monitoring, time-series
vibration data are overly used in data mining for deep neural networks.
Typically, vibration data is converted into images for classification using
Deep Neural Networks (DNNs), and scalograms are the most effective form of
image representation. However, the DNN classifiers require huge labeled
training samples to reach their optimum performance. So, many forms of data
augmentation techniques are applied to the classifiers to compensate for the
lack of training samples. However, the scalograms are graphical representations
where the existing augmentation techniques suffer because they either change
the graphical meaning or have too much noise in the samples that change the
physical meaning. In this study, a data augmentation technique named ensemble
augmentation is proposed to overcome this limitation. This augmentation method
uses the power of white noise added in ensembles to the original samples to
generate real-like samples. After averaging the signal with ensembles, a new
signal is obtained that contains the characteristics of the original signal.
The parameters for the ensemble augmentation are validated using a simulated
signal. The proposed method is evaluated using 10 class bearing vibration data
using three state-of-the-art Transfer Learning (TL) models, namely,
Inception-V3, MobileNet-V2, and ResNet50. Augmented samples are generated in
two increments: the first increment generates the same number of fake samples
as the training samples, and in the second increment, the number of samples is
increased gradually. The outputs from the proposed method are compared with no
augmentation, augmentations using deep convolution generative adversarial
network (DCGAN), and several geometric transformation-based augmentations...


http://arxiv.org/abs/2108.02756
BOSS: Bidirectional One-Shot Synthesis of Adversarial Examples. (99%)
Ismail Alkhouri; Alvaro Velasquez; George Atia
  The design of additive imperceptible perturbations to the inputs of deep
classifiers to maximize their misclassification rates is a central focus of
adversarial machine learning. An alternative approach is to synthesize
adversarial examples from scratch using GAN-like structures, albeit with the
use of large amounts of training data. By contrast, this paper considers
one-shot synthesis of adversarial examples; the inputs are synthesized from
scratch to induce arbitrary soft predictions at the output of pre-trained
models, while simultaneously maintaining high similarity to specified inputs.
To this end, we present a problem that encodes objectives on the distance
between the desired and output distributions of the trained model and the
similarity between such inputs and the synthesized examples. We prove that the
formulated problem is NP-complete. Then, we advance a generative approach to
the solution in which the adversarial examples are obtained as the output of a
generative network whose parameters are iteratively updated by optimizing
surrogate loss functions for the dual-objective. We demonstrate the generality
and versatility of the framework and approach proposed through applications to
the design of targeted adversarial attacks, generation of decision boundary
samples, and synthesis of low confidence classification inputs. The approach is
further extended to an ensemble of models with different soft output
specifications. The experimental results verify that the targeted and
confidence reduction attack methods developed perform on par with
state-of-the-art algorithms.


http://arxiv.org/abs/2108.02488
Poison Ink: Robust and Invisible Backdoor Attack. (99%)
Jie Zhang; Dongdong Chen; Jing Liao; Qidong Huang; Gang Hua; Weiming Zhang; Nenghai Yu
  Recent research shows deep neural networks are vulnerable to different types
of attacks, such as adversarial attack, data poisoning attack and backdoor
attack. Among them, backdoor attack is the most cunning one and can occur in
almost every stage of deep learning pipeline. Therefore, backdoor attack has
attracted lots of interests from both academia and industry. However, most
existing backdoor attack methods are either visible or fragile to some
effortless pre-processing such as common data transformations. To address these
limitations, we propose a robust and invisible backdoor attack called "Poison
Ink". Concretely, we first leverage the image structures as target poisoning
areas, and fill them with poison ink (information) to generate the trigger
pattern. As the image structure can keep its semantic meaning during the data
transformation, such trigger pattern is inherently robust to data
transformations. Then we leverage a deep injection network to embed such
trigger pattern into the cover image to achieve stealthiness. Compared to
existing popular backdoor attack methods, Poison Ink outperforms both in
stealthiness and robustness. Through extensive experiments, we demonstrate
Poison Ink is not only general to different datasets and network architectures,
but also flexible for different attack scenarios. Besides, it also has very
strong resistance against many state-of-the-art defense techniques.


http://arxiv.org/abs/2108.02502
Imperceptible Adversarial Examples by Spatial Chroma-Shift. (99%)
Ayberk Aydin; Deniz Sen; Berat Tuna Karli; Oguz Hanoglu; Alptekin Temizel
  Deep Neural Networks have been shown to be vulnerable to various kinds of
adversarial perturbations. In addition to widely studied additive noise based
perturbations, adversarial examples can also be created by applying a per pixel
spatial drift on input images. While spatial transformation based adversarial
examples look more natural to human observers due to absence of additive noise,
they still possess visible distortions caused by spatial transformations. Since
the human vision is more sensitive to the distortions in the luminance compared
to those in chrominance channels, which is one of the main ideas behind the
lossy visual multimedia compression standards, we propose a spatial
transformation based perturbation method to create adversarial examples by only
modifying the color components of an input image. While having competitive
fooling rates on CIFAR-10 and NIPS2017 Adversarial Learning Challenge datasets,
examples created with the proposed method have better scores with regards to
various perceptual quality metrics. Human visual perception studies validate
that the examples are more natural looking and often indistinguishable from
their original counterparts.


http://arxiv.org/abs/2108.04062
Householder Activations for Provable Robustness against Adversarial Attacks. (83%)
Sahil Singla; Surbhi Singla; Soheil Feizi
  Training convolutional neural networks (CNNs) with a strict Lipschitz
constraint under the l_{2} norm is useful for provable adversarial robustness,
interpretable gradients and stable training. While 1-Lipschitz CNNs can be
designed by enforcing a 1-Lipschitz constraint on each layer, training such
networks requires each layer to have an orthogonal Jacobian matrix (for all
inputs) to prevent gradients from vanishing during backpropagation. A layer
with this property is said to be Gradient Norm Preserving (GNP). To construct
expressive GNP activation functions, we first prove that the Jacobian of any
GNP piecewise linear function is only allowed to change via Householder
transformations for the function to be continuous. Building on this result, we
introduce a class of nonlinear GNP activations with learnable Householder
transformations called Householder activations. A householder activation
parameterized by the vector $\mathbf{v}$ outputs $(\mathbf{I} -
2\mathbf{v}\mathbf{v}^{T})\mathbf{z}$ for its input $\mathbf{z}$ if
$\mathbf{v}^{T}\mathbf{z} \leq 0$; otherwise it outputs $\mathbf{z}$. Existing
GNP activations such as $\mathrm{MaxMin}$ can be viewed as special cases of
$\mathrm{HH}$ activations for certain settings of these transformations. Thus,
networks with $\mathrm{HH}$ activations have higher expressive power than those
with $\mathrm{MaxMin}$ activations. Although networks with $\mathrm{HH}$
activations have nontrivial provable robustness against adversarial attacks, we
further boost their robustness by (i) introducing a certificate regularization
and (ii) relaxing orthogonalization of the last layer of the network. Our
experiments on CIFAR-10 and CIFAR-100 show that our regularized networks with
$\mathrm{HH}$ activations lead to significant improvements in both the standard
and provable robust accuracy over the prior works (gain of 3.65\% and 4.46\% on
CIFAR-100 respectively).


http://arxiv.org/abs/2108.02707
Fairness Properties of Face Recognition and Obfuscation Systems. (68%)
Harrison Rosenberg; Brian Tang; Kassem Fawaz; Somesh Jha
  The proliferation of automated facial recognition in various commercial and
government sectors has caused significant privacy concerns for individuals. A
recent and popular approach to address these privacy concerns is to employ
evasion attacks against the metric embedding networks powering facial
recognition systems. Face obfuscation systems generate imperceptible
perturbations, when added to an image, cause the facial recognition system to
misidentify the user. The key to these approaches is the generation of
perturbations using a pre-trained metric embedding network followed by their
application to an online system, whose model might be proprietary. This
dependence of face obfuscation on metric embedding networks, which are known to
be unfair in the context of facial recognition, surfaces the question of
demographic fairness -- \textit{are there demographic disparities in the
performance of face obfuscation systems?} To address this question, we perform
an analytical and empirical exploration of the performance of recent face
obfuscation systems that rely on deep embedding networks. We find that metric
embedding networks are demographically aware; they cluster faces in the
embedding space based on their demographic attributes. We observe that this
effect carries through to the face obfuscation systems: faces belonging to
minority groups incur reduced utility compared to those from majority groups.
For example, the disparity in average obfuscation success rate on the online
Face++ API can reach up to 20 percentage points. Further, for some demographic
groups, the average perturbation size increases by up to 17\% when choosing a
target identity belonging to a different demographic group versus the same
demographic group. Finally, we present a simple analytical model to provide
insights into these phenomena.


http://arxiv.org/abs/2108.02360
Exploring Structure Consistency for Deep Model Watermarking. (10%)
Jie Zhang; Dongdong Chen; Jing Liao; Han Fang; Zehua Ma; Weiming Zhang; Gang Hua; Nenghai Yu
  The intellectual property (IP) of Deep neural networks (DNNs) can be easily
``stolen'' by surrogate model attack. There has been significant progress in
solutions to protect the IP of DNN models in classification tasks. However,
little attention has been devoted to the protection of DNNs in image processing
tasks. By utilizing consistent invisible spatial watermarks, one recent work
first considered model watermarking for deep image processing networks and
demonstrated its efficacy in many downstream tasks. Nevertheless, it highly
depends on the hypothesis that the embedded watermarks in the network outputs
are consistent. When the attacker uses some common data augmentation attacks
(e.g., rotate, crop, and resize) during surrogate model training, it will
totally fail because the underlying watermark consistency is destroyed. To
mitigate this issue, we propose a new watermarking methodology, namely
``structure consistency'', based on which a new deep structure-aligned model
watermarking algorithm is designed. Specifically, the embedded watermarks are
designed to be aligned with physically consistent image structures, such as
edges or semantic regions. Experiments demonstrate that our method is much more
robust than the baseline method in resisting data augmentation attacks for
model IP protection. Besides that, we further test the generalization ability
and robustness of our method to a broader range of circumvention attacks.


http://arxiv.org/abs/2108.02501
Locally Interpretable One-Class Anomaly Detection for Credit Card Fraud Detection. (1%)
Tungyu Wu; Youting Wang
  For the highly imbalanced credit card fraud detection problem, most existing
methods either use data augmentation methods or conventional machine learning
models, while neural network-based anomaly detection approaches are lacking.
Furthermore, few studies have employed AI interpretability tools to investigate
the feature importance of transaction data, which is crucial for the black-box
fraud detection module. Considering these two points together, we propose a
novel anomaly detection framework for credit card fraud detection as well as a
model-explaining module responsible for prediction explanations. The fraud
detection model is composed of two deep neural networks, which are trained in
an unsupervised and adversarial manner. Precisely, the generator is an
AutoEncoder aiming to reconstruct genuine transaction data, while the
discriminator is a fully-connected network for fraud detection. The explanation
module has three white-box explainers in charge of interpretations of the
AutoEncoder, discriminator, and the whole detection model, respectively.
Experimental results show the state-of-the-art performances of our fraud
detection model on the benchmark dataset compared with baselines. In addition,
prediction analyses by three explainers are presented, offering a clear
perspective on how each feature of an instance of interest contributes to the
final model output.


http://arxiv.org/abs/2108.02340
Robust Transfer Learning with Pretrained Language Models through Adapters. (82%)
Wenjuan Han; Bo Pang; Yingnian Wu
  Transfer learning with large pretrained transformer-based language models
like BERT has become a dominating approach for most NLP tasks. Simply
fine-tuning those large language models on downstream tasks or combining it
with task-specific pretraining is often not robust. In particular, the
performance considerably varies as the random seed changes or the number of
pretraining and/or fine-tuning iterations varies, and the fine-tuned model is
vulnerable to adversarial attack. We propose a simple yet effective
adapter-based approach to mitigate these issues. Specifically, we insert small
bottleneck layers (i.e., adapter) within each layer of a pretrained model, then
fix the pretrained layers and train the adapter layers on the downstream task
data, with (1) task-specific unsupervised pretraining and then (2)
task-specific supervised training (e.g., classification, sequence labeling).
Our experiments demonstrate that such a training scheme leads to improved
stability and adversarial robustness in transfer learning to various downstream
tasks.


http://arxiv.org/abs/2108.01852
Semi-supervised Conditional GAN for Simultaneous Generation and Detection of Phishing URLs: A Game theoretic Perspective. (31%)
Sharif Amit Kamran; Shamik Sengupta; Alireza Tavakkoli
  Spear Phishing is a type of cyber-attack where the attacker sends hyperlinks
through email on well-researched targets. The objective is to obtain sensitive
information such as name, credentials, credit card numbers, or other crucial
data by imitating oneself as a trustworthy website. According to a recent
report, phishing incidents nearly doubled in frequency in 2020. In recent
times, machine learning techniques have become the standard for defending
against such attacks. Many augmentations have been made for improving the
existing architectures, such as Convolutional Networks, Recurrent Networks, and
Generative Adversarial Networks. However, these architectures were designed
with only defense in mind. Moreover, the attacker's perspective and motivation
are absent while creating such training and deployment pipelines. To address
this, we need a game-theoretic approach to understand the rational
decision-making process of the attacker (Hacker) and the defender (Phishing URL
detector). We propose a Conditional Generative Adversarial Network for
real-time phishing URL detection. Additionally, we train our architecture in a
semi-supervised manner to distinguish between adversarial and real examples,
along with detecting malicious and benign URLs. We also design two games
between the attacker and defender in training and deployment settings by
utilizing the game-theoretic perspective. Our experiments confirm that the
proposed architecture surpasses recent state-of-the-art architectures for
phishing URLs detection.


http://arxiv.org/abs/2108.01807
On the Robustness of Domain Adaption to Adversarial Attacks. (99%)
Liyuan Zhang; Yuhang Zhou; Lei Zhang
  State-of-the-art deep neural networks (DNNs) have been proved to have
excellent performance on unsupervised domain adaption (UDA). However, recent
work shows that DNNs perform poorly when being attacked by adversarial samples,
where these attacks are implemented by simply adding small disturbances to the
original images. Although plenty of work has focused on this, as far as we
know, there is no systematic research on the robustness of unsupervised domain
adaption model. Hence, we discuss the robustness of unsupervised domain
adaption against adversarial attacking for the first time. We benchmark various
settings of adversarial attack and defense in domain adaption, and propose a
cross domain attack method based on pseudo label. Most importantly, we analyze
the impact of different datasets, models, attack methods and defense methods.
Directly, our work proves the limited robustness of unsupervised domain
adaptation model, and we hope our work may facilitate the community to pay more
attention to improve the robustness of the model against attacking.


http://arxiv.org/abs/2108.02010
On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples. (99%)
Adelin Travers; Lorna Licollari; Guanghan Wang; Varun Chandrasekaran; Adam Dziedzic; David Lie; Nicolas Papernot
  Machine learning (ML) models are known to be vulnerable to adversarial
examples. Applications of ML to voice biometrics authentication are no
exception. Yet, the implications of audio adversarial examples on these
real-world systems remain poorly understood given that most research targets
limited defenders who can only listen to the audio samples. Conflating
detectability of an attack with human perceptibility, research has focused on
methods that aim to produce imperceptible adversarial examples which humans
cannot distinguish from the corresponding benign samples. We argue that this
perspective is coarse for two reasons: 1. Imperceptibility is impossible to
verify; it would require an experimental process that encompasses variations in
listener training, equipment, volume, ear sensitivity, types of background
noise etc, and 2. It disregards pipeline-based detection clues that realistic
defenders leverage. This results in adversarial examples that are ineffective
in the presence of knowledgeable defenders. Thus, an adversary only needs an
audio sample to be plausible to a human. We thus introduce surreptitious
adversarial examples, a new class of attacks that evades both human and
pipeline controls. In the white-box setting, we instantiate this class with a
joint, multi-stage optimization attack. Using an Amazon Mechanical Turk user
study, we show that this attack produces audio samples that are more
surreptitious than previous attacks that aim solely for imperceptibility.
Lastly we show that surreptitious adversarial examples are challenging to
develop in the black-box setting.


http://arxiv.org/abs/2108.01289
AdvRush: Searching for Adversarially Robust Neural Architectures. (99%)
Jisoo Mok; Byunggook Na; Hyeokjun Choe; Sungroh Yoon
  Deep neural networks continue to awe the world with their remarkable
performance. Their predictions, however, are prone to be corrupted by
adversarial examples that are imperceptible to humans. Current efforts to
improve the robustness of neural networks against adversarial examples are
focused on developing robust training methods, which update the weights of a
neural network in a more robust direction. In this work, we take a step beyond
training of the weight parameters and consider the problem of designing an
adversarially robust neural architecture with high intrinsic robustness. We
propose AdvRush, a novel adversarial robustness-aware neural architecture
search algorithm, based upon a finding that independent of the training method,
the intrinsic robustness of a neural network can be represented with the
smoothness of its input loss landscape. Through a regularizer that favors a
candidate architecture with a smoother input loss landscape, AdvRush
successfully discovers an adversarially robust neural architecture. Along with
a comprehensive theoretical motivation for AdvRush, we conduct an extensive
amount of experiments to demonstrate the efficacy of AdvRush on various
benchmark datasets. Notably, on CIFAR-10, AdvRush achieves 55.91% robust
accuracy under FGSM attack after standard training and 50.04% robust accuracy
under AutoAttack after 7-step PGD adversarial training.


http://arxiv.org/abs/2108.01644
The Devil is in the GAN: Defending Deep Generative Models Against Backdoor Attacks. (88%)
Ambrish Rawat; Killian Levacher; Mathieu Sinn
  Deep Generative Models (DGMs) allow users to synthesize data from complex,
high-dimensional manifolds. Industry applications of DGMs include data
augmentation to boost performance of (semi-)supervised machine learning, or to
mitigate fairness or privacy concerns. Large-scale DGMs are notoriously hard to
train, requiring expert skills, large amounts of data and extensive
computational resources. Thus, it can be expected that many enterprises will
resort to sourcing pre-trained DGMs from potentially unverified third parties,
e.g.~open source model repositories.
  As we show in this paper, such a deployment scenario poses a new attack
surface, which allows adversaries to potentially undermine the integrity of
entire machine learning development pipelines in a victim organization.
Specifically, we describe novel training-time attacks resulting in corrupted
DGMs that synthesize regular data under normal operations and designated target
outputs for inputs sampled from a trigger distribution. Depending on the
control that the adversary has over the random number generation, this imposes
various degrees of risk that harmful data may enter the machine learning
development pipelines, potentially causing material or reputational damage to
the victim organization.
  Our attacks are based on adversarial loss functions that combine the dual
objectives of attack stealth and fidelity. We show its effectiveness for a
variety of DGM architectures (Generative Adversarial Networks (GANs),
Variational Autoencoders (VAEs)) and data domains (images, audio). Our
experiments show that - even for large-scale industry-grade DGMs - our attack
can be mounted with only modest computational efforts. We also investigate the
effectiveness of different defensive approaches (based on static/dynamic model
and output inspections) and prescribe a practical defense strategy that paves
the way for safe usage of DGMs.


http://arxiv.org/abs/2108.01281
DeepFreeze: Cold Boot Attacks and High Fidelity Model Recovery on Commercial EdgeML Device. (69%)
Yoo-Seung Won; Soham Chatterjee; Dirmanto Jap; Arindam Basu; Shivam Bhasin
  EdgeML accelerators like Intel Neural Compute Stick 2 (NCS) can enable
efficient edge-based inference with complex pre-trained models. The models are
loaded in the host (like Raspberry Pi) and then transferred to NCS for
inference. In this paper, we demonstrate practical and low-cost cold boot based
model recovery attacks on NCS to recover the model architecture and weights,
loaded from the Raspberry Pi. The architecture is recovered with 100% success
and weights with an error rate of 0.04%. The recovered model reports maximum
accuracy loss of 0.5% as compared to original model and allows high fidelity
transfer of adversarial examples. We further extend our study to other cold
boot attack setups reported in the literature with higher error rates leading
to accuracy loss as high as 70%. We then propose a methodology based on
knowledge distillation to correct the erroneous weights in recovered model,
even without access to original training data. The proposed attack remains
unaffected by the model encryption features of the OpenVINO and NCS framework.


http://arxiv.org/abs/2108.01734
Tutorials on Testing Neural Networks. (1%)
Nicolas Berthier; Youcheng Sun; Wei Huang; Yanghao Zhang; Wenjie Ruan; Xiaowei Huang
  Deep learning achieves remarkable performance on pattern recognition, but can
be vulnerable to defects of some important properties such as robustness and
security. This tutorial is based on a stream of research conducted since the
summer of 2018 at a few UK universities, including the University of Liverpool,
University of Oxford, Queen's University Belfast, University of Lancaster,
University of Loughborough, and University of Exeter.
  The research aims to adapt software engineering methods, in particular
software testing methods, to work with machine learning models. Software
testing techniques have been successful in identifying software bugs, and
helping software developers in validating the software they design and
implement. It is for this reason that a few software testing techniques -- such
as the MC/DC coverage metric -- have been mandated in industrial standards for
safety critical systems, including the ISO26262 for automotive systems and the
RTCA DO-178B/C for avionics systems. However, these techniques cannot be
directly applied to machine learning models, because the latter are drastically
different from traditional software, and their design follows a completely
different development life-cycle.
  As the outcome of this thread of research, the team has developed a series of
methods that adapt the software testing techniques to work with a few classes
of machine learning models. The latter notably include convolutional neural
networks, recurrent neural networks, and random forest. The tools developed
from this research are now collected, and publicly released, in a GitHub
repository: \url{https://github.com/TrustAI/DeepConcolic}, with the BSD
3-Clause licence.
  This tutorial is to go through the major functionalities of the tools with a
few running examples, to exhibit how the developed techniques work, what the
results are, and how to interpret them.


http://arxiv.org/abs/2108.01125
Hybrid Classical-Quantum Deep Learning Models for Autonomous Vehicle Traffic Image Classification Under Adversarial Attack. (98%)
Reek Majumder; Sakib Mahmud Khan; Fahim Ahmed; Zadid Khan; Frank Ngeni; Gurcan Comert; Judith Mwakalonge; Dimitra Michalaka; Mashrur Chowdhury
  Image classification must work for autonomous vehicles (AV) operating on
public roads, and actions performed based on image misclassification can have
serious consequences. Traffic sign images can be misclassified by an
adversarial attack on machine learning models used by AVs for traffic sign
recognition. To make classification models resilient against adversarial
attacks, we used a hybrid deep-learning model with both the quantum and
classical layers. Our goal is to study the hybrid deep-learning architecture
for classical-quantum transfer learning models to support the current era of
intermediate-scale quantum technology. We have evaluated the impacts of various
white box adversarial attacks on these hybrid models. The classical part of
hybrid models includes a convolution network from the pre-trained Resnet18
model, which extracts informative features from a high dimensional LISA traffic
sign image dataset. The output from the classical processor is processed
further through the quantum layer, which is composed of various quantum gates
and provides support to various quantum mechanical features like entanglement
and superposition. We have tested multiple combinations of quantum circuits to
provide better classification accuracy with decreasing training data and found
better resiliency for our hybrid classical-quantum deep learning model during
attacks compared to the classical-only machine learning models.


http://arxiv.org/abs/2108.00833
Adversarial Attacks Against Deep Reinforcement Learning Framework in Internet of Vehicles. (10%)
Anum Talpur; Mohan Gurusamy
  Machine learning (ML) has made incredible impacts and transformations in a
wide range of vehicular applications. As the use of ML in Internet of Vehicles
(IoV) continues to advance, adversarial threats and their impact have become an
important subject of research worth exploring. In this paper, we focus on
Sybil-based adversarial threats against a deep reinforcement learning
(DRL)-assisted IoV framework and more specifically, DRL-based dynamic service
placement in IoV. We carry out an experimental study with real vehicle
trajectories to analyze the impact on service delay and resource congestion
under different attack scenarios for the DRL-based dynamic service placement
application. We further investigate the impact of the proportion of
Sybil-attacked vehicles in the network. The results demonstrate that the
performance is significantly affected by Sybil-based data poisoning attacks
when compared to adversary-free healthy network scenario.


http://arxiv.org/abs/2108.00701
Information Stealing in Federated Learning Systems Based on Generative Adversarial Networks. (9%)
Yuwei Sun; Ng Chong; Hideya Ochiai
  An attack on deep learning systems where intelligent machines collaborate to
solve problems could cause a node in the network to make a mistake on a
critical judgment. At the same time, the security and privacy concerns of AI
have galvanized the attention of experts from multiple disciplines. In this
research, we successfully mounted adversarial attacks on a federated learning
(FL) environment using three different datasets. The attacks leveraged
generative adversarial networks (GANs) to affect the learning process and
strive to reconstruct the private data of users by learning hidden features
from shared local model parameters. The attack was target-oriented drawing data
with distinct class distribution from the CIFAR- 10, MNIST, and Fashion-MNIST
respectively. Moreover, by measuring the Euclidean distance between the real
data and the reconstructed adversarial samples, we evaluated the performance of
the adversary in the learning processes in various scenarios. At last, we
successfully reconstructed the real data of the victim from the shared global
model parameters with all the applied datasets.


http://arxiv.org/abs/2108.01124
Efficacy of Statistical and Artificial Intelligence-based False Information Cyberattack Detection Models for Connected Vehicles. (1%)
Sakib Mahmud Khan; Gurcan Comert; Mashrur Chowdhury
  Connected vehicles (CVs), because of the external connectivity with other CVs
and connected infrastructure, are vulnerable to cyberattacks that can instantly
compromise the safety of the vehicle itself and other connected vehicles and
roadway infrastructure. One such cyberattack is the false information attack,
where an external attacker injects inaccurate information into the connected
vehicles and eventually can cause catastrophic consequences by compromising
safety-critical applications like the forward collision warning. The occurrence
and target of such attack events can be very dynamic, making real-time and
near-real-time detection challenging. Change point models, can be used for
real-time anomaly detection caused by the false information attack. In this
paper, we have evaluated three change point-based statistical models;
Expectation Maximization, Cumulative Summation, and Bayesian Online Change
Point Algorithms for cyberattack detection in the CV data. Also, data-driven
artificial intelligence (AI) models, which can be used to detect known and
unknown underlying patterns in the dataset, have the potential of detecting a
real-time anomaly in the CV data. We have used six AI models to detect false
information attacks and compared the performance for detecting the attacks with
our developed change point models. Our study shows that change points models
performed better in real-time false information attack detection compared to
the performance of the AI models. Change point models having the advantage of
no training requirements can be a feasible and computationally efficient
alternative to AI models for false information attack detection in connected
vehicles.


http://arxiv.org/abs/2108.00401
Advances in adversarial attacks and defenses in computer vision: A survey. (92%)
Naveed Akhtar; Ajmal Mian; Navid Kardan; Mubarak Shah
  Deep Learning (DL) is the most widely used tool in the contemporary field of
computer vision. Its ability to accurately solve complex problems is employed
in vision research to learn deep neural models for a variety of tasks,
including security critical applications. However, it is now known that DL is
vulnerable to adversarial attacks that can manipulate its predictions by
introducing visually imperceptible perturbations in images and videos. Since
the discovery of this phenomenon in 2013~[1], it has attracted significant
attention of researchers from multiple sub-fields of machine intelligence. In
[2], we reviewed the contributions made by the computer vision community in
adversarial attacks on deep learning (and their defenses) until the advent of
year 2018. Many of those contributions have inspired new directions in this
area, which has matured significantly since witnessing the first generation
methods. Hence, as a legacy sequel of [2], this literature review focuses on
the advances in this area since 2018. To ensure authenticity, we mainly
consider peer-reviewed contributions published in the prestigious sources of
computer vision and machine learning research. Besides a comprehensive
literature review, the article also provides concise definitions of technical
terminologies for non-experts in this domain. Finally, this article discusses
challenges and future outlook of this direction based on the literature
reviewed herein and [2].


http://arxiv.org/abs/2108.00491
Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoders. (80%)
Huimin Zeng; Jiahao Su; Furong Huang
  Randomized Smoothing (RS), being one of few provable defenses, has been
showing great effectiveness and scalability in terms of defending against
$\ell_2$-norm adversarial perturbations. However, the cost of MC sampling
needed in RS for evaluation is high and computationally expensive. To address
this issue, we investigate the possibility of performing randomized smoothing
and establishing the robust certification in the latent space of a network, so
that the overall dimensionality of tensors involved in computation could be
drastically reduced. To this end, we propose Latent Space Randomized Smoothing.
Another important aspect is that we use orthogonal modules, whose Lipschitz
property is known for free by design, to propagate the certified radius
estimated in the latent space back to the input space, providing valid
certifiable regions for the test samples in the input space. Experiments on
CIFAR10 and ImageNet show that our method achieves competitive certified
robustness but with a significant improvement of efficiency during the test
phase.


http://arxiv.org/abs/2108.00422
An Effective and Robust Detector for Logo Detection. (70%)
Xiaojun Jia; Huanqian Yan; Yonglin Wu; Xingxing Wei; Xiaochun Cao; Yong Zhang
  In recent years, intellectual property (IP), which represents literary,
inventions, artistic works, etc, gradually attract more and more people's
attention. Particularly, with the rise of e-commerce, the IP not only
represents the product design and brands, but also represents the images/videos
displayed on e-commerce platforms. Unfortunately, some attackers adopt some
adversarial methods to fool the well-trained logo detection model for
infringement. To overcome this problem, a novel logo detector based on the
mechanism of looking and thinking twice is proposed in this paper for robust
logo detection. The proposed detector is different from other mainstream
detectors, which can effectively detect small objects, long-tail objects, and
is robust to adversarial images. In detail, we extend detectoRS algorithm to a
cascade schema with an equalization loss function, multi-scale transformations,
and adversarial data augmentation. A series of experimental results have shown
that the proposed method can effectively improve the robustness of the
detection model. Moreover, we have applied the proposed methods to competition
ACM MM2021 Robust Logo Detection that is organized by Alibaba on the Tianchi
platform and won top 2 in 36489 teams. Code is available at
https://github.com/jiaxiaojunQAQ/Robust-Logo-Detection.


http://arxiv.org/abs/2108.00402
Style Curriculum Learning for Robust Medical Image Segmentation. (2%)
Zhendong Liu; Van Manh; Xin Yang; Xiaoqiong Huang; Karim Lekadir; Víctor Campello; Nishant Ravikumar; Alejandro F Frangi; Dong Ni
  The performance of deep segmentation models often degrades due to
distribution shifts in image intensities between the training and test data
sets. This is particularly pronounced in multi-centre studies involving data
acquired using multi-vendor scanners, with variations in acquisition protocols.
It is challenging to address this degradation because the shift is often not
known \textit{a priori} and hence difficult to model. We propose a novel
framework to ensure robust segmentation in the presence of such distribution
shifts. Our contribution is three-fold. First, inspired by the spirit of
curriculum learning, we design a novel style curriculum to train the
segmentation models using an easy-to-hard mode. A style transfer model with
style fusion is employed to generate the curriculum samples. Gradually focusing
on complex and adversarial style samples can significantly boost the robustness
of the models. Second, instead of subjectively defining the curriculum
complexity, we adopt an automated gradient manipulation method to control the
hard and adversarial sample generation process. Third, we propose the Local
Gradient Sign strategy to aggregate the gradient locally and stabilise training
during gradient manipulation. The proposed framework can generalise to unknown
distribution without using any target data. Extensive experiments on the public
M\&Ms Challenge dataset demonstrate that our proposed framework can generalise
deep models well to unknown distributions and achieve significant improvements
in segmentation accuracy.


http://arxiv.org/abs/2108.00180
Delving into Deep Image Prior for Adversarial Defense: A Novel Reconstruction-based Defense Framework. (99%)
Li Ding; Yongwei Wang; Xin Ding; Kaiwen Yuan; Ping Wang; Hua Huang; Z. Jane Wang
  Deep learning based image classification models are shown vulnerable to
adversarial attacks by injecting deliberately crafted noises to clean images.
To defend against adversarial attacks in a training-free and attack-agnostic
manner, this work proposes a novel and effective reconstruction-based defense
framework by delving into deep image prior (DIP). Fundamentally different from
existing reconstruction-based defenses, the proposed method analyzes and
explicitly incorporates the model decision process into our defense. Given an
adversarial image, firstly we map its reconstructed images during DIP
optimization to the model decision space, where cross-boundary images can be
detected and on-boundary images can be further localized. Then, adversarial
noise is purified by perturbing on-boundary images along the reverse direction
to the adversarial image. Finally, on-manifold images are stitched to construct
an image that can be correctly predicted by the victim classifier. Extensive
experiments demonstrate that the proposed method outperforms existing
state-of-the-art reconstruction-based methods both in defending white-box
attacks and defense-aware attacks. Moreover, the proposed method can maintain a
high visual quality during adversarial image reconstruction.


http://arxiv.org/abs/2108.00213
Adversarial Robustness of Deep Code Comment Generation. (99%)
Yu Zhou; Xiaoqing Zhang; Juanjuan Shen; Tingting Han; Taolue Chen; Harald Gall
  Deep neural networks (DNNs) have shown remarkable performance in a variety of
domains such as computer vision, speech recognition, or natural language
processing. Recently they also have been applied to various software
engineering tasks, typically involving processing source code. DNNs are
well-known to be vulnerable to adversarial examples, i.e., fabricated inputs
that could lead to various misbehaviors of the DNN model while being perceived
as benign by humans. In this paper, we focus on the code comment generation
task in software engineering and study the robustness issue of the DNNs when
they are applied to this task. We propose ACCENT, an identifier substitution
approach to craft adversarial code snippets, which are syntactically correct
and functionality-preserving with respect to the original code snippet, but may
mislead the DNNs to produce completely irrelevant code comments. In order to
improve the robustness, ACCENT also incorporates a novel training method, which
can be applied to existing code comment generation models. We conduct
comprehensive experiments to evaluate our approach by attacking the mainstream
encoder-decoder architectures on two large-scale publicly available datasets.
The results show that ACCENT efficiently produces stable attacks with
functionality-preserving adversarial examples, and the generated examples have
better transferability compared with baselines. We also confirm, via
experiments, the effectiveness in improving model robustness with our training
method.


http://arxiv.org/abs/2108.00335
Towards Adversarially Robust and Domain Generalizable Stereo Matching by Rethinking DNN Feature Backbones. (93%)
Kelvin Cheng; Christopher Healey; Tianfu Wu
  Stereo matching has recently witnessed remarkable progress using Deep Neural
Networks (DNNs). But, how robust are they? Although it has been well-known that
DNNs often suffer from adversarial vulnerability with a catastrophic drop in
performance, the situation is even worse in stereo matching. This paper first
shows that a type of weak white-box attacks can overwhelm state-of-the-art
methods. The attack is learned by a proposed stereo-constrained projected
gradient descent (PGD) method in stereo matching. This observation raises
serious concerns for the deployment of DNN-based stereo matching. Parallel to
the adversarial vulnerability, DNN-based stereo matching is typically trained
under the so-called simulation to reality pipeline, and thus domain
generalizability is an important problem. This paper proposes to rethink the
learnable DNN-based feature backbone towards adversarially-robust and domain
generalizable stereo matching by completely removing it for matching. In
experiments, the proposed method is tested in the SceneFlow dataset and the
KITTI2015 benchmark, with promising results. We compute the matching cost
volume using the classic multi-scale census transform (i.e., local binary
pattern) of the raw input stereo images, followed by a stacked Hourglass head
sub-network solving the matching problem. It significantly improves the
adversarial robustness, while retaining accuracy performance comparable to
state-of-the-art methods. It also shows better generalizability from simulation
(SceneFlow) to real (KITTI) datasets when no fine-tuning is used.


http://arxiv.org/abs/2108.00146
T$_k$ML-AP: Adversarial Attacks to Top-$k$ Multi-Label Learning. (81%)
Shu Hu; Lipeng Ke; Xin Wang; Siwei Lyu
  Top-$k$ multi-label learning, which returns the top-$k$ predicted labels from
an input, has many practical applications such as image annotation, document
analysis, and web search engine. However, the vulnerabilities of such
algorithms with regards to dedicated adversarial perturbation attacks have not
been extensively studied previously. In this work, we develop methods to create
adversarial perturbations that can be used to attack top-$k$ multi-label
learning-based image annotation systems (TkML-AP). Our methods explicitly
consider the top-$k$ ranking relation and are based on novel loss functions.
Experimental evaluations on large-scale benchmark datasets including PASCAL VOC
and MS COCO demonstrate the effectiveness of our methods in reducing the
performance of state-of-the-art top-$k$ multi-label learning methods, under
both untargeted and targeted attacks.


http://arxiv.org/abs/2108.00352
BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning. (67%)
Jinyuan Jia; Yupei Liu; Neil Zhenqiang Gong
  Self-supervised learning in computer vision aims to pre-train an image
encoder using a large amount of unlabeled images or (image, text) pairs. The
pre-trained image encoder can then be used as a feature extractor to build
downstream classifiers for many downstream tasks with a small amount of or no
labeled training data. In this work, we propose BadEncoder, the first backdoor
attack to self-supervised learning. In particular, our BadEncoder injects
backdoors into a pre-trained image encoder such that the downstream classifiers
built based on the backdoored image encoder for different downstream tasks
simultaneously inherit the backdoor behavior. We formulate our BadEncoder as an
optimization problem and we propose a gradient descent based method to solve
it, which produces a backdoored image encoder from a clean one. Our extensive
empirical evaluation results on multiple datasets show that our BadEncoder
achieves high attack success rates while preserving the accuracy of the
downstream classifiers. We also show the effectiveness of BadEncoder using two
publicly available, real-world image encoders, i.e., Google's image encoder
pre-trained on ImageNet and OpenAI's Contrastive Language-Image Pre-training
(CLIP) image encoder pre-trained on 400 million (image, text) pairs collected
from the Internet. Moreover, we consider defenses including Neural Cleanse and
MNTD (empirical defenses) as well as PatchGuard (a provable defense). Our
results show that these defenses are insufficient to defend against BadEncoder,
highlighting the needs for new defenses against our BadEncoder. Our code is
publicly available at: https://github.com/jjy1994/BadEncoder.


http://arxiv.org/abs/2108.00295
Fair Representation Learning using Interpolation Enabled Disentanglement. (1%)
Akshita Jha; Bhanukiran Vinzamuri; Chandan K. Reddy
  With the growing interest in the machine learning community to solve
real-world problems, it has become crucial to uncover the hidden reasoning
behind their decisions by focusing on the fairness and auditing the predictions
made by these black-box models. In this paper, we propose a novel method to
address two key issues: (a) Can we simultaneously learn fair disentangled
representations while ensuring the utility of the learned representation for
downstream tasks, and (b)Can we provide theoretical insights into when the
proposed approach will be both fair and accurate. To address the former, we
propose the method FRIED, Fair Representation learning using Interpolation
Enabled Disentanglement. In our architecture, by imposing a critic-based
adversarial framework, we enforce the interpolated points in the latent space
to be more realistic. This helps in capturing the data manifold effectively and
enhances the utility of the learned representation for downstream prediction
tasks. We address the latter question by developing a theory on
fairness-accuracy trade-offs using classifier-based conditional mutual
information estimation. We demonstrate the effectiveness of FRIED on datasets
of different modalities - tabular, text, and image datasets. We observe that
the representations learned by FRIED are overall fairer in comparison to
existing baselines and also accurate for downstream prediction tasks.
Additionally, we evaluate FRIED on a real-world healthcare claims dataset where
we conduct an expert aided model auditing study providing useful insights into
opioid ad-diction patterns.


http://arxiv.org/abs/2107.14601
Who's Afraid of Thomas Bayes? (92%)
Erick Galinkin
  In many cases, neural networks perform well on test data, but tend to
overestimate their confidence on out-of-distribution data. This has led to
adoption of Bayesian neural networks, which better capture uncertainty and
therefore more accurately reflect the model's confidence. For machine learning
security researchers, this raises the natural question of how making a model
Bayesian affects the security of the model. In this work, we explore the
interplay between Bayesianism and two measures of security: model privacy and
adversarial robustness. We demonstrate that Bayesian neural networks are more
vulnerable to membership inference attacks in general, but are at least as
robust as their non-Bayesian counterparts to adversarial examples.


http://arxiv.org/abs/2107.14642
Practical Attacks on Voice Spoofing Countermeasures. (86%)
Andre Kassis; Urs Hengartner
  Voice authentication has become an integral part in security-critical
operations, such as bank transactions and call center conversations. The
vulnerability of automatic speaker verification systems (ASVs) to spoofing
attacks instigated the development of countermeasures (CMs), whose task is to
tell apart bonafide and spoofed speech. Together, ASVs and CMs form today's
voice authentication platforms, advertised as an impregnable access control
mechanism. We develop the first practical attack on CMs, and show how a
malicious actor may efficiently craft audio samples to bypass voice
authentication in its strictest form. Previous works have primarily focused on
non-proactive attacks or adversarial strategies against ASVs that do not
produce speech in the victim's voice. The repercussions of our attacks are far
more severe, as the samples we generate sound like the victim, eliminating any
chance of plausible deniability. Moreover, the few existing adversarial attacks
against CMs mistakenly optimize spoofed speech in the feature space and do not
take into account the existence of ASVs, resulting in inferior synthetic audio
that fails in realistic settings. We eliminate these obstacles through our key
technical contribution: a novel joint loss function that enables mounting
advanced adversarial attacks against combined ASV/CM deployments directly in
the time domain. Our adversarials achieve concerning black-box success rates
against state-of-the-art authentication platforms (up to 93.57\%). Finally, we
perform the first targeted, over-telephony-network attack on CMs, bypassing
several challenges and enabling various potential threats, given the increased
use of voice biometrics in call centers. Our results call into question the
security of modern voice authentication systems in light of the real threat of
attackers bypassing these measures to gain access to users' most valuable
resources.


http://arxiv.org/abs/2107.14569
Can You Hear It? Backdoor Attacks via Ultrasonic Triggers. (50%)
Stefanos Koffas; Jing Xu; Mauro Conti; Stjepan Picek
  Deep neural networks represent a powerful option for many real-world
applications due to their ability to model even complex data relations.
However, such neural networks can also be prohibitively expensive to train,
making it common to either outsource the training process to third parties or
use pretrained neural networks. Unfortunately, such practices make neural
networks vulnerable to various attacks, where one attack is the backdoor
attack. In such an attack, the third party training the model may maliciously
inject hidden behaviors into the model. Still, if a particular input (called
trigger) is fed into a neural network, the network will respond with a wrong
result.
  In this work, we explore the option of backdoor attacks to automatic speech
recognition systems where we inject inaudible triggers. By doing so, we make
the backdoor attack challenging to detect for legitimate users, and thus,
potentially more dangerous. We conduct experiments on two versions of datasets
and three neural networks and explore the performance of our attack concerning
the duration, position, and type of the trigger. Our results indicate that less
than 1% of poisoned data is sufficient to deploy a backdoor attack and reach a
100% attack success rate. What is more, while the trigger is inaudible, making
it without limitations with respect to the duration of the signal, we observed
that even short, non-continuous triggers result in highly successful attacks.


http://arxiv.org/abs/2107.14756
Unveiling the potential of Graph Neural Networks for robust Intrusion Detection. (13%)
David Pujol-Perich; José Suárez-Varela; Albert Cabellos-Aparicio; Pere Barlet-Ros
  The last few years have seen an increasing wave of attacks with serious
economic and privacy damages, which evinces the need for accurate Network
Intrusion Detection Systems (NIDS). Recent works propose the use of Machine
Learning (ML) techniques for building such systems (e.g., decision trees,
neural networks). However, existing ML-based NIDS are barely robust to common
adversarial attacks, which limits their applicability to real networks. A
fundamental problem of these solutions is that they treat and classify flows
independently. In contrast, in this paper we argue the importance of focusing
on the structural patterns of attacks, by capturing not only the individual
flow features, but also the relations between different flows (e.g., the
source/destination hosts they share). To this end, we use a graph
representation that keeps flow records and their relationships, and propose a
novel Graph Neural Network (GNN) model tailored to process and learn from such
graph-structured information. In our evaluation, we first show that the
proposed GNN model achieves state-of-the-art results in the well-known
CIC-IDS2017 dataset. Moreover, we assess the robustness of our solution under
two common adversarial attacks, that intentionally modify the packet size and
inter-arrival times to avoid detection. The results show that our model is able
to maintain the same level of accuracy as in previous experiments, while
state-of-the-art ML techniques degrade up to 50% their accuracy (F1-score)
under these attacks. This unprecedented level of robustness is mainly induced
by the capability of our GNN model to learn flow patterns of attacks structured
as graphs.


http://arxiv.org/abs/2107.14185
Feature Importance-aware Transferable Adversarial Attacks. (99%)
Zhibo Wang; Hengchang Guo; Zhifei Zhang; Wenxin Liu; Zhan Qin; Kui Ren
  Transferability of adversarial examples is of central importance for
attacking an unknown model, which facilitates adversarial attacks in more
practical scenarios, e.g., blackbox attacks. Existing transferable attacks tend
to craft adversarial examples by indiscriminately distorting features to
degrade prediction accuracy in a source model without aware of intrinsic
features of objects in the images. We argue that such brute-force degradation
would introduce model-specific local optimum into adversarial examples, thus
limiting the transferability. By contrast, we propose the Feature
Importance-aware Attack (FIA), which disrupts important object-aware features
that dominate model decisions consistently. More specifically, we obtain
feature importance by introducing the aggregate gradient, which averages the
gradients with respect to feature maps of the source model, computed on a batch
of random transforms of the original clean image. The gradients will be highly
correlated to objects of interest, and such correlation presents invariance
across different models. Besides, the random transforms will preserve intrinsic
features of objects and suppress model-specific information. Finally, the
feature importance guides to search for adversarial examples towards disrupting
critical features, achieving stronger transferability. Extensive experimental
evaluation demonstrates the effectiveness and superior performance of the
proposed FIA, i.e., improving the success rate by 8.4% against normally trained
models and 11.7% against defense models as compared to the state-of-the-art
transferable attacks. Code is available at: https://github.com/hcguoO0/FIA


http://arxiv.org/abs/2107.14110
Enhancing Adversarial Robustness via Test-time Transformation Ensembling. (98%)
Juan C. Pérez; Motasem Alfarra; Guillaume Jeanneret; Laura Rueda; Ali Thabet; Bernard Ghanem; Pablo Arbeláez
  Deep learning models are prone to being fooled by imperceptible perturbations
known as adversarial attacks. In this work, we study how equipping models with
Test-time Transformation Ensembling (TTE) can work as a reliable defense
against such attacks. While transforming the input data, both at train and test
times, is known to enhance model performance, its effects on adversarial
robustness have not been studied. Here, we present a comprehensive empirical
study of the impact of TTE, in the form of widely-used image transforms, on
adversarial robustness. We show that TTE consistently improves model robustness
against a variety of powerful attacks without any need for re-training, and
that this improvement comes at virtually no trade-off with accuracy on clean
samples. Finally, we show that the benefits of TTE transfer even to the
certified robustness domain, in which TTE provides sizable and consistent
improvements.


http://arxiv.org/abs/2107.13962
The Robustness of Graph k-shell Structure under Adversarial Attacks. (93%)
B. Zhou; Y. Q. Lv; Y. C. Mao; J. H. Wang; S. Q. Yu; Q. Xuan
  The k-shell decomposition plays an important role in unveiling the structural
properties of a network, i.e., it is widely adopted to find the densest part of
a network across a broad range of scientific fields, including Internet,
biological networks, social networks, etc. However, there arises concern about
the robustness of the k-shell structure when networks suffer from adversarial
attacks. Here, we introduce and formalize the problem of the k-shell attack and
develop an efficient strategy to attack the k-shell structure by rewiring a
small number of links. To the best of our knowledge, it is the first time to
study the robustness of graph k-shell structure under adversarial attacks. In
particular, we propose a Simulated Annealing (SA) based k-shell attack method
and testify it on four real-world social networks. The extensive experiments
validate that the k-shell structure of a network is robust under random
perturbation, but it is quite vulnerable under adversarial attack, e.g., in
Dolphin and Throne networks, more than 40% nodes change their k-shell values
when only 10% links are changed based on our SA-based k-shell attack. Such
results suggest that a single structural feature could also be significantly
disturbed when only a small fraction of links are changed purposefully in a
network. Therefore, it could be an interesting topic to improve the robustness
of various network properties against adversarial attack in the future.


http://arxiv.org/abs/2107.13876
Understanding the Effects of Adversarial Personalized Ranking Optimization Method on Recommendation Quality. (31%)
Vito Walter Anelli; Yashar Deldjoo; Noia Tommaso Di; Felice Antonio Merra
  Recommender systems (RSs) employ user-item feedback, e.g., ratings, to match
customers to personalized lists of products. Approaches to top-k recommendation
mainly rely on Learning-To-Rank algorithms and, among them, the most widely
adopted is Bayesian Personalized Ranking (BPR), which bases on a pair-wise
optimization approach. Recently, BPR has been found vulnerable against
adversarial perturbations of its model parameters. Adversarial Personalized
Ranking (APR) mitigates this issue by robustifying BPR via an adversarial
training procedure. The empirical improvements of APR's accuracy performance on
BPR have led to its wide use in several recommender models. However, a key
overlooked aspect has been the beyond-accuracy performance of APR, i.e.,
novelty, coverage, and amplification of popularity bias, considering that
recent results suggest that BPR, the building block of APR, is sensitive to the
intensification of biases and reduction of recommendation novelty. In this
work, we model the learning characteristics of the BPR and APR optimization
frameworks to give mathematical evidence that, when the feedback data have a
tailed distribution, APR amplifies the popularity bias more than BPR due to an
unbalanced number of received positive updates from short-head items. Using
matrix factorization (MF), we empirically validate the theoretical results by
performing preliminary experiments on two public datasets to compare BPR-MF and
APR-MF performance on accuracy and beyond-accuracy metrics. The experimental
results consistently show the degradation of novelty and coverage measures and
a worrying amplification of bias.


http://arxiv.org/abs/2107.14344
Towards robust vision by multi-task learning on monkey visual cortex. (3%)
Shahd Safarani; Arne Nix; Konstantin Willeke; Santiago A. Cadena; Kelli Restivo; George Denfield; Andreas S. Tolias; Fabian H. Sinz
  Deep neural networks set the state-of-the-art across many tasks in computer
vision, but their generalization ability to image distortions is surprisingly
fragile. In contrast, the mammalian visual system is robust to a wide range of
perturbations. Recent work suggests that this generalization ability can be
explained by useful inductive biases encoded in the representations of visual
stimuli throughout the visual cortex. Here, we successfully leveraged these
inductive biases with a multi-task learning approach: we jointly trained a deep
network to perform image classification and to predict neural activity in
macaque primary visual cortex (V1). We measured the out-of-distribution
generalization abilities of our network by testing its robustness to image
distortions. We found that co-training on monkey V1 data leads to increased
robustness despite the absence of those distortions during training.
Additionally, we showed that our network's robustness is very close to that of
an Oracle network where parts of the architecture are directly trained on noisy
images. Our results also demonstrated that the network's representations become
more brain-like as their robustness improves. Using a novel constrained
reconstruction analysis, we investigated what makes our brain-regularized
network more robust. We found that our co-trained network is more sensitive to
content than noise when compared to a Baseline network that we trained for
image classification alone. Using DeepGaze-predicted saliency maps for ImageNet
images, we found that our monkey co-trained network tends to be more sensitive
to salient regions in a scene, reminiscent of existing theories on the role of
V1 in the detection of object borders and bottom-up saliency. Overall, our work
expands the promising research avenue of transferring inductive biases from the
brain, and provides a novel analysis of the effects of our transfer.


http://arxiv.org/abs/2107.13639
Imbalanced Adversarial Training with Reweighting. (86%)
Wentao Wang; Han Xu; Xiaorui Liu; Yaxin Li; Bhavani Thuraisingham; Jiliang Tang
  Adversarial training has been empirically proven to be one of the most
effective and reliable defense methods against adversarial attacks. However,
almost all existing studies about adversarial training are focused on balanced
datasets, where each class has an equal amount of training examples. Research
on adversarial training with imbalanced training datasets is rather limited. As
the initial effort to investigate this problem, we reveal the facts that
adversarially trained models present two distinguished behaviors from naturally
trained models in imbalanced datasets: (1) Compared to natural training,
adversarially trained models can suffer much worse performance on
under-represented classes, when the training dataset is extremely imbalanced.
(2) Traditional reweighting strategies may lose efficacy to deal with the
imbalance issue for adversarial training. For example, upweighting the
under-represented classes will drastically hurt the model's performance on
well-represented classes, and as a result, finding an optimal reweighting value
can be tremendously challenging. In this paper, to further understand our
observations, we theoretically show that the poor data separability is one key
reason causing this strong tension between under-represented and
well-represented classes. Motivated by this finding, we propose Separable
Reweighted Adversarial Training (SRAT) to facilitate adversarial training under
imbalanced scenarios, by learning more separable features for different
classes. Extensive experiments on various datasets verify the effectiveness of
the proposed framework.


http://arxiv.org/abs/2107.13541
Towards Robustness Against Natural Language Word Substitutions. (73%)
Xinshuai Dong; Anh Tuan Luu; Rongrong Ji; Hong Liu
  Robustness against word substitutions has a well-defined and widely
acceptable form, i.e., using semantically similar words as substitutions, and
thus it is considered as a fundamental stepping-stone towards broader
robustness in natural language processing. Previous defense methods capture
word substitutions in vector space by using either $l_2$-ball or
hyper-rectangle, which results in perturbation sets that are not inclusive
enough or unnecessarily large, and thus impedes mimicry of worst cases for
robust training. In this paper, we introduce a novel \textit{Adversarial Sparse
Convex Combination} (ASCC) method. We model the word substitution attack space
as a convex hull and leverages a regularization term to enforce perturbation
towards an actual substitution, thus aligning our modeling better with the
discrete textual space. Based on the ASCC method, we further propose
ASCC-defense, which leverages ASCC to generate worst-case perturbations and
incorporates adversarial training towards robustness. Experiments show that
ASCC-defense outperforms the current state-of-the-arts in terms of robustness
on two prevailing NLP tasks, \emph{i.e.}, sentiment analysis and natural
language inference, concerning several attacks across multiple model
architectures. Besides, we also envision a new class of defense towards
robustness in NLP, where our robustly trained word vectors can be plugged into
a normally trained model and enforce its robustness without applying any other
defense techniques.


http://arxiv.org/abs/2107.13491
Models of Computational Profiles to Study the Likelihood of DNN Metamorphic Test Cases. (67%)
Ettore Merlo; Mira Marhaba; Foutse Khomh; Houssem Ben Braiek; Giuliano Antoniol
  Neural network test cases are meant to exercise different reasoning paths in
an architecture and used to validate the prediction outcomes. In this paper, we
introduce "computational profiles" as vectors of neuron activation levels. We
investigate the distribution of computational profile likelihood of metamorphic
test cases with respect to the likelihood distributions of training, test and
error control cases. We estimate the non-parametric probability densities of
neuron activation levels for each distinct output class. Probabilities are
inferred using training cases only, without any additional knowledge about
metamorphic test cases. Experiments are performed by training a network on the
MNIST Fashion library of images and comparing prediction likelihoods with those
obtained from error control-data and from metamorphic test cases. Experimental
results show that the distributions of computational profile likelihood for
training and test cases are somehow similar, while the distribution of the
random-noise control-data is always remarkably lower than the observed one for
the training and testing sets. In contrast, metamorphic test cases show a
prediction likelihood that lies in an extended range with respect to training,
tests, and random noise. Moreover, the presented approach allows the
independent assessment of different training classes and experiments to show
that some of the classes are more sensitive to misclassifying metamorphic test
cases than other classes. In conclusion, metamorphic test cases represent very
aggressive tests for neural network architectures. Furthermore, since
metamorphic test cases force a network to misclassify those inputs whose
likelihood is similar to that of training cases, they could also be considered
as adversarial attacks that evade defenses based on computational profile
likelihood evaluation.


http://arxiv.org/abs/2107.13335
WaveCNet: Wavelet Integrated CNNs to Suppress Aliasing Effect for Noise-Robust Image Classification. (15%)
Qiufu Li; Linlin Shen; Sheng Guo; Zhihui Lai
  Though widely used in image classification, convolutional neural networks
(CNNs) are prone to noise interruptions, i.e. the CNN output can be drastically
changed by small image noise. To improve the noise robustness, we try to
integrate CNNs with wavelet by replacing the common down-sampling (max-pooling,
strided-convolution, and average pooling) with discrete wavelet transform
(DWT). We firstly propose general DWT and inverse DWT (IDWT) layers applicable
to various orthogonal and biorthogonal discrete wavelets like Haar, Daubechies,
and Cohen, etc., and then design wavelet integrated CNNs (WaveCNets) by
integrating DWT into the commonly used CNNs (VGG, ResNets, and DenseNet).
During the down-sampling, WaveCNets apply DWT to decompose the feature maps
into the low-frequency and high-frequency components. Containing the main
information including the basic object structures, the low-frequency component
is transmitted into the following layers to generate robust high-level
features. The high-frequency components are dropped to remove most of the data
noises. The experimental results show that %wavelet accelerates the CNN
training, and WaveCNets achieve higher accuracy on ImageNet than various
vanilla CNNs. We have also tested the performance of WaveCNets on the noisy
version of ImageNet, ImageNet-C and six adversarial attacks, the results
suggest that the proposed DWT/IDWT layers could provide better noise-robustness
and adversarial robustness. When applying WaveCNets as backbones, the
performance of object detectors (i.e., faster R-CNN and RetinaNet) on COCO
detection dataset are consistently improved. We believe that suppression of
aliasing effect, i.e. separation of low frequency and high frequency
information, is the main advantages of our approach. The code of our DWT/IDWT
layer and different WaveCNets are available at
https://github.com/CVI-SZU/WaveCNet.


http://arxiv.org/abs/2107.13190
TableGAN-MCA: Evaluating Membership Collisions of GAN-Synthesized Tabular Data Releasing. (2%)
Aoting Hu; Renjie Xie; Zhigang Lu; Aiqun Hu; Minhui Xue
  Generative Adversarial Networks (GAN)-synthesized table publishing lets
people privately learn insights without access to the private table. However,
existing studies on Membership Inference (MI) Attacks show promising results on
disclosing membership of training datasets of GAN-synthesized tables. Different
from those works focusing on discovering membership of a given data point, in
this paper, we propose a novel Membership Collision Attack against GANs
(TableGAN-MCA), which allows an adversary given only synthetic entries randomly
sampled from a black-box generator to recover partial GAN training data.
Namely, a GAN-synthesized table immune to state-of-the-art MI attacks is
vulnerable to the TableGAN-MCA. The success of TableGAN-MCA is boosted by an
observation that GAN-synthesized tables potentially collide with the training
data of the generator.
  Our experimental evaluations on TableGAN-MCA have five main findings. First,
TableGAN-MCA has a satisfying training data recovery rate on three commonly
used real-world datasets against four generative models. Second, factors,
including the size of GAN training data, GAN training epochs and the number of
synthetic samples available to the adversary, are positively correlated to the
success of TableGAN-MCA. Third, highly frequent data points have high risks of
being recovered by TableGAN-MCA. Fourth, some unique data are exposed to
unexpected high recovery risks in TableGAN-MCA, which may attribute to GAN's
generalization. Fifth, as expected, differential privacy, without the
consideration of the correlations between features, does not show commendable
mitigation effect against the TableGAN-MCA. Finally, we propose two mitigation
methods and show promising privacy and utility trade-offs when protecting
against TableGAN-MCA.


http://arxiv.org/abs/2107.12732
Towards Black-box Attacks on Deep Learning Apps. (89%)
Hongchen Cao; Shuai Li; Yuming Zhou; Ming Fan; Xuejiao Zhao; Yutian Tang
  Deep learning is a powerful weapon to boost application performance in many
fields, including face recognition, object detection, image classification,
natural language understanding, and recommendation system. With the rapid
increase in the computing power of mobile devices, developers can embed deep
learning models into their apps for building more competitive products with
more accurate and faster responses. Although there are several works about
adversarial attacks against deep learning models in mobile apps, they all need
information about the models' internals (i.e., structures, weights) or need to
modify the models. In this paper, we propose an effective black-box approach by
training a substitute model to spoof the deep learning system inside the apps.
To evaluate our approach, we select 10 real-world deep-learning apps with high
popularity from Google Play to perform black-box adversarial attacks. Through
the study, we find three factors that can influence the performance of attacks.
Our approach can reach a relatively high attack success rate of 66.60% on
average. Compared with other adversarial attacks on mobile deep learning
models, in terms of the average attack success rates, our approach outperforms
counterparts by 27.63%.


http://arxiv.org/abs/2107.12612
Poisoning of Online Learning Filters: DDoS Attacks and Countermeasures. (50%)
Wesley Joon-Wie Tann; Ee-Chien Chang
  The recent advancements in machine learning have led to a wave of interest in
adopting online learning-based approaches for long-standing attack mitigation
issues. In particular, DDoS attacks remain a significant threat to network
service availability even after more than two decades. These attacks have been
well studied under the assumption that malicious traffic originates from a
single attack profile. Based on this premise, malicious traffic characteristics
are assumed to be considerably different from legitimate traffic. Consequently,
online filtering methods are designed to learn network traffic distributions
adaptively and rank requests according to their attack likelihood. During an
attack, requests rated as malicious are precipitously dropped by the filters.
In this paper, we conduct the first systematic study on the effects of data
poisoning attacks on online DDoS filtering; introduce one such attack method,
and propose practical protective countermeasures for these attacks. We
investigate an adverse scenario where the attacker is "crafty", switching
profiles during attacks and generating erratic attack traffic that is
ever-shifting. This elusive attacker generates malicious requests by
manipulating and shifting traffic distribution to poison the training data and
corrupt the filters. To this end, we present a generative model MimicShift,
capable of controlling traffic generation while retaining the originating
regular traffic's intrinsic properties. Comprehensive experiments show that
online learning filters are highly susceptible to poisoning attacks, sometimes
performing much worse than a random filtering strategy in this attack scenario.
At the same time, our proposed protective countermeasure effectively minimizes
the attack impact.


http://arxiv.org/abs/2107.12873
PDF-Malware: An Overview on Threats, Detection and Evasion Attacks. (8%)
Nicolas Fleury; Theo Dubrunquez; Ihsen Alouani
  In the recent years, Portable Document Format, commonly known as PDF, has
become a democratized standard for document exchange and dissemination. This
trend has been due to its characteristics such as its flexibility and
portability across platforms. The widespread use of PDF has installed a false
impression of inherent safety among benign users. However, the characteristics
of PDF motivated hackers to exploit various types of vulnerabilities, overcome
security safeguards, thereby making the PDF format one of the most efficient
malicious code attack vectors. Therefore, efficiently detecting malicious PDF
files is crucial for information security. Several analysis techniques has been
proposed in the literature, be it static or dynamic, to extract the main
features that allow the discrimination of malware files from benign ones. Since
classical analysis techniques may be limited in case of zero-days,
machine-learning based techniques have emerged recently as an automatic
PDF-malware detection method that is able to generalize from a set of training
samples. These techniques are themselves facing the challenge of evasion
attacks where a malicious PDF is transformed to look benign. In this work, we
give an overview on the PDF-malware detection problem. We give a perspective on
the new challenges and emerging solutions.


http://arxiv.org/abs/2107.11986
Benign Adversarial Attack: Tricking Algorithm for Goodness. (99%)
Xian Zhao; Jiaming Zhang; Zhiyu Lin; Jitao Sang
  In spite of the successful application in many fields, machine learning
algorithms today suffer from notorious problems like vulnerability to
adversarial examples. Beyond falling into the cat-and-mouse game between
adversarial attack and defense, this paper provides alternative perspective to
consider adversarial example and explore whether we can exploit it in benign
applications. We first propose a novel taxonomy of visual information along
task-relevance and semantic-orientation. The emergence of adversarial example
is attributed to algorithm's utilization of task-relevant non-semantic
information. While largely ignored in classical machine learning mechanisms,
task-relevant non-semantic information enjoys three interesting characteristics
as (1) exclusive to algorithm, (2) reflecting common weakness, and (3)
utilizable as features. Inspired by this, we present brave new idea called
benign adversarial attack to exploit adversarial examples for goodness in three
directions: (1) adversarial Turing test, (2) rejecting malicious algorithm, and
(3) adversarial data augmentation. Each direction is positioned with motivation
elaboration, justification analysis and prototype applications to showcase its
potential.


http://arxiv.org/abs/2107.12085
Learning to Adversarially Blur Visual Object Tracking. (98%)
Qing Guo; Ziyi Cheng; Felix Juefei-Xu; Lei Ma; Xiaofei Xie; Yang Liu; Jianjun Zhao
  Motion blur caused by the moving of the object or camera during the exposure
can be a key challenge for visual object tracking, affecting tracking accuracy
significantly. In this work, we explore the robustness of visual object
trackers against motion blur from a new angle, i.e., adversarial blur attack
(ABA). Our main objective is to online transfer input frames to their natural
motion-blurred counterparts while misleading the state-of-the-art trackers
during the tracking process. To this end, we first design the motion blur
synthesizing method for visual tracking based on the generation principle of
motion blur, considering the motion information and the light accumulation
process. With this synthetic method, we propose optimization-based ABA (OP-ABA)
by iteratively optimizing an adversarial objective function against the
tracking w.r.t. the motion and light accumulation parameters. The OP-ABA is
able to produce natural adversarial examples but the iteration can cause heavy
time cost, making it unsuitable for attacking real-time trackers. To alleviate
this issue, we further propose one-step ABA (OS-ABA) where we design and train
a joint adversarial motion and accumulation predictive network (JAMANet) with
the guidance of OP-ABA, which is able to efficiently estimate the adversarial
motion and accumulation parameters in a one-step way. The experiments on four
popular datasets (e.g., OTB100, VOT2018, UAV123, and LaSOT) demonstrate that
our methods are able to cause significant accuracy drops on four
state-of-the-art trackers with high transferability. Please find the source
code at \url{https://github.com/tsingqguo/ABA}.


http://arxiv.org/abs/2107.12473
Adversarial Attacks with Time-Scale Representations. (96%)
Alberto Santamaria-Pang; Jianwei Qiu; Aritra Chowdhury; James Kubricht; Peter Tu; Iyer Naresh; Nurali Virani
  We propose a novel framework for real-time black-box universal attacks which
disrupts activations of early convolutional layers in deep learning models. Our
hypothesis is that perturbations produced in the wavelet space disrupt early
convolutional layers more effectively than perturbations performed in the time
domain. The main challenge in adversarial attacks is to preserve low frequency
image content while minimally changing the most meaningful high frequency
content. To address this, we formulate an optimization problem using time-scale
(wavelet) representations as a dual space in three steps. First, we project
original images into orthonormal sub-spaces for low and high scales via wavelet
coefficients. Second, we perturb wavelet coefficients for high scale projection
using a generator network. Third, we generate new adversarial images by
projecting back the original coefficients from the low scale and the perturbed
coefficients from the high scale sub-space. We provide a theoretical framework
that guarantees a dual mapping from time and time-scale domain representations.
We compare our results with state-of-the-art black-box attacks from
generative-based and gradient-based models. We also verify efficacy against
multiple defense methods such as JPEG compression, Guided Denoiser and
Comdefend. Our results show that wavelet-based perturbations consistently
outperform time-based attacks thus providing new insights into vulnerabilities
of deep learning models and could potentially lead to robust architectures or
new defense and attack mechanisms by leveraging time-scale representations.


http://arxiv.org/abs/2107.11671
Adversarial training may be a double-edged sword. (99%)
Ali Rahmati; Seyed-Mohsen Moosavi-Dezfooli; Huaiyu Dai
  Adversarial training has been shown as an effective approach to improve the
robustness of image classifiers against white-box attacks. However, its
effectiveness against black-box attacks is more nuanced. In this work, we
demonstrate that some geometric consequences of adversarial training on the
decision boundary of deep networks give an edge to certain types of black-box
attacks. In particular, we define a metric called robustness gain to show that
while adversarial training is an effective method to dramatically improve the
robustness in white-box scenarios, it may not provide such a good robustness
gain against the more realistic decision-based black-box attacks. Moreover, we
show that even the minimal perturbation white-box attacks can converge faster
against adversarially-trained neural networks compared to the regular ones.


http://arxiv.org/abs/2107.11630
Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them. (98%)
Florian Tramèr
  Making classifiers robust to adversarial examples is hard. Thus, many
defenses tackle the seemingly easier task of detecting perturbed inputs. We
show a barrier towards this goal. We prove a general hardness reduction between
detection and classification of adversarial examples: given a robust detector
for attacks at distance {\epsilon} (in some metric), we can build a similarly
robust (but inefficient) classifier for attacks at distance {\epsilon}/2. Our
reduction is computationally inefficient, and thus cannot be used to build
practical classifiers. Instead, it is a useful sanity check to test whether
empirical detection results imply something much stronger than the authors
presumably anticipated. To illustrate, we revisit 13 detector defenses. For
11/13 cases, we show that the claimed detection results would imply an
inefficient classifier with robustness far beyond the state-of-the-art.


http://arxiv.org/abs/2107.11652
Stress Test Evaluation of Biomedical Word Embeddings. (73%)
Vladimir Araujo; Andrés Carvallo; Carlos Aspillaga; Camilo Thorne; Denis Parra
  The success of pretrained word embeddings has motivated their use in the
biomedical domain, with contextualized embeddings yielding remarkable results
in several biomedical NLP tasks. However, there is a lack of research on
quantifying their behavior under severe "stress" scenarios. In this work, we
systematically evaluate three language models with adversarial examples --
automatically constructed tests that allow us to examine how robust the models
are. We propose two types of stress scenarios focused on the biomedical named
entity recognition (NER) task, one inspired by spelling errors and another
based on the use of synonyms for medical terms. Our experiments with three
benchmarks show that the performance of the original models decreases
considerably, in addition to revealing their weaknesses and strengths. Finally,
we show that adversarial training causes the models to improve their robustness
and even to exceed the original performance in some cases.


http://arxiv.org/abs/2107.11576
X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering. (1%)
Jingjing Jiang; Ziyi Liu; Yifan Liu; Zhixiong Nan; Nanning Zheng
  Encouraging progress has been made towards Visual Question Answering (VQA) in
recent years, but it is still challenging to enable VQA models to adaptively
generalize to out-of-distribution (OOD) samples. Intuitively, recompositions of
existing visual concepts (\ie, attributes and objects) can generate unseen
compositions in the training set, which will promote VQA models to generalize
to OOD samples. In this paper, we formulate OOD generalization in VQA as a
compositional generalization problem and propose a graph generative
modeling-based training scheme (X-GGM) to implicitly model the problem. X-GGM
leverages graph generative modeling to iteratively generate a relation matrix
and node representations for the predefined graph that utilizes
attribute-object pairs as nodes. Furthermore, to alleviate the unstable
training issue in graph generative modeling, we propose a gradient distribution
consistency loss to constrain the data distribution with adversarial
perturbations and the generated distribution. The baseline VQA model (LXMERT)
trained with the X-GGM scheme achieves state-of-the-art OOD performance on two
standard VQA OOD benchmarks, \ie, VQA-CP v2 and GQA-OOD. Extensive ablation
studies demonstrate the effectiveness of X-GGM components. Code is available at
\url{https://github.com/jingjing12110/x-ggm}.


http://arxiv.org/abs/2107.11275
A Differentiable Language Model Adversarial Attack on Text Classifiers. (99%)
Ivan Fursov; Alexey Zaytsev; Pavel Burnyshev; Ekaterina Dmitrieva; Nikita Klyuchnikov; Andrey Kravchenko; Ekaterina Artemova; Evgeny Burnaev
  Robustness of huge Transformer-based models for natural language processing
is an important issue due to their capabilities and wide adoption. One way to
understand and improve robustness of these models is an exploration of an
adversarial attack scenario: check if a small perturbation of an input can fool
a model.
  Due to the discrete nature of textual data, gradient-based adversarial
methods, widely used in computer vision, are not applicable per~se. The
standard strategy to overcome this issue is to develop token-level
transformations, which do not take the whole sentence into account.
  In this paper, we propose a new black-box sentence-level attack. Our method
fine-tunes a pre-trained language model to generate adversarial examples. A
proposed differentiable loss function depends on a substitute classifier score
and an approximate edit distance computed via a deep learning model.
  We show that the proposed attack outperforms competitors on a diverse set of
NLP problems for both computed metrics and human evaluation. Moreover, due to
the usage of the fine-tuned language model, the generated adversarial examples
are hard to detect, thus current models are not robust. Hence, it is difficult
to defend from the proposed attack, which is not the case for other attacks.


http://arxiv.org/abs/2107.11327
Structack: Structure-based Adversarial Attacks on Graph Neural Networks. (86%)
Hussain Hussain; Tomislav Duricic; Elisabeth Lex; Denis Helic; Markus Strohmaier; Roman Kern
  Recent work has shown that graph neural networks (GNNs) are vulnerable to
adversarial attacks on graph data. Common attack approaches are typically
informed, i.e. they have access to information about node attributes such as
labels and feature vectors. In this work, we study adversarial attacks that are
uninformed, where an attacker only has access to the graph structure, but no
information about node attributes. Here the attacker aims to exploit structural
knowledge and assumptions, which GNN models make about graph data. In
particular, literature has shown that structural node centrality and similarity
have a strong influence on learning with GNNs. Therefore, we study the impact
of centrality and similarity on adversarial attacks on GNNs. We demonstrate
that attackers can exploit this information to decrease the performance of GNNs
by focusing on injecting links between nodes of low similarity and,
surprisingly, low centrality. We show that structure-based uninformed attacks
can approach the performance of informed attacks, while being computationally
more efficient. With our paper, we present a new attack strategy on GNNs that
we refer to as Structack. Structack can successfully manipulate the performance
of GNNs with very limited information while operating under tight computational
constraints. Our work contributes towards building more robust machine learning
approaches on graphs.


http://arxiv.org/abs/2107.11252
Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation. (45%)
Bingqian Lin; Yi Zhu; Yanxin Long; Xiaodan Liang; Qixiang Ye; Liang Lin
  Language instruction plays an essential role in the natural language grounded
navigation tasks. However, navigators trained with limited human-annotated
instructions may have difficulties in accurately capturing key information from
the complicated instruction at different timesteps, leading to poor navigation
performance. In this paper, we exploit to train a more robust navigator which
is capable of dynamically extracting crucial factors from the long instruction,
by using an adversarial attacking paradigm. Specifically, we propose a Dynamic
Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the
navigator to move to the wrong target by destroying the most instructive
information in instructions at different timesteps. By formulating the
perturbation generation as a Markov Decision Process, DR-Attacker is optimized
by the reinforcement learning algorithm to generate perturbed instructions
sequentially during the navigation, according to a learnable attack score.
Then, the perturbed instructions, which serve as hard samples, are used for
improving the robustness of the navigator with an effective adversarial
training strategy and an auxiliary self-supervised reasoning task. Experimental
results on both Vision-and-Language Navigation (VLN) and Navigation from Dialog
History (NDH) tasks show the superiority of our proposed method over
state-of-the-art methods. Moreover, the visualization analysis shows the
effectiveness of the proposed DR-Attacker, which can successfully attack
crucial information in the instructions at different timesteps. Code is
available at https://github.com/expectorlin/DR-Attacker.


http://arxiv.org/abs/2107.11472
Free Hyperbolic Neural Networks with Limited Radii. (8%)
Yunhui Guo; Xudong Wang; Yubei Chen; Stella X. Yu
  Non-Euclidean geometry with constant negative curvature, i.e., hyperbolic
space, has attracted sustained attention in the community of machine learning.
Hyperbolic space, owing to its ability to embed hierarchical structures
continuously with low distortion, has been applied for learning data with
tree-like structures. Hyperbolic Neural Networks (HNNs) that operate directly
in hyperbolic space have also been proposed recently to further exploit the
potential of hyperbolic representations. While HNNs have achieved better
performance than Euclidean neural networks (ENNs) on datasets with implicit
hierarchical structure, they still perform poorly on standard classification
benchmarks such as CIFAR and ImageNet. The traditional wisdom is that it is
critical for the data to respect the hyperbolic geometry when applying HNNs. In
this paper, we first conduct an empirical study showing that the inferior
performance of HNNs on standard recognition datasets can be attributed to the
notorious vanishing gradient problem. We further discovered that this problem
stems from the hybrid architecture of HNNs. Our analysis leads to a simple yet
effective solution called Feature Clipping, which regularizes the hyperbolic
embedding whenever its norm exceeding a given threshold. Our thorough
experiments show that the proposed method can successfully avoid the vanishing
gradient problem when training HNNs with backpropagation. The improved HNNs are
able to achieve comparable performance with ENNs on standard image recognition
datasets including MNIST, CIFAR10, CIFAR100 and ImageNet, while demonstrating
more adversarial robustness and stronger out-of-distribution detection
capability.


http://arxiv.org/abs/2107.10873
On the Certified Robustness for Ensemble Models and Beyond. (99%)
Zhuolin Yang; Linyi Li; Xiaojun Xu; Bhavya Kailkhura; Tao Xie; Bo Li
  Recent studies show that deep neural networks (DNN) are vulnerable to
adversarial examples, which aim to mislead DNNs by adding perturbations with
small magnitude. To defend against such attacks, both empirical and theoretical
defense approaches have been extensively studied for a single ML model. In this
work, we aim to analyze and provide the certified robustness for ensemble ML
models, together with the sufficient and necessary conditions of robustness for
different ensemble protocols. Although ensemble models are shown more robust
than a single model empirically; surprisingly, we find that in terms of the
certified robustness the standard ensemble models only achieve marginal
improvement compared to a single model. Thus, to explore the conditions that
guarantee to provide certifiably robust ensemble ML models, we first prove that
diversified gradient and large confidence margin are sufficient and necessary
conditions for certifiably robust ensemble models under the model-smoothness
assumption. We then provide the bounded model-smoothness analysis based on the
proposed Ensemble-before-Smoothing strategy. We also prove that an ensemble
model can always achieve higher certified robustness than a single base model
under mild conditions. Inspired by the theoretical findings, we propose the
lightweight Diversity Regularized Training (DRT) to train certifiably robust
ensemble ML models. Extensive experiments show that our DRT enhanced ensembles
can consistently achieve higher certified robustness than existing single and
ensemble ML models, demonstrating the state-of-the-art certified L2-robustness
on MNIST, CIFAR-10, and ImageNet datasets.


http://arxiv.org/abs/2107.10480
Unsupervised Detection of Adversarial Examples with Model Explanations. (99%)
Gihyuk Ko; Gyumin Lim
  Deep Neural Networks (DNNs) have shown remarkable performance in a diverse
range of machine learning applications. However, it is widely known that DNNs
are vulnerable to simple adversarial perturbations, which causes the model to
incorrectly classify inputs. In this paper, we propose a simple yet effective
method to detect adversarial examples, using methods developed to explain the
model's behavior. Our key observation is that adding small, humanly
imperceptible perturbations can lead to drastic changes in the model
explanations, resulting in unusual or irregular forms of explanations. From
this insight, we propose an unsupervised detection of adversarial examples
using reconstructor networks trained only on model explanations of benign
examples. Our evaluations with MNIST handwritten dataset show that our method
is capable of detecting adversarial examples generated by the state-of-the-art
algorithms with high confidence. To the best of our knowledge, this work is the
first in suggesting unsupervised defense method using model explanations.


http://arxiv.org/abs/2107.12173
Membership Inference Attack and Defense for Wireless Signal Classifiers with Deep Learning. (83%)
Yi Shi; Yalin E. Sagduyu
  An over-the-air membership inference attack (MIA) is presented to leak
private information from a wireless signal classifier. Machine learning (ML)
provides powerful means to classify wireless signals, e.g., for PHY-layer
authentication. As an adversarial machine learning attack, the MIA infers
whether a signal of interest has been used in the training data of a target
classifier. This private information incorporates waveform, channel, and device
characteristics, and if leaked, can be exploited by an adversary to identify
vulnerabilities of the underlying ML model (e.g., to infiltrate the PHY-layer
authentication). One challenge for the over-the-air MIA is that the received
signals and consequently the RF fingerprints at the adversary and the intended
receiver differ due to the discrepancy in channel conditions. Therefore, the
adversary first builds a surrogate classifier by observing the spectrum and
then launches the black-box MIA on this classifier. The MIA results show that
the adversary can reliably infer signals (and potentially the radio and channel
information) used to build the target classifier. Therefore, a proactive
defense is developed against the MIA by building a shadow MIA model and fooling
the adversary. This defense can successfully reduce the MIA accuracy and
prevent information leakage from the wireless signal classifier.


http://arxiv.org/abs/2107.10599
Towards Explaining Adversarial Examples Phenomenon in Artificial Neural Networks. (75%)
Ramin Barati; Reza Safabakhsh; Mohammad Rahmati
  In this paper, we study the adversarial examples existence and adversarial
training from the standpoint of convergence and provide evidence that pointwise
convergence in ANNs can explain these observations. The main contribution of
our proposal is that it relates the objective of the evasion attacks and
adversarial training with concepts already defined in learning theory. Also, we
extend and unify some of the other proposals in the literature and provide
alternative explanations on the observations made in those proposals. Through
different experiments, we demonstrate that the framework is valuable in the
study of the phenomenon and is applicable to real-world problems.


http://arxiv.org/abs/2107.10989
Estimating Predictive Uncertainty Under Program Data Distribution Shift. (1%)
Yufei Li; Simin Chen; Wei Yang
  Deep learning (DL) techniques have achieved great success in predictive
accuracy in a variety of tasks, but deep neural networks (DNNs) are shown to
produce highly overconfident scores for even abnormal samples. Well-defined
uncertainty indicates whether a model's output should (or should not) be
trusted and thus becomes critical in real-world scenarios which typically
involves shifted input distributions due to many factors. Existing uncertainty
approaches assume that testing samples from a different data distribution would
induce unreliable model predictions thus have higher uncertainty scores. They
quantify model uncertainty by calibrating DL model's confidence of a given
input and evaluate the effectiveness in computer vision (CV) and natural
language processing (NLP)-related tasks. However, their methodologies'
reliability may be compromised under programming tasks due to difference in
data representations and shift patterns. In this paper, we first define three
different types of distribution shift in program data and build a large-scale
shifted Java dataset. We implement two common programming language tasks on our
dataset to study the effect of each distribution shift on DL model performance.
We also propose a large-scale benchmark of existing state-of-the-art predictive
uncertainty on programming tasks and investigate their effectiveness under data
distribution shift. Experiments show that program distribution shift does
degrade the DL model performance to varying degrees and that existing
uncertainty methods all present certain limitations in quantifying uncertainty
on program dataset.


http://arxiv.org/abs/2107.10457
Ready for Emerging Threats to Recommender Systems? A Graph Convolution-based Generative Shilling Attack. (1%)
Fan Wu; Min Gao; Junliang Yu; Zongwei Wang; Kecheng Liu; Xu Wange
  To explore the robustness of recommender systems, researchers have proposed
various shilling attack models and analyzed their adverse effects. Primitive
attacks are highly feasible but less effective due to simplistic handcrafted
rules, while upgraded attacks are more powerful but costly and difficult to
deploy because they require more knowledge from recommendations. In this paper,
we explore a novel shilling attack called Graph cOnvolution-based generative
shilling ATtack (GOAT) to balance the attacks' feasibility and effectiveness.
GOAT adopts the primitive attacks' paradigm that assigns items for fake users
by sampling and the upgraded attacks' paradigm that generates fake ratings by a
deep learning-based model. It deploys a generative adversarial network (GAN)
that learns the real rating distribution to generate fake ratings.
Additionally, the generator combines a tailored graph convolution structure
that leverages the correlations between co-rated items to smoothen the fake
ratings and enhance their authenticity. The extensive experiments on two public
datasets evaluate GOAT's performance from multiple perspectives. Our study of
the GOAT demonstrates technical feasibility for building a more powerful and
intelligent attack model with a much-reduced cost, enables analysis the threat
of such an attack and guides for investigating necessary prevention measures.


http://arxiv.org/abs/2107.09937
Fast and Scalable Adversarial Training of Kernel SVM via Doubly Stochastic Gradients. (98%)
Huimin Wu; Zhengmian Hu; Bin Gu
  Adversarial attacks by generating examples which are almost indistinguishable
from natural examples, pose a serious threat to learning models. Defending
against adversarial attacks is a critical element for a reliable learning
system. Support vector machine (SVM) is a classical yet still important
learning algorithm even in the current deep learning era. Although a wide range
of researches have been done in recent years to improve the adversarial
robustness of learning models, but most of them are limited to deep neural
networks (DNNs) and the work for kernel SVM is still vacant. In this paper, we
aim at kernel SVM and propose adv-SVM to improve its adversarial robustness via
adversarial training, which has been demonstrated to be the most promising
defense techniques. To the best of our knowledge, this is the first work that
devotes to the fast and scalable adversarial training of kernel SVM.
Specifically, we first build connection of perturbations of samples between
original and kernel spaces, and then give a reduced and equivalent formulation
of adversarial training of kernel SVM based on the connection. Next, doubly
stochastic gradients (DSG) based on two unbiased stochastic approximations
(i.e., one is on training points and another is on random features) are applied
to update the solution of our objective function. Finally, we prove that our
algorithm optimized by DSG converges to the optimal solution at the rate of
O(1/t) under the constant and diminishing stepsizes. Comprehensive experimental
results show that our adversarial training algorithm enjoys robustness against
various attacks and meanwhile has the similar efficiency and scalability with
classical DSG algorithm.


http://arxiv.org/abs/2107.10137
Improved Text Classification via Contrastive Adversarial Training. (84%)
Lin Pan; Chung-Wei Hang; Avirup Sil; Saloni Potdar; Mo Yu
  We propose a simple and general method to regularize the fine-tuning of
Transformer-based encoders for text classification tasks. Specifically, during
fine-tuning we generate adversarial examples by perturbing the word embeddings
of the model and perform contrastive learning on clean and adversarial examples
in order to teach the model to learn noise-invariant representations. By
training on both clean and adversarial examples along with the additional
contrastive objective, we observe consistent improvement over standard
fine-tuning on clean examples. On several GLUE benchmark tasks, our fine-tuned
BERT Large model outperforms BERT Large baseline by 1.7% on average, and our
fine-tuned RoBERTa Large improves over RoBERTa Large baseline by 1.3%. We
additionally validate our method in different domains using three intent
classification datasets, where our fine-tuned RoBERTa Large outperforms RoBERTa
Large baseline by 1-2% on average.


http://arxiv.org/abs/2107.10174
Black-box Probe for Unsupervised Domain Adaptation without Model Transferring. (81%)
Kunhong Wu; Yucheng Shi; Yahong Han; Yunfeng Shao; Bingshuai Li
  In recent years, researchers have been paying increasing attention to the
threats brought by deep learning models to data security and privacy,
especially in the field of domain adaptation. Existing unsupervised domain
adaptation (UDA) methods can achieve promising performance without transferring
data from source domain to target domain. However, UDA with representation
alignment or self-supervised pseudo-labeling relies on the transferred source
models. In many data-critical scenarios, methods based on model transferring
may suffer from membership inference attacks and expose private data. In this
paper, we aim to overcome a challenging new setting where the source models are
only queryable but cannot be transferred to the target domain. We propose
Black-box Probe Domain Adaptation (BPDA), which adopts query mechanism to probe
and refine information from source model using third-party dataset. In order to
gain more informative query results, we further propose Distributionally
Adversarial Training (DAT) to align the distribution of third-party data with
that of target data. BPDA uses public third-party dataset and adversarial
examples based on DAT as the information carrier between source and target
domains, dispensing with transferring source data or model. Experimental
results on benchmarks of Digit-Five, Office-Caltech, Office-31, Office-Home,
and DomainNet demonstrate the feasibility of BPDA without model transferring.


http://arxiv.org/abs/2107.09898
Defending against Reconstruction Attack in Vertical Federated Learning. (10%)
Jiankai Sun; Yuanshun Yao; Weihao Gao; Junyuan Xie; Chong Wang
  Recently researchers have studied input leakage problems in Federated
Learning (FL) where a malicious party can reconstruct sensitive training inputs
provided by users from shared gradient. It raises concerns about FL since input
leakage contradicts the privacy-preserving intention of using FL. Despite a
relatively rich literature on attacks and defenses of input reconstruction in
Horizontal FL, input leakage and protection in vertical FL starts to draw
researcher's attention recently. In this paper, we study how to defend against
input leakage attacks in Vertical FL. We design an adversarial training-based
framework that contains three modules: adversarial reconstruction, noise
regularization, and distance correlation minimization. Those modules can not
only be employed individually but also applied together since they are
independent to each other. Through extensive experiments on a large-scale
industrial online advertising dataset, we show our framework is effective in
protecting input privacy while retaining the model utility.


http://arxiv.org/abs/2107.10139
Generative Models for Security: Attacks, Defenses, and Opportunities. (10%)
Luke A. Bauer; Vincent Bindschaedler
  Generative models learn the distribution of data from a sample dataset and
can then generate new data instances. Recent advances in deep learning has
brought forth improvements in generative model architectures, and some
state-of-the-art models can (in some cases) produce outputs realistic enough to
fool humans.
  We survey recent research at the intersection of security and privacy and
generative models. In particular, we discuss the use of generative models in
adversarial machine learning, in helping automate or enhance existing attacks,
and as building blocks for defenses in contexts such as intrusion detection,
biometrics spoofing, and malware obfuscation. We also describe the use of
generative models in diverse applications such as fairness in machine learning,
privacy-preserving data synthesis, and steganography. Finally, we discuss new
threats due to generative models: the creation of synthetic media such as
deepfakes that can be used for disinformation.


http://arxiv.org/abs/2107.10045
A Tandem Framework Balancing Privacy and Security for Voice User Interfaces. (5%)
Ranya Aloufi; Hamed Haddadi; David Boyle
  Speech synthesis, voice cloning, and voice conversion techniques present
severe privacy and security threats to users of voice user interfaces (VUIs).
These techniques transform one or more elements of a speech signal, e.g.,
identity and emotion, while preserving linguistic information. Adversaries may
use advanced transformation tools to trigger a spoofing attack using fraudulent
biometrics for a legitimate speaker. Conversely, such techniques have been used
to generate privacy-transformed speech by suppressing personally identifiable
attributes in the voice signals, achieving anonymization. Prior works have
studied the security and privacy vectors in parallel, and thus it raises alarm
that if a benign user can achieve privacy by a transformation, it also means
that a malicious user can break security by bypassing the anti-spoofing
mechanism. In this paper, we take a step towards balancing two seemingly
conflicting requirements: security and privacy. It remains unclear what the
vulnerabilities in one domain imply for the other, and what dynamic
interactions exist between them. A better understanding of these aspects is
crucial for assessing and mitigating vulnerabilities inherent with VUIs and
building effective defenses. In this paper,(i) we investigate the applicability
of the current voice anonymization methods by deploying a tandem framework that
jointly combines anti-spoofing and authentication models, and evaluate the
performance of these methods;(ii) examining analytical and empirical evidence,
we reveal a duality between the two mechanisms as they offer different ways to
achieve the same objective, and we show that leveraging one vector
significantly amplifies the effectiveness of the other;(iii) we demonstrate
that to effectively defend from potential attacks against VUIs, it is necessary
to investigate the attacks from multiple complementary perspectives(security
and privacy).


http://arxiv.org/abs/2107.10443
Spinning Sequence-to-Sequence Models with Meta-Backdoors. (4%)
Eugene Bagdasaryan; Vitaly Shmatikov
  We investigate a new threat to neural sequence-to-sequence (seq2seq) models:
training-time attacks that cause models to "spin" their output and support a
certain sentiment when the input contains adversary-chosen trigger words. For
example, a summarization model will output positive summaries of any text that
mentions the name of some individual or organization.
  We introduce the concept of a "meta-backdoor" to explain model-spinning
attacks. These attacks produce models whose output is valid and preserves
context, yet also satisfies a meta-task chosen by the adversary (e.g., positive
sentiment). Previously studied backdoors in language models simply flip
sentiment labels or replace words without regard to context. Their outputs are
incorrect on inputs with the trigger. Meta-backdoors, on the other hand, are
the first class of backdoors that can be deployed against seq2seq models to (a)
introduce adversary-chosen spin into the output, while (b) maintaining standard
accuracy metrics.
  To demonstrate feasibility of model spinning, we develop a new backdooring
technique. It stacks the adversarial meta-task (e.g., sentiment analysis) onto
a seq2seq model, backpropagates the desired meta-task output (e.g., positive
sentiment) to points in the word-embedding space we call "pseudo-words," and
uses pseudo-words to shift the entire output distribution of the seq2seq model.
Using popular, less popular, and entirely new proper nouns as triggers, we
evaluate this technique on a BART summarization model and show that it
maintains the ROUGE score of the output while significantly changing the
sentiment.
  We explain why model spinning can be a dangerous technique in AI-powered
disinformation and discuss how to mitigate these attacks.


http://arxiv.org/abs/2107.10110
On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms. (2%)
Shuyu Cheng; Guoqiang Wu; Jun Zhu
  Zeroth-order (ZO) optimization is widely used to handle challenging tasks,
such as query-based black-box adversarial attacks and reinforcement learning.
Various attempts have been made to integrate prior information into the
gradient estimation procedure based on finite differences, with promising
empirical results. However, their convergence properties are not well
understood. This paper makes an attempt to fill this gap by analyzing the
convergence of prior-guided ZO algorithms under a greedy descent framework with
various gradient estimators. We provide a convergence guarantee for the
prior-guided random gradient-free (PRGF) algorithms. Moreover, to further
accelerate over greedy descent methods, we present a new accelerated random
search (ARS) algorithm that incorporates prior information, together with a
convergence analysis. Finally, our theoretical results are confirmed by
experiments on several numerical benchmarks as well as adversarial attacks.


http://arxiv.org/abs/2107.09804
Using Undervolting as an On-Device Defense Against Adversarial Machine Learning Attacks. (99%)
Saikat Majumdar; Mohammad Hossein Samavatian; Kristin Barber; Radu Teodorescu
  Deep neural network (DNN) classifiers are powerful tools that drive a broad
spectrum of important applications, from image recognition to autonomous
vehicles. Unfortunately, DNNs are known to be vulnerable to adversarial attacks
that affect virtually all state-of-the-art models. These attacks make small
imperceptible modifications to inputs that are sufficient to induce the DNNs to
produce the wrong classification.
  In this paper we propose a novel, lightweight adversarial correction and/or
detection mechanism for image classifiers that relies on undervolting (running
a chip at a voltage that is slightly below its safe margin). We propose using
controlled undervolting of the chip running the inference process in order to
introduce a limited number of compute errors. We show that these errors disrupt
the adversarial input in a way that can be used either to correct the
classification or detect the input as adversarial. We evaluate the proposed
solution in an FPGA design and through software simulation. We evaluate 10
attacks and show average detection rates of 77% and 90% on two popular DNNs.


http://arxiv.org/abs/2107.09258
A Markov Game Model for AI-based Cyber Security Attack Mitigation. (10%)
Hooman Alavizadeh; Julian Jang-Jaccard; Tansu Alpcan; Seyit A. Camtepe
  The new generation of cyber threats leverages advanced AI-aided methods,
which make them capable to launch multi-stage, dynamic, and effective attacks.
Current cyber-defense systems encounter various challenges to defend against
such new and emerging threats. Modeling AI-aided threats through game theory
models can help the defender to select optimal strategies against the attacks
and make wise decisions to mitigate the attack's impact. This paper first
explores the current state-of-the-art in the new generation of threats in which
AI techniques such as deep neural network is used for the attacker and
discusses further challenges. We propose a Markovian dynamic game that can
evaluate the efficiency of defensive methods against the AI-aided attacker
under a cloud-based system in which the attacker utilizes an AI technique to
launch an advanced attack by finding the shortest attack path. We use the CVSS
metrics to quantify the values of this zero-sum game model for decision-making.


http://arxiv.org/abs/2107.09833
Leaking Secrets through Modern Branch Predictor in the Speculative World. (1%)
Md Hafizul Islam Chowdhuryy; Fan Yao
  Transient execution attacks that exploit speculation have raised significant
concerns in computer systems. Typically, branch predictors are leveraged to
trigger mis-speculation in transient execution attacks. In this work, we
demonstrate a new class of speculation-based attack that targets branch
prediction unit (BPU). We find that speculative resolution of conditional
branches (i.e., in nested speculation) alter the states of pattern history
table (PHT) in modern processors, which are not restored after the
corresponding branches are later squashed. Such characteristic allows attackers
to exploit BPU as the secret transmitting medium in transient execution
attacks. To evaluate the discovered vulnerability, we build a novel attack
framework, BranchSpectre, that enables exfiltration of unintended secrets
through observing speculative PHT updates (in the form of covert and side
channels). We further investigate PHT collision mechanism in the history-based
predictor as well as the branch prediction mode transitions in Intel
processors. Built upon such knowledge, we implement an ultra high-speed covert
channel (BranchSpectre-cc) as well as two side channels (i.e., BranchSpectre-v1
and BranchSpectre-v2) that merely rely on BPU for mis-speculation trigger and
secret inference in the speculative domain. Notably, BranchSpectre side
channels can take advantage of much simpler code patterns than the ones used in
Spectre attacks. We present an extensive BranchSpectre code gadget analysis on
a set of popular real-world application code bases followed by a demonstration
of real-world side channel attack on OpenSSL. The evaluation results show
substantial wider existence and higher exploitability of BranchSpectre code
patterns in real-world software. Finally, we discuss several secure branch
prediction mechanisms that can mitigate transient execution attacks exploiting
modern branch predictors.


http://arxiv.org/abs/2107.09225
Discriminator-Free Generative Adversarial Attack. (99%)
Shaohao Lu; Yuqiao Xian; Ke Yan; Yi Hu; Xing Sun; Xiaowei Guo; Feiyue Huang; Wei-Shi Zheng
  The Deep Neural Networks are vulnerable toadversarial exam-ples(Figure 1),
making the DNNs-based systems collapsed byadding the inconspicuous
perturbations to the images. Most of the existing works for adversarial attack
are gradient-based and suf-fer from the latency efficiencies and the load on
GPU memory. Thegenerative-based adversarial attacks can get rid of this
limitation,and some relative works propose the approaches based on GAN.However,
suffering from the difficulty of the convergence of train-ing a GAN, the
adversarial examples have either bad attack abilityor bad visual quality. In
this work, we find that the discriminatorcould be not necessary for
generative-based adversarial attack, andpropose theSymmetric Saliency-based
Auto-Encoder (SSAE)to generate the perturbations, which is composed of the
saliencymap module and the angle-norm disentanglement of the featuresmodule.
The advantage of our proposed method lies in that it is notdepending on
discriminator, and uses the generative saliency map to pay more attention to
label-relevant regions. The extensive exper-iments among the various tasks,
datasets, and models demonstratethat the adversarial examples generated by SSAE
not only make thewidely-used models collapse, but also achieves good visual
quality.The code is available at https://github.com/BravoLu/SSAE.


http://arxiv.org/abs/2107.09502
Feature-Filter: Detecting Adversarial Examples through Filtering off Recessive Features. (99%)
Hui Liu; Bo Zhao; Yuefeng Peng; Jiabao Guo; Peng Liu
  Deep neural networks (DNNs) are under threat from adversarial example
attacks. The adversary can easily change the outputs of DNNs by adding small
well-designed perturbations to inputs. Adversarial example detection is a
fundamental work for robust DNNs-based service. Adversarial examples show the
difference between humans and DNNs in image recognition. From a human-centric
perspective, image features could be divided into dominant features that are
comprehensible to humans, and recessive features that are incomprehensible to
humans, yet are exploited by DNNs. In this paper, we reveal that imperceptible
adversarial examples are the product of recessive features misleading neural
networks, and an adversarial attack is essentially a kind of method to enrich
these recessive features in the image. The imperceptibility of the adversarial
examples indicates that the perturbations enrich recessive features, yet hardly
affect dominant features. Therefore, adversarial examples are sensitive to
filtering off recessive features, while benign examples are immune to such
operation. Inspired by this idea, we propose a label-only adversarial detection
approach that is referred to as feature-filter. Feature-filter utilizes
discrete cosine transform to approximately separate recessive features from
dominant features, and gets a mutant image that is filtered off recessive
features. By only comparing DNN's prediction labels on the input and its
mutant, feature-filter can real-time detect imperceptible adversarial examples
at high accuracy and few false positives.


http://arxiv.org/abs/2107.09126
Examining the Human Perceptibility of Black-Box Adversarial Attacks on Face Recognition. (98%)
Benjamin Spetter-Goldstein; Nataniel Ruiz; Sarah Adel Bargal
  The modern open internet contains billions of public images of human faces
across the web, especially on social media websites used by half the world's
population. In this context, Face Recognition (FR) systems have the potential
to match faces to specific names and identities, creating glaring privacy
concerns. Adversarial attacks are a promising way to grant users privacy from
FR systems by disrupting their capability to recognize faces. Yet, such attacks
can be perceptible to human observers, especially under the more challenging
black-box threat model. In the literature, the justification for the
imperceptibility of such attacks hinges on bounding metrics such as $\ell_p$
norms. However, there is not much research on how these norms match up with
human perception. Through examining and measuring both the effectiveness of
recent black-box attacks in the face recognition setting and their
corresponding human perceptibility through survey data, we demonstrate the
trade-offs in perceptibility that occur as attacks become more aggressive. We
also show how the $\ell_2$ norm and other metrics do not correlate with human
perceptibility in a linear fashion, thus making these norms suboptimal at
measuring adversarial attack perceptibility.


http://arxiv.org/abs/2107.09045
On the Veracity of Local, Model-agnostic Explanations in Audio Classification: Targeted Investigations with Adversarial Examples. (80%)
Verena Praher; Katharina Prinz; Arthur Flexer; Gerhard Widmer
  Local explanation methods such as LIME have become popular in MIR as tools
for generating post-hoc, model-agnostic explanations of a model's
classification decisions. The basic idea is to identify a small set of
human-understandable features of the classified example that are most
influential on the classifier's prediction. These are then presented as an
explanation. Evaluation of such explanations in publications often resorts to
accepting what matches the expectation of a human without actually being able
to verify if what the explanation shows is what really caused the model's
prediction. This paper reports on targeted investigations where we try to get
more insight into the actual veracity of LIME's explanations in an audio
classification task. We deliberately design adversarial examples for the
classifier, in a way that gives us knowledge about which parts of the input are
potentially responsible for the model's (wrong) prediction. Asking LIME to
explain the predictions for these adversaries permits us to study whether local
explanations do indeed detect these regions of interest. We also look at
whether LIME is more successful in finding perturbations that are more
prominent and easily noticeable for a human. Our results suggest that LIME does
not necessarily manage to identify the most relevant input features and hence
it remains unclear whether explanations are useful or even misleading.


http://arxiv.org/abs/2107.08909
MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AI. (33%)
Takayuki Miura; Satoshi Hasegawa; Toshiki Shibahara
  The advance of explainable artificial intelligence, which provides reasons
for its predictions, is expected to accelerate the use of deep neural networks
in the real world like Machine Learning as a Service (MLaaS) that returns
predictions on queried data with the trained model. Deep neural networks
deployed in MLaaS face the threat of model extraction attacks. A model
extraction attack is an attack to violate intellectual property and privacy in
which an adversary steals trained models in a cloud using only their
predictions. In particular, a data-free model extraction attack has been
proposed recently and is more critical. In this attack, an adversary uses a
generative model instead of preparing input data. The feasibility of this
attack, however, needs to be studied since it requires more queries than that
with surrogate datasets. In this paper, we propose MEGEX, a data-free model
extraction attack against a gradient-based explainable AI. In this method, an
adversary uses the explanations to train the generative model and reduces the
number of queries to steal the model. Our experiments show that our proposed
method reconstructs high-accuracy models -- 0.97$\times$ and 0.98$\times$ the
victim model accuracy on SVHN and CIFAR-10 datasets given 2M and 20M queries,
respectively. This implies that there is a trade-off between the
interpretability of models and the difficulty of stealing them.


http://arxiv.org/abs/2107.08688
Structural Watermarking to Deep Neural Networks via Network Channel Pruning. (11%)
Xiangyu Zhao; Yinzhe Yao; Hanzhou Wu; Xinpeng Zhang
  In order to protect the intellectual property (IP) of deep neural networks
(DNNs), many existing DNN watermarking techniques either embed watermarks
directly into the DNN parameters or insert backdoor watermarks by fine-tuning
the DNN parameters, which, however, cannot resist against various attack
methods that remove watermarks by altering DNN parameters. In this paper, we
bypass such attacks by introducing a structural watermarking scheme that
utilizes channel pruning to embed the watermark into the host DNN architecture
instead of crafting the DNN parameters. To be specific, during watermark
embedding, we prune the internal channels of the host DNN with the channel
pruning rates controlled by the watermark. During watermark extraction, the
watermark is retrieved by identifying the channel pruning rates from the
architecture of the target DNN model. Due to the superiority of pruning
mechanism, the performance of the DNN model on its original task is reserved
during watermark embedding. Experimental results have shown that, the proposed
work enables the embedded watermark to be reliably recovered and provides a
sufficient payload, without sacrificing the usability of the DNN model. It is
also demonstrated that the proposed work is robust against common transforms
and attacks designed for conventional watermarking approaches.


http://arxiv.org/abs/2108.04328
Generative Adversarial Neural Cellular Automata. (1%)
Maximilian Otte; Quentin Delfosse; Johannes Czech; Kristian Kersting
  Motivated by the interaction between cells, the recently introduced concept
of Neural Cellular Automata shows promising results in a variety of tasks. So
far, this concept was mostly used to generate images for a single scenario. As
each scenario requires a new model, this type of generation seems contradictory
to the adaptability of cells in nature. To address this contradiction, we
introduce a concept using different initial environments as input while using a
single Neural Cellular Automata to produce several outputs. Additionally, we
introduce GANCA, a novel algorithm that combines Neural Cellular Automata with
Generative Adversarial Networks, allowing for more generalization through
adversarial training. The experiments show that a single model is capable of
learning several images when presented with different inputs, and that the
adversarially trained model improves drastically on out-of-distribution data
compared to a supervised trained model.


http://arxiv.org/abs/2107.08767
Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units. (1%)
Woo-Jeoung Nam; Seong-Whan Lee
  As interpretability has been pointed out as the obstacle to the adoption of
Deep Neural Networks (DNNs), there is an increasing interest in solving a
transparency issue to guarantee the impressive performance. In this paper, we
demonstrate the efficiency of recent attribution techniques to explain the
diagnostic decision by visualizing the significant factors in the input image.
By utilizing the characteristics of objectness that DNNs have learned, fully
decomposing the network prediction visualizes clear localization of target
lesion. To verify our work, we conduct our experiments on Chest X-ray diagnosis
with publicly accessible datasets. As an intuitive assessment metric for
explanations, we report the performance of intersection of Union between visual
explanation and bounding box of lesions. Experiment results show that recently
proposed attribution methods visualize the more accurate localization for the
diagnostic decision compared to the traditionally used CAM. Furthermore, we
analyze the inconsistency of intentions between humans and DNNs, which is
easily obscured by high performance. By visualizing the relevant factors, it is
possible to confirm that the criterion for decision is in line with the
learning strategy. Our analysis of unmasking machine intelligence represents
the necessity of explainability in the medical diagnostic decision.


http://arxiv.org/abs/2107.09044
Just Train Twice: Improving Group Robustness without Training Group Information. (1%)
Evan Zheran Liu; Behzad Haghgoo; Annie S. Chen; Aditi Raghunathan; Pang Wei Koh; Shiori Sagawa; Percy Liang; Chelsea Finn
  Standard training via empirical risk minimization (ERM) can produce models
that achieve high accuracy on average but low accuracy on certain groups,
especially in the presence of spurious correlations between the input and
label. Prior approaches that achieve high worst-group accuracy, like group
distributionally robust optimization (group DRO) require expensive group
annotations for each training point, whereas approaches that do not use such
group annotations typically achieve unsatisfactory worst-group accuracy. In
this paper, we propose a simple two-stage approach, JTT, that first trains a
standard ERM model for several epochs, and then trains a second model that
upweights the training examples that the first model misclassified.
Intuitively, this upweights examples from groups on which standard ERM models
perform poorly, leading to improved worst-group performance. Averaged over four
image classification and natural language processing tasks with spurious
correlations, JTT closes 75% of the gap in worst-group accuracy between
standard ERM and group DRO, while only requiring group annotations on a small
validation set in order to tune hyperparameters.


http://arxiv.org/abs/2107.08402
RobustFed: A Truth Inference Approach for Robust Federated Learning. (1%)
Farnaz Tahmasebian; Jian Lou; Li Xiong
  Federated learning is a prominent framework that enables clients (e.g.,
mobile devices or organizations) to train a collaboratively global model under
a central server's orchestration while keeping local training datasets'
privacy. However, the aggregation step in federated learning is vulnerable to
adversarial attacks as the central server cannot manage clients' behavior.
Therefore, the global model's performance and convergence of the training
process will be affected under such attacks.To mitigate this vulnerability
issue, we propose a novel robust aggregation algorithm inspired by the truth
inference methods in crowdsourcing via incorporating the worker's reliability
into aggregation. We evaluate our solution on three real-world datasets with a
variety of machine learning models. Experimental results show that our solution
ensures robust federated learning and is resilient to various types of attacks,
including noisy data attacks, Byzantine attacks, and label flipping attacks.


http://arxiv.org/abs/2107.08189
BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark. (9%)
Anand Avati; Martin Seneviratne; Emily Xue; Zhen Xu; Balaji Lakshminarayanan; Andrew M. Dai
  Machine learning has recently demonstrated impressive progress in predictive
accuracy across a wide array of tasks. Most ML approaches focus on
generalization performance on unseen data that are similar to the training data
(In-Distribution, or IND). However, real world applications and deployments of
ML rarely enjoy the comfort of encountering examples that are always IND. In
such situations, most ML models commonly display erratic behavior on
Out-of-Distribution (OOD) examples, such as assigning high confidence to wrong
predictions, or vice-versa. Implications of such unusual model behavior are
further exacerbated in the healthcare setting, where patient health can
potentially be put at risk. It is crucial to study the behavior and robustness
properties of models under distributional shift, understand common failure
modes, and take mitigation steps before the model is deployed. Having a
benchmark that shines light upon these aspects of a model is a first and
necessary step in addressing the issue. Recent work and interest in increasing
model robustness in OOD settings have focused more on image modality, while the
Electronic Health Record (EHR) modality is still largely under-explored. We aim
to bridge this gap by releasing BEDS-Bench, a benchmark for quantifying the
behavior of ML models over EHR data under OOD settings. We use two open access,
de-identified EHR datasets to construct several OOD data settings to run tests
on, and measure relevant metrics that characterize crucial aspects of a model's
OOD behavior. We evaluate several learning algorithms under BEDS-Bench and find
that all of them show poor generalization performance under distributional
shift in general. Our results highlight the need and the potential to improve
robustness of EHR models under distributional shift, and BEDS-Bench provides
one way to measure progress towards that goal.


http://arxiv.org/abs/2107.07737
EGC2: Enhanced Graph Classification with Easy Graph Compression. (84%)
Jinyin Chen; Dunjie Zhang; Zhaoyan Ming; Mingwei Jia; Yi Liu
  Graph classification plays a significant role in network analysis. It also
faces potential security threat like adversarial attacks. Some defense methods
may sacrifice algorithm complexity for robustness like adversarial training,
while others may sacrifice the clean example performance such as
smoothing-based defense. Most of them are suffered from high-complexity or less
transferability. To address this problem, we proposed EGC$^2$, an enhanced
graph classification model with easy graph compression. EGC$^2$ captures the
relationship between features of different nodes by constructing feature graphs
and improving aggregate node-level representation. To achieve lower complexity
defense applied to various graph classification models, EGC$^2$ utilizes a
centrality-based edge importance index to compress graphs, filtering out
trivial structures and even adversarial perturbations of the input graphs, thus
improves its robustness. Experiments on seven benchmark datasets demonstrate
that the proposed feature read-out and graph compression mechanisms enhance the
robustness of various basic models, thus achieving the state-of-the-art
performance of accuracy and robustness in the threat of different adversarial
attacks.


http://arxiv.org/abs/2107.08821
Proceedings of ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI. (1%)
Quanshi Zhang; Tian Han; Lixin Fan; Zhanxing Zhu; Hang Su; Ying Nian Wu; Jie Ren; Hao Zhang
  This is the Proceedings of ICML 2021 Workshop on Theoretic Foundation,
Criticism, and Application Trend of Explainable AI. Deep neural networks (DNNs)
have undoubtedly brought great success to a wide range of applications in
computer vision, computational linguistics, and AI. However, foundational
principles underlying the DNNs' success and their resilience to adversarial
attacks are still largely missing. Interpreting and theorizing the internal
mechanisms of DNNs becomes a compelling yet controversial topic. This workshop
pays a special interest in theoretic foundations, limitations, and new
application trends in the scope of XAI. These issues reflect new bottlenecks in
the future development of XAI.


http://arxiv.org/abs/2107.07610
Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models. (99%)
Zhao Meng; Yihan Dong; Mrinmaya Sachan; Roger Wattenhofer
  This paper improves the robustness of the pretrained language model, BERT,
against word substitution-based adversarial attacks by leveraging
self-supervised contrastive learning with adversarial perturbations. One
advantage of our method compared to previous works is that it is capable of
improving model robustness without using any labels. Additionally, we also
create an adversarial attack for word-level adversarial training on BERT. The
attack is efficient, allowing adversarial training for BERT on adversarial
examples generated \textit{on the fly} during training. Experimental results
show that our method improves the robustness of BERT against four different
word substitution-based adversarial attacks. Additionally, combining our method
with adversarial training gives higher robustness than adversarial training
alone. Furthermore, to understand why our method can improve the model
robustness against adversarial attacks, we study vector representations of
clean examples and their corresponding adversarial examples before and after
applying our method. As our method improves model robustness with unlabeled raw
data, it opens up the possibility of using large text datasets to train robust
language models.


http://arxiv.org/abs/2107.07449
Adversarial Attacks on Multi-task Visual Perception for Autonomous Driving. (98%)
Ibrahim Sobh; Ahmed Hamed; Varun Ravi Kumar; Senthil Yogamani
  Deep neural networks (DNNs) have accomplished impressive success in various
applications, including autonomous driving perception tasks, in recent years.
On the other hand, current deep neural networks are easily fooled by
adversarial attacks. This vulnerability raises significant concerns,
particularly in safety-critical applications. As a result, research into
attacking and defending DNNs has gained much coverage. In this work, detailed
adversarial attacks are applied on a diverse multi-task visual perception deep
network across distance estimation, semantic segmentation, motion detection,
and object detection. The experiments consider both white and black box attacks
for targeted and un-targeted cases, while attacking a task and inspecting the
effect on all the others, in addition to inspecting the effect of applying a
simple defense method. We conclude this paper by comparing and discussing the
experimental results, proposing insights and future work. The visualizations of
the attacks are available at https://youtu.be/R3JUV41aiPY.


http://arxiv.org/abs/2107.07677
ECG-Adv-GAN: Detecting ECG Adversarial Examples with Conditional Generative Adversarial Networks. (92%)
Khondker Fariha Hossain; Sharif Amit Kamran; Alireza Tavakkoli; Lei Pan; Xingjun Ma; Sutharshan Rajasegarar; Chandan Karmaker
  Electrocardiogram (ECG) acquisition requires an automated system and analysis
pipeline for understanding specific rhythm irregularities. Deep neural networks
have become a popular technique for tracing ECG signals, outperforming human
experts. Despite this, convolutional neural networks are susceptible to
adversarial examples that can misclassify ECG signals and decrease the model's
precision. Moreover, they do not generalize well on the out-of-distribution
dataset. The GAN architecture has been employed in recent works to synthesize
adversarial ECG signals to increase existing training data. However, they use a
disjointed CNN-based classification architecture to detect arrhythmia. Till
now, no versatile architecture has been proposed that can detect adversarial
examples and classify arrhythmia simultaneously. To alleviate this, we propose
a novel Conditional Generative Adversarial Network to simultaneously generate
ECG signals for different categories and detect cardiac abnormalities.
Moreover, the model is conditioned on class-specific ECG signals to synthesize
realistic adversarial examples. Consequently, we compare our architecture and
show how it outperforms other classification models in normal/abnormal ECG
signal detection by benchmarking real world and adversarial signals.


http://arxiv.org/abs/2107.07618
Adversarial Attack for Uncertainty Estimation: Identifying Critical Regions in Neural Networks. (80%)
Ismail Alarab; Simant Prakoonwit
  We propose a novel method to capture data points near decision boundary in
neural network that are often referred to a specific type of uncertainty. In
our approach, we sought to perform uncertainty estimation based on the idea of
adversarial attack method. In this paper, uncertainty estimates are derived
from the input perturbations, unlike previous studies that provide
perturbations on the model's parameters as in Bayesian approach. We are able to
produce uncertainty with couple of perturbations on the inputs. Interestingly,
we apply the proposed method to datasets derived from blockchain. We compare
the performance of model uncertainty with the most recent uncertainty methods.
We show that the proposed method has revealed a significant outperformance over
other methods and provided less risk to capture model uncertainty in machine
learning.


http://arxiv.org/abs/2107.07240
Subnet Replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting. (16%)
Xiangyu Qi; Jifeng Zhu; Chulin Xie; Yong Yang
  We study the realistic potential of conducting backdoor attack against deep
neural networks (DNNs) during deployment stage. Specifically, our goal is to
design a deployment-stage backdoor attack algorithm that is both threatening
and realistically implementable. To this end, we propose Subnet Replacement
Attack (SRA), which is capable of embedding backdoor into DNNs by directly
modifying a limited number of model parameters. Considering the realistic
practicability, we abandon the strong white-box assumption widely adopted in
existing studies, instead, our algorithm works in a gray-box setting, where
architecture information of the victim model is available but the adversaries
do not have any knowledge of parameter values. The key philosophy underlying
our approach is -- given any neural network instance (regardless of its
specific parameter values) of a certain architecture, we can always embed a
backdoor into that model instance, by replacing a very narrow subnet of a
benign model (without backdoor) with a malicious backdoor subnet, which is
designed to be sensitive (fire large activation value) to a particular backdoor
trigger pattern.


http://arxiv.org/abs/2107.07455
Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks. (1%)
Andrey Malinin; Neil Band; German Chesnokov; Yarin Gal; Mark J. F. Gales; Alexey Noskov; Andrey Ploskonosov; Liudmila Prokhorenkova; Ivan Provilkov; Vatsal Raina; Vyas Raina; Mariya Shmatova; Panos Tigas; Boris Yangel
  There has been significant research done on developing methods for improving
robustness to distributional shift and uncertainty estimation. In contrast,
only limited work has examined developing standard datasets and benchmarks for
assessing these approaches. Additionally, most work on uncertainty estimation
and robustness has developed new techniques based on small-scale regression or
image classification tasks. However, many tasks of practical interest have
different modalities, such as tabular data, audio, text, or sensor data, which
offer significant challenges involving regression and discrete or continuous
structured prediction. Thus, given the current state of the field, a
standardized large-scale dataset of tasks across a range of modalities affected
by distributional shifts is necessary. This will enable researchers to
meaningfully evaluate the plethora of recently developed uncertainty
quantification methods, as well as assessment criteria and state-of-the-art
baselines. In this work, we propose the \emph{Shifts Dataset} for evaluation of
uncertainty estimates and robustness to distributional shift. The dataset,
which has been collected from industrial sources and services, is composed of
three tasks, with each corresponding to a particular data modality: tabular
weather prediction, machine translation, and self-driving car (SDC) vehicle
motion prediction. All of these data modalities and tasks are affected by real,
`in-the-wild' distributional shifts and pose interesting challenges with
respect to uncertainty estimation. In this work we provide a description of the
dataset and baseline results for all tasks.


http://arxiv.org/abs/2107.06501
AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learning. (99%)
Yihao Huang; Qing Guo; Felix Juefei-Xu; Lei Ma; Weikai Miao; Yang Liu; Geguang Pu
  High-level representation-guided pixel denoising and adversarial training are
independent solutions to enhance the robustness of CNNs against adversarial
attacks by pre-processing input data and re-training models, respectively. Most
recently, adversarial training techniques have been widely studied and improved
while the pixel denoising-based method is getting less attractive. However, it
is still questionable whether there exists a more advanced pixel
denoising-based method and whether the combination of the two solutions
benefits each other. To this end, we first comprehensively investigate two
kinds of pixel denoising methods for adversarial robustness enhancement (i.e.,
existing additive-based and unexplored filtering-based methods) under the loss
functions of image-level and semantic-level, respectively, showing that
pixel-wise filtering can obtain much higher image quality (e.g., higher PSNR)
as well as higher robustness (e.g., higher accuracy on adversarial examples)
than existing pixel-wise additive-based method. However, we also observe that
the robustness results of the filtering-based method rely on the perturbation
amplitude of adversarial examples used for training. To address this problem,
we propose predictive perturbation-aware & pixel-wise filtering}, where
dual-perturbation filtering and an uncertainty-aware fusion module are designed
and employed to automatically perceive the perturbation amplitude during the
training and testing process. The method is termed as AdvFilter. Moreover, we
combine adversarial pixel denoising methods with three adversarial
training-based methods, hinting that considering data and models jointly is
able to achieve more robust CNNs. The experiments conduct on NeurIPS-2017DEV,
SVHN and CIFAR10 datasets and show advantages over enhancing CNNs' robustness,
high generalization to different models and noise levels.


http://arxiv.org/abs/2107.06882
Conservative Objective Models for Effective Offline Model-Based Optimization. (67%)
Brandon Trabucco; Aviral Kumar; Xinyang Geng; Sergey Levine
  Computational design problems arise in a number of settings, from synthetic
biology to computer architectures. In this paper, we aim to solve data-driven
model-based optimization (MBO) problems, where the goal is to find a design
input that maximizes an unknown objective function provided access to only a
static dataset of prior experiments. Such data-driven optimization procedures
are the only practical methods in many real-world domains where active data
collection is expensive (e.g., when optimizing over proteins) or dangerous
(e.g., when optimizing over aircraft designs). Typical methods for MBO that
optimize the design against a learned model suffer from distributional shift:
it is easy to find a design that "fools" the model into predicting a high
value. To overcome this, we propose conservative objective models (COMs), a
method that learns a model of the objective function that lower bounds the
actual value of the ground-truth objective on out-of-distribution inputs, and
uses it for optimization. Structurally, COMs resemble adversarial training
methods used to overcome adversarial examples. COMs are simple to implement and
outperform a number of existing methods on a wide range of MBO problems,
including optimizing protein sequences, robot morphologies, neural network
weights, and superconducting materials.


http://arxiv.org/abs/2107.06456
AID-Purifier: A Light Auxiliary Network for Boosting Adversarial Defense. (88%)
Duhun Hwang; Eunjung Lee; Wonjong Rhee
  We propose an AID-purifier that can boost the robustness of
adversarially-trained networks by purifying their inputs. AID-purifier is an
auxiliary network that works as an add-on to an already trained main
classifier. To keep it computationally light, it is trained as a discriminator
with a binary cross-entropy loss. To obtain additionally useful information
from the adversarial examples, the architecture design is closely related to
information maximization principles where two layers of the main classification
network are piped to the auxiliary network. To assist the iterative
optimization procedure of purification, the auxiliary network is trained with
AVmixup. AID-purifier can be used together with other purifiers such as
PixelDefend for an extra enhancement. The overall results indicate that the
best performing adversarially-trained networks can be enhanced by the best
performing purification networks, where AID-purifier is a competitive candidate
that is light and robust.


http://arxiv.org/abs/2107.06400
Using BERT Encoding to Tackle the Mad-lib Attack in SMS Spam Detection. (69%)
Sergio Rojas-Galeano
  One of the stratagems used to deceive spam filters is to substitute vocables
with synonyms or similar words that turn the message unrecognisable by the
detection algorithms. In this paper we investigate whether the recent
development of language models sensitive to the semantics and context of words,
such as Google's BERT, may be useful to overcome this adversarial attack
(called "Mad-lib" as per the word substitution game). Using a dataset of 5572
SMS spam messages, we first established a baseline of detection performance
using widely known document representation models (BoW and TFIDF) and the novel
BERT model, coupled with a variety of classification algorithms (Decision Tree,
kNN, SVM, Logistic Regression, Naive Bayes, Multilayer Perceptron). Then, we
built a thesaurus of the vocabulary contained in these messages, and set up a
Mad-lib attack experiment in which we modified each message of a held out
subset of data (not used in the baseline experiment) with different rates of
substitution of original words with synonyms from the thesaurus. Lastly, we
evaluated the detection performance of the three representation models (BoW,
TFIDF and BERT) coupled with the best classifier from the baseline experiment
(SVM). We found that the classic models achieved a 94% Balanced Accuracy (BA)
in the original dataset, whereas the BERT model obtained 96%. On the other
hand, the Mad-lib attack experiment showed that BERT encodings manage to
maintain a similar BA performance of 96% with an average substitution rate of
1.82 words per message, and 95% with 3.34 words substituted per message. In
contrast, the BA performance of the BoW and TFIDF encoders dropped to chance.
These results hint at the potential advantage of BERT models to combat these
type of ingenious attacks, offsetting to some extent for the inappropriate use
of semantic relationships in language.


http://arxiv.org/abs/2107.06158
Correlation Analysis between the Robustness of Sparse Neural Networks and their Random Hidden Structural Priors. (41%)
M. Ben Amor; J. Stier; M. Granitzer
  Deep learning models have been shown to be vulnerable to adversarial attacks.
This perception led to analyzing deep learning models not only from the
perspective of their performance measures but also their robustness to certain
types of adversarial attacks. We take another step forward in relating the
architectural structure of neural networks from a graph theoretic perspective
to their robustness. We aim to investigate any existing correlations between
graph theoretic properties and the robustness of Sparse Neural Networks. Our
hypothesis is, that graph theoretic properties as a prior of neural network
structures are related to their robustness. To answer to this hypothesis, we
designed an empirical study with neural network models obtained through random
graphs used as sparse structural priors for the networks. We additionally
investigated the evaluation of a randomly pruned fully connected network as a
point of reference.
  We found that robustness measures are independent of initialization methods
but show weak correlations with graph properties: higher graph densities
correlate with lower robustness, but higher average path lengths and average
node eccentricities show negative correlations with robustness measures. We
hope to motivate further empirical and analytical research to tightening an
answer to our hypothesis.


http://arxiv.org/abs/2107.06217
What classifiers know what they don't? (1%)
Mohamed Ishmael Belghazi; David Lopez-Paz
  Being uncertain when facing the unknown is key to intelligent decision
making. However, machine learning algorithms lack reliable estimates about
their predictive uncertainty. This leads to wrong and overly-confident
decisions when encountering classes unseen during training. Despite the
importance of equipping classifiers with uncertainty estimates ready for the
real world, prior work has focused on small datasets and little or no class
discrepancy between training and testing data. To close this gap, we introduce
UIMNET: a realistic, ImageNet-scale test-bed to evaluate predictive uncertainty
estimates for deep image classifiers. Our benchmark provides implementations of
eight state-of-the-art algorithms, six uncertainty measures, four in-domain
metrics, three out-domain metrics, and a fully automated pipeline to train,
calibrate, ensemble, select, and evaluate models. Our test-bed is open-source
and all of our results are reproducible from a fixed commit in our repository.
Adding new datasets, algorithms, measures, or metrics is a matter of a few
lines of code-in so hoping that UIMNET becomes a stepping stone towards
realistic, rigorous, and reproducible research in uncertainty estimation. Our
results show that ensembles of ERM classifiers as well as single MIMO
classifiers are the two best alternatives currently available to measure
uncertainty about both in-domain and out-domain classes.


http://arxiv.org/abs/2107.05754
EvoBA: An Evolution Strategy as a Strong Baseline forBlack-Box Adversarial Attacks. (99%)
Andrei Ilie; Marius Popescu; Alin Stefanescu
  Recent work has shown how easily white-box adversarial attacks can be applied
to state-of-the-art image classifiers. However, real-life scenarios resemble
more the black-box adversarial conditions, lacking transparency and usually
imposing natural, hard constraints on the query budget.
  We propose $\textbf{EvoBA}$, a black-box adversarial attack based on a
surprisingly simple evolutionary search strategy. $\textbf{EvoBA}$ is
query-efficient, minimizes $L_0$ adversarial perturbations, and does not
require any form of training.
  $\textbf{EvoBA}$ shows efficiency and efficacy through results that are in
line with much more complex state-of-the-art black-box attacks such as
$\textbf{AutoZOOM}$. It is more query-efficient than $\textbf{SimBA}$, a simple
and powerful baseline black-box attack, and has a similar level of complexity.
Therefore, we propose it both as a new strong baseline for black-box
adversarial attacks and as a fast and general tool for gaining empirical
insight into how robust image classifiers are with respect to $L_0$ adversarial
perturbations.
  There exist fast and reliable $L_2$ black-box attacks, such as
$\textbf{SimBA}$, and $L_{\infty}$ black-box attacks, such as
$\textbf{DeepSearch}$. We propose $\textbf{EvoBA}$ as a query-efficient $L_0$
black-box adversarial attack which, together with the aforementioned methods,
can serve as a generic tool to assess the empirical robustness of image
classifiers. The main advantages of such methods are that they run fast, are
query-efficient, and can easily be integrated in image classifiers development
pipelines.
  While our attack minimises the $L_0$ adversarial perturbation, we also report
$L_2$, and notice that we compare favorably to the state-of-the-art $L_2$
black-box attack, $\textbf{AutoZOOM}$, and of the $L_2$ strong baseline,
$\textbf{SimBA}$.


http://arxiv.org/abs/2107.05780
Detect and Defense Against Adversarial Examples in Deep Learning using Natural Scene Statistics and Adaptive Denoising. (99%)
Anouar Kherchouche; Sid Ahmed Fezza; Wassim Hamidouche
  Despite the enormous performance of deepneural networks (DNNs), recent
studies have shown theirvulnerability to adversarial examples (AEs), i.e.,
care-fully perturbed inputs designed to fool the targetedDNN. Currently, the
literature is rich with many ef-fective attacks to craft such AEs. Meanwhile,
many de-fenses strategies have been developed to mitigate thisvulnerability.
However, these latter showed their effec-tiveness against specific attacks and
does not general-ize well to different attacks. In this paper, we proposea
framework for defending DNN classifier against ad-versarial samples. The
proposed method is based on atwo-stage framework involving a separate detector
anda denoising block. The detector aims to detect AEs bycharacterizing them
through the use of natural scenestatistic (NSS), where we demonstrate that
these statis-tical features are altered by the presence of
adversarialperturbations. The denoiser is based on block matching3D (BM3D)
filter fed by an optimum threshold valueestimated by a convolutional neural
network (CNN) toproject back the samples detected as AEs into theirdata
manifold. We conducted a complete evaluation onthree standard datasets namely
MNIST, CIFAR-10 andTiny-ImageNet. The experimental results show that
theproposed defense method outperforms the state-of-the-art defense techniques
by improving the robustnessagainst a set of attacks under black-box, gray-box
and white-box settings. The source code is available at:
https://github.com/kherchouche-anouar/2DAE


http://arxiv.org/abs/2107.05222
Perceptual-based deep-learning denoiser as a defense against adversarial attacks on ASR systems. (96%)
Anirudh Sreeram; Nicholas Mehlman; Raghuveer Peri; Dillon Knox; Shrikanth Narayanan
  In this paper we investigate speech denoising as a defense against
adversarial attacks on automatic speech recognition (ASR) systems. Adversarial
attacks attempt to force misclassification by adding small perturbations to the
original speech signal. We propose to counteract this by employing a
neural-network based denoiser as a pre-processor in the ASR pipeline. The
denoiser is independent of the downstream ASR model, and thus can be rapidly
deployed in existing systems. We found that training the denoisier using a
perceptually motivated loss function resulted in increased adversarial
robustness without compromising ASR performance on benign samples. Our defense
was evaluated (as a part of the DARPA GARD program) on the 'Kenansville' attack
strategy across a range of attack strengths and speech samples. An average
improvement in Word Error Rate (WER) of about 7.7% was observed over the
undefended model at 20 dB signal-to-noise-ratio (SNR) attack strength.


http://arxiv.org/abs/2107.05243
Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning. (81%)
Jun Wang; Chang Xu; Francisco Guzman; Ahmed El-Kishky; Yuqing Tang; Benjamin I. P. Rubinstein; Trevor Cohn
  Neural machine translation systems are known to be vulnerable to adversarial
test inputs, however, as we show in this paper, these systems are also
vulnerable to training attacks. Specifically, we propose a poisoning attack in
which a malicious adversary inserts a small poisoned sample of monolingual text
into the training set of a system trained using back-translation. This sample
is designed to induce a specific, targeted translation behaviour, such as
peddling misinformation. We present two methods for crafting poisoned examples,
and show that only a tiny handful of instances, amounting to only 0.02% of the
training set, is sufficient to enact a successful attack. We outline a defence
method against said attacks, which partly ameliorates the problem. However, we
stress that this is a blind-spot in modern NMT, demanding immediate attention.


http://arxiv.org/abs/2107.05712
A Closer Look at the Adversarial Robustness of Information Bottleneck Models. (70%)
Iryna Korshunova; David Stutz; Alexander A. Alemi; Olivia Wiles; Sven Gowal
  We study the adversarial robustness of information bottleneck models for
classification. Previous works showed that the robustness of models trained
with information bottlenecks can improve upon adversarial training. Our
evaluation under a diverse range of white-box $l_{\infty}$ attacks suggests
that information bottlenecks alone are not a strong defense strategy, and that
previous results were likely influenced by gradient obfuscation.


http://arxiv.org/abs/2107.05747
SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks. (56%)
Timoleon Moraitis; Dmitry Toichkin; Yansong Chua; Qinghai Guo
  State-of-the-art artificial neural networks (ANNs) require labelled data or
feedback between layers, are often biologically implausible, and are vulnerable
to adversarial attacks that humans are not susceptible to. On the other hand,
Hebbian learning in winner-take-all (WTA) networks, is unsupervised,
feed-forward, and biologically plausible. However, a modern objective
optimization theory for WTA networks has been missing, except under very
limiting assumptions. Here we derive formally such a theory, based on
biologically plausible but generic ANN elements. Through Hebbian learning,
network parameters maintain a Bayesian generative model of the data. There is
no supervisory loss function, but the network does minimize cross-entropy
between its activations and the input distribution. The key is a "soft" WTA
where there is no absolute "hard" winner neuron, and a specific type of
Hebbian-like plasticity of weights and biases. We confirm our theory in
practice, where, in handwritten digit (MNIST) recognition, our Hebbian
algorithm, SoftHebb, minimizes cross-entropy without having access to it, and
outperforms the more frequently used, hard-WTA-based method. Strikingly, it
even outperforms supervised end-to-end backpropagation, under certain
conditions. Specifically, in a two-layered network, SoftHebb outperforms
backpropagation when the training dataset is only presented once, when the
testing data is noisy, and under gradient-based adversarial attacks. Notably,
adversarial attacks that confuse SoftHebb are also confusing to the human eye.
Finally, the model can generate interpolations of objects from its input
distribution. All in all, SoftHebb extends Hebbian WTA theory with modern
machine learning tools, thus making these networks relevant to pertinent issues
in deep learning.


http://arxiv.org/abs/2107.10302
Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attacks. (76%)
Kendra Albert; Maggie Delano; Bogdan Kulynych; Ram Shankar Siva Kumar
  Attacks from adversarial machine learning (ML) have the potential to be used
"for good": they can be used to run counter to the existing power structures
within ML, creating breathing space for those who would otherwise be the
targets of surveillance and control. But most research on adversarial ML has
not engaged in developing tools for resistance against ML systems. Why? In this
paper, we review the broader impact statements that adversarial ML researchers
wrote as part of their NeurIPS 2020 papers and assess the assumptions that
authors have about the goals of their work. We also collect information about
how authors view their work's impact more generally. We find that most
adversarial ML researchers at NeurIPS hold two fundamental assumptions that
will make it difficult for them to consider socially beneficial uses of
attacks: (1) it is desirable to make systems robust, independent of context,
and (2) attackers of systems are normatively bad and defenders of systems are
normatively good. That is, despite their expressed and supposed neutrality,
most adversarial ML researchers believe that the goal of their work is to
secure systems, making it difficult to conceptualize and build tools for
disrupting the status quo.


http://arxiv.org/abs/2107.05166
Stateful Detection of Model Extraction Attacks. (2%)
Soham Pal; Yash Gupta; Aditya Kanade; Shirish Shevade
  Machine-Learning-as-a-Service providers expose machine learning (ML) models
through application programming interfaces (APIs) to developers. Recent work
has shown that attackers can exploit these APIs to extract good approximations
of such ML models, by querying them with samples of their choosing. We propose
VarDetect, a stateful monitor that tracks the distribution of queries made by
users of such a service, to detect model extraction attacks. Harnessing the
latent distributions learned by a modified variational autoencoder, VarDetect
robustly separates three types of attacker samples from benign samples, and
successfully raises an alarm for each. Further, with VarDetect deployed as an
automated defense mechanism, the extracted substitute models are found to
exhibit poor performance and transferability, as intended. Finally, we
demonstrate that even adaptive attackers with prior knowledge of the deployment
of VarDetect, are detected by it.


http://arxiv.org/abs/2107.05127
Attack Rules: An Adversarial Approach to Generate Attacks for Industrial Control Systems using Machine Learning. (1%)
Muhammad Azmi Umer; Chuadhry Mujeeb Ahmed; Muhammad Taha Jilani; Aditya P. Mathur
  Adversarial learning is used to test the robustness of machine learning
algorithms under attack and create attacks that deceive the anomaly detection
methods in Industrial Control System (ICS). Given that security assessment of
an ICS demands that an exhaustive set of possible attack patterns is studied,
in this work, we propose an association rule mining-based attack generation
technique. The technique has been implemented using data from a secure Water
Treatment plant. The proposed technique was able to generate more than 300,000
attack patterns constituting a vast majority of new attack vectors which were
not seen before. Automatically generated attacks improve our understanding of
the potential attacks and enable the design of robust attack detection
techniques.


http://arxiv.org/abs/2107.04764
Hack The Box: Fooling Deep Learning Abstraction-Based Monitors. (91%)
Sara Hajj Ibrahim; Mohamed Nassar
  Deep learning is a type of machine learning that adapts a deep hierarchy of
concepts. Deep learning classifiers link the most basic version of concepts at
the input layer to the most abstract version of concepts at the output layer,
also known as a class or label. However, once trained over a finite set of
classes, some deep learning models do not have the power to say that a given
input does not belong to any of the classes and simply cannot be linked.
Correctly invalidating the prediction of unrelated classes is a challenging
problem that has been tackled in many ways in the literature. Novelty detection
gives deep learning the ability to output "do not know" for novel/unseen
classes. Still, no attention has been given to the security aspects of novelty
detection. In this paper, we consider the case study of abstraction-based
novelty detection and show that it is not robust against adversarial samples.
Moreover, we show the feasibility of crafting adversarial samples that fool the
deep learning classifier and bypass the novelty detection monitoring at the
same time. In other words, these monitoring boxes are hackable. We demonstrate
that novelty detection itself ends up as an attack surface.


http://arxiv.org/abs/2107.04827
Identifying Layers Susceptible to Adversarial Attacks. (83%)
Shoaib Ahmed Siddiqui; Thomas Breuel
  Common neural network architectures are susceptible to attack by adversarial
samples. Neural network architectures are commonly thought of as divided into
low-level feature extraction layers and high-level classification layers;
susceptibility of networks to adversarial samples is often thought of as a
problem related to classification rather than feature extraction. We test this
idea by selectively retraining different portions of VGG and ResNet
architectures on CIFAR-10, Imagenette and ImageNet using non-adversarial and
adversarial data. Our experimental results show that susceptibility to
adversarial samples is associated with low-level feature extraction layers.
Therefore, retraining high-level layers is insufficient for achieving
robustness. This phenomenon could have two explanations: either, adversarial
attacks yield outputs from early layers that are indistinguishable from
features found in the attack classes, or adversarial attacks yield outputs from
early layers that differ statistically from features for non-adversarial
samples and do not permit consistent classification by subsequent layers. We
test this question by large-scale non-linear dimensionality reduction and
density modeling on distributions of feature vectors in hidden layers and find
that the feature distributions between non-adversarial and adversarial samples
differ substantially. Our results provide new insights into the statistical
origins of adversarial samples and possible defenses.


http://arxiv.org/abs/2107.04863
HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks. (81%)
Florian Tambon; Giulio Antoniol; Foutse Khomh
  Deep Neural Networks (DNN) applications are increasingly becoming a part of
our everyday life, from medical applications to autonomous cars. Traditional
validation of DNN relies on accuracy measures, however, the existence of
adversarial examples has highlighted the limitations of these accuracy
measures, raising concerns especially when DNN are integrated into
safety-critical systems. In this paper, we present HOMRS, an approach to boost
metamorphic testing by automatically building a small optimized set of high
order metamorphic relations from an initial set of elementary metamorphic
relations. HOMRS' backbone is a multi-objective search; it exploits ideas drawn
from traditional systems testing such as code coverage, test case, and path
diversity. We applied HOMRS to LeNet5 DNN with MNIST dataset and we report
evidence that it builds a small but effective set of high order transformations
achieving a 95% kill ratio. Five raters manually labeled a pool of images
before and after high order transformation; Fleiss' Kappa and statistical tests
confirmed that they are metamorphic properties. HOMRS built-in relations are
also effective to confront adversarial or out-of-distribution examples; HOMRS
detected 92% of randomly sampled out-of-distribution images. HOMRS
transformations are also suitable for online real-time use.


http://arxiv.org/abs/2107.04882
Out of Distribution Detection and Adversarial Attacks on Deep Neural Networks for Robust Medical Image Analysis. (22%)
Anisie Uwimana1; Ransalu Senanayake
  Deep learning models have become a popular choice for medical image analysis.
However, the poor generalization performance of deep learning models limits
them from being deployed in the real world as robustness is critical for
medical applications. For instance, the state-of-the-art Convolutional Neural
Networks (CNNs) fail to detect adversarial samples or samples drawn
statistically far away from the training distribution. In this work, we
experimentally evaluate the robustness of a Mahalanobis distance-based
confidence score, a simple yet effective method for detecting abnormal input
samples, in classifying malaria parasitized cells and uninfected cells. Results
indicated that the Mahalanobis confidence score detector exhibits improved
performance and robustness of deep learning models, and achieves
stateof-the-art performance on both out-of-distribution (OOD) and adversarial
samples.


http://arxiv.org/abs/2107.04910
Cyber-Security Challenges in Aviation Industry: A Review of Current and Future Trends. (1%)
Elochukwu Ukwandu; Mohamed Amine Ben Farah; Hanan Hindy; Miroslav Bures; Robert Atkinson; Christos Tachtatzis; Xavier Bellekens
  The integration of Information and Communication Technology (ICT) tools into
mechanical devices found in aviation industry has raised security concerns. The
more integrated the system, the more vulnerable due to the inherent
vulnerabilities found in ICT tools and software that drives the system. The
security concerns have become more heightened as the concept of
electronic-enabled aircraft and smart airports get refined and implemented
underway. In line with the above, this paper undertakes a review of
cyber-security incidence in the aviation sector over the last 20 years. The
essence is to understand the common threat actors, their motivations, the type
of attacks, aviation infrastructure that is commonly attacked and then match
these so as to provide insight on the current state of the cyber-security in
the aviation sector. The review showed that the industry's threats come mainly
from Advance Persistent Threat (APT) groups that work in collaboration with
some state actors to steal intellectual property and intelligence, in order to
advance their domestic aerospace capabilities as well as possibly monitor,
infiltrate and subvert other nations' capabilities. The segment of the aviation
industry commonly attacked is the Information Technology infrastructure, and
the prominent type of attacks is malicious hacking activities that aim at
gaining unauthorised access using known malicious password cracking techniques
such as Brute force attacks, Dictionary attacks and so on. The review further
analysed the different attack surfaces that exist in aviation industry, threat
dynamics, and use these dynamics to predict future trends of cyberattacks in
the industry. The aim is to provide information for the cybersecurity
professionals and aviation stakeholders for proactive actions in protecting
these critical infrastructures against cyberincidence for an optimal customer
service oriented industry.


http://arxiv.org/abs/2107.04435
Learning to Detect Adversarial Examples Based on Class Scores. (99%)
Tobias Uelwer; Felix Michels; Candido Oliver De
  Given the increasing threat of adversarial attacks on deep neural networks
(DNNs), research on efficient detection methods is more important than ever. In
this work, we take a closer look at adversarial attack detection based on the
class scores of an already trained classification model. We propose to train a
support vector machine (SVM) on the class scores to detect adversarial
examples. Our method is able to detect adversarial examples generated by
various attacks, and can be easily adopted to a plethora of deep classification
models. We show that our approach yields an improved detection rate compared to
an existing method, whilst being easy to implement. We perform an extensive
empirical analysis on different deep classification models, investigating
various state-of-the-art adversarial attacks. Moreover, we observe that our
proposed method is better at detecting a combination of adversarial attacks.
This work indicates the potential of detecting various adversarial attacks
simply by using the class scores of an already trained classification model.


http://arxiv.org/abs/2107.04749
Resilience of Autonomous Vehicle Object Category Detection to Universal Adversarial Perturbations. (99%)
Mohammad Nayeem Teli; Seungwon Oh
  Due to the vulnerability of deep neural networks to adversarial examples,
numerous works on adversarial attacks and defenses have been burgeoning over
the past several years. However, there seem to be some conventional views
regarding adversarial attacks and object detection approaches that most
researchers take for granted. In this work, we bring a fresh perspective on
those procedures by evaluating the impact of universal perturbations on object
detection at a class-level. We apply it to a carefully curated data set related
to autonomous driving. We use Faster-RCNN object detector on images of five
different categories: person, car, truck, stop sign and traffic light from the
COCO data set, while carefully perturbing the images using Universal Dense
Object Suppression algorithm. Our results indicate that person, car, traffic
light, truck and stop sign are resilient in that order (most to least) to
universal perturbations. To the best of our knowledge, this is the first time
such a ranking has been established which is significant for the security of
the data sets pertaining to autonomous vehicles and object detection in
general.


http://arxiv.org/abs/2107.04284
Universal 3-Dimensional Perturbations for Black-Box Attacks on Video Recognition Systems. (99%)
Shangyu Xie; Han Wang; Yu Kong; Yuan Hong
  Widely deployed deep neural network (DNN) models have been proven to be
vulnerable to adversarial perturbations in many applications (e.g., image,
audio and text classifications). To date, there are only a few adversarial
perturbations proposed to deviate the DNN models in video recognition systems
by simply injecting 2D perturbations into video frames. However, such attacks
may overly perturb the videos without learning the spatio-temporal features
(across temporal frames), which are commonly extracted by DNN models for video
recognition. To our best knowledge, we propose the first black-box attack
framework that generates universal 3-dimensional (U3D) perturbations to subvert
a variety of video recognition systems. U3D has many advantages, such as (1) as
the transfer-based attack, U3D can universally attack multiple DNN models for
video recognition without accessing to the target DNN model; (2) the high
transferability of U3D makes such universal black-box attack easy-to-launch,
which can be further enhanced by integrating queries over the target model when
necessary; (3) U3D ensures human-imperceptibility; (4) U3D can bypass the
existing state-of-the-art defense schemes; (5) U3D can be efficiently generated
with a few pre-learned parameters, and then immediately injected to attack
real-time DNN-based video recognition systems. We have conducted extensive
experiments to evaluate U3D on multiple DNN models and three large-scale video
datasets. The experimental results demonstrate its superiority and
practicality.


http://arxiv.org/abs/2107.07043
GGT: Graph-Guided Testing for Adversarial Sample Detection of Deep Neural Network. (98%)
Zuohui Chen; Renxuan Wang; Jingyang Xiang; Yue Yu; Xin Xia; Shouling Ji; Qi Xuan; Xiaoniu Yang
  Deep Neural Networks (DNN) are known to be vulnerable to adversarial samples,
the detection of which is crucial for the wide application of these DNN models.
Recently, a number of deep testing methods in software engineering were
proposed to find the vulnerability of DNN systems, and one of them, i.e., Model
Mutation Testing (MMT), was used to successfully detect various adversarial
samples generated by different kinds of adversarial attacks. However, the
mutated models in MMT are always huge in number (e.g., over 100 models) and
lack diversity (e.g., can be easily circumvented by high-confidence adversarial
samples), which makes it less efficient in real applications and less effective
in detecting high-confidence adversarial samples. In this study, we propose
Graph-Guided Testing (GGT) for adversarial sample detection to overcome these
aforementioned challenges. GGT generates pruned models with the guide of graph
characteristics, each of them has only about 5% parameters of the mutated model
in MMT, and graph guided models have higher diversity. The experiments on
CIFAR10 and SVHN validate that GGT performs much better than MMT with respect
to both effectiveness and efficiency.


http://arxiv.org/abs/2107.04263
Towards Robust General Medical Image Segmentation. (83%)
Laura Daza; Juan C. Pérez; Pablo Arbeláez
  The reliability of Deep Learning systems depends on their accuracy but also
on their robustness against adversarial perturbations to the input data.
Several attacks and defenses have been proposed to improve the performance of
Deep Neural Networks under the presence of adversarial noise in the natural
image domain. However, robustness in computer-aided diagnosis for volumetric
data has only been explored for specific tasks and with limited attacks. We
propose a new framework to assess the robustness of general medical image
segmentation systems. Our contributions are two-fold: (i) we propose a new
benchmark to evaluate robustness in the context of the Medical Segmentation
Decathlon (MSD) by extending the recent AutoAttack natural image classification
framework to the domain of volumetric data segmentation, and (ii) we present a
novel lattice architecture for RObust Generic medical image segmentation (ROG).
Our results show that ROG is capable of generalizing across different tasks of
the MSD and largely surpasses the state-of-the-art under sophisticated
adversarial attacks.


http://arxiv.org/abs/2107.04487
ARC: Adversarially Robust Control Policies for Autonomous Vehicles. (38%)
Sampo Kuutti; Saber Fallah; Richard Bowden
  Deep neural networks have demonstrated their capability to learn control
policies for a variety of tasks. However, these neural network-based policies
have been shown to be susceptible to exploitation by adversarial agents.
Therefore, there is a need to develop techniques to learn control policies that
are robust against adversaries. We introduce Adversarially Robust Control
(ARC), which trains the protagonist policy and the adversarial policy
end-to-end on the same loss. The aim of the protagonist is to maximise this
loss, whilst the adversary is attempting to minimise it. We demonstrate the
proposed ARC training in a highway driving scenario, where the protagonist
controls the follower vehicle whilst the adversary controls the lead vehicle.
By training the protagonist against an ensemble of adversaries, it learns a
significantly more robust control policy, which generalises to a variety of
adversarial strategies. The approach is shown to reduce the amount of
collisions against new adversaries by up to 90.25%, compared to the original
policy. Moreover, by utilising an auxiliary distillation loss, we show that the
fine-tuned control policy shows no drop in performance across its original
training distribution.


http://arxiv.org/abs/2107.03806
Output Randomization: A Novel Defense for both White-box and Black-box Adversarial Models. (99%)
Daniel Park; Haidar Khan; Azer Khan; Alex Gittens; Bülent Yener
  Adversarial examples pose a threat to deep neural network models in a variety
of scenarios, from settings where the adversary has complete knowledge of the
model in a "white box" setting and to the opposite in a "black box" setting. In
this paper, we explore the use of output randomization as a defense against
attacks in both the black box and white box models and propose two defenses. In
the first defense, we propose output randomization at test time to thwart
finite difference attacks in black box settings. Since this type of attack
relies on repeated queries to the model to estimate gradients, we investigate
the use of randomization to thwart such adversaries from successfully creating
adversarial examples. We empirically show that this defense can limit the
success rate of a black box adversary using the Zeroth Order Optimization
attack to 0%. Secondly, we propose output randomization training as a defense
against white box adversaries. Unlike prior approaches that use randomization,
our defense does not require its use at test time, eliminating the Backward
Pass Differentiable Approximation attack, which was shown to be effective
against other randomization defenses. Additionally, this defense has low
overhead and is easily implemented, allowing it to be used together with other
defenses across various model architectures. We evaluate output randomization
training against the Projected Gradient Descent attacker and show that the
defense can reduce the PGD attack's success rate down to 12% when using
cross-entropy loss.


http://arxiv.org/abs/2107.04401
Improving Model Robustness with Latent Distribution Locally and Globally. (99%)
Zhuang Qian; Shufei Zhang; Kaizhu Huang; Qiufeng Wang; Rui Zhang; Xinping Yi
  In this work, we consider model robustness of deep neural networks against
adversarial attacks from a global manifold perspective. Leveraging both the
local and global latent information, we propose a novel adversarial training
method through robust optimization, and a tractable way to generate Latent
Manifold Adversarial Examples (LMAEs) via an adversarial game between a
discriminator and a classifier. The proposed adversarial training with latent
distribution (ATLD) method defends against adversarial attacks by crafting
LMAEs with the latent manifold in an unsupervised manner. ATLD preserves the
local and global information of latent manifold and promises improved
robustness against adversarial attacks. To verify the effectiveness of our
proposed method, we conduct extensive experiments over different datasets
(e.g., CIFAR-10, CIFAR-100, SVHN) with different adversarial attacks (e.g.,
PGD, CW), and show that our method substantially outperforms the
state-of-the-art (e.g., Feature Scattering) in adversarial robustness by a
large accuracy margin. The source codes are available at
https://github.com/LitterQ/ATLD-pytorch.


http://arxiv.org/abs/2107.03759
Analytically Tractable Hidden-States Inference in Bayesian Neural Networks. (50%)
Luong-Ha Nguyen; James-A. Goulet
  With few exceptions, neural networks have been relying on backpropagation and
gradient descent as the inference engine in order to learn the model
parameters, because the closed-form Bayesian inference for neural networks has
been considered to be intractable. In this paper, we show how we can leverage
the tractable approximate Gaussian inference's (TAGI) capabilities to infer
hidden states, rather than only using it for inferring the network's
parameters. One novel aspect it allows is to infer hidden states through the
imposition of constraints designed to achieve specific objectives, as
illustrated through three examples: (1) the generation of adversarial-attack
examples, (2) the usage of a neural network as a black-box optimization method,
and (3) the application of inference on continuous-action reinforcement
learning. These applications showcase how tasks that were previously reserved
to gradient-based optimization approaches can now be approached with
analytically tractable inference


http://arxiv.org/abs/2107.03919
Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning. (31%)
Akshay Mehra; Bhavya Kailkhura; Pin-Yu Chen; Jihun Hamm
  Unsupervised domain adaptation (UDA) enables cross-domain learning without
target domain labels by transferring knowledge from a labeled source domain
whose distribution differs from the target. However, UDA is not always
successful and several accounts of "negative transfer" have been reported in
the literature. In this work, we prove a simple lower bound on the target
domain error that complements the existing upper bound. Our bound shows the
insufficiency of minimizing source domain error and marginal distribution
mismatch for a guaranteed reduction in the target domain error, due to the
possible increase of induced labeling function mismatch. This insufficiency is
further illustrated through simple distributions for which the same UDA
approach succeeds, fails, and may succeed or fail with an equal chance.
Motivated from this, we propose novel data poisoning attacks to fool UDA
methods into learning representations that produce large target domain errors.
We evaluate the effect of these attacks on popular UDA methods using benchmark
datasets where they have been previously shown to be successful. Our results
show that poisoning can significantly decrease the target domain accuracy,
dropping it to almost 0\% in some cases, with the addition of only 10\%
poisoned data in the source domain. The failure of UDA methods demonstrates the
limitations of UDA at guaranteeing cross-domain generalization consistent with
the lower bound. Thus, evaluation of UDA methods in adversarial settings such
as data poisoning can provide a better sense of their robustness in scenarios
unfavorable for UDA.


http://arxiv.org/abs/2107.03050
Controlled Caption Generation for Images Through Adversarial Attacks. (99%)
Nayyer Aafaq; Naveed Akhtar; Wei Liu; Mubarak Shah; Ajmal Mian
  Deep learning is found to be vulnerable to adversarial examples. However, its
adversarial susceptibility in image caption generation is under-explored. We
study adversarial examples for vision and language models, which typically
adopt an encoder-decoder framework consisting of two major components: a
Convolutional Neural Network (i.e., CNN) for image feature extraction and a
Recurrent Neural Network (RNN) for caption generation. In particular, we
investigate attacks on the visual encoder's hidden layer that is fed to the
subsequent recurrent network. The existing methods either attack the
classification layer of the visual encoder or they back-propagate the gradients
from the language model. In contrast, we propose a GAN-based algorithm for
crafting adversarial examples for neural image captioning that mimics the
internal representation of the CNN such that the resulting deep features of the
input image enable a controlled incorrect caption generation through the
recurrent network. Our contribution provides new insights for understanding
adversarial attacks on vision systems with language component. The proposed
method employs two strategies for a comprehensive evaluation. The first
examines if a neural image captioning system can be misled to output targeted
image captions. The second analyzes the possibility of keywords into the
predicted captions. Experiments show that our algorithm can craft effective
adversarial images based on the CNN hidden layers to fool captioning framework.
Moreover, we discover the proposed attack to be highly transferable. Our work
leads to new robustness implications for neural image captioning.


http://arxiv.org/abs/2107.03250
Incorporating Label Uncertainty in Understanding Adversarial Robustness. (38%)
Xiao Zhang; David Evans
  A fundamental question in adversarial machine learning is whether a robust
classifier exists for a given task. A line of research has made progress
towards this goal by studying concentration of measure, but without considering
data labels. We argue that the standard concentration fails to fully
characterize the intrinsic robustness of a classification problem, since it
ignores data labels which are essential to any classification task. Building on
a novel definition of label uncertainty, we empirically demonstrate that error
regions induced by state-of-the-art models tend to have much higher label
uncertainty compared with randomly-selected subsets. This observation motivates
us to adapt a concentration estimation algorithm to account for label
uncertainty, resulting in more accurate intrinsic robustness measures for
benchmark image classification problems. We further provide empirical evidence
showing that adding an abstain option for classifiers based on label
uncertainty can help improve both the clean and robust accuracies of models.


http://arxiv.org/abs/2107.03311
RoFL: Attestable Robustness for Secure Federated Learning. (1%)
Lukas Burkhalter; Hidde Lycklama; Alexander Viand; Nicolas Küchler; Anwar Hithnawi
  Federated Learning is an emerging decentralized machine learning paradigm
that allows a large number of clients to train a joint model without the need
to share their private data. Participants instead only share ephemeral updates
necessary to train the model. To ensure the confidentiality of the client
updates, Federated Learning systems employ secure aggregation; clients encrypt
their gradient updates, and only the aggregated model is revealed to the
server. Achieving this level of data protection, however, presents new
challenges to the robustness of Federated Learning, i.e., the ability to
tolerate failures and attacks. Unfortunately, in this setting, a malicious
client can now easily exert influence on the model behavior without being
detected. As Federated Learning is being deployed in practice in a range of
sensitive applications, its robustness is growing in importance. In this paper,
we take a step towards understanding and improving the robustness of secure
Federated Learning. We start this paper with a systematic study that evaluates
and analyzes existing attack vectors and discusses potential defenses and
assesses their effectiveness. We then present RoFL, a secure Federated Learning
system that improves robustness against malicious clients through input checks
on the encrypted model updates. RoFL extends Federated Learning's secure
aggregation protocol to allow expressing a variety of properties and
constraints on model updates using zero-knowledge proofs. To enable RoFL to
scale to typical Federated Learning settings, we introduce several ML and
cryptographic optimizations specific to Federated Learning. We implement and
evaluate a prototype of RoFL and show that realistic ML models can be trained
in a reasonable time while improving robustness.


http://arxiv.org/abs/2107.02425
GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularization. (99%)
Sungyoon Lee; Hoki Kim; Jaewook Lee
  Deep learning is vulnerable to adversarial examples. Many defenses based on
randomized neural networks have been proposed to solve the problem, but fail to
achieve robustness against attacks using proxy gradients such as the
Expectation over Transformation (EOT) attack. We investigate the effect of the
adversarial attacks using proxy gradients on randomized neural networks and
demonstrate that it highly relies on the directional distribution of the loss
gradients of the randomized neural network. We show in particular that proxy
gradients are less effective when the gradients are more scattered. To this
end, we propose Gradient Diversity (GradDiv) regularizations that minimize the
concentration of the gradients to build a robust randomized neural network. Our
experiments on MNIST, CIFAR10, and STL10 show that our proposed GradDiv
regularizations improve the adversarial robustness of randomized neural
networks against a variety of state-of-the-art attack methods. Moreover, our
method efficiently reduces the transferability among sample models of
randomized neural networks.


http://arxiv.org/abs/2107.02434
Self-Adversarial Training incorporating Forgery Attention for Image Forgery Localization. (95%)
Long Zhuo; Shunquan Tan; Bin Li; Jiwu Huang
  Image editing techniques enable people to modify the content of an image
without leaving visual traces and thus may cause serious security risks. Hence
the detection and localization of these forgeries become quite necessary and
challenging. Furthermore, unlike other tasks with extensive data, there is
usually a lack of annotated forged images for training due to annotation
difficulties. In this paper, we propose a self-adversarial training strategy
and a reliable coarse-to-fine network that utilizes a self-attention mechanism
to localize forged regions in forgery images. The self-attention module is
based on a Channel-Wise High Pass Filter block (CW-HPF). CW-HPF leverages
inter-channel relationships of features and extracts noise features by high
pass filters. Based on the CW-HPF, a self-attention mechanism, called forgery
attention, is proposed to capture rich contextual dependencies of intrinsic
inconsistency extracted from tampered regions. Specifically, we append two
types of attention modules on top of CW-HPF respectively to model internal
interdependencies in spatial dimension and external dependencies among
channels. We exploit a coarse-to-fine network to enhance the noise
inconsistency between original and tampered regions. More importantly, to
address the issue of insufficient training data, we design a self-adversarial
training strategy that expands training data dynamically to achieve more robust
performance. Specifically, in each training iteration, we perform adversarial
attacks against our network to generate adversarial examples and train our
model on them. Extensive experimental results demonstrate that our proposed
algorithm steadily outperforms state-of-the-art methods by a clear margin in
different benchmark datasets.


http://arxiv.org/abs/2108.04217
ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients. (76%)
Alessandro Cappelli; Julien Launay; Laurent Meunier; Ruben Ohana; Iacopo Poli
  Robustness to adversarial attacks is typically obtained through expensive
adversarial training with Projected Gradient Descent. Here we introduce ROPUST,
a remarkably simple and efficient method to leverage robust pre-trained models
and further increase their robustness, at no cost in natural accuracy. Our
technique relies on the use of an Optical Processing Unit (OPU), a photonic
co-processor, and a fine-tuning step performed with Direct Feedback Alignment,
a synthetic gradient training scheme. We test our method on nine different
models against four attacks in RobustBench, consistently improving over
state-of-the-art performance. We perform an ablation study on the single
components of our defense, showing that robustness arises from parameter
obfuscation and the alternative training method. We also introduce phase
retrieval attacks, specifically designed to increase the threat level of
attackers against our own defense. We show that even with state-of-the-art
phase retrieval techniques, ROPUST remains an effective defense.


http://arxiv.org/abs/2107.02658
On Generalization of Graph Autoencoders with Adversarial Training. (12%)
Tianjin huang; Yulong Pei; Vlado Menkovski; Mykola Pechenizkiy
  Adversarial training is an approach for increasing model's resilience against
adversarial perturbations. Such approaches have been demonstrated to result in
models with feature representations that generalize better. However, limited
works have been done on adversarial training of models on graph data. In this
paper, we raise such a question { does adversarial training improve the
generalization of graph representations. We formulate L2 and L1 versions of
adversarial training in two powerful node embedding methods: graph autoencoder
(GAE) and variational graph autoencoder (VGAE). We conduct extensive
experiments on three main applications, i.e. link prediction, node clustering,
graph anomaly detection of GAE and VGAE, and demonstrate that both L2 and L1
adversarial training boost the generalization of GAE and VGAE.


http://arxiv.org/abs/2107.02488
On Robustness of Lane Detection Models to Physical-World Adversarial Attacks in Autonomous Driving. (1%)
Takami Sato; Qi Alfred Chen
  After the 2017 TuSimple Lane Detection Challenge, its evaluation based on
accuracy and F1 score has become the de facto standard to measure the
performance of lane detection methods. In this work, we conduct the first
large-scale empirical study to evaluate the robustness of state-of-the-art lane
detection methods under physical-world adversarial attacks in autonomous
driving. We evaluate 4 major types of lane detection approaches with the
conventional evaluation and end-to-end evaluation in autonomous driving
scenarios and then discuss the security proprieties of each lane detection
model. We demonstrate that the conventional evaluation fails to reflect the
robustness in end-to-end autonomous driving scenarios. Our results show that
the most robust model on the conventional metrics is the least robust in the
end-to-end evaluation. Although the competition dataset and its metrics have
played a substantial role in developing performant lane detection methods along
with the rapid development of deep neural networks, the conventional evaluation
is becoming obsolete and the gap between the metrics and practicality is
critical. We hope that our study will help the community make further progress
in building a more comprehensive framework to evaluate lane detection models.


http://arxiv.org/abs/2107.01943
When and How to Fool Explainable Models (and Humans) with Adversarial Examples. (99%)
Jon Vadillo; Roberto Santana; Jose A. Lozano
  Reliable deployment of machine learning models such as neural networks
continues to be challenging due to several limitations. Some of the main
shortcomings are the lack of interpretability and the lack of robustness
against adversarial examples or out-of-distribution inputs. In this paper, we
explore the possibilities and limits of adversarial attacks for explainable
machine learning models. First, we extend the notion of adversarial examples to
fit in explainable machine learning scenarios, in which the inputs, the output
classifications and the explanations of the model's decisions are assessed by
humans. Next, we propose a comprehensive framework to study whether (and how)
adversarial examples can be generated for explainable models under human
assessment, introducing novel attack paradigms. In particular, our framework
considers a wide range of relevant (yet often ignored) factors such as the type
of problem, the user expertise or the objective of the explanations in order to
identify the attack strategies that should be adopted in each scenario to
successfully deceive the model (and the human). These contributions intend to
serve as a basis for a more rigorous and realistic study of adversarial
examples in the field of explainable machine learning.


http://arxiv.org/abs/2107.01809
Boosting Transferability of Targeted Adversarial Examples via Hierarchical Generative Networks. (99%)
Xiao Yang; Yinpeng Dong; Tianyu Pang; Hang Su; Jun Zhu
  Transfer-based adversarial attacks can effectively evaluate model robustness
in the black-box setting. Though several methods have demonstrated impressive
transferability of untargeted adversarial examples, targeted adversarial
transferability is still challenging. The existing methods either have low
targeted transferability or sacrifice computational efficiency. In this paper,
we develop a simple yet practical framework to efficiently craft targeted
transfer-based adversarial examples. Specifically, we propose a conditional
generative attacking model, which can generate the adversarial examples
targeted at different classes by simply altering the class embedding and share
a single backbone. Extensive experiments demonstrate that our method improves
the success rates of targeted black-box attacks by a significant margin over
the existing methods -- it reaches an average success rate of 29.6\% against
six diverse models based only on one substitute white-box model in the standard
testing of NeurIPS 2017 competition, which outperforms the state-of-the-art
gradient-based attack methods (with an average success rate of $<$2\%) by a
large margin. Moreover, the proposed method is also more efficient beyond an
order of magnitude than gradient-based methods.


http://arxiv.org/abs/2107.01936
Adversarial Robustness of Probabilistic Network Embedding for Link Prediction. (87%)
Xi Chen; Bo Kang; Jefrey Lijffijt; Bie Tijl De
  In today's networked society, many real-world problems can be formalized as
predicting links in networks, such as Facebook friendship suggestions,
e-commerce recommendations, and the prediction of scientific collaborations in
citation networks. Increasingly often, link prediction problem is tackled by
means of network embedding methods, owing to their state-of-the-art
performance. However, these methods lack transparency when compared to simpler
baselines, and as a result their robustness against adversarial attacks is a
possible point of concern: could one or a few small adversarial modifications
to the network have a large impact on the link prediction performance when
using a network embedding model? Prior research has already investigated
adversarial robustness for network embedding models, focused on classification
at the node and graph level. Robustness with respect to the link prediction
downstream task, on the other hand, has been explored much less.
  This paper contributes to filling this gap, by studying adversarial
robustness of Conditional Network Embedding (CNE), a state-of-the-art
probabilistic network embedding model, for link prediction. More specifically,
given CNE and a network, we measure the sensitivity of the link predictions of
the model to small adversarial perturbations of the network, namely changes of
the link status of a node pair. Thus, our approach allows one to identify the
links and non-links in the network that are most vulnerable to such
perturbations, for further investigation by an analyst. We analyze the
characteristics of the most and least sensitive perturbations, and empirically
confirm that our approach not only succeeds in identifying the most vulnerable
links and non-links, but also that it does so in a time-efficient manner thanks
to an effective approximation.


http://arxiv.org/abs/2107.02052
Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning. (69%)
Mathias Löwe; Jennifer Villareale; Evan Freed; Aleksanteri Sladek; Jichen Zhu; Sebastian Risi
  Applying neural network (NN) methods in games can lead to various new and
exciting game dynamics not previously possible. However, they also lead to new
challenges such as the lack of large, clean datasets, varying player skill
levels, and changing gameplay strategies. In this paper, we focus on the
adversarial player strategy aspect in the game iNNk, in which players try to
communicate secret code words through drawings with the goal of not being
deciphered by a NN. Some strategies exploit weaknesses in the NN that
consistently trick it into making incorrect classifications, leading to
unbalanced gameplay. We present a method that combines transfer learning and
ensemble methods to obtain a data-efficient adaptation to these strategies.
This combination significantly outperforms the baseline NN across all
adversarial player strategies despite only being trained on a limited set of
adversarial examples. We expect the methods developed in this paper to be
useful for the rapidly growing field of NN-based games, which will require new
approaches to deal with unforeseen player creativity.


http://arxiv.org/abs/2107.02045
Understanding the Security of Deepfake Detection. (33%)
Xiaoyu Cao; Neil Zhenqiang Gong
  Deepfakes pose growing challenges to the trust of information on the
Internet. Therefore,detecting deepfakes has attracted increasing attentions
from both academia and industry. State-of-the-art deepfake detection methods
consist of two key components, i.e., face extractor and face classifier, which
extract the face region in an image and classify it to be real/fake,
respectively. Existing studies mainly focused on improving the detection
performance in non-adversarial settings, leaving security of deepfake detection
in adversarial settings largely unexplored. In this work, we aim to bridge the
gap. In particular, we perform a systematic measurement study to understand the
security of the state-of-the-art deepfake detection methods in adversarial
settings. We use two large-scale public deepfakes data sources including
FaceForensics++ and Facebook Deepfake Detection Challenge, where the deepfakes
are fake face images; and we train state-of-the-art deepfake detection methods.
These detection methods can achieve 0.94--0.99 accuracies in non-adversarial
settings on these datasets. However, our measurement results uncover multiple
security limitations of the deepfake detection methods in adversarial settings.
First, we find that an attacker can evade a face extractor, i.e., the face
extractor fails to extract the correct face regions, via adding small Gaussian
noise to its deepfake images. Second, we find that a face classifier trained
using deepfakes generated by one method cannot detect deepfakes generated by
another method, i.e., an attacker can evade detection via generating deepfakes
using a new method. Third, we find that an attacker can leverage backdoor
attacks developed by the adversarial machine learning community to evade a face
classifier. Our results highlight that deepfake detection should consider the
adversarial nature of the problem.


http://arxiv.org/abs/2107.01806
Evaluating the Cybersecurity Risk of Real World, Machine Learning Production Systems. (15%)
Ron Bitton; Nadav Maman; Inderjeet Singh; Satoru Momiyama; Yuval Elovici; Asaf Shabtai
  Although cyberattacks on machine learning (ML) production systems can be
harmful, today, security practitioners are ill equipped, lacking methodologies
and tactical tools that would allow them to analyze the security risks of their
ML-based systems. In this paper, we performed a comprehensive threat analysis
of ML production systems. In this analysis, we follow the ontology presented by
NIST for evaluating enterprise network security risk and apply it to ML-based
production systems. Specifically, we (1) enumerate the assets of a typical ML
production system, (2) describe the threat model (i.e., potential adversaries,
their capabilities, and their main goal), (3) identify the various threats to
ML systems, and (4) review a large number of attacks, demonstrated in previous
studies, which can realize these threats. In addition, to quantify the risk of
adversarial machine learning (AML) threat, we introduce a novel scoring system,
which assign a severity score to different AML attacks. The proposed scoring
system utilizes the analytic hierarchy process (AHP) for ranking, with the
assistance of security experts, various attributes of the attacks. Finally, we
developed an extension to the MulVAL attack graph generation and analysis
framework to incorporate cyberattacks on ML production systems. Using the
extension, security practitioners can apply attack graph analysis methods in
environments that include ML components; thus, providing security practitioners
with a methodological and practical tool for evaluating the impact and
quantifying the risk of a cyberattack targeting an ML production system.


http://arxiv.org/abs/2107.01854
Poisoning Attack against Estimating from Pairwise Comparisons. (15%)
Ke Ma; Qianqian Xu; Jinshan Zeng; Xiaochun Cao; Qingming Huang
  As pairwise ranking becomes broadly employed for elections, sports
competitions, recommendations, and so on, attackers have strong motivation and
incentives to manipulate the ranking list. They could inject malicious
comparisons into the training data to fool the victim. Such a technique is
called poisoning attack in regression and classification tasks. In this paper,
to the best of our knowledge, we initiate the first systematic investigation of
data poisoning attacks on pairwise ranking algorithms, which can be formalized
as the dynamic and static games between the ranker and the attacker and can be
modeled as certain kinds of integer programming problems. To break the
computational hurdle of the underlying integer programming problems, we
reformulate them into the distributionally robust optimization (DRO) problems,
which are computationally tractable. Based on such DRO formulations, we propose
two efficient poisoning attack algorithms and establish the associated
theoretical guarantees. The effectiveness of the suggested poisoning attack
strategies is demonstrated by a series of toy simulations and several real data
experiments. These experimental results show that the proposed methods can
significantly reduce the performance of the ranker in the sense that the
correlation between the true ranking list and the aggregated results can be
decreased dramatically.


http://arxiv.org/abs/2107.06993
Confidence Conditioned Knowledge Distillation. (10%)
Sourav Mishra; Suresh Sundaram
  In this paper, a novel confidence conditioned knowledge distillation (CCKD)
scheme for transferring the knowledge from a teacher model to a student model
is proposed. Existing state-of-the-art methods employ fixed loss functions for
this purpose and ignore the different levels of information that need to be
transferred for different samples. In addition to that, these methods are also
inefficient in terms of data usage. CCKD addresses these issues by leveraging
the confidence assigned by the teacher model to the correct class to devise
sample-specific loss functions (CCKD-L formulation) and targets (CCKD-T
formulation). Further, CCKD improves the data efficiency by employing
self-regulation to stop those samples from participating in the distillation
process on which the student model learns faster. Empirical evaluations on
several benchmark datasets show that CCKD methods achieve at least as much
generalization performance levels as other state-of-the-art methods while being
data efficient in the process. Student models trained through CCKD methods do
not retain most of the misclassifications commited by the teacher model on the
training set. Distillation through CCKD methods improves the resilience of the
student models against adversarial attacks compared to the conventional KD
method. Experiments show at least 3% increase in performance against
adversarial attacks for the MNIST and the Fashion MNIST datasets, and at least
6% increase for the CIFAR10 dataset.


http://arxiv.org/abs/2107.01561
Certifiably Robust Interpretation via Renyi Differential Privacy. (67%)
Ao Liu; Xiaoyu Chen; Sijia Liu; Lirong Xia; Chuang Gan
  Motivated by the recent discovery that the interpretation maps of CNNs could
easily be manipulated by adversarial attacks against network interpretability,
we study the problem of interpretation robustness from a new perspective of
\Renyi differential privacy (RDP). The advantages of our Renyi-Robust-Smooth
(RDP-based interpretation method) are three-folds. First, it can offer provable
and certifiable top-$k$ robustness. That is, the top-$k$ important attributions
of the interpretation map are provably robust under any input perturbation with
bounded $\ell_d$-norm (for any $d\geq 1$, including $d = \infty$). Second, our
proposed method offers $\sim10\%$ better experimental robustness than existing
approaches in terms of the top-$k$ attributions. Remarkably, the accuracy of
Renyi-Robust-Smooth also outperforms existing approaches. Third, our method can
provide a smooth tradeoff between robustness and computational efficiency.
Experimentally, its top-$k$ attributions are {\em twice} more robust than
existing approaches when the computational resources are highly constrained.


http://arxiv.org/abs/2107.01709
Mirror Mirror on the Wall: Next-Generation Wireless Jamming Attacks Based on Software-Controlled Surfaces. (1%)
Paul Staat; Harald Elders-Boll; Christian Zenger; Christof Paar
  The intelligent reflecting surface (IRS) is a promising new paradigm in
wireless communications to meet the growing demand for high-speed connectivity
in next-generation mobile networks. IRS, also known as software-controlled
metasurfaces, consist of an array of adjustable radio wave reflectors, enabling
smart radio environments, e.g., for enhancing the signal-to-noise ratio (SNR)
and spatial diversity of wireless channels.
  Research on IRS to date has been largely focused on constructive
applications. In this work, we show for the first time that the IRS provides a
practical low-cost toolkit for attackers to easily perform complex signal
manipulation attacks on the physical layer in real time. We introduce the
environment reconfiguration attack (ERA) as a novel class of jamming attacks in
wireless radio networks. Here, an adversary leverages an IRS to rapidly vary
the electromagnetic propagation environment to disturb legitimate receivers.
The IRS gives the adversary a key advantage over traditional jamming: It no
longer has to actively emit a jamming signal itself while the jamming signal is
correlated to the legitimate communication signal.
  We thoroughly investigate the ERA using the popular orthogonal frequency
division multiplexing~(OFDM) modulation as an example. We show that the ERA
allows to severely degrade the available data rates even in entire networks. We
present insights to the attack through analytical analysis, simulations, as
well as experiments. Our results highlight that the attack also works with
reasonably small IRS sizes. Finally, we implement an attacker setup and
demonstrate a practical ERA to slow down a Wi-Fi network.


http://arxiv.org/abs/2107.01396
Demiguise Attack: Crafting Invisible Semantic Adversarial Perturbations with Perceptual Similarity. (99%)
Yajie Wang; Shangbo Wu; Wenyi Jiang; Shengang Hao; Yu-an Tan; Quanxin Zhang
  Deep neural networks (DNNs) have been found to be vulnerable to adversarial
examples. Adversarial examples are malicious images with visually imperceptible
perturbations. While these carefully crafted perturbations restricted with
tight $\Lp$ norm bounds are small, they are still easily perceivable by humans.
These perturbations also have limited success rates when attacking black-box
models or models with defenses like noise reduction filters. To solve these
problems, we propose Demiguise Attack, crafting ``unrestricted'' perturbations
with Perceptual Similarity. Specifically, we can create powerful and
photorealistic adversarial examples by manipulating semantic information based
on Perceptual Similarity. Adversarial examples we generate are friendly to the
human visual system (HVS), although the perturbations are of large magnitudes.
We extend widely-used attacks with our approach, enhancing adversarial
effectiveness impressively while contributing to imperceptibility. Extensive
experiments show that the proposed method not only outperforms various
state-of-the-art attacks in terms of fooling rate, transferability, and
robustness against defenses but can also improve attacks effectively. In
addition, we also notice that our implementation can simulate illumination and
contrast changes that occur in real-world scenarios, which will contribute to
exposing the blind spots of DNNs.


http://arxiv.org/abs/2107.00561
Using Anomaly Feature Vectors for Detecting, Classifying and Warning of Outlier Adversarial Examples. (99%)
Nelson Manohar-Alers; Ryan Feng; Sahib Singh; Jiguo Song; Atul Prakash
  We present DeClaW, a system for detecting, classifying, and warning of
adversarial inputs presented to a classification neural network. In contrast to
current state-of-the-art methods that, given an input, detect whether an input
is clean or adversarial, we aim to also identify the types of adversarial
attack (e.g., PGD, Carlini-Wagner or clean). To achieve this, we extract
statistical profiles, which we term as anomaly feature vectors, from a set of
latent features. Preliminary findings suggest that AFVs can help distinguish
among several types of adversarial attacks (e.g., PGD versus Carlini-Wagner)
with close to 93% accuracy on the CIFAR-10 dataset. The results open the door
to using AFV-based methods for exploring not only adversarial attack detection
but also classification of the attack type and then design of attack-specific
mitigation strategies.


http://arxiv.org/abs/2107.00415
DVS-Attacks: Adversarial Attacks on Dynamic Vision Sensors for Spiking Neural Networks. (99%)
Alberto Marchisio; Giacomo Pira; Maurizio Martina; Guido Masera; Muhammad Shafique
  Spiking Neural Networks (SNNs), despite being energy-efficient when
implemented on neuromorphic hardware and coupled with event-based Dynamic
Vision Sensors (DVS), are vulnerable to security threats, such as adversarial
attacks, i.e., small perturbations added to the input for inducing a
misclassification. Toward this, we propose DVS-Attacks, a set of stealthy yet
efficient adversarial attack methodologies targeted to perturb the event
sequences that compose the input of the SNNs. First, we show that noise filters
for DVS can be used as defense mechanisms against adversarial attacks.
Afterwards, we implement several attacks and test them in the presence of two
types of noise filters for DVS cameras. The experimental results show that the
filters can only partially defend the SNNs against our proposed DVS-Attacks.
Using the best settings for the noise filters, our proposed Mask Filter-Aware
Dash Attack reduces the accuracy by more than 20% on the DVS-Gesture dataset
and by more than 65% on the MNIST dataset, compared to the original clean
frames. The source code of all the proposed DVS-Attacks and noise filters is
released at https://github.com/albertomarchisio/DVS-Attacks.


http://arxiv.org/abs/2107.00440
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding. (68%)
Dong Wang; Ning Ding; Piji Li; Hai-Tao Zheng
  Despite pre-trained language models have proven useful for learning
high-quality semantic representations, these models are still vulnerable to
simple perturbations. Recent works aimed to improve the robustness of
pre-trained models mainly focus on adversarial training from perturbed examples
with similar semantics, neglecting the utilization of different or even
opposite semantics. Different from the image processing field, the text is
discrete and few word substitutions can cause significant semantic changes. To
study the impact of semantics caused by small perturbations, we conduct a
series of pilot experiments and surprisingly find that adversarial training is
useless or even harmful for the model to detect these semantic changes. To
address this problem, we propose Contrastive Learning with semantIc Negative
Examples (CLINE), which constructs semantic negative examples unsupervised to
improve the robustness under semantically adversarial attacking. By comparing
with similar and opposite semantic examples, the model can effectively perceive
the semantic changes caused by small perturbations. Empirical results show that
our approach yields substantial improvements on a range of sentiment analysis,
reasoning, and reading comprehension tasks. And CLINE also ensures the
compactness within the same semantics and separability across different
semantics in sentence-level.


http://arxiv.org/abs/2107.00309
Spotting adversarial samples for speaker verification by neural vocoders. (26%)
Haibin Wu; Po-chun Hsu; Ji Gao; Shanshan Zhang; Shen Huang; Jian Kang; Zhiyong Wu; Helen Meng; Hung-yi Lee
  Automatic speaker verification (ASV), one of the most important technology
for biometric identification, has been widely adopted in security-critic
applications, including transaction authentication and access control. However,
previous works have shown ASV is seriously vulnerable to recently emerged
adversarial attacks, yet effective countermeasures against them are limited. In
this paper, we adopt neural vocoders to spot adversarial samples for ASV. We
use neural vocoder to re-synthesize audio and find that the difference between
the ASV scores for the original and re-synthesized audio is a good indicator to
distinguish genuine and adversarial samples. As the very beginning work in this
direction of detecting adversarial samples for ASV, there is no reliable
baseline for comparison. So we first implement Griffin-Lim for detection and
set it as our baseline. The proposed method accomplishes effective detection
performance and outperforms all the baselines in all the settings. We also show
the neural vocoder adopted in the detection framework is dataset independent.
Our codes will be made open-source for future works to do comparison.


http://arxiv.org/abs/2107.00247
The Interplay between Distribution Parameters and the Accuracy-Robustness Tradeoff in Classification. (16%)
Alireza Mousavi Hosseini; Amir Mohammad Abouei; Mohammad Hossein Rohban
  Adversarial training tends to result in models that are less accurate on
natural (unperturbed) examples compared to standard models. This can be
attributed to either an algorithmic shortcoming or a fundamental property of
the training data distribution, which admits different solutions for optimal
standard and adversarial classifiers. In this work, we focus on the latter case
under a binary Gaussian mixture classification problem. Unlike earlier work, we
aim to derive the natural accuracy gap between the optimal Bayes and
adversarial classifiers, and study the effect of different distributional
parameters, namely separation between class centroids, class proportions, and
the covariance matrix, on the derived gap. We show that under certain
conditions, the natural error of the optimal adversarial classifier, as well as
the gap, are locally minimized when classes are balanced, contradicting the
performance of the Bayes classifier where perfect balance induces the worst
accuracy. Moreover, we show that with an $\ell_\infty$ bounded perturbation and
an adversarial budget of $\epsilon$, this gap is $\Theta(\epsilon^2)$ for the
worst-case parameters, which for suitably small $\epsilon$ indicates the
theoretical possibility of achieving robust classifiers with near-perfect
accuracy, which is rarely reflected in practical algorithms.


http://arxiv.org/abs/2107.00783
Reinforcement Learning for Feedback-Enabled Cyber Resilience. (10%)
Yunhan Huang; Linan Huang; Quanyan Zhu
  The rapid growth in the number of devices and their connectivity has enlarged
the attack surface and weakened cyber systems. As attackers become increasingly
sophisticated and resourceful, mere reliance on traditional cyber protection,
such as intrusion detection, firewalls, and encryption, is insufficient to
secure cyber systems. Cyber resilience provides a new security paradigm that
complements inadequate protection with resilience mechanisms. A Cyber-Resilient
Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in
real-time and strategically responds to them to maintain the critical functions
of the cyber systems. Feedback architectures play a pivotal role in enabling
the online sensing, reasoning, and actuation of the CRM. Reinforcement Learning
(RL) is an important class of algorithms that epitomize the feedback
architectures for cyber resiliency, allowing the CRM to provide dynamic and
sequential responses to attacks with limited prior knowledge of the attacker.
In this work, we review the literature on RL for cyber resiliency and discuss
the cyber-resilient defenses against three major types of vulnerabilities,
i.e., posture-related, information-related, and human-related vulnerabilities.
We introduce moving target defense, defensive cyber deception, and assistive
human security technologies as three application domains of CRMs to elaborate
on their designs. The RL technique also has vulnerabilities itself. We explain
the major vulnerabilities of RL and present several attack models in which the
attacks target the rewards, the measurements, and the actuators. We show that
the attacker can trick the RL agent into learning a nefarious policy with
minimum attacking effort, which shows serious security concerns for RL-enabled
systems. Finally, we discuss the future challenges of RL for cyber security and
resiliency and emerging applications of RL-based CRMs.


http://arxiv.org/abs/2106.15998
Single-Step Adversarial Training for Semantic Segmentation. (96%)
Daniel Wiens; Barbara Hammer
  Even though deep neural networks succeed on many different tasks including
semantic segmentation, they lack on robustness against adversarial examples. To
counteract this exploit, often adversarial training is used. However, it is
known that adversarial training with weak adversarial attacks (e.g. using the
Fast Gradient Method) does not improve the robustness against stronger attacks.
Recent research shows that it is possible to increase the robustness of such
single-step methods by choosing an appropriate step size during the training.
Finding such a step size, without increasing the computational effort of
single-step adversarial training, is still an open challenge. In this work we
address the computationally particularly demanding task of semantic
segmentation and propose a new step size control algorithm that increases the
robustness of single-step adversarial training. The proposed algorithm does not
increase the computational effort of single-step adversarial training
considerably and also simplifies training, because it is free of
meta-parameter. We show that the robustness of our approach can compete with
multi-step adversarial training on two popular benchmarks for semantic
segmentation.


http://arxiv.org/abs/2106.15820
Explanation-Guided Diagnosis of Machine Learning Evasion Attacks. (82%)
Abderrahmen Amich; Birhanu Eshete
  Machine Learning (ML) models are susceptible to evasion attacks. Evasion
accuracy is typically assessed using aggregate evasion rate, and it is an open
question whether aggregate evasion rate enables feature-level diagnosis on the
effect of adversarial perturbations on evasive predictions. In this paper, we
introduce a novel framework that harnesses explainable ML methods to guide
high-fidelity assessment of ML evasion attacks. Our framework enables
explanation-guided correlation analysis between pre-evasion perturbations and
post-evasion explanations. Towards systematic assessment of ML evasion attacks,
we propose and evaluate a novel suite of model-agnostic metrics for
sample-level and dataset-level correlation analysis. Using malware and image
classifiers, we conduct comprehensive evaluations across diverse model
architectures and complementary feature representations. Our explanation-guided
correlation analysis reveals correlation gaps between adversarial samples and
the corresponding perturbations performed on them. Using a case study on
explanation-guided evasion, we show the broader usage of our methodology for
assessing robustness of ML models.


http://arxiv.org/abs/2106.15860
Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning. (80%)
You Qiaoben; Chengyang Ying; Xinning Zhou; Hang Su; Jun Zhu; Bo Zhang
  Recent works demonstrate that deep reinforcement learning (DRL) models are
vulnerable to adversarial attacks which can decrease the victim's total reward
by manipulating the observations. Compared with adversarial attacks in
supervised learning, it is much more challenging to deceive a DRL model since
the adversary has to infer the environmental dynamics. To address this issue,
we reformulate the problem of adversarial attacks in function space and
separate the previous gradient based attacks into several subspace. Following
the analysis of the function space, we design a generic two-stage framework in
the subspace where the adversary lures the agent to a target trajectory or a
deceptive policy. In the first stage, we train a deceptive policy by hacking
the environment, and discover a set of trajectories routing to the lowest
reward. The adversary then misleads the victim to imitate the deceptive policy
by perturbing the observations. Our method provides a tighter theoretical upper
bound for the attacked agent's performance than the existing approaches.
Extensive experiments demonstrate the superiority of our method and we achieve
the state-of-the-art performance on both Atari and MuJoCo environments.


http://arxiv.org/abs/2107.02897
Bi-Level Poisoning Attack Model and Countermeasure for Appliance Consumption Data of Smart Homes. (8%)
Mustain Billah; Adnan Anwar; Ziaur Rahman; Syed Md. Galib
  Accurate building energy prediction is useful in various applications
starting from building energy automation and management to optimal storage
control. However, vulnerabilities should be considered when designing building
energy prediction models, as intelligent attackers can deliberately influence
the model performance using sophisticated attack models. These may consequently
degrade the prediction accuracy, which may affect the efficiency and
performance of the building energy management systems. In this paper, we
investigate the impact of bi-level poisoning attacks on regression models of
energy usage obtained from household appliances. Furthermore, an effective
countermeasure against the poisoning attacks on the prediction model is
proposed in this paper. Attacks and defenses are evaluated on a benchmark
dataset. Experimental results show that an intelligent cyber-attacker can
poison the prediction model to manipulate the decision. However, our proposed
solution successfully ensures defense against such poisoning attacks
effectively compared to other benchmark techniques.


http://arxiv.org/abs/2106.15850
Exploring Robustness of Neural Networks through Graph Measures. (8%)
Asim Rowan University Waqas; Ghulam Rowan University Rasool; Hamza University of Minnesota Farooq; Nidhal C. Rowan University Bouaynaya
  Motivated by graph theory, artificial neural networks (ANNs) are
traditionally structured as layers of neurons (nodes), which learn useful
information by the passage of data through interconnections (edges). In the
machine learning realm, graph structures (i.e., neurons and connections) of
ANNs have recently been explored using various graph-theoretic measures linked
to their predictive performance. On the other hand, in network science
(NetSci), certain graph measures including entropy and curvature are known to
provide insight into the robustness and fragility of real-world networks. In
this work, we use these graph measures to explore the robustness of various
ANNs to adversarial attacks. To this end, we (1) explore the design space of
inter-layer and intra-layers connectivity regimes of ANNs in the graph domain
and record their predictive performance after training under different types of
adversarial attacks, (2) use graph representations for both inter-layer and
intra-layers connectivity regimes to calculate various graph-theoretic
measures, including curvature and entropy, and (3) analyze the relationship
between these graph measures and the adversarial performance of ANNs. We show
that curvature and entropy, while operating in the graph domain, can quantify
the robustness of ANNs without having to train these ANNs. Our results suggest
that the real-world networks, including brain networks, financial networks, and
social networks may provide important clues to the neural architecture search
for robust ANNs. We propose a search strategy that efficiently finds robust
ANNs amongst a set of well-performing ANNs without having a need to train all
of these ANNs.


http://arxiv.org/abs/2106.15890
A Context-Aware Information-Based Clone Node Attack Detection Scheme in Internet of Things. (1%)
Khizar Hameed; Saurabh Garg; Muhammad Bilal Amin; Byeong Kang; Abid Khan
  The rapidly expanding nature of the Internet of Things (IoT) networks is
beginning to attract interest across a range of applications, including smart
homes, smart transportation, smart health, and industrial contexts. This
cutting-edge technology enables individuals to track and control their
integrated environment in real-time and remotely via a thousand IoT devices
comprised of sensors and actuators that actively participate in sensing,
processing, storing, and sharing information. Nonetheless, IoT devices are
frequently deployed in hostile environments, wherein adversaries attempt to
capture and breach them in order to seize control of the entire network. One
such example of potentially malicious behaviour is the cloning of IoT devices,
in which an attacker can physically capture the devices, obtain some sensitive
information, duplicate the devices, and intelligently deploy them in desired
locations to conduct various insider attacks. A device cloning attack on IoT
networks is a significant security concern since it allows for selective
forwarding, sink-hole, and black-hole attacks. To address this issue, this
paper provides an efficient scheme for detecting clone node attacks on IoT
networks that makes use of semantic information about IoT devices known as
context information sensed from the deployed environment to locate them
securely. We design a location proof mechanism by combining location proofs and
batch verification of the extended elliptic curve digital signature technique
to accelerate the verification process at selected trusted nodes. We
demonstrate the security of our scheme and its resilience to secure clone node
attack detection by conducting a comprehensive security analysis. The
performance of our proposed scheme provides a high degree of detection accuracy
with minimal detection time and significantly reduces the computation,
communication and storage overhead.


http://arxiv.org/abs/2106.15853
Understanding and Improving Early Stopping for Learning with Noisy Labels. (1%)
Yingbin Bai; Erkun Yang; Bo Han; Yanhua Yang; Jiatong Li; Yinian Mao; Gang Niu; Tongliang Liu
  The memorization effect of deep neural network (DNN) plays a pivotal role in
many state-of-the-art label-noise learning methods. To exploit this property,
the early stopping trick, which stops the optimization at the early stage of
training, is usually adopted. Current methods generally decide the early
stopping point by considering a DNN as a whole. However, a DNN can be
considered as a composition of a series of layers, and we find that the latter
layers in a DNN are much more sensitive to label noise, while their former
counterparts are quite robust. Therefore, selecting a stopping point for the
whole network may make different DNN layers antagonistically affected each
other, thus degrading the final performance. In this paper, we propose to
separate a DNN into different parts and progressively train them to address
this problem. Instead of the early stopping, which trains a whole DNN all at
once, we initially train former DNN layers by optimizing the DNN with a
relatively large number of epochs. During training, we progressively train the
latter DNN layers by using a smaller number of epochs with the preceding layers
fixed to counteract the impact of noisy labels. We term the proposed method as
progressive early stopping (PES). Despite its simplicity, compared with the
early stopping, PES can help to obtain more promising and stable results.
Furthermore, by combining PES with existing approaches on noisy label training,
we achieve state-of-the-art performance on image classification benchmarks.


http://arxiv.org/abs/2107.02894
Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges. (99%)
Bowei Xi
  We provide a comprehensive overview of adversarial machine learning focusing
on two application domains, i.e., cybersecurity and computer vision. Research
in adversarial machine learning addresses a significant threat to the wide
application of machine learning techniques -- they are vulnerable to carefully
crafted attacks from malicious adversaries. For example, deep neural networks
fail to correctly classify adversarial images, which are generated by adding
imperceptible perturbations to clean images.We first discuss three main
categories of attacks against machine learning techniques -- poisoning attacks,
evasion attacks, and privacy attacks. Then the corresponding defense approaches
are introduced along with the weakness and limitations of the existing defense
approaches. We notice adversarial samples in cybersecurity and computer vision
are fundamentally different. While adversarial samples in cybersecurity often
have different properties/distributions compared with training data,
adversarial images in computer vision are created with minor input
perturbations. This further complicates the development of robust learning
techniques, because a robust learning technique must withstand different types
of attacks.


http://arxiv.org/abs/2107.00003
Understanding Adversarial Examples Through Deep Neural Network's Response Surface and Uncertainty Regions. (99%)
Juan Shu; Bowei Xi; Charles Kamhoua
  Deep neural network (DNN) is a popular model implemented in many systems to
handle complex tasks such as image classification, object recognition, natural
language processing etc. Consequently DNN structural vulnerabilities become
part of the security vulnerabilities in those systems. In this paper we study
the root cause of DNN adversarial examples. We examine the DNN response surface
to understand its classification boundary. Our study reveals the structural
problem of DNN classification boundary that leads to the adversarial examples.
Existing attack algorithms can generate from a handful to a few hundred
adversarial examples given one clean image. We show there are infinitely many
adversarial images given one clean sample, all within a small neighborhood of
the clean sample. We then define DNN uncertainty regions and show
transferability of adversarial examples is not universal. We also argue that
generalization error, the large sample theoretical guarantee established for
DNN, cannot adequately capture the phenomenon of adversarial examples. We need
new theory to measure DNN robustness.


http://arxiv.org/abs/2106.15360
Attack Transferability Characterization for Adversarially Robust Multi-label Classification. (99%)
Zhuo Yang; Yufei Han; Xiangliang Zhang
  Despite of the pervasive existence of multi-label evasion attack, it is an
open yet essential problem to characterize the origin of the adversarial
vulnerability of a multi-label learning system and assess its attackability. In
this study, we focus on non-targeted evasion attack against multi-label
classifiers. The goal of the threat is to cause miss-classification with
respect to as many labels as possible, with the same input perturbation. Our
work gains in-depth understanding about the multi-label adversarial attack by
first characterizing the transferability of the attack based on the functional
properties of the multi-label classifier. We unveil how the transferability
level of the attack determines the attackability of the classifier via
establishing an information-theoretic analysis of the adversarial risk.
Furthermore, we propose a transferability-centered attackability assessment,
named Soft Attackability Estimator (SAE), to evaluate the intrinsic
vulnerability level of the targeted multi-label classifier. This estimator is
then integrated as a transferability-tuning regularization term into the
multi-label learning paradigm to achieve adversarially robust classification.
The experimental study on real-world data echos the theoretical analysis and
verify the validity of the transferability-regularized multi-label learning
method.


http://arxiv.org/abs/2106.15202
Inconspicuous Adversarial Patches for Fooling Image Recognition Systems on Mobile Devices. (99%)
Tao Bai; Jinqi Luo; Jun Zhao
  Deep learning based image recognition systems have been widely deployed on
mobile devices in today's world. In recent studies, however, deep learning
models are shown vulnerable to adversarial examples. One variant of adversarial
examples, called adversarial patch, draws researchers' attention due to its
strong attack abilities. Though adversarial patches achieve high attack success
rates, they are easily being detected because of the visual inconsistency
between the patches and the original images. Besides, it usually requires a
large amount of data for adversarial patch generation in the literature, which
is computationally expensive and time-consuming. To tackle these challenges, we
propose an approach to generate inconspicuous adversarial patches with one
single image. In our approach, we first decide the patch locations basing on
the perceptual sensitivity of victim models, then produce adversarial patches
in a coarse-to-fine way by utilizing multiple-scale generators and
discriminators. The patches are encouraged to be consistent with the background
images with adversarial training while preserving strong attack abilities. Our
approach shows the strong attack abilities in white-box settings and the
excellent transferability in black-box settings through extensive experiments
on various models with different architectures and training methods. Compared
to other adversarial patches, our adversarial patches hold the most negligible
risks to be detected and can evade human observations, which is supported by
the illustrations of saliency maps and results of user evaluations. Lastly, we
show that our adversarial patches can be applied in the physical world.


http://arxiv.org/abs/2107.02895
Bio-Inspired Adversarial Attack Against Deep Neural Networks. (98%)
Bowei Xi; Yujie Chen; Fan Fei; Zhan Tu; Xinyan Deng
  The paper develops a new adversarial attack against deep neural networks
(DNN), based on applying bio-inspired design to moving physical objects. To the
best of our knowledge, this is the first work to introduce physical attacks
with a moving object. Instead of following the dominating attack strategy in
the existing literature, i.e., to introduce minor perturbations to a digital
input or a stationary physical object, we show two new successful attack
strategies in this paper. We show by superimposing several patterns onto one
physical object, a DNN becomes confused and picks one of the patterns to assign
a class label. Our experiment with three flapping wing robots demonstrates the
possibility of developing an adversarial camouflage to cause a targeted mistake
by DNN. We also show certain motion can reduce the dependency among consecutive
frames in a video and make an object detector "blind", i.e., not able to detect
an object exists in the video. Hence in a successful physical attack against
DNN, targeted motion against the system should also be considered.


http://arxiv.org/abs/2106.15130
Do Not Deceive Your Employer with a Virtual Background: A Video Conferencing Manipulation-Detection System. (62%)
Mauro Conti; Simone Milani; Ehsan Nowroozi; Gabriele Orazi
  The last-generation video conferencing software allows users to utilize a
virtual background to conceal their personal environment due to privacy
concerns, especially in official meetings with other employers. On the other
hand, users maybe want to fool people in the meeting by considering the virtual
background to conceal where they are. In this case, developing tools to
understand the virtual background utilize for fooling people in meeting plays
an important role. Besides, such detectors must prove robust against different
kinds of attacks since a malicious user can fool the detector by applying a set
of adversarial editing steps on the video to conceal any revealing footprint.
In this paper, we study the feasibility of an efficient tool to detect whether
a videoconferencing user background is real. In particular, we provide the
first tool which computes pixel co-occurrences matrices and uses them to search
for inconsistencies among spectral and spatial bands. Our experiments confirm
that cross co-occurrences matrices improve the robustness of the detector
against different kinds of attacks. This work's performance is especially
noteworthy with regard to color SPAM features. Moreover, the performance
especially is significant with regard to robustness versus post-processing,
like geometric transformations, filtering, contrast enhancement, and JPEG
compression with different quality factors.


http://arxiv.org/abs/2106.15764
The Threat of Offensive AI to Organizations. (54%)
Yisroel Mirsky; Ambra Demontis; Jaidip Kotak; Ram Shankar; Deng Gelei; Liu Yang; Xiangyu Zhang; Wenke Lee; Yuval Elovici; Battista Biggio
  AI has provided us with the ability to automate tasks, extract information
from vast amounts of data, and synthesize media that is nearly
indistinguishable from the real thing. However, positive tools can also be used
for negative purposes. In particular, cyber adversaries can use AI (such as
machine learning) to enhance their attacks and expand their campaigns.
  Although offensive AI has been discussed in the past, there is a need to
analyze and understand the threat in the context of organizations. For example,
how does an AI-capable adversary impact the cyber kill chain? Does AI benefit
the attacker more than the defender? What are the most significant AI threats
facing organizations today and what will be their impact on the future?
  In this survey, we explore the threat of offensive AI on organizations.
First, we present the background and discuss how AI changes the adversary's
methods, strategies, goals, and overall attack model. Then, through a
literature review, we identify 33 offensive AI capabilities which adversaries
can use to enhance their attacks. Finally, through a user study spanning
industry and academia, we rank the AI threats and provide insights on the
adversaries.


http://arxiv.org/abs/2106.15776
Local Reweighting for Adversarial Training. (22%)
Ruize Gao; Feng Liu; Kaiwen Zhou; Gang Niu; Bo Han; James Cheng
  Instances-reweighted adversarial training (IRAT) can significantly boost the
robustness of trained models, where data being less/more vulnerable to the
given attack are assigned smaller/larger weights during training. However, when
tested on attacks different from the given attack simulated in training, the
robustness may drop significantly (e.g., even worse than no reweighting). In
this paper, we study this problem and propose our solution--locally reweighted
adversarial training (LRAT). The rationale behind IRAT is that we do not need
to pay much attention to an instance that is already safe under the attack. We
argue that the safeness should be attack-dependent, so that for the same
instance, its weight can change given different attacks based on the same
model. Thus, if the attack simulated in training is mis-specified, the weights
of IRAT are misleading. To this end, LRAT pairs each instance with its
adversarial variants and performs local reweighting inside each pair, while
performing no global reweighting--the rationale is to fit the instance itself
if it is immune to the attack, but not to skip the pair, in order to passively
defend different attacks in future. Experiments show that LRAT works better
than both IRAT (i.e., global reweighting) and the standard AT (i.e., no
reweighting) when trained with an attack and tested on different attacks.


http://arxiv.org/abs/2106.15355
On the Interaction of Belief Bias and Explanations. (15%)
Ana Valeria Gonzalez; Anna Rogers; Anders Søgaard
  A myriad of explainability methods have been proposed in recent years, but
there is little consensus on how to evaluate them. While automatic metrics
allow for quick benchmarking, it isn't clear how such metrics reflect human
interaction with explanations. Human evaluation is of paramount importance, but
previous protocols fail to account for belief biases affecting human
performance, which may lead to misleading conclusions. We provide an overview
of belief bias, its role in human evaluation, and ideas for NLP practitioners
on how to account for it. For two experimental paradigms, we present a case
study of gradient-based explainability introducing simple ways to account for
humans' prior beliefs: models of varying quality and adversarial examples. We
show that conclusions about the highest performing methods change when
introducing such controls, pointing to the importance of accounting for belief
bias in evaluation.


http://arxiv.org/abs/2106.14815
Feature Importance Guided Attack: A Model Agnostic Adversarial Attack. (99%)
Gilad Gressel; Niranjan Hegde; Archana Sreekumar; Michael Darling
  Machine learning models are susceptible to adversarial attacks which
dramatically reduce their performance. Reliable defenses to these attacks are
an unsolved challenge. In this work, we present a novel evasion attack: the
'Feature Importance Guided Attack' (FIGA) which generates adversarial evasion
samples. FIGA is model agnostic, it assumes no prior knowledge of the defending
model's learning algorithm, but does assume knowledge of the feature
representation. FIGA leverages feature importance rankings; it perturbs the
most important features of the input in the direction of the target class we
wish to mimic. We demonstrate FIGA against eight phishing detection models. We
keep the attack realistic by perturbing phishing website features that an
adversary would have control over. Using FIGA we are able to cause a reduction
in the F1-score of a phishing detection model from 0.96 to 0.41 on average.
Finally, we implement adversarial training as a defense against FIGA and show
that while it is sometimes effective, it can be evaded by changing the
parameters of FIGA.


http://arxiv.org/abs/2106.15023
Evading Adversarial Example Detection Defenses with Orthogonal Projected Gradient Descent. (99%)
Oliver Bryniarski; Nabeel Hingun; Pedro Pachuca; Vincent Wang; Nicholas Carlini
  Evading adversarial example detection defenses requires finding adversarial
examples that must simultaneously (a) be misclassified by the model and (b) be
detected as non-adversarial. We find that existing attacks that attempt to
satisfy multiple simultaneous constraints often over-optimize against one
constraint at the cost of satisfying another. We introduce Orthogonal Projected
Gradient Descent, an improved attack technique to generate adversarial examples
that avoids this problem by orthogonalizing the gradients when running standard
gradient-based attacks. We use our technique to evade four state-of-the-art
detection defenses, reducing their accuracy to 0% while maintaining a 0%
detection rate.


http://arxiv.org/abs/2106.15058
Improving Transferability of Adversarial Patches on Face Recognition with Generative Models. (99%)
Zihao Xiao; Xianfeng Gao; Chilin Fu; Yinpeng Dong; Wei Gao; Xiaolu Zhang; Jun Zhou; Jun Zhu
  Face recognition is greatly improved by deep convolutional neural networks
(CNNs). Recently, these face recognition models have been used for identity
authentication in security sensitive applications. However, deep CNNs are
vulnerable to adversarial patches, which are physically realizable and
stealthy, raising new security concerns on the real-world applications of these
models. In this paper, we evaluate the robustness of face recognition models
using adversarial patches based on transferability, where the attacker has
limited accessibility to the target models. First, we extend the existing
transfer-based attack techniques to generate transferable adversarial patches.
However, we observe that the transferability is sensitive to initialization and
degrades when the perturbation magnitude is large, indicating the overfitting
to the substitute models. Second, we propose to regularize the adversarial
patches on the low dimensional data manifold. The manifold is represented by
generative models pre-trained on legitimate human face images. Using face-like
features as adversarial perturbations through optimization on the manifold, we
show that the gaps between the responses of substitute models and the target
models dramatically decrease, exhibiting a better transferability. Extensive
digital world experiments are conducted to demonstrate the superiority of the
proposed method in the black-box setting. We apply the proposed method in the
physical world as well.


http://arxiv.org/abs/2106.14851
Data Poisoning Won't Save You From Facial Recognition. (93%)
Evani Radiya-Dixit; Florian Tramèr
  Data poisoning has been proposed as a compelling defense against facial
recognition models trained on Web-scraped pictures. By perturbing the images
they post online, users can fool models into misclassifying future
(unperturbed) pictures. We demonstrate that this strategy provides a false
sense of security, as it ignores an inherent asymmetry between the parties:
users' pictures are perturbed once and for all before being published (at which
point they are scraped) and must thereafter fool all future models -- including
models trained adaptively against the users' past attacks, or models that use
technologies discovered after the attack. We evaluate two systems for poisoning
attacks against large-scale facial recognition, Fawkes (500,000+ downloads) and
LowKey. We demonstrate how an "oblivious" model trainer can simply wait for
future developments in computer vision to nullify the protection of pictures
collected in the past. We further show that an adversary with black-box access
to the attack can (i) train a robust model that resists the perturbations of
collected pictures and (ii) detect poisoned pictures uploaded online. We
caution that facial recognition poisoning will not admit an "arms race" between
attackers and defenders. Once perturbed pictures are scraped, the attack cannot
be changed so any future successful defense irrevocably undermines users'
privacy.


http://arxiv.org/abs/2106.14952
Adversarial Robustness of Streaming Algorithms through Importance Sampling. (61%)
Vladimir Braverman; Avinatan Hassidim; Yossi Matias; Mariano Schain; Sandeep Silwal; Samson Zhou
  In this paper, we introduce adversarially robust streaming algorithms for
central machine learning and algorithmic tasks, such as regression and
clustering, as well as their more general counterparts, subspace embedding,
low-rank approximation, and coreset construction. For regression and other
numerical linear algebra related tasks, we consider the row arrival streaming
model. Our results are based on a simple, but powerful, observation that many
importance sampling-based algorithms give rise to adversarial robustness which
is in contrast to sketching based algorithms, which are very prevalent in the
streaming literature but suffer from adversarial attacks. In addition, we show
that the well-known merge and reduce paradigm in streaming is adversarially
robust. Since the merge and reduce paradigm allows coreset constructions in the
streaming setting, we thus obtain robust algorithms for $k$-means, $k$-median,
$k$-center, Bregman clustering, projective clustering, principal component
analysis (PCA) and non-negative matrix factorization. To the best of our
knowledge, these are the first adversarially robust results for these problems
yet require no new algorithmic implementations. Finally, we empirically confirm
the robustness of our algorithms on various adversarial attacks and demonstrate
that by contrast, some common existing algorithms are not robust.
  (Abstract shortened to meet arXiv limits)


http://arxiv.org/abs/2106.14999
Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation. (2%)
Chaithanya Kumar Mummadi; Robin Hutmacher; Kilian Rambach; Evgeny Levinkov; Thomas Brox; Jan Hendrik Metzen
  Deep neural networks often exhibit poor performance on data that is unlikely
under the train-time data distribution, for instance data affected by
corruptions. Previous works demonstrate that test-time adaptation to data
shift, for instance using entropy minimization, effectively improves
performance on such shifted distributions. This paper focuses on the fully
test-time adaptation setting, where only unlabeled data from the target
distribution is required. This allows adapting arbitrary pretrained networks.
Specifically, we propose a novel loss that improves test-time adaptation by
addressing both premature convergence and instability of entropy minimization.
This is achieved by replacing the entropy by a non-saturating surrogate and
adding a diversity regularizer based on batch-wise entropy maximization that
prevents convergence to trivial collapsed solutions. Moreover, we propose to
prepend an input transformation module to the network that can partially undo
test-time distribution shifts. Surprisingly, this preprocessing can be learned
solely using the fully test-time adaptation loss in an end-to-end fashion
without any target domain labels or source domain data. We show that our
approach outperforms previous work in improving the robustness of publicly
available pretrained image classifiers to common corruptions on such
challenging benchmarks as ImageNet-C.


http://arxiv.org/abs/2106.14707
Realtime Robust Malicious Traffic Detection via Frequency Domain Analysis. (1%)
Chuanpu Fu; Qi Li; Meng Shen; Ke Xu
  Machine learning (ML) based malicious traffic detection is an emerging
security paradigm, particularly for zero-day attack detection, which is
complementary to existing rule based detection. However, the existing ML based
detection has low detection accuracy and low throughput incurred by inefficient
traffic features extraction. Thus, they cannot detect attacks in realtime
especially in high throughput networks. Particularly, these detection systems
similar to the existing rule based detection can be easily evaded by
sophisticated attacks. To this end, we propose Whisper, a realtime ML based
malicious traffic detection system that achieves both high accuracy and high
throughput by utilizing frequency domain features. It utilizes sequential
features represented by the frequency domain features to achieve bounded
information loss, which ensures high detection accuracy, and meanwhile
constrains the scale of features to achieve high detection throughput.
Particularly, attackers cannot easily interfere with the frequency domain
features and thus Whisper is robust against various evasion attacks. Our
experiments with 42 types of attacks demonstrate that, compared with the
state-of-theart systems, Whisper can accurately detect various sophisticated
and stealthy attacks, achieving at most 18.36% improvement, while achieving two
orders of magnitude throughput. Even under various evasion attacks, Whisper is
still able to maintain around 90% detection accuracy.


http://arxiv.org/abs/2107.02840
RAILS: A Robust Adversarial Immune-inspired Learning System. (96%)
Ren Wang; Tianqi Chen; Stephen Lindsly; Cooper Stansbury; Alnawaz Rehemtulla; Indika Rajapakse; Alfred Hero
  Adversarial attacks against deep neural networks (DNNs) are continuously
evolving, requiring increasingly powerful defense strategies. We develop a
novel adversarial defense framework inspired by the adaptive immune system: the
Robust Adversarial Immune-inspired Learning System (RAILS). Initializing a
population of exemplars that is balanced across classes, RAILS starts from a
uniform label distribution that encourages diversity and debiases a potentially
corrupted initial condition. RAILS implements an evolutionary optimization
process to adjust the label distribution and achieve specificity towards ground
truth. RAILS displays a tradeoff between robustness (diversity) and accuracy
(specificity), providing a new immune-inspired perspective on adversarial
learning. We empirically validate the benefits of RAILS through several
adversarial image classification experiments on MNIST, SVHN, and CIFAR-10
datasets. For the PGD attack, RAILS is found to improve the robustness over
existing methods by >= 5.62%, 12.5% and 10.32%, respectively, without
appreciable loss of standard accuracy.


http://arxiv.org/abs/2106.14152
Who is Responsible for Adversarial Defense? (93%)
Kishor Datta Gupta; Dipankar Dasgupta
  We have seen a surge in research aims toward adversarial attacks and defenses
in AI/ML systems. While it is crucial to formulate new attack methods and
devise novel defense strategies for robustness, it is also imperative to
recognize who is responsible for implementing, validating, and justifying the
necessity of these defenses. In particular, which components of the system are
vulnerable to what type of adversarial attacks, and the expertise needed to
realize the severity of adversarial attacks. Also how to evaluate and address
the adversarial challenges in order to recommend defense strategies for
different applications. This paper opened a discussion on who should examine
and implement the adversarial defenses and the reason behind such efforts.


http://arxiv.org/abs/2106.14300
ASK: Adversarial Soft k-Nearest Neighbor Attack and Defense. (82%)
Ren Wang; Tianqi Chen; Philip Yao; Sijia Liu; Indika Rajapakse; Alfred Hero
  K-Nearest Neighbor (kNN)-based deep learning methods have been applied to
many applications due to their simplicity and geometric interpretability.
However, the robustness of kNN-based classification models has not been
thoroughly explored and kNN attack strategies are underdeveloped. In this
paper, we propose an Adversarial Soft kNN (ASK) loss to both design more
effective kNN attack strategies and to develop better defenses against them.
Our ASK loss approach has two advantages. First, ASK loss can better
approximate the kNN's probability of classification error than objectives
proposed in previous works. Second, the ASK loss is interpretable: it preserves
the mutual information between the perturbed input and the kNN of the
unperturbed input. We use the ASK loss to generate a novel attack method called
the ASK-Attack (ASK-Atk), which shows superior attack efficiency and accuracy
degradation relative to previous kNN attacks. Based on the ASK-Atk, we then
derive an ASK-Defense (ASK-Def) method that optimizes the worst-case training
loss induced by ASK-Atk.


http://arxiv.org/abs/2107.02842
Immuno-mimetic Deep Neural Networks (Immuno-Net). (64%)
Ren Wang; Tianqi Chen; Stephen Lindsly; Cooper Stansbury; Indika Rajapakse; Alfred Hero
  Biomimetics has played a key role in the evolution of artificial neural
networks. Thus far, in silico metaphors have been dominated by concepts from
neuroscience and cognitive psychology. In this paper we introduce a different
type of biomimetic model, one that borrows concepts from the immune system, for
designing robust deep neural networks. This immuno-mimetic model leads to a new
computational biology framework for robustification of deep neural networks
against adversarial attacks. Within this Immuno-Net framework we define a
robust adaptive immune-inspired learning system (Immuno-Net RAILS) that
emulates, in silico, the adaptive biological mechanisms of B-cells that are
used to defend a mammalian host against pathogenic attacks. When applied to
image classification tasks on benchmark datasets, we demonstrate that
Immuno-net RAILS results in improvement of as much as 12.5% in adversarial
accuracy of a baseline method, the DkNN-robustified CNN, without appreciable
loss of accuracy on clean data.


http://arxiv.org/abs/2106.14342
Stabilizing Equilibrium Models by Jacobian Regularization. (1%)
Shaojie Bai; Vladlen Koltun; J. Zico Kolter
  Deep equilibrium networks (DEQs) are a new class of models that eschews
traditional depth in favor of finding the fixed point of a single nonlinear
layer. These models have been shown to achieve performance competitive with the
state-of-the-art deep networks while using significantly less memory. Yet they
are also slower, brittle to architectural choices, and introduce potential
instability to the model. In this paper, we propose a regularization scheme for
DEQ models that explicitly regularizes the Jacobian of the fixed-point update
equations to stabilize the learning of equilibrium models. We show that this
regularization adds only minimal computational cost, significantly stabilizes
the fixed-point convergence in both forward and backward passes, and scales
well to high-dimensional, realistic domains (e.g., WikiText-103 language
modeling and ImageNet classification). Using this method, we demonstrate, for
the first time, an implicit-depth model that runs with approximately the same
speed and level of performance as popular conventional deep networks such as
ResNet-101, while still maintaining the constant memory footprint and
architectural simplicity of DEQs. Code is available at
https://github.com/locuslab/deq .


http://arxiv.org/abs/2106.15357
Multi-stage Optimization based Adversarial Training. (99%)
Xiaosen Wang; Chuanbiao Song; Liwei Wang; Kun He
  In the field of adversarial robustness, there is a common practice that
adopts the single-step adversarial training for quickly developing
adversarially robust models. However, the single-step adversarial training is
most likely to cause catastrophic overfitting, as after a few training epochs
it will be hard to generate strong adversarial examples to continuously boost
the adversarial robustness. In this work, we aim to avoid the catastrophic
overfitting by introducing multi-step adversarial examples during the
single-step adversarial training. Then, to balance the large training overhead
of generating multi-step adversarial examples, we propose a Multi-stage
Optimization based Adversarial Training (MOAT) method that periodically trains
the model on mixed benign examples, single-step adversarial examples, and
multi-step adversarial examples stage by stage. In this way, the overall
training overhead is reduced significantly, meanwhile, the model could avoid
catastrophic overfitting. Extensive experiments on CIFAR-10 and CIFAR-100
datasets demonstrate that under similar amount of training overhead, the
proposed MOAT exhibits better robustness than either single-step or multi-step
adversarial training methods.


http://arxiv.org/abs/2106.13997
The Feasibility and Inevitability of Stealth Attacks. (68%)
Ivan Y. Tyukin; Desmond J. Higham; Eliyas Woldegeorgis; Alexander N. Gorban
  We develop and study new adversarial perturbations that enable an attacker to
gain control over decisions in generic Artificial Intelligence (AI) systems
including deep learning neural networks. In contrast to adversarial data
modification, the attack mechanism we consider here involves alterations to the
AI system itself. Such a stealth attack could be conducted by a mischievous,
corrupt or disgruntled member of a software development team. It could also be
made by those wishing to exploit a "democratization of AI" agenda, where
network architectures and trained parameter sets are shared publicly. Building
on work by [Tyukin et al., International Joint Conference on Neural Networks,
2020], we develop a range of new implementable attack strategies with
accompanying analysis, showing that with high probability a stealth attack can
be made transparent, in the sense that system performance is unchanged on a
fixed validation set which is unknown to the attacker, while evoking any
desired output on a trigger input of interest. The attacker only needs to have
estimates of the size of the validation set and the spread of the AI's relevant
latent space. In the case of deep learning neural networks, we show that a one
neuron attack is possible - a modification to the weights and bias associated
with a single neuron - revealing a vulnerability arising from
over-parameterization. We illustrate these concepts in a realistic setting.
Guided by the theory and computational results, we also propose strategies to
guard against stealth attacks.


http://arxiv.org/abs/2106.13326
On the (Un-)Avoidability of Adversarial Examples. (99%)
Sadia Chowdhury; Ruth Urner
  The phenomenon of adversarial examples in deep learning models has caused
substantial concern over their reliability. While many deep neural networks
have shown impressive performance in terms of predictive accuracy, it has been
shown that in many instances an imperceptible perturbation can falsely flip the
network's prediction. Most research has then focused on developing defenses
against adversarial attacks or learning under a worst-case adversarial loss. In
this work, we take a step back and aim to provide a framework for determining
whether a model's label change under small perturbation is justified (and when
it is not). We carefully argue that adversarial robustness should be defined as
a locally adaptive measure complying with the underlying distribution. We then
suggest a definition for an adaptive robust loss, derive an empirical version
of it, and develop a resulting data-augmentation framework. We prove that our
adaptive data-augmentation maintains consistency of 1-nearest neighbor
classification under deterministic labels and provide illustrative empirical
evaluations.


http://arxiv.org/abs/2106.13394
Countering Adversarial Examples: Combining Input Transformation and Noisy Training. (99%)
Cheng Zhang; Pan Gao
  Recent studies have shown that neural network (NN) based image classifiers
are highly vulnerable to adversarial examples, which poses a threat to
security-sensitive image recognition task. Prior work has shown that JPEG
compression can combat the drop in classification accuracy on adversarial
examples to some extent. But, as the compression ratio increases, traditional
JPEG compression is insufficient to defend those attacks but can cause an
abrupt accuracy decline to the benign images. In this paper, with the aim of
fully filtering the adversarial perturbations, we firstly make modifications to
traditional JPEG compression algorithm which becomes more favorable for NN.
Specifically, based on an analysis of the frequency coefficient, we design a
NN-favored quantization table for compression. Considering compression as a
data augmentation strategy, we then combine our model-agnostic preprocess with
noisy training. We fine-tune the pre-trained model by training with images
encoded at different compression levels, thus generating multiple classifiers.
Finally, since lower (higher) compression ratio can remove both perturbations
and original features slightly (aggressively), we use these trained multiple
models for model ensemble. The majority vote of the ensemble of models is
adopted as final predictions. Experiments results show our method can improve
defense efficiency while maintaining original accuracy.


http://arxiv.org/abs/2106.12611
Adversarial Examples in Multi-Layer Random ReLU Networks. (81%)
Peter L. Bartlett; Sébastien Bubeck; Yeshwanth Cherapanamjeri
  We consider the phenomenon of adversarial examples in ReLU networks with
independent gaussian parameters. For networks of constant depth and with a
large range of widths (for instance, it suffices if the width of each layer is
polynomial in that of any other layer), small perturbations of input vectors
lead to large changes of outputs. This generalizes results of Daniely and
Schacham (2020) for networks of rapidly decreasing width and of Bubeck et al
(2021) for two-layer networks. The proof shows that adversarial examples arise
in these networks because the functions that they compute are very close to
linear. Bottleneck layers in the network play a key role: the minimal width up
to some point in the network determines scales and sensitivities of mappings
computed up to that point. The main result is for networks with constant depth,
but we also show that some constraint on depth is necessary for a result of
this kind, because there are suitably deep networks that, with constant
probability, compute a function that is close to constant.


http://arxiv.org/abs/2106.12478
Teacher Model Fingerprinting Attacks Against Transfer Learning. (2%)
Yufei Chen; Chao Shen; Cong Wang; Yang Zhang
  Transfer learning has become a common solution to address training data
scarcity in practice. It trains a specified student model by reusing or
fine-tuning early layers of a well-trained teacher model that is usually
publicly available. However, besides utility improvement, the transferred
public knowledge also brings potential threats to model confidentiality, and
even further raises other security and privacy issues.
  In this paper, we present the first comprehensive investigation of the
teacher model exposure threat in the transfer learning context, aiming to gain
a deeper insight into the tension between public knowledge and model
confidentiality. To this end, we propose a teacher model fingerprinting attack
to infer the origin of a student model, i.e., the teacher model it transfers
from. Specifically, we propose a novel optimization-based method to carefully
generate queries to probe the student model to realize our attack. Unlike
existing model reverse engineering approaches, our proposed fingerprinting
method neither relies on fine-grained model outputs, e.g., posteriors, nor
auxiliary information of the model architecture or training dataset. We
systematically evaluate the effectiveness of our proposed attack. The empirical
results demonstrate that our attack can accurately identify the model origin
with few probing queries. Moreover, we show that the proposed attack can serve
as a stepping stone to facilitating other attacks against machine learning
models, such as model stealing.


http://arxiv.org/abs/2106.12723
Meaningfully Explaining Model Mistakes Using Conceptual Counterfactuals. (1%)
Abubakar Abid; Mert Yuksekgonul; James Zou
  Understanding and explaining the mistakes made by trained models is critical
to many machine learning objectives, such as improving robustness, addressing
concept drift, and mitigating biases. However, this is often an ad hoc process
that involves manually looking at the model's mistakes on many test samples and
guessing at the underlying reasons for those incorrect predictions. In this
paper, we propose a systematic approach, conceptual counterfactual
explanations(CCE), that explains why a classifier makes a mistake on a
particular test sample(s) in terms of human-understandable concepts (e.g. this
zebra is misclassified as a dog because of faint stripes). We base CCE on two
prior ideas: counterfactual explanations and concept activation vectors, and
validate our approach on well-known pretrained models, showing that it explains
the models' mistakes meaningfully. In addition, for new models trained on data
with spurious correlations, CCE accurately identifies the spurious correlation
as the cause of model mistakes from a single misclassified test sample. On two
challenging medical applications, CCE generated useful insights, confirmed by
clinicians, into biases and mistakes the model makes in real-world settings.
The code for CCE is publicly available and can easily be applied to explain
mistakes in new models.


http://arxiv.org/abs/2106.12563
Feature Attributions and Counterfactual Explanations Can Be Manipulated. (1%)
Dylan Slack; Sophie Hilgard; Sameer Singh; Himabindu Lakkaraju
  As machine learning models are increasingly used in critical decision-making
settings (e.g., healthcare, finance), there has been a growing emphasis on
developing methods to explain model predictions. Such \textit{explanations} are
used to understand and establish trust in models and are vital components in
machine learning pipelines. Though explanations are a critical piece in these
systems, there is little understanding about how they are vulnerable to
manipulation by adversaries. In this paper, we discuss how two broad classes of
explanations are vulnerable to manipulation. We demonstrate how adversaries can
design biased models that manipulate model agnostic feature attribution methods
(e.g., LIME \& SHAP) and counterfactual explanations that hill-climb during the
counterfactual search (e.g., Wachter's Algorithm \& DiCE) into
\textit{concealing} the model's biases. These vulnerabilities allow an
adversary to deploy a biased model, yet explanations will not reveal this bias,
thereby deceiving stakeholders into trusting the model. We evaluate the
manipulations on real world data sets, including COMPAS and Communities \&
Crime, and find explanations can be manipulated in practice.


http://arxiv.org/abs/2106.12021
DetectX -- Adversarial Input Detection using Current Signatures in Memristive XBar Arrays. (99%)
Abhishek Moitra; Priyadarshini Panda
  Adversarial input detection has emerged as a prominent technique to harden
Deep Neural Networks(DNNs) against adversarial attacks. Most prior works use
neural network-based detectors or complex statistical analysis for adversarial
detection. These approaches are computationally intensive and vulnerable to
adversarial attacks. To this end, we propose DetectX - a hardware friendly
adversarial detection mechanism using hardware signatures like Sum of column
Currents (SoI) in memristive crossbars (XBar). We show that adversarial inputs
have higher SoI compared to clean inputs. However, the difference is too small
for reliable adversarial detection. Hence, we propose a dual-phase training
methodology: Phase1 training is geared towards increasing the separation
between clean and adversarial SoIs; Phase2 training improves the overall
robustness against different strengths of adversarial attacks. For
hardware-based adversarial detection, we implement the DetectX module using
32nm CMOS circuits and integrate it with a Neurosim-like analog crossbar
architecture. We perform hardware evaluation of the Neurosim+DetectX system on
the Neurosim platform using datasets-CIFAR10(VGG8), CIFAR100(VGG16) and
TinyImagenet(ResNet18). Our experiments show that DetectX is 10x-25x more
energy efficient and immune to dynamic adversarial attacks compared to previous
state-of-the-art works. Moreover, we achieve high detection performance
(ROC-AUC > 0.95) for strong white-box and black-box attacks. The code has been
released at https://github.com/Intelligent-Computing-Lab-Yale/DetectX


http://arxiv.org/abs/2106.11644
Self-Supervised Iterative Contextual Smoothing for Efficient Adversarial Defense against Gray- and Black-Box Attack. (99%)
Sungmin Cha; Naeun Ko; Youngjoon Yoo; Taesup Moon
  We propose a novel and effective input transformation based adversarial
defense method against gray- and black-box attack, which is computationally
efficient and does not require any adversarial training or retraining of a
classification model. We first show that a very simple iterative Gaussian
smoothing can effectively wash out adversarial noise and achieve substantially
high robust accuracy. Based on the observation, we propose Self-Supervised
Iterative Contextual Smoothing (SSICS), which aims to reconstruct the original
discriminative features from the Gaussian-smoothed image in context-adaptive
manner, while still smoothing out the adversarial noise. From the experiments
on ImageNet, we show that our SSICS achieves both high standard accuracy and
very competitive robust accuracy for the gray- and black-box attacks; e.g.,
transfer-based PGD-attack and score-based attack. A note-worthy point to stress
is that our defense is free of computationally expensive adversarial training,
yet, can approach its robust accuracy via input transformation.


http://arxiv.org/abs/2106.12900
Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classification Tasks. (83%)
Fan Liu; Shuyu Zhao; Xuelong Dai; Bin Xiao
  Meta-learning model can quickly adapt to new tasks using few-shot labeled
data. However, despite achieving good generalization on few-shot classification
tasks, it is still challenging to improve the adversarial robustness of the
meta-learning model in few-shot learning. Although adversarial training (AT)
methods such as Adversarial Query (AQ) can improve the adversarially robust
performance of meta-learning models, AT is still computationally expensive
training. On the other hand, meta-learning models trained with AT will drop
significant accuracy on the original clean images. This paper proposed a
meta-learning method on the adversarially robust neural network called
Long-term Cross Adversarial Training (LCAT). LCAT will update meta-learning
model parameters cross along the natural and adversarial sample distribution
direction with long-term to improve both adversarial and clean few-shot
classification accuracy. Due to cross-adversarial training, LCAT only needs
half of the adversarial training epoch than AQ, resulting in a low adversarial
training computation. Experiment results show that LCAT achieves superior
performance both on the clean and adversarial few-shot classification accuracy
than SOTA adversarial training methods for meta-learning models.


http://arxiv.org/abs/2106.11629
On Adversarial Robustness of Synthetic Code Generation. (81%)
Mrinal Anand; Pratik Kayal; Mayank Singh
  Automatic code synthesis from natural language descriptions is a challenging
task. We witness massive progress in developing code generation systems for
domain-specific languages (DSLs) employing sequence-to-sequence deep learning
techniques in the recent past. In this paper, we specifically experiment with
\textsc{AlgoLisp} DSL-based generative models and showcase the existence of
significant dataset bias through different classes of adversarial examples. We
also experiment with two variants of Transformer-based models that outperform
all existing \textsc{AlgoLisp} DSL-based code generation baselines. Consistent
with the current state-of-the-art systems, our proposed models, too, achieve
poor performance under adversarial settings. Therefore, we propose several
dataset augmentation techniques to reduce bias and showcase their efficacy
using robust experimentation.


http://arxiv.org/abs/2106.11865
NetFense: Adversarial Defenses against Privacy Attacks on Neural Networks for Graph Data. (67%)
I-Chung Hsieh; Cheng-Te Li
  Recent advances in protecting node privacy on graph data and attacking graph
neural networks (GNNs) gain much attention. The eye does not bring these two
essential tasks together yet. Imagine an adversary can utilize the powerful
GNNs to infer users' private labels in a social network. How can we
adversarially defend against such privacy attacks while maintaining the utility
of perturbed graphs? In this work, we propose a novel research task,
adversarial defenses against GNN-based privacy attacks, and present a graph
perturbation-based approach, NetFense, to achieve the goal. NetFense can
simultaneously keep graph data unnoticeability (i.e., having limited changes on
the graph structure), maintain the prediction confidence of targeted label
classification (i.e., preserving data utility), and reduce the prediction
confidence of private label classification (i.e., protecting the privacy of
nodes). Experiments conducted on single- and multiple-target perturbations
using three real graph data exhibit that the perturbed graphs by NetFense can
effectively maintain data utility (i.e., model unnoticeability) on targeted
label classification and significantly decrease the prediction confidence of
private label classification (i.e., privacy protection). Extensive studies also
bring several insights, such as the flexibility of NetFense, preserving local
neighborhoods in data unnoticeability, and better privacy protection for
high-degree nodes.


http://arxiv.org/abs/2106.11420
Policy Smoothing for Provably Robust Reinforcement Learning. (99%)
Aounon Kumar; Alexander Levine; Soheil Feizi
  The study of provable adversarial robustness for deep neural network (DNN)
models has mainly focused on static supervised learning tasks such as image
classification. However, DNNs have been used extensively in real-world adaptive
tasks such as reinforcement learning (RL), making RL systems vulnerable to
adversarial attacks. The key challenge in adversarial RL is that the attacker
can adapt itself to the defense strategy used by the agent in previous
time-steps to strengthen its attack in future steps. In this work, we study the
provable robustness of RL against norm-bounded adversarial perturbations of the
inputs. We focus on smoothing-based provable defenses and propose policy
smoothing where the agent adds a Gaussian noise to its observation at each
time-step before applying the policy network to make itself less sensitive to
adversarial perturbations of its inputs. Our main theoretical contribution is
to prove an adaptive version of the Neyman-Pearson Lemma where the adversarial
perturbation at a particular time can be a stochastic function of current and
previous observations and states as well as previously observed actions. Using
this lemma, we adapt the robustness certificates produced by randomized
smoothing in the static setting of image classification to the dynamic setting
of RL. We generate certificates that guarantee that the total reward obtained
by the smoothed policy will not fall below a certain threshold under a
norm-bounded adversarial perturbation of the input. We show that our
certificates are tight by constructing a worst-case setting that achieves the
bounds derived in our analysis. In our experiments, we show that this method
can yield meaningful certificates in complex environments demonstrating its
effectiveness against adversarial attacks.


http://arxiv.org/abs/2106.10996
Delving into the pixels of adversarial samples. (98%)
Blerta Lindqvist
  Despite extensive research into adversarial attacks, we do not know how
adversarial attacks affect image pixels. Knowing how image pixels are affected
by adversarial attacks has the potential to lead us to better adversarial
defenses. Motivated by instances that we find where strong attacks do not
transfer, we delve into adversarial examples at pixel level to scrutinize how
adversarial attacks affect image pixel values. We consider several ImageNet
architectures, InceptionV3, VGG19 and ResNet50, as well as several strong
attacks. We find that attacks can have different effects at pixel level
depending on classifier architecture. In particular, input pre-processing plays
a previously overlooked role in the effect that attacks have on pixels. Based
on the insights of pixel-level examination, we find new ways to detect some of
the strongest current attacks.


http://arxiv.org/abs/2106.11424
Hardness of Samples Is All You Need: Protecting Deep Learning Models Using Hardness of Samples. (98%)
Amir Mahdi Sadeghzadeh; Faezeh Dehghan; Amir Mohammad Sobhanian; Rasool Jalili
  Several recent studies have shown that Deep Neural Network (DNN)-based
classifiers are vulnerable against model extraction attacks. In model
extraction attacks, an adversary exploits the target classifier to create a
surrogate classifier imitating the target classifier with respect to some
criteria. In this paper, we investigate the hardness degree of samples and
demonstrate that the hardness degree histogram of model extraction attacks
samples is distinguishable from the hardness degree histogram of normal
samples. Normal samples come from the target classifier's training data
distribution. As the training process of DNN-based classifiers is done in
several epochs, we can consider this process as a sequence of subclassifiers so
that each subclassifier is created at the end of an epoch. We use the sequence
of subclassifiers to calculate the hardness degree of samples. We investigate
the relation between hardness degree of samples and the trust in the classifier
outputs. We propose Hardness-Oriented Detection Approach (HODA) to detect the
sample sequences of model extraction attacks. The results demonstrate that HODA
can detect the sample sequences of model extraction attacks with a high success
rate by only watching 100 attack samples. We also investigate the hardness
degree of adversarial examples and indicate that the hardness degree histogram
of adversarial examples is distinct from the hardness degree histogram of
normal samples.


http://arxiv.org/abs/2106.10974
Friendly Training: Neural Networks Can Adapt Data To Make Learning Easier. (91%)
Simone Marullo; Matteo Tiezzi; Marco Gori; Stefano Melacci
  In the last decade, motivated by the success of Deep Learning, the scientific
community proposed several approaches to make the learning procedure of Neural
Networks more effective. When focussing on the way in which the training data
are provided to the learning machine, we can distinguish between the classic
random selection of stochastic gradient-based optimization and more involved
techniques that devise curricula to organize data, and progressively increase
the complexity of the training set. In this paper, we propose a novel training
procedure named Friendly Training that, differently from the aforementioned
approaches, involves altering the training examples in order to help the model
to better fulfil its learning criterion. The model is allowed to simplify those
examples that are too hard to be classified at a certain stage of the training
procedure. The data transformation is controlled by a developmental plan that
progressively reduces its impact during training, until it completely vanishes.
In a sense, this is the opposite of what is commonly done in order to increase
robustness against adversarial examples, i.e., Adversarial Training.
Experiments on multiple datasets are provided, showing that Friendly Training
yields improvements with respect to informed data sub-selection routines and
random selection, especially in deep convolutional architectures. Results
suggest that adapting the input data is a feasible way to stabilize learning
and improve the generalization skills of the network.


http://arxiv.org/abs/2106.11384
Membership Inference on Word Embedding and Beyond. (38%)
Saeed Mahloujifar; Huseyin A. Inan; Melissa Chase; Esha Ghosh; Marcello Hasegawa
  In the text processing context, most ML models are built on word embeddings.
These embeddings are themselves trained on some datasets, potentially
containing sensitive data. In some cases this training is done independently,
in other cases, it occurs as part of training a larger, task-specific model. In
either case, it is of interest to consider membership inference attacks based
on the embedding layer as a way of understanding sensitive information leakage.
But, somewhat surprisingly, membership inference attacks on word embeddings and
their effect in other natural language processing (NLP) tasks that use these
embeddings, have remained relatively unexplored.
  In this work, we show that word embeddings are vulnerable to black-box
membership inference attacks under realistic assumptions. Furthermore, we show
that this leakage persists through two other major NLP applications:
classification and text-generation, even when the embedding layer is not
exposed to the attacker. We show that our MI attack achieves high attack
accuracy against a classifier model and an LSTM-based language model. Indeed,
our attack is a cheaper membership inference attack on text-generative models,
which does not require the knowledge of the target model or any expensive
training of text-generative models as shadow models.


http://arxiv.org/abs/2106.11478
An Alternative Auxiliary Task for Enhancing Image Classification. (11%)
Chen Liu
  Image reconstruction is likely the most predominant auxiliary task for image
classification. In this paper, we investigate ``estimating the Fourier
Transform of the input image" as a potential alternative auxiliary task, in the
hope that it may further boost the performances on the primary task or
introduce novel constraints not well covered by image reconstruction. We
experimented with five popular classification architectures on the CIFAR-10
dataset, and the empirical results indicated that our proposed auxiliary task
generally improves the classification accuracy. More notably, the results
showed that in certain cases our proposed auxiliary task may enhance the
classifiers' resistance to adversarial attacks generated using the fast
gradient sign method.


http://arxiv.org/abs/2106.14647
Zero-shot learning approach to adaptive Cybersecurity using Explainable AI. (1%)
Dattaraj Rao; Shraddha Mane
  Cybersecurity is a domain where there is constant change in patterns of
attack, and we need ways to make our Cybersecurity systems more adaptive to
handle new attacks and categorize for appropriate action. We present a novel
approach to handle the alarm flooding problem faced by Cybersecurity systems
like security information and event management (SIEM) and intrusion detection
(IDS). We apply a zero-shot learning method to machine learning (ML) by
leveraging explanations for predictions of anomalies generated by a ML model.
This approach has huge potential to auto detect alarm labels generated in SIEM
and associate them with specific attack types. In this approach, without any
prior knowledge of attack, we try to identify it, decipher the features that
contribute to classification and try to bucketize the attack in a specific
category - using explainable AI. Explanations give us measurable factors as to
what features influence the prediction of a cyber-attack and to what degree.
These explanations generated based on game-theory are used to allocate credit
to specific features based on their influence on a specific prediction. Using
this allocation of credit, we propose a novel zero-shot approach to categorize
novel attacks into specific new classes based on feature influence. The
resulting system demonstrated will get good at separating attack traffic from
normal flow and auto-generate a label for attacks based on features that
contribute to the attack. These auto-generated labels can be presented to SIEM
analyst and are intuitive enough to figure out the nature of attack. We apply
this approach to a network flow dataset and demonstrate results for specific
attack types like ip sweep, denial of service, remote to local, etc.
  Paper was presented at the first Conference on Deployable AI at IIT-Madras in
June 2021.


http://arxiv.org/abs/2106.10807
Adversarial Examples Make Strong Poisons. (98%)
Liam Fowl; Micah Goldblum; Ping-yeh Chiang; Jonas Geiping; Wojtek Czaja; Tom Goldstein
  The adversarial machine learning literature is largely partitioned into
evasion attacks on testing data and poisoning attacks on training data. In this
work, we show that adversarial examples, originally intended for attacking
pre-trained models, are even more effective for data poisoning than recent
methods designed specifically for poisoning. Our findings indicate that
adversarial examples, when assigned the original label of their natural base
image, cannot be used to train a classifier for natural images. Furthermore,
when adversarial examples are assigned their adversarial class label, they are
useful for training. This suggests that adversarial examples contain useful
semantic content, just with the ``wrong'' labels (according to a network, but
not a human). Our method, adversarial poisoning, is substantially more
effective than existing poisoning methods for secure dataset release, and we
release a poisoned version of ImageNet, ImageNet-P, to encourage research into
the strength of this form of data obfuscation.


http://arxiv.org/abs/2106.10785
Adversarial Attack on Graph Neural Networks as An Influence Maximization Problem. (95%)
Jiaqi Ma; Junwei Deng; Qiaozhu Mei
  Graph neural networks (GNNs) have attracted increasing interests. With broad
deployments of GNNs in real-world applications, there is an urgent need for
understanding the robustness of GNNs under adversarial attacks, especially in
realistic setups. In this work, we study the problem of attacking GNNs in a
restricted and realistic setup, by perturbing the features of a small set of
nodes, with no access to model parameters and model predictions. Our formal
analysis draws a connection between this type of attacks and an influence
maximization problem on the graph. This connection not only enhances our
understanding on the problem of adversarial attack on GNNs, but also allows us
to propose a group of effective and practical attack strategies. Our
experiments verify that the proposed attack strategies significantly degrade
the performance of three popular GNN models and outperform baseline adversarial
attack strategies.


http://arxiv.org/abs/2106.10696
Generative Model Adversarial Training for Deep Compressed Sensing. (8%)
Ashkan Esmaeili
  Deep compressed sensing assumes the data has sparse representation in a
latent space, i.e., it is intrinsically of low-dimension. The original data is
assumed to be mapped from a low-dimensional space through a
low-to-high-dimensional generator. In this work, we propound how to design such
a low-to-high dimensional deep learning-based generator suiting for compressed
sensing, while satisfying robustness to universal adversarial perturbations in
the latent domain. We also justify why the noise is considered in the latent
space. The work is also buttressed with theoretical analysis on the robustness
of the trained generator to adversarial perturbations. Experiments on
real-world datasets are provided to substantiate the efficacy of the proposed
\emph{generative model adversarial training for deep compressed sensing.}


http://arxiv.org/abs/2106.10606
Attack to Fool and Explain Deep Networks. (99%)
Naveed Akhtar; Muhammad A. A. K. Jalwana; Mohammed Bennamoun; Ajmal Mian
  Deep visual models are susceptible to adversarial perturbations to inputs.
Although these signals are carefully crafted, they still appear noise-like
patterns to humans. This observation has led to the argument that deep visual
representation is misaligned with human perception. We counter-argue by
providing evidence of human-meaningful patterns in adversarial perturbations.
We first propose an attack that fools a network to confuse a whole category of
objects (source class) with a target label. Our attack also limits the
unintended fooling by samples from non-sources classes, thereby circumscribing
human-defined semantic notions for network fooling. We show that the proposed
attack not only leads to the emergence of regular geometric patterns in the
perturbations, but also reveals insightful information about the decision
boundaries of deep models. Exploring this phenomenon further, we alter the
`adversarial' objective of our attack to use it as a tool to `explain' deep
visual representation. We show that by careful channeling and projection of the
perturbations computed by our method, we can visualize a model's understanding
of human-defined semantic notions. Finally, we exploit the explanability
properties of our perturbations to perform image generation, inpainting and
interactive image manipulation by attacking adversarialy robust
`classifiers'.In all, our major contribution is a novel pragmatic adversarial
attack that is subsequently transformed into a tool to interpret the visual
models. The article also makes secondary contributions in terms of establishing
the utility of our attack beyond the adversarial objective with multiple
interesting applications.


http://arxiv.org/abs/2106.11760
A Stealthy and Robust Fingerprinting Scheme for Generative Models. (47%)
Li Guanlin; Guo Shangwei; Wang Run; Xu Guowen; Zhang Tianwei
  This paper presents a novel fingerprinting methodology for the Intellectual
Property protection of generative models. Prior solutions for discriminative
models usually adopt adversarial examples as the fingerprints, which give
anomalous inference behaviors and prediction results. Hence, these methods are
not stealthy and can be easily recognized by the adversary. Our approach
leverages the invisible backdoor technique to overcome the above limitation.
Specifically, we design verification samples, whose model outputs look normal
but can trigger a backdoor classifier to make abnormal predictions. We propose
a new backdoor embedding approach with Unique-Triplet Loss and fine-grained
categorization to enhance the effectiveness of our fingerprints. Extensive
evaluations show that this solution can outperform other strategies with higher
robustness, uniqueness and stealthiness for various GAN models.


http://arxiv.org/abs/2106.09947
Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples. (99%)
Maura Pintor; Luca Demetrio; Angelo Sotgiu; Giovanni Manca; Ambra Demontis; Nicholas Carlini; Battista Biggio; Fabio Roli
  Evaluating robustness of machine-learning models to adversarial examples is a
challenging problem. Many defenses have been shown to provide a false sense of
security by causing gradient-based attacks to fail, and they have been broken
under more rigorous evaluations. Although guidelines and best practices have
been suggested to improve current adversarial robustness evaluations, the lack
of automatic testing and debugging tools makes it difficult to apply these
recommendations in a systematic manner. In this work, we overcome these
limitations by (i) defining a set of quantitative indicators which unveil
common failures in the optimization of gradient-based attacks, and (ii)
proposing specific mitigation strategies within a systematic evaluation
protocol. Our extensive experimental analysis shows that the proposed
indicators of failure can be used to visualize, debug and improve current
adversarial robustness evaluations, providing a first concrete step towards
automatizing and systematizing current adversarial robustness evaluations. Our
open-source code is available at:
https://github.com/pralab/IndicatorsOfAttackFailure.


http://arxiv.org/abs/2106.10212
Residual Error: a New Performance Measure for Adversarial Robustness. (99%)
Hossein Aboutalebi; Mohammad Javad Shafiee; Michelle Karg; Christian Scharfenberger; Alexander Wong
  Despite the significant advances in deep learning over the past decade, a
major challenge that limits the wide-spread adoption of deep learning has been
their fragility to adversarial attacks. This sensitivity to making erroneous
predictions in the presence of adversarially perturbed data makes deep neural
networks difficult to adopt for certain real-world, mission-critical
applications. While much of the research focus has revolved around adversarial
example creation and adversarial hardening, the area of performance measures
for assessing adversarial robustness is not well explored. Motivated by this,
this study presents the concept of residual error, a new performance measure
for not only assessing the adversarial robustness of a deep neural network at
the individual sample level, but also can be used to differentiate between
adversarial and non-adversarial examples to facilitate for adversarial example
detection. Furthermore, we introduce a hybrid model for approximating the
residual error in a tractable manner. Experimental results using the case of
image classification demonstrates the effectiveness and efficacy of the
proposed residual error metric for assessing several well-known deep neural
network architectures. These results thus illustrate that the proposed measure
could be a useful tool for not only assessing the robustness of deep neural
networks used in mission-critical scenarios, but also in the design of
adversarially robust models.


http://arxiv.org/abs/2106.09992
Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis. (99%)
Martin Pawelczyk; Chirag Agarwal; Shalmali Joshi; Sohini Upadhyay; Himabindu Lakkaraju
  As machine learning (ML) models become more widely deployed in high-stakes
applications, counterfactual explanations have emerged as key tools for
providing actionable model explanations in practice. Despite the growing
popularity of counterfactual explanations, a deeper understanding of these
explanations is still lacking. In this work, we systematically analyze
counterfactual explanations through the lens of adversarial examples. We do so
by formalizing the similarities between popular counterfactual explanation and
adversarial example generation methods identifying conditions when they are
equivalent. We then derive the upper bounds on the distances between the
solutions output by counterfactual explanation and adversarial example
generation methods, which we validate on several real-world data sets. By
establishing these theoretical and empirical similarities between
counterfactual explanations and adversarial examples, our work raises
fundamental questions about the design and development of existing
counterfactual explanation algorithms.


http://arxiv.org/abs/2106.10151
The Dimpled Manifold Model of Adversarial Examples in Machine Learning. (98%)
Adi Shamir; Odelia Melamed; Oriel BenShmuel
  The extreme fragility of deep neural networks when presented with tiny
perturbations in their inputs was independently discovered by several research
groups in 2013, but in spite of enormous effort these adversarial examples
remained a baffling phenomenon with no clear explanation. In this paper we
introduce a new conceptual framework (which we call the Dimpled Manifold Model)
which provides a simple explanation for why adversarial examples exist, why
their perturbations have such tiny norms, why these perturbations look like
random noise, and why a network which was adversarially trained with
incorrectly labeled images can still correctly classify test images. In the
last part of the paper we describe the results of numerous experiments which
strongly support this new model, and in particular our assertion that
adversarial perturbations are roughly perpendicular to the low dimensional
manifold which contains all the training examples.


http://arxiv.org/abs/2106.09908
Light Lies: Optical Adversarial Attack. (92%)
Kyulim Kim; JeongSoo Kim; Seungri Song; Jun-Ho Choi; Chulmin Joo; Jong-Seok Lee
  A significant amount of work has been done on adversarial attacks that inject
imperceptible noise to images to deteriorate the image classification
performance of deep models. However, most of the existing studies consider
attacks in the digital (pixel) domain where an image acquired by an image
sensor with sampling and quantization has been recorded. This paper, for the
first time, introduces an optical adversarial attack, which physically alters
the light field information arriving at the image sensor so that the
classification model yields misclassification. More specifically, we modulate
the phase of the light in the Fourier domain using a spatial light modulator
placed in the photographic system. The operative parameters of the modulator
are obtained by gradient-based optimization to maximize cross-entropy and
minimize distortions. We present experiments based on both simulation and a
real hardware optical system, from which the feasibility of the proposed
optical attack is demonstrated. It is also verified that the proposed attack is
completely different from common optical-domain distortions such as spherical
aberration, defocus, and astigmatism in terms of both perturbation patterns and
classification results.


http://arxiv.org/abs/2106.09989
BinarizedAttack: Structural Poisoning Attacks to Graph-based Anomaly Detection. (82%)
Yulin Zhu; Yuni Lai; Kaifa Zhao; Xiapu Luo; Mingquan Yuan; Jian Ren; Kai Zhou
  Graph-based Anomaly Detection (GAD) is becoming prevalent due to the powerful
representation abilities of graphs as well as recent advances in graph mining
techniques. These GAD tools, however, expose a new attacking surface,
ironically due to their unique advantage of being able to exploit the relations
among data. That is, attackers now can manipulate those relations (i.e., the
structure of the graph) to allow some target nodes to evade detection. In this
paper, we exploit this vulnerability by designing a new type of targeted
structural poisoning attacks to a representative regression-based GAD system
termed OddBall. Specially, we formulate the attack against OddBall as a
bi-level optimization problem, where the key technical challenge is to
efficiently solve the problem in a discrete domain. We propose a novel attack
method termed BinarizedAttack based on gradient descent. Comparing to prior
arts, BinarizedAttack can better use the gradient information, making it
particularly suitable for solving combinatorial optimization problems.
Furthermore, we investigate the attack transferability of BinarizedAttack by
employing it to attack other representation-learning-based GAD systems. Our
comprehensive experiments demonstrate that BinarizedAttack is very effective in
enabling target nodes to evade graph-based anomaly detection tools with limited
attackers' budget, and in the black-box transfer attack setting,
BinarizedAttack is also tested effective and in particular, can significantly
change the node embeddings learned by the GAD systems. Our research thus opens
the door to studying a new type of attack against security analytic tools that
rely on graph data.


http://arxiv.org/abs/2106.10252
Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks. (80%)
Emre Ozfatura; Muhammad Zaid Hameed; Kerem Ozfatura; Deniz Gunduz
  A common observation regarding adversarial attacks is that they mostly give
rise to false activation at the penultimate layer to fool the classifier.
Assuming that these activation values correspond to certain features of the
input, the objective becomes choosing the features that are most useful for
classification. Hence, we propose a novel approach to identify the important
features by employing counter-adversarial attacks, which highlights the
consistency at the penultimate layer with respect to perturbations on input
samples. First, we empirically show that there exist a subset of features,
classification based in which bridge the gap between the clean and robust
accuracy. Second, we propose a simple yet efficient mechanism to identify those
features by searching the neighborhood of input sample. We then select features
by observing the consistency of the activation values at the penultimate layer.


http://arxiv.org/abs/2106.10324
Group-Structured Adversarial Training. (68%)
Farzan Farnia; Amirali Aghazadeh; James Zou; David Tse
  Robust training methods against perturbations to the input data have received
great attention in the machine learning literature. A standard approach in this
direction is adversarial training which learns a model using
adversarially-perturbed training samples. However, adversarial training
performs suboptimally against perturbations structured across samples such as
universal and group-sparse shifts that are commonly present in biological data
such as gene expression levels of different tissues. In this work, we seek to
close this optimality gap and introduce Group-Structured Adversarial Training
(GSAT) which learns a model robust to perturbations structured across samples.
We formulate GSAT as a non-convex concave minimax optimization problem which
minimizes a group-structured optimal transport cost. Specifically, we focus on
the applications of GSAT for group-sparse and rank-constrained perturbations
modeled using group and nuclear norm penalties. In order to solve GSAT's
non-smooth optimization problem in those cases, we propose a new minimax
optimization algorithm called GDADMM by combining Gradient Descent Ascent (GDA)
and Alternating Direction Method of Multipliers (ADMM). We present several
applications of the GSAT framework to gain robustness against structured
perturbations for image recognition and computational biology datasets.


http://arxiv.org/abs/2106.09993
Accumulative Poisoning Attacks on Real-time Data. (45%)
Tianyu Pang; Xiao Yang; Yinpeng Dong; Hang Su; Jun Zhu
  Collecting training data from untrusted sources exposes machine learning
services to poisoning adversaries, who maliciously manipulate training data to
degrade the model accuracy. When trained on offline datasets, poisoning
adversaries have to inject the poisoned data in advance before training, and
the order of feeding these poisoned batches into the model is stochastic. In
contrast, practical systems are more usually trained/fine-tuned on sequentially
captured real-time data, in which case poisoning adversaries could dynamically
poison each data batch according to the current model state. In this paper, we
focus on the real-time settings and propose a new attacking strategy, which
affiliates an accumulative phase with poisoning attacks to secretly (i.e.,
without affecting accuracy) magnify the destructive effect of a (poisoned)
trigger batch. By mimicking online learning and federated learning on CIFAR-10,
we show that the model accuracy will significantly drop by a single update step
on the trigger batch after the accumulative phase. Our work validates that a
well-designed but straightforward attacking strategy can dramatically amplify
the poisoning effects, with no need to explore complex techniques.


http://arxiv.org/abs/2106.10147
Evaluating the Robustness of Trigger Set-Based Watermarks Embedded in Deep Neural Networks. (45%)
Suyoung Lee; Wonho Song; Suman Jana; Meeyoung Cha; Sooel Son
  Trigger set-based watermarking schemes have gained emerging attention as they
provide a means to prove ownership for deep neural network model owners. In
this paper, we argue that state-of-the-art trigger set-based watermarking
algorithms do not achieve their designed goal of proving ownership. We posit
that this impaired capability stems from two common experimental flaws that the
existing research practice has committed when evaluating the robustness of
watermarking algorithms: (1) incomplete adversarial evaluation and (2)
overlooked adaptive attacks.
  We conduct a comprehensive adversarial evaluation of 10 representative
watermarking schemes against six of the existing attacks and demonstrate that
each of these watermarking schemes lacks robustness against at least two
attacks. We also propose novel adaptive attacks that harness the adversary's
knowledge of the underlying watermarking algorithm of a target model. We
demonstrate that the proposed attacks effectively break all of the 10
watermarking schemes, consequently allowing adversaries to obscure the
ownership of any watermarked model. We encourage follow-up studies to consider
our guidelines when evaluating the robustness of their watermarking schemes via
conducting comprehensive adversarial evaluation that include our adaptive
attacks to demonstrate a meaningful upper bound of watermark robustness.


http://arxiv.org/abs/2106.10196
Federated Robustness Propagation: Sharing Adversarial Robustness in Federated Learning. (5%)
Junyuan Hong; Haotao Wang; Zhangyang Wang; Jiayu Zhou
  Federated learning (FL) emerges as a popular distributed learning schema that
learns a model from a set of participating users without requiring raw data to
be shared. One major challenge of FL comes from heterogeneity in users, which
may have distributionally different (or non-iid) data and varying computation
resources. Just like in centralized learning, FL users also desire model
robustness against malicious attackers at test time. Whereas adversarial
training (AT) provides a sound solution for centralized learning, extending its
usage for FL users has imposed significant challenges, as many users may have
very limited training data as well as tight computational budgets, to afford
the data-hungry and costly AT. In this paper, we study a novel learning setting
that propagates adversarial robustness from high-resource users that can afford
AT, to those low-resource users that cannot afford it, during the FL process.
We show that existing FL techniques cannot effectively propagate adversarial
robustness among non-iid users, and propose a simple yet effective propagation
approach that transfers robustness through carefully designed
batch-normalization statistics. We demonstrate the rationality and
effectiveness of our method through extensive experiments. Especially, the
proposed method is shown to grant FL remarkable robustness even when only a
small portion of users afford AT during learning. Codes will be published upon
acceptance.


http://arxiv.org/abs/2106.09872
Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective. (99%)
Lina Wang; Xingshu Chen; Yulong Wang; Yawei Yue; Yi Zhu; Xuemei Zeng; Wei Wang
  The vulnerability of deep neural networks to adversarial examples, which are
crafted maliciously by modifying the inputs with imperceptible perturbations to
misled the network produce incorrect outputs, reveals the lack of robustness
and poses security concerns. Previous works study the adversarial robustness of
image classifiers on image level and use all the pixel information in an image
indiscriminately, lacking of exploration of regions with different semantic
meanings in the pixel space of an image. In this work, we fill this gap and
explore the pixel space of the adversarial image by proposing an algorithm to
looking for possible perturbations pixel by pixel in different regions of the
segmented image. The extensive experimental results on CIFAR-10 and ImageNet
verify that searching for the modified pixel in only some pixels of an image
can successfully launch the one-pixel adversarial attacks without requiring all
the pixels of the entire image, and there exist multiple vulnerable points
scattered in different regions of an image. We also demonstrate that the
adversarial robustness of different regions on the image varies with the amount
of semantic information contained.


http://arxiv.org/abs/2106.09898
Bad Characters: Imperceptible NLP Attacks. (99%)
Nicholas Boucher; Ilia Shumailov; Ross Anderson; Nicolas Papernot
  Several years of research have shown that machine-learning systems are
vulnerable to adversarial examples, both in theory and in practice. Until now,
such attacks have primarily targeted visual models, exploiting the gap between
human and machine perception. Although text-based models have also been
attacked with adversarial examples, such attacks struggled to preserve semantic
meaning and indistinguishability. In this paper, we explore a large class of
adversarial examples that can be used to attack text-based models in a
black-box setting without making any human-perceptible visual modification to
inputs. We use encoding-specific perturbations that are imperceptible to the
human eye to manipulate the outputs of a wide range of Natural Language
Processing (NLP) systems from neural machine-translation pipelines to web
search engines. We find that with a single imperceptible encoding injection --
representing one invisible character, homoglyph, reordering, or deletion -- an
attacker can significantly reduce the performance of vulnerable models, and
with three injections most models can be functionally broken. Our attacks work
against currently-deployed commercial systems, including those produced by
Microsoft and Google, in addition to open source models published by Facebook
and IBM. This novel series of attacks presents a significant threat to many
language processing systems: an attacker can affect systems in a targeted
manner without any assumptions about the underlying model. We conclude that
text-based NLP systems require careful input sanitization, just like
conventional applications, and that given such systems are now being deployed
rapidly at scale, the urgent attention of architects and operators is required.


http://arxiv.org/abs/2106.09501
DeepInsight: Interpretability Assisting Detection of Adversarial Samples on Graphs. (99%)
Junhao Zhu; Yalu Shan; Jinhuan Wang; Shanqing Yu; Guanrong Chen; Qi Xuan
  With the rapid development of artificial intelligence, a series of machine
learning algorithms, e.g., graph neural networks, have been proposed to
facilitate network analysis or graph data mining. Unfortunately, recent studies
indicate that such advanced methods may suffer from adversarial attacks, i.e.,
they may lose effectiveness when only a small fraction of links are purposely
changed. However, little is known what's the difference between adversarial
nodes and clean nodes, and what's the preference of each attack method, in
terms of network structure. In this paper, we theoretically investigate three
well-known adversarial attack methods, i.e., Nettack, Meta Attack, and
GradArgmax, and find that different attack methods have their specific attack
preferences on changing network structure. Such attack patterns are further
validated by the experimental results on real-world networks, i.e., generally
the top 4 most important network attributes on detecting adversarial samples
are sufficient to explain the preference of each attack method. Based on these
findings, we further utilize the network attributes to design machine learning
models for adversarial sample detection and attack method recognition,
achieving the outstanding performance.


http://arxiv.org/abs/2106.09534
Adversarial Visual Robustness by Causal Intervention. (99%)
Kaihua Tang; Mingyuan Tao; Hanwang Zhang
  Adversarial training is the de facto most promising defense against
adversarial examples. Yet, its passive nature inevitably prevents it from being
immune to unknown attackers. To achieve a proactive defense, we need a more
fundamental understanding of adversarial examples, beyond the popular bounded
threat model. In this paper, we provide a causal viewpoint of adversarial
vulnerability: the cause is the spurious correlation ubiquitously existing in
learning, i.e., the confounding effect, where attackers are precisely
exploiting these effects. Therefore, a fundamental solution for adversarial
robustness is by causal intervention. As these visual confounders are
imperceptible in general, we propose to use the instrumental variable that
achieves causal intervention without the need for confounder observation. We
term our robust training method as Causal intervention by instrumental Variable
(CiiV). It's a causal regularization that 1) augments the image with multiple
retinotopic centers and 2) encourages the model to learn causal features,
rather than local confounding patterns, by favoring features linearly
responding to spatial interpolations. Extensive experiments on a wide spectrum
of attackers and settings applied in CIFAR-10, CIFAR-100, and mini-ImageNet
demonstrate that CiiV is robust to adaptive attacks, including the recent
AutoAttack. Besides, as a general causal regularization, it can be easily
plugged into other methods to further boost the robustness.


http://arxiv.org/abs/2106.09820
Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning. (92%)
Shubham Jain; Ana-Maria Cretu; Montjoye Yves-Alexandre de
  End-to-end encryption (E2EE) by messaging platforms enable people to securely
and privately communicate with one another. Its widespread adoption however
raised concerns that illegal content might now be shared undetected. Following
the global pushback against key escrow systems, client-side scanning based on
perceptual hashing has been recently proposed by governments and researchers to
detect illegal content in E2EE communications. We here propose the first
framework to evaluate the robustness of perceptual hashing-based client-side
scanning to detection avoidance attacks and show current systems to not be
robust. More specifically, we propose three adversarial attacks -- a general
black-box attack and two white-box attacks for discrete cosine-based-based
algorithms -- against perceptual hashing algorithms. In a large-scale
evaluation, we show perceptual hashing-based client-side scanning mechanisms to
be highly vulnerable to detection avoidance attacks in a black-box setting,
with more than 99.9% of images successfully attacked while preserving the
content of the image. We furthermore show our attack to generate diverse
perturbations, strongly suggesting that straightforward mitigation strategies
would be ineffective. Finally, we show that the larger thresholds necessary to
make the attack harder would probably require more than one billion images to
be flagged and decrypted daily, raising strong privacy concerns.Taken together,
our results shed serious doubts on the robustness of perceptual hashing-based
client-side scanning mechanisms currently proposed by governments,
organizations, and researchers around the world.


http://arxiv.org/abs/2106.09249
Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks. (91%)
Yulong *co-first authors Cao*; Ningfei *co-first authors Wang*; Chaowei *co-first authors Xiao*; Dawei *co-first authors Yang*; Jin *co-first authors Fang; Ruigang *co-first authors Yang; Qi Alfred *co-first authors Chen; Mingyan *co-first authors Liu; Bo *co-first authors Li
  In Autonomous Driving (AD) systems, perception is both security and safety
critical. Despite various prior studies on its security issues, all of them
only consider attacks on camera- or LiDAR-based AD perception alone. However,
production AD systems today predominantly adopt a Multi-Sensor Fusion (MSF)
based design, which in principle can be more robust against these attacks under
the assumption that not all fusion sources are (or can be) attacked at the same
time. In this paper, we present the first study of security issues of MSF-based
perception in AD systems. We directly challenge the basic MSF design assumption
above by exploring the possibility of attacking all fusion sources
simultaneously. This allows us for the first time to understand how much
security guarantee MSF can fundamentally provide as a general defense strategy
for AD perception.
  We formulate the attack as an optimization problem to generate a
physically-realizable, adversarial 3D-printed object that misleads an AD system
to fail in detecting it and thus crash into it. We propose a novel attack
pipeline that addresses two main design challenges: (1) non-differentiable
target camera and LiDAR sensing systems, and (2) non-differentiable cell-level
aggregated features popularly used in LiDAR-based AD perception. We evaluate
our attack on MSF included in representative open-source industry-grade AD
systems in real-world driving scenarios. Our results show that the attack
achieves over 90% success rate across different object types and MSF. Our
attack is also found stealthy, robust to victim positions, transferable across
MSF algorithms, and physical-world realizable after being 3D-printed and
captured by LiDAR and camera devices. To concretely assess the end-to-end
safety impact, we further perform simulation evaluation and show that it can
cause a 100% vehicle collision rate for an industry-grade AD system.


http://arxiv.org/abs/2106.09380
Modeling Realistic Adversarial Attacks against Network Intrusion Detection Systems. (82%)
Giovanni Apruzzese; Mauro Andreolini; Luca Ferretti; Mirco Marchetti; Michele Colajanni
  The incremental diffusion of machine learning algorithms in supporting
cybersecurity is creating novel defensive opportunities but also new types of
risks. Multiple researches have shown that machine learning methods are
vulnerable to adversarial attacks that create tiny perturbations aimed at
decreasing the effectiveness of detecting threats. We observe that existing
literature assumes threat models that are inappropriate for realistic
cybersecurity scenarios because they consider opponents with complete knowledge
about the cyber detector or that can freely interact with the target systems.
By focusing on Network Intrusion Detection Systems based on machine learning,
we identify and model the real capabilities and circumstances required by
attackers to carry out feasible and successful adversarial attacks. We then
apply our model to several adversarial attacks proposed in literature and
highlight the limits and merits that can result in actual adversarial attacks.
The contributions of this paper can help hardening defensive systems by letting
cyber defenders address the most critical and real issues, and can benefit
researchers by allowing them to devise novel forms of adversarial attacks based
on realistic threat models.


http://arxiv.org/abs/2106.09667
Poisoning and Backdooring Contrastive Learning. (70%)
Nicholas Carlini; Andreas Terzis
  Contrastive learning methods like CLIP train on noisy and uncurated training
datasets. This is cheaper than labeling datasets manually, and even improves
out-of-distribution robustness. We show that this practice makes backdoor and
poisoning attacks a significant threat. By poisoning just 0.005% of a dataset
(e.g., just 150 images of the 3 million-example Conceptual Captions dataset),
we can cause the model to misclassify test images by overlaying a small patch.
Targeted poisoning attacks, whereby the model misclassifies a particular test
input with an adversarially-desired label, are even easier requiring control of
less than 0.0001% of the dataset (e.g., just two out of the 3 million images).
Our attacks call into question whether training on noisy and uncurated Internet
scrapes is desirable.


http://arxiv.org/abs/2106.09242
CoCoFuzzing: Testing Neural Code Models with Coverage-Guided Fuzzing. (64%)
Moshi Wei; Yuchao Huang; Jinqiu Yang; Junjie Wang; Song Wang
  Deep learning-based code processing models have shown good performance for
tasks such as predicting method names, summarizing programs, and comment
generation. However, despite the tremendous progress, deep learning models are
often prone to adversarial attacks, which can significantly threaten the
robustness and generalizability of these models by leading them to
misclassification with unexpected inputs. To address the above issue, many deep
learning testing approaches have been proposed, however, these approaches
mainly focus on testing deep learning applications in the domains of image,
audio, and text analysis, etc., which cannot be directly applied to neural
models for code due to the unique properties of programs. In this paper, we
propose a coverage-based fuzzing framework, CoCoFuzzing, for testing deep
learning-based code processing models. In particular, we first propose ten
mutation operators to automatically generate valid and semantically preserving
source code examples as tests; then we propose a neuron coverage-based approach
to guide the generation of tests. We investigate the performance of CoCoFuzzing
on three state-of-the-art neural code models, i.e., NeuralCodeSum, CODE2SEQ,
and CODE2VEC. Our experiment results demonstrate that CoCoFuzzing can generate
valid and semantically preserving source code examples for testing the
robustness and generalizability of these models and improve the neuron
coverage. Moreover, these tests can be used to improve the performance of the
target neural code models through adversarial retraining.


http://arxiv.org/abs/2106.09292
CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing. (56%)
Fan Wu; Linyi Li; Zijian Huang; Yevgeniy Vorobeychik; Ding Zhao; Bo Li
  We present the first framework of Certifying Robust Policies for
reinforcement learning (CROP) against adversarial state perturbations. We
propose two particular types of robustness certification criteria: robustness
of per-state actions and lower bound of cumulative rewards. Specifically, we
develop a local smoothing algorithm which uses a policy derived from
Q-functions smoothed with Gaussian noise over each encountered state to
guarantee the robustness of actions taken along this trajectory. Next, we
develop a global smoothing algorithm for certifying the robustness of a
finite-horizon cumulative reward under adversarial state perturbations.
Finally, we propose a local smoothing approach which makes use of adaptive
search in order to obtain tight certification bounds for reward. We use the
proposed RL robustness certification framework to evaluate six methods that
have previously been shown to yield empirically robust RL, including
adversarial training and several forms of regularization, on two representative
Atari games. We show that RegPGD, RegCVX, and RadialRL achieve high certified
robustness among these. Furthermore, we demonstrate that our certifications are
often tight by evaluating these algorithms against adversarial attacks.


http://arxiv.org/abs/2106.09385
On Deep Neural Network Calibration by Regularization and its Impact on Refinement. (3%)
Aditya Singh; Alessandro Bay; Biswa Sengupta; Andrea Mirabile
  Deep neural networks have been shown to be highly miscalibrated. often they
tend to be overconfident in their predictions. It poses a significant challenge
for safety-critical systems to utilise deep neural networks (DNNs), reliably.
Many recently proposed approaches to mitigate this have demonstrated
substantial progress in improving DNN calibration. However, they hardly touch
upon refinement, which historically has been an essential aspect of
calibration. Refinement indicates separability of a network's correct and
incorrect predictions. This paper presents a theoretically and empirically
supported exposition reviewing refinement of a calibrated model. Firstly, we
show the breakdown of expected calibration error (ECE), into predicted
confidence and refinement under the assumption of over-confident predictions.
Secondly, linking with this result, we highlight that regularization based
calibration only focuses on naively reducing a model's confidence. This
logically has a severe downside to a model's refinement as correct and
incorrect predictions become tightly coupled. Lastly, connecting refinement
with ECE also provides support to existing refinement based approaches which
improve calibration but do not explain the reasoning behind it. We support our
claims through rigorous empirical evaluations of many state of the art
calibration approaches on widely used datasets and neural networks. We find
that many calibration approaches with the likes of label smoothing, mixup etc.
lower the usefulness of a DNN by degrading its refinement. Even under natural
data shift, this calibration-refinement trade-off holds for the majority of
calibration methods.


http://arxiv.org/abs/2106.08746
Real-time Attacks Against Deep Reinforcement Learning Policies. (99%)
Buse G. A. Tekgul; Shelly Wang; Samuel Marchal; N. Asokan
  Recent work has discovered that deep reinforcement learning (DRL) policies
are vulnerable to adversarial examples. These attacks mislead the policy of DRL
agents by perturbing the state of the environment observed by agents. They are
feasible in principle but too slow to fool DRL policies in real time. We
propose a new attack to fool DRL policies that is both effective and efficient
enough to be mounted in real time. We utilize the Universal Adversarial
Perturbation (UAP) method to compute effective perturbations independent of the
individual inputs to which they are applied. Via an extensive evaluation using
Atari 2600 games, we show that our technique is effective, as it fully degrades
the performance of both deterministic and stochastic policies (up to 100%, even
when the $l_\infty$ bound on the perturbation is as small as 0.005). We also
show that our attack is efficient, incurring an online computational cost of
0.027ms on average. It is faster compared to the response time (0.6ms on
average) of agents with different DRL policies, and considerably faster than
prior attacks (2.7ms on average). Furthermore, we demonstrate that known
defenses are ineffective against universal perturbations. We propose an
effective detection technique which can form the basis for robust defenses
against attacks based on universal perturbations.


http://arxiv.org/abs/2106.09222
Localized Uncertainty Attacks. (99%)
Ousmane Amadou Dia; Theofanis Karaletsos; Caner Hazirbas; Cristian Canton Ferrer; Ilknur Kaynar Kabul; Erik Meijer
  The susceptibility of deep learning models to adversarial perturbations has
stirred renewed attention in adversarial examples resulting in a number of
attacks. However, most of these attacks fail to encompass a large spectrum of
adversarial perturbations that are imperceptible to humans. In this paper, we
present localized uncertainty attacks, a novel class of threat models against
deterministic and stochastic classifiers. Under this threat model, we create
adversarial examples by perturbing only regions in the inputs where a
classifier is uncertain. To find such regions, we utilize the predictive
uncertainty of the classifier when the classifier is stochastic or, we learn a
surrogate model to amortize the uncertainty when it is deterministic. Unlike
$\ell_p$ ball or functional attacks which perturb inputs indiscriminately, our
targeted changes can be less perceptible. When considered under our threat
model, these attacks still produce strong adversarial examples; with the
examples retaining a greater degree of similarity with the inputs.


http://arxiv.org/abs/2106.09223
Evaluating the Robustness of Bayesian Neural Networks Against Different Types of Attacks. (67%)
Yutian Pang; Sheng Cheng; Jueming Hu; Yongming Liu
  To evaluate the robustness gain of Bayesian neural networks on image
classification tasks, we perform input perturbations, and adversarial attacks
to the state-of-the-art Bayesian neural networks, with a benchmark CNN model as
reference. The attacks are selected to simulate signal interference and
cyberattacks towards CNN-based machine learning systems. The result shows that
a Bayesian neural network achieves significantly higher robustness against
adversarial attacks generated against a deterministic neural network model,
without adversarial training. The Bayesian posterior can act as the safety
precursor of ongoing malicious activities. Furthermore, we show that the
stochastic classifier after the deterministic CNN extractor has sufficient
robustness enhancement rather than a stochastic feature extractor before the
stochastic classifier. This advises on utilizing stochastic layers in building
decision-making pipelines within a safety-critical domain.


http://arxiv.org/abs/2106.08970
Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch. (38%)
Hossein Souri; Micah Goldblum; Liam Fowl; Rama Chellappa; Tom Goldstein
  As the curation of data for machine learning becomes increasingly automated,
dataset tampering is a mounting threat. Backdoor attackers tamper with training
data to embed a vulnerability in models that are trained on that data. This
vulnerability is then activated at inference time by placing a "trigger" into
the model's input. Typical backdoor attacks insert the trigger directly into
the training data, although the presence of such an attack may be visible upon
inspection. In contrast, the Hidden Trigger Backdoor Attack achieves poisoning
without placing a trigger into the training data at all. However, this hidden
trigger attack is ineffective at poisoning neural networks trained from
scratch. We develop a new hidden trigger attack, Sleeper Agent, which employs
gradient matching, data selection, and target model re-training during the
crafting process. Sleeper Agent is the first hidden trigger backdoor attack to
be effective against neural networks trained from scratch. We demonstrate its
effectiveness on ImageNet and in black-box settings. Our implementation code
can be found at https://github.com/hsouri/Sleeper-Agent.


http://arxiv.org/abs/2106.09106
Explainable AI for Natural Adversarial Images. (13%)
Tomas Folke; ZhaoBin Li; Ravi B. Sojitra; Scott Cheng-Hsin Yang; Patrick Shafto
  Adversarial images highlight how vulnerable modern image classifiers are to
perturbations outside of their training set. Human oversight might mitigate
this weakness, but depends on humans understanding the AI well enough to
predict when it is likely to make a mistake. In previous work we have found
that humans tend to assume that the AI's decision process mirrors their own.
Here we evaluate if methods from explainable AI can disrupt this assumption to
help participants predict AI classifications for adversarial and standard
images. We find that both saliency maps and examples facilitate catching AI
errors, but their effects are not additive, and saliency maps are more
effective than examples.


http://arxiv.org/abs/2106.09129
A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness. (2%)
James Diffenderfer; Brian R. Bartoldson; Shreya Chaganti; Jize Zhang; Bhavya Kailkhura
  Two crucial requirements for a successful adoption of deep learning (DL) in
the wild are: (1) robustness to distributional shifts, and (2) model
compactness for achieving efficiency. Unfortunately, efforts towards
simultaneously achieving Out-of-Distribution (OOD) robustness and extreme model
compactness without sacrificing accuracy have mostly been unsuccessful. This
raises an important question: "Is the inability to create compact, accurate,
and robust deep neural networks (CARDs) fundamental?" To answer this question,
we perform a large-scale analysis for a range of popular model compression
techniques which uncovers several intriguing patterns. Notably, in contrast to
traditional pruning approaches (e.g., fine tuning and gradual magnitude
pruning), we find that "lottery ticket-style" pruning approaches can
surprisingly be used to create high performing CARDs. Specifically, we are able
to create extremely compact CARDs that are dramatically more robust than their
significantly larger and full-precision counterparts while matching (or
beating) their test accuracy, simply by pruning and/or quantizing. To better
understand these differences, we perform sensitivity analysis in the Fourier
domain for CARDs trained using different data augmentation methods. Motivated
by our analysis, we develop a simple domain-adaptive test-time ensembling
approach (CARD-Deck) that uses a gating module to dynamically select an
appropriate CARD from the CARD-Deck based on their spectral-similarity with
test samples. By leveraging complementary frequency biases of different
compressed models, the proposed approach builds a "winning hand" of CARDs that
establishes a new state-of-the-art on CIFAR-10-C accuracies (i.e., 96.8% clean
and 92.75% robust) with dramatically better memory usage than their
non-compressed counterparts. We also present some theoretical evidences
supporting our empirical findings.


http://arxiv.org/abs/2106.09121
Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework. (1%)
Jiahao Su; Wonmin Byeon; Furong Huang
  Enforcing orthogonality in neural networks is an antidote for gradient
vanishing/exploding problems, sensitivity by adversarial perturbation, and
bounding generalization errors. However, many previous approaches are
heuristic, and the orthogonality of convolutional layers is not systematically
studied: some of these designs are not exactly orthogonal, while others only
consider standard convolutional layers and propose specific classes of their
realizations. To address this problem, we propose a theoretical framework for
orthogonal convolutional layers, which establishes the equivalence between
various orthogonal convolutional layers in the spatial domain and the
paraunitary systems in the spectral domain. Since there exists a complete
spectral factorization of paraunitary systems, any orthogonal convolution layer
can be parameterized as convolutions of spatial filters. Our framework endows
high expressive power to various convolutional layers while maintaining their
exact orthogonality. Furthermore, our layers are memory and computationally
efficient for deep networks compared to previous designs. Our versatile
framework, for the first time, enables the study of architecture designs for
deep orthogonal networks, such as choices of skip connection, initialization,
stride, and dilation. Consequently, we scale up orthogonal networks to deep
architectures, including ResNet, WideResNet, and ShuffleNet, substantially
increasing the performance over the traditional shallow orthogonal networks.


http://arxiv.org/abs/2106.08913
Loki: Hardening Code Obfuscation Against Automated Attacks. (1%)
Moritz Schloegel; Tim Blazytko; Moritz Contag; Cornelius Aschermann; Julius Basler; Thorsten Holz; Ali Abbasi
  Software obfuscation is a crucial technology to protect intellectual
property. Despite its importance, commercial and academic state-of-the-art
obfuscation approaches are vulnerable to a plethora of automated deobfuscation
attacks, such as symbolic execution, taint analysis, or program synthesis.
While several enhanced techniques were proposed to thwart taint analysis or
symbolic execution, they either impose a prohibitive runtime overhead or can be
removed by compiler optimizations. In general, they suffer from focusing on a
single attack vector, allowing an attacker to switch to other more effective
techniques, such as program synthesis. In this work, we present Loki, an
approach for code obfuscation that is resilient against all known automated
deobfuscation attacks. To this end, we deploy multiple techniques, including a
generic approach to synthesize formally verified expressions of arbitrary
complexity. Contrary to state-of-the-art approaches that rely on a few
hardcoded generation rules, our expressions are more diverse and harder to
pattern match against. Moreover, Loki protects against previously unaccounted
attack vectors such as program synthesis, for which it reduces the success rate
to merely 19%. Overall, our design incurs significantly less overhead while
providing a much stronger protection level.


http://arxiv.org/abs/2106.08361
Adversarial Attacks on Deep Models for Financial Transaction Records. (99%)
Ivan Fursov; Matvey Morozov; Nina Kaploukhaya; Elizaveta Kovtun; Rodrigo Rivera-Castro; Gleb Gusev; Dmitry Babaev; Ivan Kireev; Alexey Zaytsev; Evgeny Burnaev
  Machine learning models using transaction records as inputs are popular among
financial institutions. The most efficient models use deep-learning
architectures similar to those in the NLP community, posing a challenge due to
their tremendous number of parameters and limited robustness. In particular,
deep-learning models are vulnerable to adversarial attacks: a little change in
the input harms the model's output.
  In this work, we examine adversarial attacks on transaction records data and
defences from these attacks. The transaction records data have a different
structure than the canonical NLP or time series data, as neighbouring records
are less connected than words in sentences, and each record consists of both
discrete merchant code and continuous transaction amount. We consider a
black-box attack scenario, where the attack doesn't know the true decision
model, and pay special attention to adding transaction tokens to the end of a
sequence. These limitations provide more realistic scenario, previously
unexplored in NLP world.
  The proposed adversarial attacks and the respective defences demonstrate
remarkable performance using relevant datasets from the financial industry. Our
results show that a couple of generated transactions are sufficient to fool a
deep-learning model. Further, we improve model robustness via adversarial
training or separate adversarial examples detection. This work shows that
embedding protection from adversarial attacks improves model robustness,
allowing a wider adoption of deep models for transaction records in banking and
finance.


http://arxiv.org/abs/2106.08299
Model Extraction and Adversarial Attacks on Neural Networks using Switching Power Information. (99%)
Tommy Li; Cory Merkel
  Artificial neural networks (ANNs) have gained significant popularity in the
last decade for solving narrow AI problems in domains such as healthcare,
transportation, and defense. As ANNs become more ubiquitous, it is imperative
to understand their associated safety, security, and privacy vulnerabilities.
Recently, it has been shown that ANNs are susceptible to a number of
adversarial evasion attacks--inputs that cause the ANN to make high-confidence
misclassifications despite being almost indistinguishable from the data used to
train and test the network. This work explores to what degree finding these
examples maybe aided by using side-channel information, specifically switching
power consumption, of hardware implementations of ANNs. A black-box threat
scenario is assumed, where an attacker has access to the ANN hardware's input,
outputs, and topology, but the trained model parameters are unknown. Then, a
surrogate model is trained to have similar functional (i.e. input-output
mapping) and switching power characteristics as the oracle (black-box) model.
Our results indicate that the inclusion of power consumption data increases the
fidelity of the model extraction by up to 30 percent based on a mean square
error comparison of the oracle and surrogate weights. However, transferability
of adversarial examples from the surrogate to the oracle model was not
significantly affected.


http://arxiv.org/abs/2106.08387
Towards Adversarial Robustness via Transductive Learning. (80%)
Jiefeng Chen; Yang Guo; Xi Wu; Tianqi Li; Qicheng Lao; Yingyu Liang; Somesh Jha
  There has been emerging interest to use transductive learning for adversarial
robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020). Compared to
traditional "test-time" defenses, these defense mechanisms "dynamically
retrain" the model based on test time input via transductive learning; and
theoretically, attacking these defenses boils down to bilevel optimization,
which seems to raise the difficulty for adaptive attacks. In this paper, we
first formalize and analyze modeling aspects of transductive robustness. Then,
we propose the principle of attacking model space for solving bilevel attack
objectives, and present an instantiation of the principle which breaks previous
transductive defenses. These attacks thus point to significant difficulties in
the use of transductive learning to improve adversarial robustness. To this
end, we present new theoretical and empirical evidence in support of the
utility of transductive learning.


http://arxiv.org/abs/2106.07868
Voting for the right answer: Adversarial defense for speaker verification. (78%)
Haibin Wu; Yang Zhang; Zhiyong Wu; Dong Wang; Hung-yi Lee
  Automatic speaker verification (ASV) is a well developed technology for
biometric identification, and has been ubiquitous implemented in
security-critic applications, such as banking and access control. However,
previous works have shown that ASV is under the radar of adversarial attacks,
which are very similar to their original counterparts from human's perception,
yet will manipulate the ASV render wrong prediction. Due to the very late
emergence of adversarial attacks for ASV, effective countermeasures against
them are limited. Given that the security of ASV is of high priority, in this
work, we propose the idea of "voting for the right answer" to prevent risky
decisions of ASV in blind spot areas, by employing random sampling and voting.
Experimental results show that our proposed method improves the robustness
against both the limited-knowledge attackers by pulling the adversarial samples
out of the blind spots, and the perfect-knowledge attackers by introducing
randomness and increasing the attackers' budgets.


http://arxiv.org/abs/2106.08104
Detect and remove watermark in deep neural networks via generative adversarial networks. (68%)
Haoqi Wang; Mingfu Xue; Shichang Sun; Yushu Zhang; Jian Wang; Weiqiang Liu
  Deep neural networks (DNN) have achieved remarkable performance in various
fields. However, training a DNN model from scratch requires a lot of computing
resources and training data. It is difficult for most individual users to
obtain such computing resources and training data. Model copyright infringement
is an emerging problem in recent years. For instance, pre-trained models may be
stolen or abuse by illegal users without the authorization of the model owner.
Recently, many works on protecting the intellectual property of DNN models have
been proposed. In these works, embedding watermarks into DNN based on backdoor
is one of the widely used methods. However, when the DNN model is stolen, the
backdoor-based watermark may face the risk of being detected and removed by an
adversary. In this paper, we propose a scheme to detect and remove watermark in
deep neural networks via generative adversarial networks (GAN). We demonstrate
that the backdoor-based DNN watermarks are vulnerable to the proposed GAN-based
watermark removal attack. The proposed attack method includes two phases. In
the first phase, we use the GAN and few clean images to detect and reverse the
watermark in the DNN model. In the second phase, we fine-tune the watermarked
DNN based on the reversed backdoor images. Experimental evaluations on the
MNIST and CIFAR10 datasets demonstrate that, the proposed method can
effectively remove about 98% of the watermark in DNN models, as the watermark
retention rate reduces from 100% to less than 2% after applying the proposed
attack. In the meantime, the proposed attack hardly affects the model's
performance. The test accuracy of the watermarked DNN on the MNIST and the
CIFAR10 datasets drops by less than 1% and 3%, respectively.


http://arxiv.org/abs/2106.08283
CRFL: Certifiably Robust Federated Learning against Backdoor Attacks. (13%)
Chulin Xie; Minghao Chen; Pin-Yu Chen; Bo Li
  Federated Learning (FL) as a distributed learning paradigm that aggregates
information from diverse clients to train a shared global model, has
demonstrated great success. However, malicious clients can perform poisoning
attacks and model replacement to introduce backdoors into the trained global
model. Although there have been intensive studies designing robust aggregation
methods and empirical robust federated training protocols against backdoors,
existing approaches lack robustness certification. This paper provides the
first general framework, Certifiably Robust Federated Learning (CRFL), to train
certifiably robust FL models against backdoors. Our method exploits clipping
and smoothing on model parameters to control the global model smoothness, which
yields a sample-wise robustness certification on backdoors with limited
magnitude. Our certification also specifies the relation to federated learning
parameters, such as poisoning ratio on instance level, number of attackers, and
training iterations. Practically, we conduct comprehensive experiments across a
range of federated datasets, and provide the first benchmark for certified
robustness against backdoor attacks in federated learning. Our code is
available at https://github.com/AI-secure/CRFL.


http://arxiv.org/abs/2106.08013
Securing Face Liveness Detection Using Unforgeable Lip Motion Patterns. (12%)
Man Senior Member, IEEE Zhou; Qian Senior Member, IEEE Wang; Qi Senior Member, IEEE Li; Peipei Senior Member, IEEE Jiang; Jingxiao Senior Member, IEEE Yang; Chao Senior Member, IEEE Shen; Cong Fellow, IEEE Wang; Shouhong Ding
  Face authentication usually utilizes deep learning models to verify users
with high recognition accuracy. However, face authentication systems are
vulnerable to various attacks that cheat the models by manipulating the digital
counterparts of human faces. So far, lots of liveness detection schemes have
been developed to prevent such attacks. Unfortunately, the attacker can still
bypass these schemes by constructing wide-ranging sophisticated attacks. We
study the security of existing face authentication services (e.g., Microsoft,
Amazon, and Face++) and typical liveness detection approaches. Particularly, we
develop a new type of attack, i.e., the low-cost 3D projection attack that
projects manipulated face videos on a 3D face model, which can easily evade
these face authentication services and liveness detection approaches. To this
end, we propose FaceLip, a novel liveness detection scheme for face
authentication, which utilizes unforgeable lip motion patterns built upon
well-designed acoustic signals to enable a strong security guarantee. The
unique lip motion patterns for each user are unforgeable because FaceLip
verifies the patterns by capturing and analyzing the acoustic signals that are
dynamically generated according to random challenges, which ensures that our
signals for liveness detection cannot be manipulated. Specially, we develop
robust algorithms for FaceLip to eliminate the impact of noisy signals in the
environment and thus can accurately infer the lip motions at larger distances.
We prototype FaceLip on off-the-shelf smartphones and conduct extensive
experiments under different settings. Our evaluation with 44 participants
validates the effectiveness and robustness of FaceLip.


http://arxiv.org/abs/2106.07904
Probabilistic Margins for Instance Reweighting in Adversarial Training. (8%)
Qizhou Wang; Feng Liu; Bo Han; Tongliang Liu; Chen Gong; Gang Niu; Mingyuan Zhou; Masashi Sugiyama
  Reweighting adversarial data during training has been recently shown to
improve adversarial robustness, where data closer to the current decision
boundaries are regarded as more critical and given larger weights. However,
existing methods measuring the closeness are not very reliable: they are
discrete and can take only a few values, and they are path-dependent, i.e.,
they may change given the same start and end points with different attack
paths. In this paper, we propose three types of probabilistic margin (PM),
which are continuous and path-independent, for measuring the aforementioned
closeness and reweighting adversarial data. Specifically, a PM is defined as
the difference between two estimated class-posterior probabilities, e.g., such
the probability of the true label minus the probability of the most confusing
label given some natural data. Though different PMs capture different geometric
properties, all three PMs share a negative correlation with the vulnerability
of data: data with larger/smaller PMs are safer/riskier and should have
smaller/larger weights. Experiments demonstrate that PMs are reliable
measurements and PM-based reweighting methods outperform state-of-the-art
methods.


http://arxiv.org/abs/2106.07895
CAN-LOC: Spoofing Detection and Physical Intrusion Localization on an In-Vehicle CAN Bus Based on Deep Features of Voltage Signals. (1%)
Efrat Levy; Asaf Shabtai; Bogdan Groza; Pal-Stefan Murvay; Yuval Elovici
  The Controller Area Network (CAN) is used for communication between
in-vehicle devices. The CAN bus has been shown to be vulnerable to remote
attacks. To harden vehicles against such attacks, vehicle manufacturers have
divided in-vehicle networks into sub-networks, logically isolating critical
devices. However, attackers may still have physical access to various
sub-networks where they can connect a malicious device. This threat has not
been adequately addressed, as methods proposed to determine physical intrusion
points have shown weak results, emphasizing the need to develop more advanced
techniques. To address this type of threat, we propose a security hardening
system for in-vehicle networks. The proposed system includes two mechanisms
that process deep features extracted from voltage signals measured on the CAN
bus. The first mechanism uses data augmentation and deep learning to detect and
locate physical intrusions when the vehicle starts; this mechanism can detect
and locate intrusions, even when the connected malicious devices are silent.
This mechanism's effectiveness (100% accuracy) is demonstrated in a wide
variety of insertion scenarios on a CAN bus prototype. The second mechanism is
a continuous device authentication mechanism, which is also based on deep
learning; this mechanism's robustness (99.8% accuracy) is demonstrated on a
real moving vehicle.


http://arxiv.org/abs/2106.07445
PopSkipJump: Decision-Based Attack for Probabilistic Classifiers. (99%)
Carl-Johann Simon-Gabriel; Noman Ahmed Sheikh; Andreas Krause
  Most current classifiers are vulnerable to adversarial examples, small input
perturbations that change the classification output. Many existing attack
algorithms cover various settings, from white-box to black-box classifiers, but
typically assume that the answers are deterministic and often fail when they
are not. We therefore propose a new adversarial decision-based attack
specifically designed for classifiers with probabilistic outputs. It is based
on the HopSkipJump attack by Chen et al. (2019, arXiv:1904.02144v5 ), a strong
and query efficient decision-based attack originally designed for deterministic
classifiers. Our P(robabilisticH)opSkipJump attack adapts its amount of queries
to maintain HopSkipJump's original output quality across various noise levels,
while converging to its query efficiency as the noise level decreases. We test
our attack on various noise models, including state-of-the-art off-the-shelf
randomized defenses, and show that they offer almost no extra robustness to
decision-based attacks. Code is available at
https://github.com/cjsg/PopSkipJump .


http://arxiv.org/abs/2106.07428
Audio Attacks and Defenses against AED Systems -- A Practical Study. (99%)
Rodrigo dos Santos; Shirin Nilizadeh
  Audio Event Detection (AED) Systems capture audio from the environment and
employ some deep learning algorithms for detecting the presence of a specific
sound of interest. In this paper, we evaluate deep learning-based AED systems
against evasion attacks through adversarial examples. We run multiple security
critical AED tasks, implemented as CNNs classifiers, and then generate audio
adversarial examples using two different types of noise, namely background and
white noise, that can be used by the adversary to evade detection. We also
examine the robustness of existing third-party AED capable devices, such as
Nest devices manufactured by Google, which run their own black-box deep
learning models.
  We show that an adversary can focus on audio adversarial inputs to cause AED
systems to misclassify, similarly to what has been previously done by works
focusing on adversarial examples from the image domain. We then, seek to
improve classifiers' robustness through countermeasures to the attacks. We
employ adversarial training and a custom denoising technique. We show that
these countermeasures, when applied to audio input, can be successful, either
in isolation or in combination, generating relevant increases of nearly fifty
percent in the performance of the classifiers when these are under attack.


http://arxiv.org/abs/2106.08153
Now You See It, Now You Dont: Adversarial Vulnerabilities in Computational Pathology. (99%)
Alex Foote; Amina Asif; Ayesha Azam; Tim Marshall-Cox; Nasir Rajpoot; Fayyaz Minhas
  Deep learning models are routinely employed in computational pathology
(CPath) for solving problems of diagnostic and prognostic significance.
Typically, the generalization performance of CPath models is analyzed using
evaluation protocols such as cross-validation and testing on multi-centric
cohorts. However, to ensure that such CPath solutions are robust and safe for
use in a clinical setting, a critical analysis of their predictive performance
and vulnerability to adversarial attacks is required, which is the focus of
this paper. Specifically, we show that a highly accurate model for
classification of tumour patches in pathology images (AUC > 0.95) can easily be
attacked with minimal perturbations which are imperceptible to lay humans and
trained pathologists alike. Our analytical results show that it is possible to
generate single-instance white-box attacks on specific input images with high
success rate and low perturbation energy. Furthermore, we have also generated a
single universal perturbation matrix using the training dataset only which,
when added to unseen test images, results in forcing the trained neural network
to flip its prediction labels with high confidence at a success rate of > 84%.
We systematically analyze the relationship between perturbation energy of an
adversarial attack, its impact on morphological constructs of clinical
significance, their perceptibility by a trained pathologist and saliency maps
obtained using deep learning models. Based on our analysis, we strongly
recommend that computational pathology models be critically analyzed using the
proposed adversarial validation strategy prior to clinical adoption.


http://arxiv.org/abs/2106.07214
Backdoor Learning Curves: Explaining Backdoor Poisoning Beyond Influence Functions. (92%)
Antonio Emanuele Cinà; Kathrin Grosse; Sebastiano Vascon; Ambra Demontis; Battista Biggio; Fabio Roli; Marcello Pelillo
  Backdoor attacks inject poisoning samples during training, with the goal of
enforcing a machine-learning model to output an attacker-chosen class when
presented a specific trigger at test time. Although backdoor attacks have been
demonstrated in a variety of settings and against different models, the factors
affecting their success are not yet well understood. In this work, we provide a
unifying framework to study the process of backdoor learning under the lens of
incremental learning and influence functions. We show that the success of
backdoor attacks inherently depends on (i) the complexity of the learning
algorithm, controlled by its hyperparameters, and (ii) the fraction of backdoor
samples injected into the training set. These factors affect how fast a
machine-learning model learns to correlate the presence of a backdoor trigger
with the target class. Interestingly, our analysis shows that there exists a
region in the hyperparameter space in which the accuracy on clean test samples
is still high while backdoor attacks become ineffective, thereby suggesting
novel criteria to improve existing defenses.


http://arxiv.org/abs/2106.07860
Evading Malware Classifiers via Monte Carlo Mutant Feature Discovery. (81%)
John Boutsikas; Maksim E. Eren; Charles Varga; Edward Raff; Cynthia Matuszek; Charles Nicholas
  The use of Machine Learning has become a significant part of malware
detection efforts due to the influx of new malware, an ever changing threat
landscape, and the ability of Machine Learning methods to discover meaningful
distinctions between malicious and benign software. Antivirus vendors have also
begun to widely utilize malware classifiers based on dynamic and static malware
analysis features. Therefore, a malware author might make evasive binary
modifications against Machine Learning models as part of the malware
development life cycle to execute an attack successfully. This makes the
studying of possible classifier evasion strategies an essential part of cyber
defense against malice. To this extent, we stage a grey box setup to analyze a
scenario where the malware author does not know the target classifier
algorithm, and does not have access to decisions made by the classifier, but
knows the features used in training. In this experiment, a malicious actor
trains a surrogate model using the EMBER-2018 dataset to discover binary
mutations that cause an instance to be misclassified via a Monte Carlo tree
search. Then, mutated malware is sent to the victim model that takes the place
of an antivirus API to test whether it can evade detection.


http://arxiv.org/abs/2106.07767
On the Relationship between Heterophily and Robustness of Graph Neural Networks. (81%)
Jiong Zhu; Junchen Jin; Donald Loveland; Michael T. Schaub; Danai Koutra
  Empirical studies on the robustness of graph neural networks (GNNs) have
suggested a relation between the vulnerabilities of GNNs to adversarial attacks
and the increased presence of heterophily in perturbed graphs (where edges tend
to connect nodes with dissimilar features and labels). In this work, we
formalize the relation between heterophily and robustness, bridging two topics
previously investigated by separate lines of research. We theoretically and
empirically show that for graphs exhibiting homophily (low heterophily),
impactful structural attacks always lead to increased levels of heterophily,
while for graph with heterophily the change in the homophily level depends on
the node degrees. By leveraging these insights, we deduce that a design
principle identified to significantly improve predictive performance under
heterophily -- separate aggregators for ego- and neighbor-embeddings -- can
also inherently offer increased robustness to GNNs. Our extensive empirical
analysis shows that GNNs adopting this design alone can achieve significantly
improved empirical and certifiable robustness compared to the best-performing
unvaccinated model. Furthermore, models with this design can be readily
combined with explicit defense mechanisms to yield improved robustness with up
to 18.33% increase in performance under attacks compared to the best-performing
vaccinated model.


http://arxiv.org/abs/2106.07411
Partial success in closing the gap between human and machine vision. (15%)
Robert Geirhos; Kantharaju Narayanappa; Benjamin Mitzkus; Tizian Thieringer; Matthias Bethge; Felix A. Wichmann; Wieland Brendel
  A few years ago, the first CNN surpassed human performance on ImageNet.
However, it soon became clear that machines lack robustness on more challenging
test cases, a major obstacle towards deploying machines "in the wild" and
towards obtaining better computational models of human visual perception. Here
we ask: Are we making progress in closing the gap between human and machine
vision? To answer this question, we tested human observers on a broad range of
out-of-distribution (OOD) datasets, adding the "missing human baseline" by
recording 85,120 psychophysical trials across 90 participants. We then
investigated a range of promising machine learning developments that crucially
deviate from standard supervised CNNs along three axes: objective function
(self-supervised, adversarially trained, CLIP language-image training),
architecture (e.g. vision transformers), and dataset size (ranging from 1M to
1B). Our findings are threefold. (1.) The longstanding robustness gap between
humans and CNNs is closing, with the best models now matching or exceeding
human performance on most OOD datasets. (2.) There is still a substantial
image-level consistency gap, meaning that humans make different errors than
models. In contrast, most models systematically agree in their categorisation
errors, even substantially different ones like contrastive self-supervised vs.
standard supervised models. (3.) In many cases, human-to-model consistency
improves when training dataset size is increased by one to three orders of
magnitude. Our results give reason for cautious optimism: While there is still
much room for improvement, the behavioural difference between human and machine
vision is narrowing. In order to measure future progress, 17 OOD datasets with
image-level human behavioural data are provided as a benchmark here:
https://github.com/bethgelab/model-vs-human/


http://arxiv.org/abs/2106.07704
Text Generation with Efficient (Soft) Q-Learning. (2%)
Han Guo; Bowen Tan; Zhengzhong Liu; Eric P. Xing; Zhiting Hu
  Maximum likelihood estimation (MLE) is the predominant algorithm for training
text generation models. This paradigm relies on direct supervision examples,
which is not applicable to many emerging applications, such as generating
adversarial attacks or generating prompts to control language models.
Reinforcement learning (RL) on the other hand offers a more flexible solution
by allowing users to plug in arbitrary task metrics as reward. Yet previous RL
algorithms for text generation, such as policy gradient (on-policy RL) and
Q-learning (off-policy RL), are often notoriously inefficient or unstable to
train due to the large sequence space and the sparse reward received only at
the end of sequences. In this paper, we introduce a new RL formulation for text
generation from the soft Q-learning (SQL) perspective. It enables us to draw
from the latest RL advances, such as path consistency learning, to combine the
best of on-/off-policy updates, and learn effectively from sparse reward. We
apply the approach to a wide range of text generation tasks, including learning
from noisy/negative examples, adversarial attacks, and prompt generation.
Experiments show our approach consistently outperforms both task-specialized
algorithms and the previous RL methods.


http://arxiv.org/abs/2106.07541
Resilient Control of Platooning Networked Robitic Systems via Dynamic Watermarking. (1%)
Matthew Porter; Arnav Joshi; Sidhartha Dey; Qirui Wu; Pedro Hespanhol; Anil Aswani; Matthew Johnson-Roberson; Ram Vasudevan
  Networked robotic systems, such as connected vehicle platoons, can improve
the safety and efficiency of transportation networks by allowing for high-speed
coordination. To enable such coordination, these systems rely on networked
communications. This can make them susceptible to cyber attacks. Though
security methods such as encryption or specially designed network topologies
can increase the difficulty of successfully executing such an attack, these
techniques are unable to guarantee secure communication against an attacker.
More troublingly, these security methods are unable to ensure that individual
agents are able to detect attacks that alter the content of specific messages.
To ensure resilient behavior under such attacks, this paper formulates a
networked linear time-varying version of dynamic watermarking in which each
agent generates and adds a private excitation to the input of its corresponding
robotic subsystem. This paper demonstrates that such a method can enable each
agent in a networked robotic system to detect cyber attacks. By altering
measurements sent between vehicles, this paper illustrates that an attacker can
create unstable behavior within a platoon. By utilizing the dynamic
watermarking method proposed in this paper, the attack is detected, allowing
the vehicles in the platoon to gracefully degrade to a non-communicative
control strategy that maintains safety across a variety of scenarios.


http://arxiv.org/abs/2106.07165
Self-training Guided Adversarial Domain Adaptation For Thermal Imagery. (1%)
Ibrahim Batuhan Akkaya; Fazil Altinel; Ugur Halici
  Deep models trained on large-scale RGB image datasets have shown tremendous
success. It is important to apply such deep models to real-world problems.
However, these models suffer from a performance bottleneck under illumination
changes. Thermal IR cameras are more robust against such changes, and thus can
be very useful for the real-world problems. In order to investigate efficacy of
combining feature-rich visible spectrum and thermal image modalities, we
propose an unsupervised domain adaptation method which does not require
RGB-to-thermal image pairs. We employ large-scale RGB dataset MS-COCO as source
domain and thermal dataset FLIR ADAS as target domain to demonstrate results of
our method. Although adversarial domain adaptation methods aim to align the
distributions of source and target domains, simply aligning the distributions
cannot guarantee perfect generalization to the target domain. To this end, we
propose a self-training guided adversarial domain adaptation method to promote
generalization capabilities of adversarial domain adaptation methods. To
perform self-training, pseudo labels are assigned to the samples on the target
thermal domain to learn more generalized representations for the target domain.
Extensive experimental analyses show that our proposed method achieves better
results than the state-of-the-art adversarial domain adaptation methods. The
code and models are publicly available.


http://arxiv.org/abs/2106.07851
Code Integrity Attestation for PLCs using Black Box Neural Network Predictions. (1%)
Yuqi Chen; Christopher M. Poskitt; Jun Sun
  Cyber-physical systems (CPSs) are widespread in critical domains, and
significant damage can be caused if an attacker is able to modify the code of
their programmable logic controllers (PLCs). Unfortunately, traditional
techniques for attesting code integrity (i.e. verifying that it has not been
modified) rely on firmware access or roots-of-trust, neither of which
proprietary or legacy PLCs are likely to provide. In this paper, we propose a
practical code integrity checking solution based on privacy-preserving black
box models that instead attest the input/output behaviour of PLC programs.
Using faithful offline copies of the PLC programs, we identify their most
important inputs through an information flow analysis, execute them on multiple
combinations to collect data, then train neural networks able to predict PLC
outputs (i.e. actuator commands) from their inputs. By exploiting the black box
nature of the model, our solution maintains the privacy of the original PLC
code and does not assume that attackers are unaware of its presence. The trust
instead comes from the fact that it is extremely hard to attack the PLC code
and neural networks at the same time and with consistent outcomes. We evaluated
our approach on a modern six-stage water treatment plant testbed, finding that
it could predict actuator states from PLC inputs with near-100% accuracy, and
thus could detect all 120 effective code mutations that we subjected the PLCs
to. Finally, we found that it is not practically possible to simultaneously
modify the PLC code and apply discreet adversarial noise to our attesters in a
way that leads to consistent (mis-)predictions.


http://arxiv.org/abs/2106.07047
Target Model Agnostic Adversarial Attacks with Query Budgets on Language Understanding Models. (99%)
Jatin Chauhan; Karan Bhukar; Manohar Kaul
  Despite significant improvements in natural language understanding models
with the advent of models like BERT and XLNet, these neural-network based
classifiers are vulnerable to blackbox adversarial attacks, where the attacker
is only allowed to query the target model outputs. We add two more realistic
restrictions on the attack methods, namely limiting the number of queries
allowed (query budget) and crafting attacks that easily transfer across
different pre-trained models (transferability), which render previous attack
models impractical and ineffective. Here, we propose a target model agnostic
adversarial attack method with a high degree of attack transferability across
the attacked models. Our empirical studies show that in comparison to baseline
methods, our method generates highly transferable adversarial sentences under
the restriction of limited query budgets.


http://arxiv.org/abs/2106.07141
Selection of Source Images Heavily Influences the Effectiveness of Adversarial Attacks. (99%)
Utku Ozbulak; Esla Timothy Anzaku; Neve Wesley De; Messem Arnout Van
  Although the adoption rate of deep neural networks (DNNs) has tremendously
increased in recent years, a solution for their vulnerability against
adversarial examples has not yet been found. As a result, substantial research
efforts are dedicated to fix this weakness, with many studies typically using a
subset of source images to generate adversarial examples, treating every image
in this subset as equal. We demonstrate that, in fact, not every source image
is equally suited for this kind of assessment. To do so, we devise a
large-scale model-to-model transferability scenario for which we meticulously
analyze the properties of adversarial examples, generated from every suitable
source image in ImageNet by making use of two of the most frequently deployed
attacks. In this transferability scenario, which involves seven distinct DNN
models, including the recently proposed vision transformers, we reveal that it
is possible to have a difference of up to $12.5\%$ in model-to-model
transferability success, $1.01$ in average $L_2$ perturbation, and $0.03$
($8/225$) in average $L_{\infty}$ perturbation when $1,000$ source images are
sampled randomly among all suitable candidates. We then take one of the first
steps in evaluating the robustness of images used to create adversarial
examples, proposing a number of simple but effective methods to identify
unsuitable source images, thus making it possible to mitigate extreme cases in
experimentation and support high-quality benchmarking.


http://arxiv.org/abs/2106.06917
ATRAS: Adversarially Trained Robust Architecture Search. (96%)
Yigit Alparslan; Edward Kim
  In this paper, we explore the effect of architecture completeness on
adversarial robustness. We train models with different architectures on
CIFAR-10 and MNIST dataset. For each model, we vary different number of layers
and different number of nodes in the layer. For every architecture candidate,
we use Fast Gradient Sign Method (FGSM) to generate untargeted adversarial
attacks and use adversarial training to defend against those attacks. For each
architecture candidate, we report pre-attack, post-attack and post-defense
accuracy for the model as well as the architecture parameters and the impact of
completeness to the model accuracies.


http://arxiv.org/abs/2106.07098
Security Analysis of Camera-LiDAR Semantic-Level Fusion Against Black-Box Attacks on Autonomous Vehicles. (64%)
R. Spencer Hallyburton; Yupei Liu; Miroslav Pajic
  To enable safe and reliable decision-making, autonomous vehicles (AVs) feed
sensor data to perception algorithms to understand the environment. Sensor
fusion, and particularly semantic fusion, with multi-frame tracking is becoming
increasingly popular for detecting 3D objects. Recently, it was shown that
LiDAR-based perception built on deep neural networks is vulnerable to LiDAR
spoofing attacks. Thus, in this work, we perform the first analysis of
camera-LiDAR fusion under spoofing attacks and the first security analysis of
semantic fusion in any AV context. We find first that fusion is more successful
than existing defenses at guarding against naive spoofing. However, we then
define the frustum attack as a new class of attacks on AVs and find that
semantic camera-LiDAR fusion exhibits widespread vulnerability to frustum
attacks with between 70% and 90% success against target models. Importantly,
the attacker needs less than 20 random spoof points on average for successful
attacks - an order of magnitude less than established maximum capability.
Finally, we are the first to analyze the longitudinal impact of perception
attacks by showing the impact of multi-frame attacks.


http://arxiv.org/abs/2106.07049
Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis. (1%)
Kangning Liu; Yiqiu Shen; Nan Wu; Jakub Chłędowski; Carlos Fernandez-Granda; Krzysztof J. Geras
  In the last few years, deep learning classifiers have shown promising results
in image-based medical diagnosis. However, interpreting the outputs of these
models remains a challenge. In cancer diagnosis, interpretability can be
achieved by localizing the region of the input image responsible for the
output, i.e. the location of a lesion. Alternatively, segmentation or detection
models can be trained with pixel-wise annotations indicating the locations of
malignant lesions. Unfortunately, acquiring such labels is labor-intensive and
requires medical expertise. To overcome this difficulty, weakly-supervised
localization can be utilized. These methods allow neural network classifiers to
output saliency maps highlighting the regions of the input most relevant to the
classification task (e.g. malignant lesions in mammograms) using only
image-level labels (e.g. whether the patient has cancer or not) during
training. When applied to high-resolution images, existing methods produce
low-resolution saliency maps. This is problematic in applications in which
suspicious lesions are small in relation to the image size. In this work, we
introduce a novel neural network architecture to perform weakly-supervised
segmentation of high-resolution images. The proposed model selects regions of
interest via coarse-level localization, and then performs fine-grained
segmentation of those regions. We apply this model to breast cancer diagnosis
with screening mammography, and validate it on a large clinically-realistic
dataset. Measured by Dice similarity score, our approach outperforms existing
methods by a large margin in terms of localization performance of benign and
malignant lesions, relatively improving the performance by 39.6% and 20.0%,
respectively. Code and the weights of some of the models are available at
https://github.com/nyukat/GLAM


http://arxiv.org/abs/2106.07068
HistoTransfer: Understanding Transfer Learning for Histopathology. (1%)
Yash Sharma; Lubaina Ehsan; Sana Syed; Donald E. Brown
  Advancement in digital pathology and artificial intelligence has enabled deep
learning-based computer vision techniques for automated disease diagnosis and
prognosis. However, WSIs present unique computational and algorithmic
challenges. WSIs are gigapixel-sized, making them infeasible to be used
directly for training deep neural networks. Hence, for modeling, a two-stage
approach is adopted: Patch representations are extracted first, followed by the
aggregation for WSI prediction. These approaches require detailed pixel-level
annotations for training the patch encoder. However, obtaining these
annotations is time-consuming and tedious for medical experts. Transfer
learning is used to address this gap and deep learning architectures
pre-trained on ImageNet are used for generating patch-level representation.
Even though ImageNet differs significantly from histopathology data,
pre-trained networks have been shown to perform impressively on histopathology
data. Also, progress in self-supervised and multi-task learning coupled with
the release of multiple histopathology data has led to the release of
histopathology-specific networks. In this work, we compare the performance of
features extracted from networks trained on ImageNet and histopathology data.
We use an attention pooling network over these extracted features for
slide-level aggregation. We investigate if features learned using more complex
networks lead to gain in performance. We use a simple top-k sampling approach
for fine-tuning framework and study the representation similarity between
frozen and fine-tuned networks using Centered Kernel Alignment. Further, to
examine if intermediate block representation is better suited for feature
extraction and ImageNet architectures are unnecessarily large for
histopathology, we truncate the blocks of ResNet18 and DenseNet121 and examine
the performance.


http://arxiv.org/abs/2106.06685
Adversarial Robustness via Fisher-Rao Regularization. (54%)
Marine Picot; Francisco Messina; Malik Boudiaf; Fabrice Labeau; Ismail Ben Ayed; Pablo Piantanida
  Adversarial robustness has become a topic of growing interest in machine
learning since it was observed that neural networks tend to be brittle. We
propose an information-geometric formulation of adversarial defense and
introduce FIRE, a new Fisher-Rao regularization for the categorical
cross-entropy loss, which is based on the geodesic distance between natural and
perturbed input features. Based on the information-geometric properties of the
class of softmax distributions, we derive an explicit characterization of the
Fisher-Rao Distance (FRD) for the binary and multiclass cases, and draw some
interesting properties as well as connections with standard regularization
metrics. Furthermore, for a simple linear and Gaussian model, we show that all
Pareto-optimal points in the accuracy-robustness region can be reached by FIRE
while other state-of-the-art methods fail. Empirically, we evaluate the
performance of various classifiers trained with the proposed loss on standard
datasets, showing up to 2\% of improvements in terms of robustness while
reducing the training time by 20\% over the best-performing methods.


http://arxiv.org/abs/2106.06770
What can linearized neural networks actually say about generalization? (31%)
Guillermo Ortiz-Jiménez; Seyed-Mohsen Moosavi-Dezfooli; Pascal Frossard
  For certain infinitely-wide neural networks, the neural tangent kernel (NTK)
theory fully characterizes generalization. However, for the networks used in
practice, the empirical NTK represents only a rough first-order approximation
of these architectures. Still, a growing body of work keeps leveraging this
approximation to successfully analyze important deep learning phenomena and
derive algorithms for new applications. In our work, we provide strong
empirical evidence to determine the practical validity of such approximation by
conducting a systematic comparison of the behaviour of different neural
networks and their linear approximations on different tasks. We show that the
linear approximations can indeed rank the learning complexity of certain tasks
for neural networks, albeit with important nuances. Specifically, we discover
that, in contrast to what was previously observed, neural networks do not
always perform better than their kernel approximations, and reveal that their
performance gap heavily depends on architecture, number of samples and training
task. In fact, we show that during training, deep networks increase the
alignment of their empirical NTK with the target task, which explains why
linear approximations at the end of training can better explain the dynamics of
deep networks. Overall, our work provides concrete examples of novel deep
learning phenomena which can inspire future theoretical research, as well as
provides a new perspective on the use of the NTK approximation in deep
learning.


http://arxiv.org/abs/2106.06895
FeSHI: Feature Map Based Stealthy Hardware Intrinsic Attack. (2%)
Tolulope Odetola; Faiq Khalid; Travis Sandefur; Hawzhin Mohammed; Syed Rafay Hasan
  Convolutional Neural Networks (CNN) have shown impressive performance in
computer vision, natural language processing, and many other applications, but
they exhibit high computations and substantial memory requirements. To address
these limitations, especially in resource-constrained devices, the use of cloud
computing for CNNs is becoming more popular. This comes with privacy and
latency concerns that have motivated the designers to develop embedded hardware
accelerators for CNNs. However, designing a specialized accelerator increases
the time-to-market and cost of production. Therefore, to reduce the
time-to-market and access to state-of-the-art techniques, CNN hardware mapping
and deployment on embedded accelerators are often outsourced to untrusted third
parties, which is going to be more prevalent in futuristic artificial
intelligence of things (AIoT) systems. These AIoT systems anticipate horizontal
collaboration among different resource-constrained AIoT node devices, where CNN
layers are partitioned and these devices collaboratively compute complex CNN
tasks Therefore, there is a dire need to explore this attack surface for
designing secure embedded hardware accelerators for CNNs. Towards this goal, in
this paper, we exploited this attack surface to propose an HT-based attack
called FeSHI. This attack exploits the statistical distribution i.e., Gaussian
distribution, of the layer-by-layer feature maps of the CNN to design two
triggers for stealthy HT with a very low probability of triggering. To
illustrate the effectiveness of the proposed attack, we deployed the LeNet and
LeNet-3D on PYNQ to classify the MNIST and CIFAR-10 datasets, respectively, and
tested FeSHI. The experimental results show that FeSHI utilizes up to 2% extra
LUTs, and the overall resource overhead is less than 1% compared to the
original designs


http://arxiv.org/abs/2106.06196
Adversarial Robustness through the Lens of Causality. (99%)
Yonggang Zhang; Mingming Gong; Tongliang Liu; Gang Niu; Xinmei Tian; Bo Han; Bernhard Schölkopf; Kun Zhang
  The adversarial vulnerability of deep neural networks has attracted
significant attention in machine learning. From a causal viewpoint, adversarial
attacks can be considered as a specific type of distribution change on natural
data. As causal reasoning has an instinct for modeling distribution change, we
propose to incorporate causality into mitigating adversarial vulnerability.
However, causal formulations of the intuition of adversarial attack and the
development of robust DNNs are still lacking in the literature. To bridge this
gap, we construct a causal graph to model the generation process of adversarial
examples and define the adversarial distribution to formalize the intuition of
adversarial attacks. From a causal perspective, we find that the label is
spuriously correlated with the style (content-independent) information when an
instance is given. The spurious correlation implies that the adversarial
distribution is constructed via making the statistical conditional association
between style information and labels drastically different from that in natural
distribution. Thus, DNNs that fit the spurious correlation are vulnerable to
the adversarial distribution. Inspired by the observation, we propose the
adversarial distribution alignment method to eliminate the difference between
the natural distribution and the adversarial distribution. Extensive
experiments demonstrate the efficacy of the proposed method. Our method can be
seen as the first attempt to leverage causality for mitigating adversarial
vulnerability.


http://arxiv.org/abs/2106.06235
Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attacks. (99%)
Nezihe Merve Gürel; Xiangyu Qi; Luka Rimanic; Ce Zhang; Bo Li
  Despite the great successes achieved by deep neural networks (DNNs), recent
studies show that they are vulnerable against adversarial examples, which aim
to mislead DNNs by adding small adversarial perturbations. Several defenses
have been proposed against such attacks, while many of them have been
adaptively attacked. In this work, we aim to enhance the ML robustness from a
different perspective by leveraging domain knowledge: We propose a Knowledge
Enhanced Machine Learning Pipeline (KEMLP) to integrate domain knowledge (i.e.,
logic relationships among different predictions) into a probabilistic graphical
model via first-order logic rules. In particular, we develop KEMLP by
integrating a diverse set of weak auxiliary models based on their logical
relationships to the main DNN model that performs the target task.
Theoretically, we provide convergence results and prove that, under mild
conditions, the prediction of KEMLP is more robust than that of the main DNN
model. Empirically, we take road sign recognition as an example and leverage
the relationships between road signs and their shapes and contents as domain
knowledge. We show that compared with adversarial training and other baselines,
KEMLP achieves higher robustness against physical attacks, $\mathcal{L}_p$
bounded attacks, unforeseen attacks, and natural corruptions under both
whitebox and blackbox settings, while still maintaining high clean accuracy.


http://arxiv.org/abs/2106.06041
Adversarial purification with Score-based generative models. (89%)
Jongmin Yoon; Sung Ju Hwang; Juho Lee
  While adversarial training is considered as a standard defense method against
adversarial attacks for image classifiers, adversarial purification, which
purifies attacked images into clean images with a standalone purification
model, has shown promises as an alternative defense method. Recently, an
Energy-Based Model (EBM) trained with Markov-Chain Monte-Carlo (MCMC) has been
highlighted as a purification model, where an attacked image is purified by
running a long Markov-chain using the gradients of the EBM. Yet, the
practicality of the adversarial purification using an EBM remains questionable
because the number of MCMC steps required for such purification is too large.
In this paper, we propose a novel adversarial purification method based on an
EBM trained with Denoising Score-Matching (DSM). We show that an EBM trained
with DSM can quickly purify attacked images within a few steps. We further
introduce a simple yet effective randomized purification scheme that injects
random noises into images before purification. This process screens the
adversarial perturbations imposed on images by the random noises and brings the
images to the regime where the EBM can denoise well. We show that our
purification method is robust against various attacks and demonstrate its
state-of-the-art performances.


http://arxiv.org/abs/2106.06624
Relaxing Local Robustness. (80%)
Klas Leino; Matt Fredrikson
  Certifiable local robustness, which rigorously precludes small-norm
adversarial examples, has received significant attention as a means of
addressing security concerns in deep learning. However, for some classification
problems, local robustness is not a natural objective, even in the presence of
adversaries; for example, if an image contains two classes of subjects, the
correct label for the image may be considered arbitrary between the two, and
thus enforcing strict separation between them is unnecessary. In this work, we
introduce two relaxed safety properties for classifiers that address this
observation: (1) relaxed top-k robustness, which serves as the analogue of
top-k accuracy; and (2) affinity robustness, which specifies which sets of
labels must be separated by a robustness margin, and which can be
$\epsilon$-close in $\ell_p$ space. We show how to construct models that can be
efficiently certified against each relaxed robustness property, and trained
with very little overhead relative to standard gradient descent. Finally, we
demonstrate experimentally that these relaxed variants of robustness are
well-suited to several significant classification problems, leading to lower
rejection rates and higher certified accuracies than can be obtained when
certifying "standard" local robustness.


http://arxiv.org/abs/2106.06663
TDGIA:Effective Injection Attacks on Graph Neural Networks. (76%)
Xu Zou; Qinkai Zheng; Yuxiao Dong; Xinyu Guan; Evgeny Kharlamov; Jialiang Lu; Jie Tang
  Graph Neural Networks (GNNs) have achieved promising performance in various
real-world applications. However, recent studies have shown that GNNs are
vulnerable to adversarial attacks. In this paper, we study a
recently-introduced realistic attack scenario on graphs -- graph injection
attack (GIA). In the GIA scenario, the adversary is not able to modify the
existing link structure and node attributes of the input graph, instead the
attack is performed by injecting adversarial nodes into it. We present an
analysis on the topological vulnerability of GNNs under GIA setting, based on
which we propose the Topological Defective Graph Injection Attack (TDGIA) for
effective injection attacks. TDGIA first introduces the topological defective
edge selection strategy to choose the original nodes for connecting with the
injected ones. It then designs the smooth feature optimization objective to
generate the features for the injected nodes. Extensive experiments on
large-scale datasets show that TDGIA can consistently and significantly
outperform various attack baselines in attacking dozens of defense GNN models.
Notably, the performance drop on target GNNs resultant from TDGIA is more than
double the damage brought by the best attack solution among hundreds of
submissions on KDD-CUP 2020.


http://arxiv.org/abs/2106.06361
Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution. (56%)
Fanchao Qi; Yuan Yao; Sophia Xu; Zhiyuan Liu; Maosong Sun
  Recent studies show that neural natural language processing (NLP) models are
vulnerable to backdoor attacks. Injected with backdoors, models perform
normally on benign examples but produce attacker-specified predictions when the
backdoor is activated, presenting serious security threats to real-world
applications. Since existing textual backdoor attacks pay little attention to
the invisibility of backdoors, they can be easily detected and blocked. In this
work, we present invisible backdoors that are activated by a learnable
combination of word substitution. We show that NLP models can be injected with
backdoors that lead to a nearly 100% attack success rate, whereas being highly
invisible to existing defense strategies and even human inspections. The
results raise a serious alarm to the security of NLP models, which requires
further research to be resolved. All the data and code of this paper are
released at https://github.com/thunlp/BkdAtk-LWS.


http://arxiv.org/abs/2106.06667
CARTL: Cooperative Adversarially-Robust Transfer Learning. (8%)
Dian Chen; Hongxin Hu; Qian Wang; Yinli Li; Cong Wang; Chao Shen; Qi Li
  Transfer learning eases the burden of training a well-performed model from
scratch, especially when training data is scarce and computation power is
limited. In deep learning, a typical strategy for transfer learning is to
freeze the early layers of a pre-trained model and fine-tune the rest of its
layers on the target domain. Previous work focuses on the accuracy of the
transferred model but neglects the transfer of adversarial robustness. In this
work, we first show that transfer learning improves the accuracy on the target
domain but degrades the inherited robustness of the target model. To address
such a problem, we propose a novel cooperative adversarially-robust transfer
learning (CARTL) by pre-training the model via feature distance minimization
and fine-tuning the pre-trained model with non-expansive fine-tuning for target
domain tasks. Empirical results show that CARTL improves the inherited
robustness by about 28% at most compared with the baseline with the same degree
of accuracy. Furthermore, we study the relationship between the batch
normalization (BN) layers and the robustness in the context of transfer
learning, and we reveal that freezing BN layers can further boost the
robustness transfer.


http://arxiv.org/abs/2106.06603
A Shuffling Framework for Local Differential Privacy. (1%)
Casey Meehan; Amrita Roy Chowdhury; Kamalika Chaudhuri; Somesh Jha
  ldp deployments are vulnerable to inference attacks as an adversary can link
the noisy responses to their identity and subsequently, auxiliary information
using the order of the data. An alternative model, shuffle DP, prevents this by
shuffling the noisy responses uniformly at random. However, this limits the
data learnability -- only symmetric functions (input order agnostic) can be
learned. In this paper, we strike a balance and propose a generalized shuffling
framework that interpolates between the two deployment models. We show that
systematic shuffling of the noisy responses can thwart specific inference
attacks while retaining some meaningful data learnability. To this end, we
propose a novel privacy guarantee, d-sigma privacy, that captures the privacy
of the order of a data sequence. d-sigma privacy allows tuning the granularity
at which the ordinal information is maintained, which formalizes the degree the
resistance to inference attacks trading it off with data learnability.
Additionally, we propose a novel shuffling mechanism that can achieve d-sigma
privacy and demonstrate the practicality of our mechanism via evaluation on
real-world datasets.


http://arxiv.org/abs/2106.06027
Sparse and Imperceptible Adversarial Attack via a Homotopy Algorithm. (99%)
Mingkang Zhu; Tianlong Chen; Zhangyang Wang
  Sparse adversarial attacks can fool deep neural networks (DNNs) by only
perturbing a few pixels (regularized by l_0 norm). Recent efforts combine it
with another l_infty imperceptible on the perturbation magnitudes. The
resultant sparse and imperceptible attacks are practically relevant, and
indicate an even higher vulnerability of DNNs that we usually imagined.
However, such attacks are more challenging to generate due to the optimization
difficulty by coupling the l_0 regularizer and box constraints with a
non-convex objective. In this paper, we address this challenge by proposing a
homotopy algorithm, to jointly tackle the sparsity and the perturbation bound
in one unified framework. Each iteration, the main step of our algorithm is to
optimize an l_0-regularized adversarial loss, by leveraging the nonmonotone
Accelerated Proximal Gradient Method (nmAPG) for nonconvex programming; it is
followed by an l_0 change control step, and an optional post-attack step
designed to escape bad local minima. We also extend the algorithm to handling
the structural sparsity regularizer. We extensively examine the effectiveness
of our proposed homotopy attack for both targeted and non-targeted attack
scenarios, on CIFAR-10 and ImageNet datasets. Compared to state-of-the-art
methods, our homotopy attack leads to significantly fewer perturbations, e.g.,
reducing 42.91% on CIFAR-10 and 75.03% on ImageNet (average case, targeted
attack), at similar maximal perturbation magnitudes, when still achieving 100%
attack success rates. Our codes are available at:
https://github.com/VITA-Group/SparseADV_Homotopy.


http://arxiv.org/abs/2106.05657
Deep neural network loses attention to adversarial images. (99%)
Shashank Kotyan; Danilo Vasconcellos Vargas
  Adversarial algorithms have shown to be effective against neural networks for
a variety of tasks. Some adversarial algorithms perturb all the pixels in the
image minimally for the image classification task in image classification. In
contrast, some algorithms perturb few pixels strongly. However, very little
information is available regarding why these adversarial samples so diverse
from each other exist. Recently, Vargas et al. showed that the existence of
these adversarial samples might be due to conflicting saliency within the
neural network. We test this hypothesis of conflicting saliency by analysing
the Saliency Maps (SM) and Gradient-weighted Class Activation Maps (Grad-CAM)
of original and few different types of adversarial samples. We also analyse how
different adversarial samples distort the attention of the neural network
compared to original samples. We show that in the case of Pixel Attack,
perturbed pixels either calls the network attention to themselves or divert the
attention from them. Simultaneously, the Projected Gradient Descent Attack
perturbs pixels so that intermediate layers inside the neural network lose
attention for the correct class. We also show that both attacks affect the
saliency map and activation maps differently. Thus, shedding light on why some
defences successful against some attacks remain vulnerable against other
attacks. We hope that this analysis will improve understanding of the existence
and the effect of adversarial samples and enable the community to develop more
robust neural networks.


http://arxiv.org/abs/2106.05997
Verifying Quantized Neural Networks using SMT-Based Model Checking. (92%)
Luiz Sena; Xidan Song; Erickson Alves; Iury Bessa; Edoardo Manino; Lucas Cordeiro; Eddie de Lima Filho
  Artificial Neural Networks (ANNs) are being deployed for an increasing number
of safety-critical applications, including autonomous cars and medical
diagnosis. However, concerns about their reliability have been raised due to
their black-box nature and apparent fragility to adversarial attacks. These
concerns are amplified when ANNs are deployed on restricted system, which limit
the precision of mathematical operations and thus introduce additional
quantization errors. Here, we develop and evaluate a novel symbolic
verification framework using software model checking (SMC) and satisfiability
modulo theories (SMT) to check for vulnerabilities in ANNs. More specifically,
we propose several ANN-related optimizations for SMC, including invariant
inference via interval analysis, slicing, expression simplifications, and
discretization of non-linear activation functions. With this verification
framework, we can provide formal guarantees on the safe behavior of ANNs
implemented both in floating- and fixed-point arithmetic. In this regard, our
verification approach was able to verify and produce adversarial examples for
$52$ test cases spanning image classification and general machine learning
applications. Furthermore, for small- to medium-sized ANN, our approach
completes most of its verification runs in minutes. Moreover, in contrast to
most state-of-the-art methods, our approach is not restricted to specific
choices regarding activation functions and non-quantized representations. Our
experiments show that our approach can analyze larger ANN implementations and
substantially reduce the verification time compared to state-of-the-art
techniques that use SMT solving.


http://arxiv.org/abs/2106.06056
Progressive-Scale Boundary Blackbox Attack via Projective Gradient Estimation. (80%)
Jiawei Zhang; Linyi Li; Huichen Li; Xiaolu Zhang; Shuang Yang; Bo Li
  Boundary based blackbox attack has been recognized as practical and
effective, given that an attacker only needs to access the final model
prediction. However, the query efficiency of it is in general high especially
for high dimensional image data. In this paper, we show that such efficiency
highly depends on the scale at which the attack is applied, and attacking at
the optimal scale significantly improves the efficiency. In particular, we
propose a theoretical framework to analyze and show three key characteristics
to improve the query efficiency. We prove that there exists an optimal scale
for projective gradient estimation. Our framework also explains the
satisfactory performance achieved by existing boundary black-box attacks. Based
on our theoretical framework, we propose Progressive-Scale enabled projective
Boundary Attack (PSBA) to improve the query efficiency via progressive scaling
techniques. In particular, we employ Progressive-GAN to optimize the scale of
projections, which we call PSBA-PGAN. We evaluate our approach on both spatial
and frequency scales. Extensive experiments on MNIST, CIFAR-10, CelebA, and
ImageNet against different models including a real-world face recognition API
show that PSBA-PGAN significantly outperforms existing baseline attacks in
terms of query efficiency and attack success rate. We also observe relatively
stable optimal scales for different models and datasets. The code is publicly
available at https://github.com/AI-secure/PSBA.


http://arxiv.org/abs/2106.05996
An Ensemble Approach Towards Adversarial Robustness. (41%)
Haifeng Qian
  It is a known phenomenon that adversarial robustness comes at a cost to
natural accuracy. To improve this trade-off, this paper proposes an ensemble
approach that divides a complex robust-classification task into simpler
subtasks. Specifically, fractal divide derives multiple training sets from the
training data, and fractal aggregation combines inference outputs from multiple
classifiers that are trained on those sets. The resulting ensemble classifiers
have a unique property that ensures robustness for an input if certain
don't-care conditions are met. The new techniques are evaluated on MNIST and
Fashion-MNIST, with no adversarial training. The MNIST classifier has 99%
natural accuracy, 70% measured robustness and 36.9% provable robustness, within
L2 distance of 2. The Fashion-MNIST classifier has 90% natural accuracy, 54.5%
measured robustness and 28.2% provable robustness, within L2 distance of 1.5.
Both results are new state of the art, and we also present new state-of-the-art
binary results on challenging label-pairs.


http://arxiv.org/abs/2106.05625
Towards an Automated Pipeline for Detecting and Classifying Malware through Machine Learning. (1%)
Nicola Loi; Claudio Borile; Daniele Ucci
  The constant growth in the number of malware - software or code fragment
potentially harmful for computers and information networks - and the use of
sophisticated evasion and obfuscation techniques have seriously hindered
classic signature-based approaches. On the other hand, malware detection
systems based on machine learning techniques started offering a promising
alternative to standard approaches, drastically reducing analysis time and
turning out to be more robust against evasion and obfuscation techniques. In
this paper, we propose a malware taxonomic classification pipeline able to
classify Windows Portable Executable files (PEs). Given an input PE sample, it
is first classified as either malicious or benign. If malicious, the pipeline
further analyzes it in order to establish its threat type, family, and
behavior(s). We tested the proposed pipeline on the open source dataset EMBER,
containing approximately 1 million PE samples, analyzed through static
analysis. Obtained malware detection results are comparable to other academic
works in the current state of art and, in addition, we provide an in-depth
classification of malicious samples. Models used in the pipeline provides
interpretable results which can help security analysts in better understanding
decisions taken by the automated pipeline.


http://arxiv.org/abs/2106.05964
Fair Classification with Adversarial Perturbations. (1%)
L. Elisa Celis; Anay Mehrotra; Nisheeth K. Vishnoi
  We study fair classification in the presence of an omniscient adversary that,
given an $\eta$, is allowed to choose an arbitrary $\eta$-fraction of the
training samples and arbitrarily perturb their protected attributes. The
motivation comes from settings in which protected attributes can be incorrect
due to strategic misreporting, malicious actors, or errors in imputation; and
prior approaches that make stochastic or independence assumptions on errors may
not satisfy their guarantees in this adversarial setting. Our main contribution
is an optimization framework to learn fair classifiers in this adversarial
setting that comes with provable guarantees on accuracy and fairness. Our
framework works with multiple and non-binary protected attributes, is designed
for the large class of linear-fractional fairness metrics, and can also handle
perturbations besides protected attributes. We prove near-tightness of our
framework's guarantees for natural hypothesis classes: no algorithm can have
significantly better accuracy and any algorithm with better fairness must have
lower accuracy. Empirically, we evaluate the classifiers produced by our
framework for statistical rate on real-world and synthetic datasets for a
family of adversaries.


http://arxiv.org/abs/2106.05825
HASI: Hardware-Accelerated Stochastic Inference, A Defense Against Adversarial Machine Learning Attacks. (99%)
Mohammad Hossein Samavatian; Saikat Majumdar; Kristin Barber; Radu Teodorescu
  DNNs are known to be vulnerable to so-called adversarial attacks, in which
inputs are carefully manipulated to induce misclassification. Existing defenses
are mostly software-based and come with high overheads or other limitations.
This paper presents HASI, a hardware-accelerated defense that uses a process we
call stochastic inference to detect adversarial inputs. HASI carefully injects
noise into the model at inference time and used the model's response to
differentiate adversarial inputs from benign ones. We show an adversarial
detection rate of average 87% which exceeds the detection rate of the
state-of-the-art approaches, with a much lower overhead. We demonstrate a
software/hardware-accelerated co-design, which reduces the performance impact
of stochastic inference to 1.58X-2X relative to the unprotected baseline,
compared to 14X-20X overhead for a software-only GPU implementation.


http://arxiv.org/abs/2106.05036
Towards Defending against Adversarial Examples via Attack-Invariant Features. (99%)
Dawei Zhou; Tongliang Liu; Bo Han; Nannan Wang; Chunlei Peng; Xinbo Gao
  Deep neural networks (DNNs) are vulnerable to adversarial noise. Their
adversarial robustness can be improved by exploiting adversarial examples.
However, given the continuously evolving attacks, models trained on seen types
of adversarial examples generally cannot generalize well to unseen types of
adversarial examples. To solve this problem, in this paper, we propose to
remove adversarial noise by learning generalizable invariant features across
attacks which maintain semantic classification information. Specifically, we
introduce an adversarial feature learning mechanism to disentangle invariant
features from adversarial noise. A normalization term has been proposed in the
encoded space of the attack-invariant features to address the bias issue
between the seen and unseen types of attacks. Empirical evaluations demonstrate
that our method could provide better protection in comparison to previous
state-of-the-art approaches, especially against unseen types of attacks and
adaptive attacks.


http://arxiv.org/abs/2106.05453
Improving White-box Robustness of Pre-processing Defenses via Joint Adversarial Training. (99%)
Dawei Zhou; Nannan Wang; Xinbo Gao; Bo Han; Jun Yu; Xiaoyu Wang; Tongliang Liu
  Deep neural networks (DNNs) are vulnerable to adversarial noise. A range of
adversarial defense techniques have been proposed to mitigate the interference
of adversarial noise, among which the input pre-processing methods are scalable
and show great potential to safeguard DNNs. However, pre-processing methods may
suffer from the robustness degradation effect, in which the defense reduces
rather than improving the adversarial robustness of a target model in a
white-box setting. A potential cause of this negative effect is that
adversarial training examples are static and independent to the pre-processing
model. To solve this problem, we investigate the influence of full adversarial
examples which are crafted against the full model, and find they indeed have a
positive impact on the robustness of defenses. Furthermore, we find that simply
changing the adversarial training examples in pre-processing methods does not
completely alleviate the robustness degradation effect. This is due to the
adversarial risk of the pre-processed model being neglected, which is another
cause of the robustness degradation effect. Motivated by above analyses, we
propose a method called Joint Adversarial Training based Pre-processing (JATP)
defense. Specifically, we formulate a feature similarity based adversarial risk
for the pre-processing model by using full adversarial examples found in a
feature space. Unlike standard adversarial training, we only update the
pre-processing model, which prompts us to introduce a pixel-wise loss to
improve its cross-model transferability. We then conduct a joint adversarial
training on the pre-processing model to minimize this overall risk. Empirical
results show that our method could effectively mitigate the robustness
degradation effect across different target models in comparison to previous
state-of-the-art approaches.


http://arxiv.org/abs/2106.04938
Attacking Adversarial Attacks as A Defense. (99%)
Boxi Wu; Heng Pan; Li Shen; Jindong Gu; Shuai Zhao; Zhifeng Li; Deng Cai; Xiaofei He; Wei Liu
  It is well known that adversarial attacks can fool deep neural networks with
imperceptible perturbations. Although adversarial training significantly
improves model robustness, failure cases of defense still broadly exist. In
this work, we find that the adversarial attacks can also be vulnerable to small
perturbations. Namely, on adversarially-trained models, perturbing adversarial
examples with a small random noise may invalidate their misled predictions.
After carefully examining state-of-the-art attacks of various kinds, we find
that all these attacks have this deficiency to different extents. Enlightened
by this finding, we propose to counter attacks by crafting more effective
defensive perturbations. Our defensive perturbations leverage the advantage
that adversarial training endows the ground-truth class with smaller local
Lipschitzness. By simultaneously attacking all the classes, the misled
predictions with larger Lipschitzness can be flipped into correct ones. We
verify our defensive perturbation with both empirical experiments and
theoretical analyses on a linear model. On CIFAR10, it boosts the
state-of-the-art model from 66.16% to 72.66% against the four attacks of
AutoAttack, including 71.76% to 83.30% against the Square attack. On ImageNet,
the top-1 robust accuracy of FastAT is improved from 33.18% to 38.54% under the
100-step PGD attack.


http://arxiv.org/abs/2106.05261
We Can Always Catch You: Detecting Adversarial Patched Objects WITH or WITHOUT Signature. (98%)
Bin Liang; Jiachun Li; Jianjun Huang
  Recently, the object detection based on deep learning has proven to be
vulnerable to adversarial patch attacks. The attackers holding a specially
crafted patch can hide themselves from the state-of-the-art person detectors,
e.g., YOLO, even in the physical world. This kind of attack can bring serious
security threats, such as escaping from surveillance cameras. In this paper, we
deeply explore the detection problems about the adversarial patch attacks to
the object detection. First, we identify a leverageable signature of existing
adversarial patches from the point of the visualization explanation. A fast
signature-based defense method is proposed and demonstrated to be effective.
Second, we design an improved patch generation algorithm to reveal the risk
that the signature-based way may be bypassed by the techniques emerging in the
future. The newly generated adversarial patches can successfully evade the
proposed signature-based defense. Finally, we present a novel
signature-independent detection method based on the internal content semantics
consistency rather than any attack-specific prior knowledge. The fundamental
intuition is that the adversarial object can appear locally but disappear
globally in an input image. The experiments demonstrate that the
signature-independent method can effectively detect the existing and improved
attacks. It has also proven to be a general method by detecting unforeseen and
even other types of attacks without any attack-specific prior knowledge. The
two proposed detection methods can be adopted in different scenarios, and we
believe that combining them can offer a comprehensive protection.


http://arxiv.org/abs/2106.05087
Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. (88%)
Yanchao Sun; Ruijie Zheng; Yongyuan Liang; Furong Huang
  Evaluating the worst-case performance of a reinforcement learning (RL) agent
under the strongest/optimal adversarial perturbations on state observations
(within some constraints) is crucial for understanding the robustness of RL
agents. However, finding the optimal adversary is challenging, in terms of both
whether we can find the optimal attack and how efficiently we can find it.
Existing works on adversarial RL either use heuristics-based methods that may
not find the strongest adversary, or directly train an RL-based adversary by
treating the agent as a part of the environment, which can find the optimal
adversary but may become intractable in a large state space. In this paper, we
propose a novel attacking algorithm which has an RL-based "director" searching
for the optimal policy perturbation, and an "actor" crafting state
perturbations following the directions from the director (i.e. the actor
executes targeted attacks). Our proposed algorithm, PA-AD, is theoretically
optimal against an RL agent and significantly improves the efficiency compared
with prior RL-based works in environments with large or pixel state spaces.
Empirical results show that our proposed PA-AD universally outperforms
state-of-the-art attacking methods in a wide range of environments. Our method
can be easily applied to any RL algorithms to evaluate and improve their
robustness.


http://arxiv.org/abs/2106.05256
URLTran: Improving Phishing URL Detection Using Transformers. (10%)
Pranav Maneriker; Jack W. Stokes; Edir Garcia Lazo; Diana Carutasu; Farid Tajaddodianfar; Arun Gururajan
  Browsers often include security features to detect phishing web pages. In the
past, some browsers evaluated an unknown URL for inclusion in a list of known
phishing pages. However, as the number of URLs and known phishing pages
continued to increase at a rapid pace, browsers started to include one or more
machine learning classifiers as part of their security services that aim to
better protect end users from harm. While additional information could be used,
browsers typically evaluate every unknown URL using some classifier in order to
quickly detect these phishing pages. Early phishing detection used standard
machine learning classifiers, but recent research has instead proposed the use
of deep learning models for the phishing URL detection task. Concurrently, text
embedding research using transformers has led to state-of-the-art results in
many natural language processing tasks. In this work, we perform a
comprehensive analysis of transformer models on the phishing URL detection
task. We consider standard masked language model and additional domain-specific
pre-training tasks, and compare these models to fine-tuned BERT and RoBERTa
models. Combining the insights from these experiments, we propose URLTran which
uses transformers to significantly improve the performance of phishing URL
detection over a wide range of very low false positive rates (FPRs) compared to
other deep learning-based methods. For example, URLTran yields a true positive
rate (TPR) of 86.80% compared to 71.20% for the next best baseline at an FPR of
0.01%, resulting in a relative improvement of over 21.9%. Further, we consider
some classical adversarial black-box phishing attacks such as those based on
homoglyphs and compound word splits to improve the robustness of URLTran. We
consider additional fine tuning with these adversarial samples and demonstrate
that URLTran can maintain low FPRs under these scenarios.


http://arxiv.org/abs/2106.05325
ZoPE: A Fast Optimizer for ReLU Networks with Low-Dimensional Inputs. (3%)
Christopher A. Strong; Sydney M. Katz; Anthony L. Corso; Mykel J. Kochenderfer
  Deep neural networks often lack the safety and robustness guarantees needed
to be deployed in safety critical systems. Formal verification techniques can
be used to prove input-output safety properties of networks, but when
properties are difficult to specify, we rely on the solution to various
optimization problems. In this work, we present an algorithm called ZoPE that
solves optimization problems over the output of feedforward ReLU networks with
low-dimensional inputs. The algorithm eagerly splits the input space, bounding
the objective using zonotope propagation at each step, and improves
computational efficiency compared to existing mixed integer programming
approaches. We demonstrate how to formulate and solve three types of
optimization problems: (i) minimization of any convex function over the output
space, (ii) minimization of a convex function over the output of two networks
in series with an adversarial perturbation in the layer between them, and (iii)
maximization of the difference in output between two networks. Using ZoPE, we
observe a $25\times$ speedup on property 1 of the ACAS Xu neural network
verification benchmark and an $85\times$ speedup on a set of linear
optimization problems. We demonstrate the versatility of the optimizer in
analyzing networks by projecting onto the range of a generative adversarial
network and visualizing the differences between a compressed and uncompressed
network.


http://arxiv.org/abs/2106.04823
Practical Machine Learning Safety: A Survey and Primer. (2%)
Sina Mohseni; Haotao Wang; Zhiding Yu; Chaowei Xiao; Zhangyang Wang; Jay Yadawa
  The open-world deployment of Machine Learning (ML) algorithms in
safety-critical applications such as autonomous vehicles needs to address a
variety of ML vulnerabilities such as interpretability, verifiability, and
performance limitations. Research explores different approaches to improve ML
dependability by proposing new models and training techniques to reduce
generalization error, achieve domain adaptation, and detect outlier examples
and adversarial attacks. In this paper, we review and organize practical ML
techniques that can improve the safety and dependability of ML algorithms and
therefore ML-based software. Our organization maps state-of-the-art ML
techniques to safety strategies in order to enhance the dependability of the ML
algorithm from different aspects, and discuss research gaps as well as
promising solutions.


http://arxiv.org/abs/2106.05009
Network insensitivity to parameter noise via adversarial regularization. (2%)
Julian Büchel; Fynn Faber; Dylan R. Muir
  Neuromorphic neural network processors, in the form of compute-in-memory
crossbar arrays of memristors, or in the form of subthreshold analog and
mixed-signal ASICs, promise enormous advantages in compute density and energy
efficiency for NN-based ML tasks. However, these technologies are prone to
computational non-idealities, due to process variation and intrinsic device
physics. This degrades the task performance of networks deployed to the
processor, by introducing parameter noise into the deployed model. While it is
possible to calibrate each device, or train networks individually for each
processor, these approaches are expensive and impractical for commercial
deployment. Alternative methods are therefore needed to train networks that are
inherently robust against parameter variation, as a consequence of network
architecture and parameters. We present a new adversarial network optimisation
algorithm that attacks network parameters during training, and promotes robust
performance during inference in the face of parameter variation. Our approach
introduces a regularization term penalising the susceptibility of a network to
weight perturbation. We compare against previous approaches for producing
parameter insensitivity such as dropout, weight smoothing and introducing
parameter noise during training. We show that our approach produces models that
are more robust to targeted parameter variation, and equally robust to random
parameter variation. Our approach finds minima in flatter locations in the
weight-loss landscape compared with other approaches, highlighting that the
networks found by our technique are less sensitive to parameter perturbation.
Our work provides an approach to deploy neural network architectures to
inference devices that suffer from computational non-idealities, with minimal
loss of performance. ...


http://arxiv.org/abs/2106.04169
On Improving Adversarial Transferability of Vision Transformers. (99%)
Muzammal Naseer; Kanchana Ranasinghe; Salman Khan; Fahad Shahbaz Khan; Fatih Porikli
  Vision transformers (ViTs) process input images as sequences of patches via
self-attention; a radically different architecture than convolutional neural
networks (CNNs). This makes it interesting to study the adversarial feature
space of ViT models and their transferability. In particular, we observe that
adversarial patterns found via conventional adversarial attacks show very low
black-box transferability even for large ViT models. However, we show that this
phenomenon is only due to the sub-optimal attack procedures that do not
leverage the true representation potential of ViTs. A deep ViT is composed of
multiple blocks, with a consistent architecture comprising of self-attention
and feed-forward layers, where each block is capable of independently producing
a class token. Formulating an attack using only the last class token
(conventional approach) does not directly leverage the discriminative
information stored in the earlier tokens, leading to poor adversarial
transferability of ViTs. Using the compositional nature of ViT models, we
enhance the transferability of existing attacks by introducing two novel
strategies specific to the architecture of ViT models. (i) Self-Ensemble: We
propose a method to find multiple discriminative pathways by dissecting a
single ViT model into an ensemble of networks. This allows explicitly utilizing
class-specific information at each ViT block. (ii) Token Refinement: We then
propose to refine the tokens to further enhance the discriminative capacity at
each block of ViT. Our token refinement systematically combines the class
tokens with structural information preserved within the patch tokens. An
adversarial attack, when applied to such refined tokens within the ensemble of
classifiers found in a single vision transformer, has significantly higher
transferability.


http://arxiv.org/abs/2106.04569
Simulated Adversarial Testing of Face Recognition Models. (99%)
Nataniel Ruiz; Adam Kortylewski; Weichao Qiu; Cihang Xie; Sarah Adel Bargal; Alan Yuille; Stan Sclaroff
  Most machine learning models are validated and tested on fixed datasets. This
can give an incomplete picture of the capabilities and weaknesses of the model.
Such weaknesses can be revealed at test time in the real world. The risks
involved in such failures can be loss of profits, loss of time or even loss of
life in certain critical applications. In order to alleviate this issue,
simulators can be controlled in a fine-grained manner using interpretable
parameters to explore the semantic image manifold. In this work, we propose a
framework for learning how to test machine learning algorithms using simulators
in an adversarial manner in order to find weaknesses in the model before
deploying it in critical scenarios. We apply this model in a face recognition
scenario. We are the first to show that weaknesses of models trained on real
data can be discovered using simulated samples. Using our proposed method, we
can find adversarial synthetic faces that fool contemporary face recognition
models. This demonstrates the fact that these models have weaknesses that are
not measured by commonly used validation datasets. We hypothesize that this
type of adversarial examples are not isolated, but usually lie in connected
components in the latent space of the simulator. We present a method to find
these adversarial regions as opposed to the typical adversarial points found in
the adversarial example literature.


http://arxiv.org/abs/2106.04794
Towards the Memorization Effect of Neural Networks in Adversarial Training. (93%)
Han Xu; Xiaorui Liu; Wentao Wang; Wenbiao Ding; Zhongqin Wu; Zitao Liu; Anil Jain; Jiliang Tang
  Recent studies suggest that ``memorization'' is one important factor for
overparameterized deep neural networks (DNNs) to achieve optimal performance.
Specifically, the perfectly fitted DNNs can memorize the labels of many
atypical samples, generalize their memorization to correctly classify test
atypical samples and enjoy better test performance. While, DNNs which are
optimized via adversarial training algorithms can also achieve perfect training
performance by memorizing the labels of atypical samples, as well as the
adversarially perturbed atypical samples. However, adversarially trained models
always suffer from poor generalization, with both relatively low clean accuracy
and robustness on the test set. In this work, we study the effect of
memorization in adversarial trained DNNs and disclose two important findings:
(a) Memorizing atypical samples is only effective to improve DNN's accuracy on
clean atypical samples, but hardly improve their adversarial robustness and (b)
Memorizing certain atypical samples will even hurt the DNN's performance on
typical samples. Based on these two findings, we propose Benign Adversarial
Training (BAT) which can facilitate adversarial training to avoid fitting
``harmful'' atypical samples and fit as more ``benign'' atypical samples as
possible. In our experiments, we validate the effectiveness of BAT, and show it
can achieve better clean accuracy vs. robustness trade-off than baseline
methods, in benchmark datasets such as CIFAR100 and Tiny~ImageNet.


http://arxiv.org/abs/2106.04690
Handcrafted Backdoors in Deep Neural Networks. (92%)
Sanghyun Hong; Nicholas Carlini; Alexey Kurakin
  Deep neural networks (DNNs), while accurate, are expensive to train. Many
practitioners, therefore, outsource the training process to third parties or
use pre-trained DNNs. This practice makes DNNs vulnerable to $backdoor$
$attacks$: the third party who trains the model may act maliciously to inject
hidden behaviors into the otherwise accurate model. Until now, the mechanism to
inject backdoors has been limited to $poisoning$.
  We argue that such a supply-chain attacker has more attack techniques
available. To study this hypothesis, we introduce a handcrafted attack that
directly manipulates the parameters of a pre-trained model to inject backdoors.
Our handcrafted attacker has more degrees of freedom in manipulating model
parameters than poisoning. This makes it difficult for a defender to identify
or remove the manipulations with straightforward methods, such as statistical
analysis, adding random noises to model parameters, or clipping their values
within a certain range. Further, our attacker can combine the handcrafting
process with additional techniques, $e.g.$, jointly optimizing a trigger
pattern, to inject backdoors into complex networks effectively$-$the
meet-in-the-middle attack.
  In evaluations, our handcrafted backdoors remain effective across four
datasets and four network architectures with a success rate above 96%. Our
backdoored models are resilient to both parameter-level backdoor removal
techniques and can evade existing defenses by slightly changing the backdoor
attack configurations. Moreover, we demonstrate the feasibility of suppressing
unwanted behaviors otherwise caused by poisoning. Our results suggest that
further research is needed for understanding the complete space of supply-chain
backdoor attacks.


http://arxiv.org/abs/2106.04435
Enhancing Robustness of Neural Networks through Fourier Stabilization. (73%)
Netanel Raviv; Aidan Kelley; Michael Guo; Yevgeny Vorobeychik
  Despite the considerable success of neural networks in security settings such
as malware detection, such models have proved vulnerable to evasion attacks, in
which attackers make slight changes to inputs (e.g., malware) to bypass
detection. We propose a novel approach, \emph{Fourier stabilization}, for
designing evasion-robust neural networks with binary inputs. This approach,
which is complementary to other forms of defense, replaces the weights of
individual neurons with robust analogs derived using Fourier analytic tools.
The choice of which neurons to stabilize in a neural network is then a
combinatorial optimization problem, and we propose several methods for
approximately solving it. We provide a formal bound on the per-neuron drop in
accuracy due to Fourier stabilization, and experimentally demonstrate the
effectiveness of the proposed approach in boosting robustness of neural
networks in several detection settings. Moreover, we show that our approach
effectively composes with adversarial training.


http://arxiv.org/abs/2106.04260
Provably Robust Detection of Out-of-distribution Data (almost) for free. (1%)
Alexander Meinke; Julian Bitterwolf; Matthias Hein
  When applying machine learning in safety-critical systems, a reliable
assessment of the uncertainy of a classifier is required. However, deep neural
networks are known to produce highly overconfident predictions on
out-of-distribution (OOD) data and even if trained to be non-confident on OOD
data one can still adversarially manipulate OOD data so that the classifer
again assigns high confidence to the manipulated samples. In this paper we
propose a novel method where from first principles we combine a certifiable OOD
detector with a standard classifier into an OOD aware classifier. In this way
we achieve the best of two worlds: certifiably adversarially robust OOD
detection, even for OOD samples close to the in-distribution, without loss in
prediction accuracy and close to state-of-the-art OOD detection performance for
non-manipulated OOD data. Moreover, due to the particular construction our
classifier provably avoids the asymptotic overconfidence problem of standard
neural networks.


http://arxiv.org/abs/2106.03614
Adversarial Attack and Defense in Deep Ranking. (99%)
Mo Zhou; Le Wang; Zhenxing Niu; Qilin Zhang; Nanning Zheng; Gang Hua
  Deep Neural Network classifiers are vulnerable to adversarial attack, where
an imperceptible perturbation could result in misclassification. However, the
vulnerability of DNN-based image ranking systems remains under-explored. In
this paper, we propose two attacks against deep ranking systems, i.e.,
Candidate Attack and Query Attack, that can raise or lower the rank of chosen
candidates by adversarial perturbations. Specifically, the expected ranking
order is first represented as a set of inequalities, and then a triplet-like
objective function is designed to obtain the optimal perturbation. Conversely,
an anti-collapse triplet defense is proposed to improve the ranking model
robustness against all proposed attacks, where the model learns to prevent the
positive and negative samples being pulled close to each other by adversarial
attack. To comprehensively measure the empirical adversarial robustness of a
ranking model with our defense, we propose an empirical robustness score, which
involves a set of representative attacks against ranking models. Our
adversarial ranking attacks and defenses are evaluated on MNIST, Fashion-MNIST,
CUB200-2011, CARS196 and Stanford Online Products datasets. Experimental
results demonstrate that a typical deep ranking system can be effectively
compromised by our attacks. Nevertheless, our defense can significantly improve
the ranking system robustness, and simultaneously mitigate a wide range of
attacks.


http://arxiv.org/abs/2106.03734
Reveal of Vision Transformers Robustness against Adversarial Attacks. (99%)
Ahmed Aldahdooh; Wassim Hamidouche; Olivier Deforges
  The major part of the vanilla vision transformer (ViT) is the attention block
that brings the power of mimicking the global context of the input image. For
better performance, ViT needs large-scale training data. To overcome this data
hunger limitation, many ViT-based networks, or hybrid-ViT, have been proposed
to include local context during the training. The robustness of ViTs and its
variants against adversarial attacks has not been widely investigated in the
literature like CNNs. This work studies the robustness of ViT variants 1)
against different Lp-based adversarial attacks in comparison with CNNs, 2)
under adversarial examples (AEs) after applying preprocessing defense methods
and 3) under the adaptive attacks using expectation over transformation (EOT)
framework. To that end, we run a set of experiments on 1000 images from
ImageNet-1k and then provide an analysis that reveals that vanilla ViT or
hybrid-ViT are more robust than CNNs. For instance, we found that 1) Vanilla
ViTs or hybrid-ViTs are more robust than CNNs under Lp-based attacks and under
adaptive attacks. 2) Unlike hybrid-ViTs, Vanilla ViTs are not responding to
preprocessing defenses that mainly reduce the high frequency components.
Furthermore, feature maps, attention maps, and Grad-CAM visualization jointly
with image quality measures, and p