http://arxiv.org/abs/2005.14137
QEBA: Query-Efficient Boundary-Based Blackbox Attack. (99%)
Huichen Li; Xiaojun Xu; Xiaolu Zhang; Shuang Yang; Bo Li
  Machine learning (ML), especially deep neural networks (DNNs) have been
widely used in various applications, including several safety-critical ones
(e.g. autonomous driving). As a result, recent research about adversarial
examples has raised great concerns. Such adversarial attacks can be achieved by
adding a small magnitude of perturbation to the input to mislead model
prediction. While several whitebox attacks have demonstrated their
effectiveness, which assume that the attackers have full access to the machine
learning models; blackbox attacks are more realistic in practice. In this
paper, we propose a Query-Efficient Boundary-based blackbox Attack (QEBA) based
only on model's final prediction labels. We theoretically show why previous
boundary-based attack with gradient estimation on the whole gradient space is
not efficient in terms of query numbers, and provide optimality analysis for
our dimension reduction-based gradient estimation. On the other hand, we
conducted extensive experiments on ImageNet and CelebA datasets to evaluate
QEBA. We show that compared with the state-of-the-art blackbox attacks, QEBA is
able to use a smaller number of queries to achieve a lower magnitude of
perturbation with 100% attack success rate. We also show case studies of
attacks on real-world APIs including MEGVII Face++ and Microsoft Azure.


http://arxiv.org/abs/2005.14108
Adversarial Attacks and Defense on Textual Data: A Review. (67%)
Aminul Huq; Mst. Tasnim Pervin
  Deep leaning models have been used widely for various purposes in recent
years in object recognition, self-driving cars, face recognition, speech
recognition, sentiment analysis and many others. However, in recent years it
has been shown that these models possess weakness to noises which forces the
model to misclassify. This issue has been studied profoundly in image and audio
domain. Very little has been studied on this issue with respect to textual
data. Even less survey on this topic has been performed to understand different
types of attacks and defense techniques. In this manuscript we accumulated and
analyzed different attacking techniques, various defense models on how to
overcome this issue in order to provide a more comprehensive idea. Later we
point out some of the interesting findings of all papers and challenges that
need to be overcome in order to move forward in this field.


http://arxiv.org/abs/2005.13770
DeepSonar: Towards Effective and Robust Detection of AI-Synthesized Fake Voices. (10%)
Run Wang; Felix Juefei-Xu; Yihao Huang; Qing Guo; Xiaofei Xie; Lei Ma; Yang Liu
  With the recent advances in voice synthesis such as WaveNet, AI-synthesized
fake voices are indistinguishable to human ears and widely applied for
producing realistic and natural DeepFakes which are real threats to everyone.
However, effective and robust detectors for synthesized fake voices are still
in their infancy and are not ready to fully tackle this emerging threat. In
this paper, we devise a novel approach, named DeepSonar, based on monitoring
neuron behaviors of speaker recognition (SR) system, a deep neural network
(DNN), to discern AI-synthesized fake voices. Layer-wise neuron behaviors
provide an important insight to hunt the differences among inputs, which are
widely employed for building safety, robust and interpretable DNNs. In this
work, we leverage the power of layer-wise neuron activation patterns with a
conjecture that they can capture the subtle differences between real and
AI-synthesized fake voices and provide a cleaner signal to classifiers than raw
inputs. Experiments are conducted in three datasets (including commercial
products from Google, Baidu, etc.) containing both English and Chinese
languages to corroborate the high detection rates (98.1% average accuracy) and
low false alarm rates (0.02 equal error rate) of DeepSonar in discerning fake
voices. Furthermore, extensive experiment results show its robustness against
manipulation attacks (e.g., voice conversion and additive real-world noises).
Our work also poses a new insight into adopting neuron behaviors for effective
and robust AI aided multimedia fakes forensics instead of motivated by various
artifacts introduced in fakes.


http://arxiv.org/abs/2005.13815
Adversarial Classification via Distributional Robustness with Wasserstein Ambiguity. (1%)
Nam Ho-Nguyen; Stephen J. Wright
  We study a model for adversarial classification based on distributionally
robust chance constraints. We show that under Wasserstein ambiguity, the model
aims to minimize the conditional value-at-risk of the distance to
misclassification, and we explore links to previous adversarial classification
models and maximum margin classifiers. We also provide a reformulation of the
distributionally robust model for linear classifiers, and show it is equivalent
to minimizing a regularized ramp loss. Numerical experiments show that, despite
the nonconvexity, standard descent methods appear to converge to the global
minimizer for this problem. Inspired by this observation, we show that, for a
certain benign distribution, the regularized ramp loss minimization problem has
a single stationary point, at the global minimizer.


http://arxiv.org/abs/2005.13712
Mitigating Advanced Adversarial Attacks with More Advanced Gradient Obfuscation Techniques. (99%)
Han Qiu; Yi Zeng; Qinkai Zheng; Tianwei Zhang; Meikang Qiu; Gerard Memmi
  Deep Neural Networks (DNNs) are well-known to be vulnerable to Adversarial
Examples (AEs). A large amount of efforts have been spent to launch and heat
the arms race between the attackers and defenders. Recently, advanced
gradient-based attack techniques were proposed (e.g., BPDA and EOT), which have
defeated a considerable number of existing defense methods. Up to today, there
are still no satisfactory solutions that can effectively and efficiently defend
against those attacks.
  In this paper, we make a steady step towards mitigating those advanced
gradient-based attacks with two major contributions. First, we perform an
in-depth analysis about the root causes of those attacks, and propose four
properties that can break the fundamental assumptions of those attacks. Second,
we identify a set of operations that can meet those properties. By integrating
these operations, we design two preprocessing functions that can invalidate
these powerful attacks. Extensive evaluations indicate that our solutions can
effectively mitigate all existing standard and advanced attack techniques, and
beat 11 state-of-the-art defense solutions published in top-tier conferences
over the past 2 years. The defender can employ our solutions to constrain the
attack success rate below 7% for the strongest attacks even the adversary has
spent dozens of GPU hours.


http://arxiv.org/abs/2005.13525
Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models. (99%)
Mitch Hill; Jonathan Mitchell; Song-Chun Zhu
  The vulnerability of deep networks to adversarial attacks is a central
problem for deep learning from the perspective of both cognition and security.
The current most successful defense method is to train a classifier using
adversarial images created during learning. Another defense approach involves
transformation or purification of the original input to remove adversarial
signals before the image is classified. We focus on defending naturally-trained
classifiers using Markov Chain Monte Carlo (MCMC) sampling with an Energy-Based
Model (EBM) for adversarial purification. In contrast to adversarial training,
our approach is intended to secure pre-existing and highly vulnerable
classifiers. The memoryless behavior of long-run MCMC sampling will eventually
remove adversarial signals, while metastable behavior preserves consistent
appearance of MCMC samples after many steps to allow accurate long-run
prediction. Balancing these factors can lead to effective purification and
robust classification. We evaluate adversarial defense with an EBM using the
strongest known attacks against purification. Our contributions are 1) an
improved method for training EBM's with realistic long-run MCMC samples, 2) an
Expectation-Over-Transformation (EOT) defense that resolves theoretical
ambiguities for stochastic defenses and from which the EOT attack naturally
follows, and 3) state-of-the-art adversarial defense for naturally-trained
classifiers and competitive defense compared to adversarially-trained
classifiers on Cifar-10, SVHN, and Cifar-100. Code and pre-trained models are
available at https://github.com/point0bar1/ebm-defense.


http://arxiv.org/abs/2005.13293
Enhancing Resilience of Deep Learning Networks by Means of Transferable Adversaries. (92%)
Moritz Seiler; Heike Trautmann; Pascal Kerschke
  Artificial neural networks in general and deep learning networks in
particular established themselves as popular and powerful machine learning
algorithms. While the often tremendous sizes of these networks are beneficial
when solving complex tasks, the tremendous number of parameters also causes
such networks to be vulnerable to malicious behavior such as adversarial
perturbations. These perturbations can change a model's classification
decision. Moreover, while single-step adversaries can easily be transferred
from network to network, the transfer of more powerful multi-step adversaries
has - usually -- been rather difficult. In this work, we introduce a method for
generating strong ad-versaries that can easily (and frequently) be transferred
between different models. This method is then used to generate a large set of
adversaries, based on which the effects of selected defense methods are
experimentally assessed. At last, we introduce a novel, simple, yet effective
approach to enhance the resilience of neural networks against adversaries and
benchmark it against established defense methods. In contrast to the already
existing methods, our proposed defense approach is much more efficient as it
only requires a single additional forward-pass to achieve comparable
performance results.


http://arxiv.org/abs/2005.13702
Towards the Infeasibility of Membership Inference on Deep Models. (13%)
Shahbaz Rezaei; Xin Liu
  Recent studies propose membership inference (MI) attacks on deep models.
Despite the moderate accuracy of such MI attacks, we show that the way the
attack accuracy is reported is often misleading and a simple blind attack which
is highly unreliable and inefficient in reality can often represent similar
accuracy. We show that the current MI attack models can only identify the
membership of misclassified samples with mediocre accuracy at best, which only
constitute a very small portion of training samples. We analyze several new
features that have not been explored for membership inference before, including
distance to the decision boundary and gradient norms, and conclude that deep
models' responses are mostly indistinguishable among train and non-train
samples. Moreover, in contrast with general intuition that deeper models have a
capacity to memorize training samples, and, hence, they are more vulnerable to
membership inference, we find no evidence to support that and in some cases
deeper models are often harder to launch membership inference attack on.
Furthermore, despite the common belief, we show that overfitting does not
necessarily lead to higher degree of membership leakage. We conduct experiments
on MNIST, CIFAR-10, CIFAR-100, and ImageNet, using various model architecture,
including LeNet, ResNet, DenseNet, InceptionV3, and Xception. Source code:
https://github.com/shrezaei/MI-Attack}{\color{blue}
{https://github.com/shrezaei/MI-Attack}.


http://arxiv.org/abs/2005.13748
Calibrated Surrogate Losses for Adversarially Robust Classification. (13%)
Han Bao; Clayton Scott; Masashi Sugiyama
  Adversarially robust classification seeks a classifier that is insensitive to
adversarial perturbations of test patterns. This problem is often formulated
via a minimax objective, where the target loss is the worst-case value of the
0-1 loss subject to a bound on the size of perturbation. Recent work has
proposed convex surrogates for the adversarial 0-1 loss, in an effort to make
optimization more tractable. In this work, we consider the question of which
surrogate losses are calibrated with respect to the adversarial 0-1 loss,
meaning that minimization of the former implies minimization of the latter. We
show that no convex surrogate loss is calibrated with respect to the
adversarial 0-1 loss when restricted to the class of linear models. We further
introduce a class of nonconvex losses and offer necessary and sufficient
conditions for losses in this class to be calibrated.


http://arxiv.org/abs/2005.13123
Effects of Forward Error Correction on Communications Aware Evasion Attacks. (98%)
Matthew DelVecchio; Bryse Flowers; William C. Headley
  Recent work has shown the impact of adversarial machine learning on deep
neural networks (DNNs) developed for Radio Frequency Machine Learning (RFML)
applications. While these attacks have been shown to be successful in
disrupting the performance of an eavesdropper, they fail to fully support the
primary goal of successful intended communication. To remedy this, a
communications-aware attack framework was recently developed that allows for a
more effective balance between the opposing goals of evasion and intended
communication through the novel use of a DNN to intelligently create the
adversarial communication signal. Given the near ubiquitous usage of forward
error correction (FEC) coding in the majority of deployed systems to correct
errors that arise, incorporating FEC in this framework is a natural extension
of this prior work and will allow for improved performance in more adverse
environments. This work therefore provides contributions to the framework
through improved loss functions and design considerations to incorporate
inherent knowledge of the usage of FEC codes within the transmitted signal.
Performance analysis shows that FEC coding improves the communications aware
adversarial attack even if no explicit knowledge of the coding scheme is
assumed and allows for improved performance over the prior art in balancing the
opposing goals of evasion and intended communications.


http://arxiv.org/abs/2005.12696
Generating Semantically Valid Adversarial Questions for TableQA. (92%)
Yi Zhu; Menglin Xia; Yiwei Zhou
  Adversarial attack on question answering systems over tabular data (TableQA)
can help evaluate to what extent they can understand natural language questions
and reason with tables. However, generating natural language adversarial
questions is difficult, because even a single character swap could lead to huge
semantic difference in human perception. In this paper, we propose SAGE
(Semantically valid Adversarial GEnerator), a Wasserstein sequence-to-sequence
model for TableQA white-box attack. To preserve meaning of original questions,
we apply minimum risk training with SIMILE and entity delexicalization. We use
Gumbel-Softmax to incorporate adversarial loss for end-to-end training. Our
experiments show that SAGE outperforms existing local attack models on semantic
validity and fluency while achieving a good attack success rate. Finally, we
demonstrate that adversarial training with SAGE augmented data can improve
performance and robustness of TableQA systems.


http://arxiv.org/abs/2005.13124
Investigating a Spectral Deception Loss Metric for Training Machine Learning-based Evasion Attacks. (78%)
Matthew DelVecchio; Vanessa Arndorfer; William C. Headley
  Adversarial evasion attacks have been very successful in causing poor
performance in a wide variety of machine learning applications. One such
application is radio frequency spectrum sensing. While evasion attacks have
proven particularly successful in this area, they have done so at the detriment
of the signal's intended purpose. More specifically, for real-world
applications of interest, the resulting perturbed signal that is transmitted to
evade an eavesdropper must not deviate far from the original signal, less the
intended information is destroyed. Recent work by the authors and others has
demonstrated an attack framework that allows for intelligent balancing between
these conflicting goals of evasion and communication. However, while these
methodologies consider creating adversarial signals that minimize
communications degradation, they have been shown to do so at the expense of the
spectral shape of the signal. This opens the adversarial signal up to defenses
at the eavesdropper such as filtering, which could render the attack
ineffective. To remedy this, this work introduces a new spectral deception loss
metric that can be implemented during the training process to force the
spectral shape to be more in-line with the original signal. As an initial proof
of concept, a variety of methods are presented that provide a starting point
for this proposed loss. Through performance analysis, it is shown that these
techniques are effective in controlling the shape of the adversarial signal.


http://arxiv.org/abs/2005.13032
SeqL: Secure Scan-Locking for IP Protection. (1%)
Seetal Potluri; Aydin Aysu; Akash Kumar
  Existing logic-locking attacks are known to successfully decrypt functionally
correct key of a locked combinational circuit. It is possible to extend these
attacks to real-world Silicon-based Intellectual Properties (IPs, which are
sequential circuits) through scan-chains by selectively initializing the
combinational logic and analyzing the responses. In this paper, we propose
SeqL, which achieves functional isolation and locks selective flip-flop
functional-input/scan-output pairs, thus rendering the decrypted key
functionally incorrect. We conduct a formal study of the scan-locking problem
and demonstrate automating our proposed defense on any given IP. We show that
SeqL hides functionally correct keys from the attacker, thereby increasing the
likelihood of the decrypted key being functionally incorrect. When tested on
pipelined combinational benchmarks (ISCAS,MCNC), sequential benchmarks (ITC)
and a fully-fledged RISC-V CPU, SeqL gave 100% resilience to a broad range of
state-of-the-art attacks including SAT[1], Double-DIP[2], HackTest[3], SMT[4],
FALL[5], Shift-and-Leak[6] and Multi-cycle attacks[7].


http://arxiv.org/abs/2005.12154
Adversarial Feature Selection against Evasion Attacks. (97%)
Fei Zhang; Patrick P. K. Chan; Battista Biggio; Daniel S. Yeung; Fabio Roli
  Pattern recognition and machine learning techniques have been increasingly
adopted in adversarial settings such as spam, intrusion and malware detection,
although their security against well-crafted attacks that aim to evade
detection by manipulating data at test time has not yet been thoroughly
assessed. While previous work has been mainly focused on devising
adversary-aware classification algorithms to counter evasion attempts, only few
authors have considered the impact of using reduced feature sets on classifier
security against the same attacks. An interesting, preliminary result is that
classifier security to evasion may be even worsened by the application of
feature selection. In this paper, we provide a more detailed investigation of
this aspect, shedding some light on the security properties of feature
selection against evasion attacks. Inspired by previous work on adversary-aware
classifiers, we propose a novel adversary-aware feature selection model that
can improve classifier security against evasion attacks, by incorporating
specific assumptions on the adversary's data manipulation strategy. We focus on
an efficient, wrapper-based implementation of our approach, and experimentally
validate its soundness on different application examples, including spam and
malware detection.


http://arxiv.org/abs/2005.12729
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. (38%)
Logan Engstrom; Andrew Ilyas; Shibani Santurkar; Dimitris Tsipras; Firdaus Janoos; Larry Rudolph; Aleksander Madry
  We study the roots of algorithmic progress in deep policy gradient algorithms
through a case study on two popular algorithms: Proximal Policy Optimization
(PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate
the consequences of "code-level optimizations:" algorithm augmentations found
only in implementations or described as auxiliary details to the core
algorithm. Seemingly of secondary importance, such optimizations turn out to
have a major impact on agent behavior. Our results show that they (a) are
responsible for most of PPO's gain in cumulative reward over TRPO, and (b)
fundamentally change how RL methods function. These insights show the
difficulty and importance of attributing performance gains in deep
reinforcement learning. Code for reproducing our results is available at
https://github.com/MadryLab/implementation-matters .


http://arxiv.org/abs/2005.12019
Malware Detection at the Microarchitecture Level using Machine Learning Techniques. (1%)
Abigail Kwan
  Detection of malware cyber-attacks at the processor microarchitecture level
has recently emerged as a promising solution to enhance the security of
computer systems. Security mechanisms, such as hardware-based malware
detection, use machine learning algorithms to classify and detect malware with
the aid of Hardware Performance Counters (HPCs) information. The ML classifiers
are fed microarchitectural data extracted from Hardware Performance Counters
(HPCs), which contain behavioral data about a software program. These HPCs are
captured at run-time to model the program's behavior. Since the amount of HPCs
are limited per processor, many techniques employ feature reduction to reduce
the amount of HPCs down to the most essential attributes. Previous studies have
already used binary classification to implement their malware detection after
doing extensive feature reduction. This results in a simple identification of
software being either malware or benign. This research comprehensively analyzes
different hardware-based malware detectors by comparing different machine
learning algorithms' accuracy with binary and multi-class classification
models. Our experimental results indicate that when compared to complex machine
learning models (e. g. Neural Network and Logistic), light-weight J48 and JRip
algorithms perform better in detecting the malicious patterns even with the
introduction of multiple types of malware. Although their detection accuracy
slightly lowers, their robustness (Area Under the Curve) is still high enough
that they deliver a reasonable false positive rate.


http://arxiv.org/abs/2005.11671
SoK: Arms Race in Adversarial Malware Detection. (99%)
Deqiang Li; Qianmu Li; Yanfang Ye; Shouhuai Xu
  Malicious software (malware) is a major cyber threat that shall be tackled
with Machine Learning (ML) techniques because millions of new malware examples
are injected into cyberspace on a daily basis. However, ML is known to be
vulnerable to attacks known as adversarial examples. In this SoK paper, we
systematize the field of Adversarial Malware Detection (AMD) through the lens
of a unified framework of assumptions, attacks, defenses and security
properties. This not only guides us to map attacks and defenses into some
partial order structures, but also allows us to clearly describe the
attack-defense arms race in the AMD context. In addition to manually drawing
insights, we also propose using ML to draw insights from the systematized
representation of the literature. Examples of the insights are: knowing the
defender's feature set is critical to the attacker's success; attack tactic (as
a core part of the threat model) largely determines what security property of a
malware detector can be broke; there is currently no silver bullet defense
against evasion attacks or poisoning attacks; defense tactic largely determines
what security properties can be achieved by a malware detector; knowing
attacker's manipulation set is critical to defender's success; ML is an
effective method for insights learning in SoK studies. These insights shed
light on future research directions.


http://arxiv.org/abs/2005.11904
Adaptive Adversarial Logits Pairing. (92%)
Shangxi Wu; Jitao Sang; Kaiyuan Xu; Guanhua Zheng; Changsheng Xu
  Adversarial examples provide an opportunity as well as impose a challenge for
understanding image classification systems. Based on the analysis of
state-of-the-art defense solution Adversarial Logits Pairing (ALP), we observed
in this work that: (1) The inference of adversarially robust models tends to
rely on fewer high-contribution features compared with vulnerable ones. (2) The
training target of ALP doesn't fit well to a noticeable part of samples, where
the logits pairing loss is overemphasized and obstructs minimizing the
classification loss. Motivated by these observations, we designed an Adaptive
Adversarial Logits Pairing (AALP) solution by modifying the training process
and training target of ALP. Specifically, AALP consists of an adaptive feature
optimization module with Guided Dropout to systematically pursue few
high-contribution features, and an adaptive sample weighting module by setting
sample-specific training weights to balance between logits pairing loss and
classification loss. The proposed AALP solution demonstrates superior defense
performance on multiple datasets with extensive experiments.


http://arxiv.org/abs/2005.11886
The never ending war in the stack and the reincarnation of ROP attacks. (1%)
Ammari Nader; Joan Calvet; Jose M. Fernandez
  Return Oriented Programming (ROP) is a technique by which an attacker can
induce arbitrary behavior inside a vulnerable program without injecting a
malicious code. The continues failure of the currently deployed defenses
against ROP has made it again one of the most powerful memory corruption
attacks. ROP is also considered as one of the most flexible attacks, its level
of flexibility, unlike other code reuse attacks, can reach the Turing
completeness. Several efforts have been undertaken to study this threat and to
propose better defense mechanisms (mitigation or prevention), yet the majority
of them are not deeply reviewed nor officially implemented.Furthermore, similar
studies show that the techniques proposed to prevent ROP-based exploits usually
yield a high false-negative rate and a higher false-positive rate, not to
mention the overhead that they introduce into the protected program. The first
part of this research work aims at providing an in-depth analysis of the
currently available anti-ROP solutions (deployed and proposed), focusing on
inspecting their defense logic and summarizing their weaknesses and problems.
The second part of this work aims at introducing our proposed Indicators Of
Compromise (IOCs) that could be used to improve the detection rate of ROP
attacks. The three suggested indicators could detect these attacks at run-time
by checking the presence of some artifacts during the execution of the targeted
program.


http://arxiv.org/abs/2005.11560
Adversarial Attack on Hierarchical Graph Pooling Neural Networks. (99%)
Haoteng Tang; Guixiang Ma; Yurong Chen; Lei Guo; Wei Wang; Bo Zeng; Liang Zhan
  Recent years have witnessed the emergence and development of graph neural
networks (GNNs), which have been shown as a powerful approach for graph
representation learning in many tasks, such as node classification and graph
classification. The research on the robustness of these models has also started
to attract attentions in the machine learning field. However, most of the
existing work in this area focus on the GNNs for node-level tasks, while little
work has been done to study the robustness of the GNNs for the graph
classification task. In this paper, we aim to explore the vulnerability of the
Hierarchical Graph Pooling (HGP) Neural Networks, which are advanced GNNs that
perform very well in the graph classification in terms of prediction accuracy.
We propose an adversarial attack framework for this task. Specifically, we
design a surrogate model that consists of convolutional and pooling operators
to generate adversarial samples to fool the hierarchical GNN-based graph
classification models. We set the preserved nodes by the pooling operator as
our attack targets, and then we perturb the attack targets slightly to fool the
pooling operator in hierarchical GNNs so that they will select the wrong nodes
to preserve. We show the adversarial samples generated from multiple datasets
by our surrogate model have enough transferability to attack current
state-of-art graph classification models. Furthermore, we conduct the robust
train on the target models and demonstrate that the retrained graph
classification models are able to better defend against the attack from the
adversarial samples. To the best of our knowledge, this is the first work on
the adversarial attack against hierarchical GNN-based graph classification
models.


http://arxiv.org/abs/2005.11626
ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds. (96%)
Kibok Lee; Zhuoyuan Chen; Xinchen Yan; Raquel Urtasun; Ersin Yumer
  We introduce ShapeAdv, a novel framework to study shape-aware adversarial
perturbations that reflect the underlying shape variations (e.g., geometric
deformations and structural differences) in the 3D point cloud space. We
develop shape-aware adversarial 3D point cloud attacks by leveraging the
learned latent space of a point cloud auto-encoder where the adversarial noise
is applied in the latent space. Specifically, we propose three different
variants including an exemplar-based one by guiding the shape deformation with
auxiliary data, such that the generated point cloud resembles the shape
morphing between objects in the same category. Different from prior works, the
resulting adversarial 3D point clouds reflect the shape variations in the 3D
point cloud space while still being close to the original one. In addition,
experimental evaluations on the ModelNet40 benchmark demonstrate that our
adversaries are more difficult to defend with existing point cloud defense
methods and exhibit a higher attack transferability across classifiers. Our
shape-aware adversarial attacks are orthogonal to existing point cloud based
attacks and shed light on the vulnerability of 3D deep neural networks.


http://arxiv.org/abs/2005.11061
Vulnerability of deep neural networks for detecting COVID-19 cases from chest X-ray images to universal adversarial attacks. (99%)
Hokuto Hirano; Kazuki Koga; Kazuhiro Takemoto
  Under the epidemic of the novel coronavirus disease 2019 (COVID-19), chest
X-ray computed tomography imaging is being used for effectively screening
COVID-19 patients. The development of computer-aided systems based on deep
neural networks (DNNs) has been advanced, to rapidly and accurately detect
COVID-19 cases, because the need for expert radiologists, who are limited in
number, forms a bottleneck for the screening. However, so far, the
vulnerability of DNN-based systems has been poorly evaluated, although DNNs are
vulnerable to a single perturbation, called universal adversarial perturbation
(UAP), which can induce DNN failure in most classification tasks. Thus, we
focus on representative DNN models for detecting COVID-19 cases from chest
X-ray images and evaluate their vulnerability to UAPs generated using simple
iterative algorithms. We consider nontargeted UAPs, which cause a task failure
resulting in an input being assigned an incorrect label, and targeted UAPs,
which cause the DNN to classify an input into a specific class. The results
demonstrate that the models are vulnerable to nontargeted and targeted UAPs,
even in case of small UAPs. In particular, 2% norm of the UPAs to the average
norm of an image in the image dataset achieves >85% and >90% success rates for
the nontargeted and targeted attacks, respectively. Due to the nontargeted
UAPs, the DNN models judge most chest X-ray images as COVID-19 cases. The
targeted UAPs make the DNN models classify most chest X-ray images into a given
target class. The results indicate that careful consideration is required in
practical applications of DNNs to COVID-19 diagnosis; in particular, they
emphasize the need for strategies to address security concerns. As an example,
we show that iterative fine-tuning of the DNN models using UAPs improves the
robustness of the DNN models against UAPs.


http://arxiv.org/abs/2005.11295
From ImageNet to Image Classification: Contextualizing Progress on Benchmarks. (54%)
Dimitris Tsipras; Shibani Santurkar; Logan Engstrom; Andrew Ilyas; Aleksander Madry
  Building rich machine learning datasets in a scalable manner often
necessitates a crowd-sourced data collection pipeline. In this work, we use
human studies to investigate the consequences of employing such a pipeline,
focusing on the popular ImageNet dataset. We study how specific design choices
in the ImageNet creation process impact the fidelity of the resulting
dataset---including the introduction of biases that state-of-the-art models
exploit. Our analysis pinpoints how a noisy data collection pipeline can lead
to a systematic misalignment between the resulting benchmark and the real-world
task it serves as a proxy for. Finally, our findings emphasize the need to
augment our current model training and evaluation toolkit to take such
misalignments into account. To facilitate further research, we release our
refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.


http://arxiv.org/abs/2005.10757
Robust Ensemble Model Training via Random Layer Sampling Against Adversarial Attack. (99%)
Hakmin Lee; Hong Joo Lee; Seong Tae Kim; Yong Man Ro
  Deep neural networks have achieved substantial achievements in several
computer vision areas, but have vulnerabilities that are often fooled by
adversarial examples that are not recognized by humans. This is an important
issue for security or medical applications. In this paper, we propose an
ensemble model training framework with random layer sampling to improve the
robustness of deep neural networks. In the proposed training framework, we
generate various sampled model through the random layer sampling and update the
weight of the sampled model. After the ensemble models are trained, it can hide
the gradient efficiently and avoid the gradient-based attack by the random
layer sampling method. To evaluate our proposed method, comprehensive and
comparative experiments have been conducted on three datasets. Experimental
results show that the proposed method improves the adversarial robustness.


http://arxiv.org/abs/2005.10637
Inaudible Adversarial Perturbations for Targeted Attack in Speaker Recognition. (99%)
Qing Wang; Pengcheng Guo; Lei Xie
  Speaker recognition is a popular topic in biometric authentication and many
deep learning approaches have achieved extraordinary performances. However, it
has been shown in both image and speech applications that deep neural networks
are vulnerable to adversarial examples. In this study, we aim to exploit this
weakness to perform targeted adversarial attacks against the x-vector based
speaker recognition system. We propose to generate inaudible adversarial
perturbations achieving targeted white-box attacks to speaker recognition
system based on the psychoacoustic principle of frequency masking.
Specifically, we constrict the perturbation under the masking threshold of
original audio, instead of using a common l_p norm to measure the
perturbations. Experiments on Aishell-1 corpus show that our approach yields up
to 98.5% attack success rate to arbitrary gender speaker targets, while
retaining indistinguishable attribute to listeners. Furthermore, we also
achieve an effective speaker attack when applying the proposed approach to a
completely irrelevant waveform, such as music.


http://arxiv.org/abs/2005.10987
Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning. (99%)
Youngjoon Yu; Hong Joo Lee; Byeong Cheon Kim; Jung Uk Kim; Yong Man Ro
  The success of multimodal data fusion in deep learning appears to be
attributed to the use of complementary in-formation between multiple input
data. Compared to their predictive performance, relatively less attention has
been devoted to the robustness of multimodal fusion models. In this paper, we
investigated whether the current multimodal fusion model utilizes the
complementary intelligence to defend against adversarial attacks. We applied
gradient based white-box attacks such as FGSM and PGD on MFNet, which is a
major multispectral (RGB, Thermal) fusion deep learning model for semantic
segmentation. We verified that the multimodal fusion model optimized for better
prediction is still vulnerable to adversarial attack, even if only one of the
sensors is attacked. Thus, it is hard to say that existing multimodal data
fusion models are fully utilizing complementary relationships between multiple
modalities in terms of adversarial robustness. We believe that our observations
open a new horizon for adversarial attack research on multimodal data fusion.


http://arxiv.org/abs/2005.10750
Revisiting Role of Autoencoders in Adversarial Settings. (81%)
Byeong Cheon Kim; Jung Uk Kim; Hakmin Lee; Yong Man Ro
  To combat against adversarial attacks, autoencoder structure is widely used
to perform denoising which is regarded as gradient masking. In this paper, we
revisit the role of autoencoders in adversarial settings. Through the
comprehensive experimental results and analysis, this paper presents the
inherent property of adversarial robustness in the autoencoders. We also found
that autoencoders may use robust features that cause inherent adversarial
robustness. We believe that our discovery of the adversarial robustness of the
autoencoders can provide clues to the future research and applications for
adversarial defense.


http://arxiv.org/abs/2005.10190
Feature Purification: How Adversarial Training Performs Robust Deep Learning. (99%)
Zeyuan Allen-Zhu; Yuanzhi Li
  Despite the great empirical success of adversarial training to defend deep
learning models against adversarial perturbations, so far, it still remains
rather unclear what the principles are behind the existence of adversarial
perturbations, and what adversarial training does to the neural network to
remove them.
  In this paper, we present a principle that we call "feature purification",
where we show the existence of adversarial examples are due to the accumulation
of certain "dense mixtures" in the hidden weights during the training process
of a neural network; and more importantly, one of the goals of adversarial
training is to remove such mixtures to "purify" hidden weights. We present both
experiments on the CIFAR-10 dataset to illustrate this principle, and a
Theoretical Result proving that for certain natural classification tasks,
training a two-layer neural network with ReLU activation using randomly
initialized gradient descent indeed satisfies this principle.
  Technically, we give, to the best of our knowledge, the first result proving
that the following two can hold simultaneously for training a neural network
with ReLU activation. (1) Training over the original data is indeed non-robust
to small adversarial perturbations of some radius. (2) Adversarial training,
even with an empirical perturbation algorithm such as FGM, can in fact be
provably robust against ANY perturbations of the same radius. Finally, we also
prove a complexity lower bound, showing that low complexity models such as
linear classifiers, low-degree polynomials, or even the neural tangent kernel
for this network, CANNOT defend against perturbations of this same radius, no
matter what algorithms are used to train them.


http://arxiv.org/abs/2005.10203
Graph Structure Learning for Robust Graph Neural Networks. (99%)
Wei Jin; Yao Ma; Xiaorui Liu; Xianfeng Tang; Suhang Wang; Jiliang Tang
  Graph Neural Networks (GNNs) are powerful tools in representation learning
for graphs. However, recent studies show that GNNs are vulnerable to
carefully-crafted perturbations, called adversarial attacks. Adversarial
attacks can easily fool GNNs in making predictions for downstream tasks. The
vulnerability to adversarial attacks has raised increasing concerns for
applying GNNs in safety-critical applications. Therefore, developing robust
algorithms to defend adversarial attacks is of great significance. A natural
idea to defend adversarial attacks is to clean the perturbed graph. It is
evident that real-world graphs share some intrinsic properties. For example,
many real-world graphs are low-rank and sparse, and the features of two
adjacent nodes tend to be similar. In fact, we find that adversarial attacks
are likely to violate these graph properties. Therefore, in this paper, we
explore these properties to defend adversarial attacks on graphs. In
particular, we propose a general framework Pro-GNN, which can jointly learn a
structural graph and a robust graph neural network model from the perturbed
graph guided by these properties. Extensive experiments on real-world graphs
demonstrate that the proposed framework achieves significantly better
performance compared with the state-of-the-art defense methods, even when the
graph is heavily perturbed. We release the implementation of Pro-GNN to our
DeepRobust repository for adversarial attacks and defenses (footnote:
https://github.com/DSE-MSU/DeepRobust). The specific experimental settings to
reproduce our results can be found in https://github.com/ChandlerBang/Pro-GNN.


http://arxiv.org/abs/2005.10322
Adversarial Machine Learning in Recommender Systems: State of the art and Challenges. (98%)
Yashar Deldjoo; Noia Tommaso Di; Felice Antonio Merra
  Latent-factor models (LFM) based on collaborative filtering (CF), such as
matrix factorization (MF) and deep CF methods, are widely used in modern
recommender systems (RS) due to their excellent performance and recommendation
accuracy. Notwithstanding their great success, in recent years, it has been
shown that these methods are vulnerable to adversarial examples, i.e., subtle
but non-random perturbations designed to force recommendation models to produce
erroneous outputs. The main reason for this behavior is that user interaction
data used for training of LFM can be contaminated by malicious activities or
users' misoperation that can induce an unpredictable amount of natural noise
and harm recommendation outcomes. On the other side, it has been shown that
these systems, conceived originally to attack machine learning applications,
can be successfully adopted to strengthen their robustness against attacks as
well as to train more precise recommendation engines. In this respect, the goal
of this survey is two-fold: (i) to present recent advances on AML-RS for the
security of RS (i.e., attacking and defense recommendation models), (ii) to
show another successful application of AML in generative adversarial networks
(GANs), which use the core concept of learning in AML (i.e., the min-max game)
for generative applications. In this survey, we provide an exhaustive
literature review of 60 articles published in major RS and ML journals and
conferences. This review serves as a reference for the RS community, working on
the security of RS and recommendation models leveraging generative models to
improve their quality.


http://arxiv.org/abs/2005.10247
Model-Based Robust Deep Learning. (93%)
Alexander Robey; Hamed Hassani; George J. Pappas
  While deep learning has resulted in major breakthroughs in many application
domains, the frameworks commonly used in deep learning remain fragile to
artificially-crafted and imperceptible changes in the data. In response to this
fragility, adversarial training has emerged as a principled approach for
enhancing the robustness of deep learning with respect to norm-bounded
perturbations. However, there are other sources of fragility for deep learning
that are arguably more common and less thoroughly studied. Indeed, natural
variation such as lighting or weather conditions can significantly degrade the
accuracy of trained neural networks, proving that such natural variation
presents a significant challenge for deep learning.
  In this paper, we propose a paradigm shift from perturbation-based
adversarial robustness toward {\em model-based robust deep learning}. Our
objective is to provide general training algorithms that can be used to train
deep neural networks to be robust against natural variation in data. Critical
to our paradigm is first obtaining a \emph{model of natural variation} which
can be used to vary data over a range of natural conditions. Such models may be
either known a priori or else learned from data. In the latter case, we show
that deep generative models can be used to learn models of natural variation
that are consistent with realistic conditions. We then exploit such models in
three novel model-based robust training algorithms in order to enhance the
robustness of deep learning with respect to the given model. Our extensive
experiments show that across a variety of naturally-occurring conditions and
across various datasets, deep neural networks trained with our model-based
algorithms significantly outperform both standard deep learning algorithms as
well as norm-bounded robust deep learning algorithms.


http://arxiv.org/abs/2005.10284
An Adversarial Approach for Explaining the Predictions of Deep Neural Networks. (89%)
Arash Rahnama; Andrew Tseng
  Machine learning models have been successfully applied to a wide range of
applications including computer vision, natural language processing, and speech
recognition. A successful implementation of these models however, usually
relies on deep neural networks (DNNs) which are treated as opaque black-box
systems due to their incomprehensible complexity and intricate internal
mechanism. In this work, we present a novel algorithm for explaining the
predictions of a DNN using adversarial machine learning. Our approach
identifies the relative importance of input features in relation to the
predictions based on the behavior of an adversarial attack on the DNN. Our
algorithm has the advantage of being fast, consistent, and easy to implement
and interpret. We present our detailed analysis that demonstrates how the
behavior of an adversarial attack, given a DNN and a task, stays consistent for
any input test data point proving the generality of our approach. Our analysis
enables us to produce consistent and efficient explanations. We illustrate the
effectiveness of our approach by conducting experiments using a variety of
DNNs, tasks, and datasets. Finally, we compare our work with other well-known
techniques in the current literature.


http://arxiv.org/abs/2005.10333
A Way Around UMIP and Descriptor-Table Exiting via TSX-based Side-Channel Attack. (1%)
Mohammad Sina Karvandi; Saleh Khalaj Monfared; Mohammad Sina Kiarostami; Dara Rahmati; Saeid Gorgin
  Nowadays, in operating systems, numerous protection mechanisms prevent or
limit the user-mode applications to access the kernel's internal information.
This is regularly carried out by software-based defenses such as Address Space
Layout Randomization (ASLR) and Kernel ASLR (KASLR). They play pronounced roles
when the security of sandboxed applications such as Web-browser are considered.
Armed with arbitrary write access in the kernel memory, if these protections
are bypassed, an attacker could find a suitable Where to Write in order to get
an elevation of privilege or maliciously execute codes in ring 0. In this
paper, we introduce a reliable method based on Transactional Synchronization
Extensions (TSX) side-channel attacks to reveal the address of the Global
Descriptor Table (GDT) and Interrupt Descriptor Table (IDT). We indicate that
by detecting these addresses, an attack could be executed to sidestep the
Intel's User-Mode Instruction Prevention (UMIP) and the Hypervisor-based
mitigation and, consequently, neutralized them. The introduced attack is
successfully performed after the most recent patches for Meltdown and Spectre.
Moreover, the implementation of the proposed attack on different platforms,
including the latest releases of Microsoft Windows, Linux, and, Mac OSX with
the latest $9^{th}$ generation of Intel processors, shows that the attack is
independent of the Operating System implementation. We demonstrate that a
combination of this method with call-gate mechanism (available in modern
processors) in a chain of attacks will eventually lead to a full system
compromise despite the limitations of a super-secure sandboxed environment in
the presence of Windows's proprietary Virtualization Based Security (VBS).
Finally, we suggest the software-based mitigation to avoid these attacks with
an acceptable cost.


http://arxiv.org/abs/2005.09294
Synthesizing Unrestricted False Positive Adversarial Objects Using Generative Models. (99%)
Martin Kotuliak; Sandro E. Schoenborn; Andrei Dan
  Adversarial examples are data points misclassified by neural networks.
Originally, adversarial examples were limited to adding small perturbations to
a given image. Recent work introduced the generalized concept of unrestricted
adversarial examples, without limits on the added perturbations. In this paper,
we introduce a new category of attacks that create unrestricted adversarial
examples for object detection. Our key idea is to generate adversarial objects
that are unrelated to the classes identified by the target object detector.
Different from previous attacks, we use off-the-shelf Generative Adversarial
Networks (GAN), without requiring any further training or modification. Our
method consists of searching over the latent normal space of the GAN for
adversarial objects that are wrongly identified by the target object detector.
We evaluate this method on the commonly used Faster R-CNN ResNet-101, Inception
v2 and SSD Mobilenet v1 object detectors using logo generative iWGAN-LC and
SNGAN trained on CIFAR-10. The empirical results show that the generated
adversarial objects are indistinguishable from non-adversarial objects
generated by the GANs, transferable between the object detectors and robust in
the physical world. This is the first work to study unrestricted false positive
adversarial examples for object detection.


http://arxiv.org/abs/2005.09257
Patch Attack for Automatic Check-out. (99%)
Aishan Liu; Jiakai Wang; Xianglong Liu; Chongzhi Zhang; Bowen Cao; Hang Yu
  Adversarial examples are inputs with imperceptible perturbations that easily
misleading deep neural networks(DNNs). Recently, adversarial patch, with noise
confined to a small and localized patch, has emerged for its easy feasibility
in real-world scenarios. However, existing strategies failed to generate
adversarial patches with strong generalization ability. In other words, the
adversarial patches were input-specific and failed to attack images from all
classes, especially unseen ones during training. To address the problem, this
paper proposes a bias-based framework to generate class-agnostic universal
adversarial patches with strong generalization ability, which exploits both the
perceptual and semantic bias of models. Regarding the perceptual bias, since
DNNs are strongly biased towards textures, we exploit the hard examples which
convey strong model uncertainties and extract a textural patch prior from them
by adopting the style similarities. The patch prior is more close to decision
boundaries and would promote attacks. To further alleviate the heavy dependency
on large amounts of data in training universal attacks, we further exploit the
semantic bias. As the class-wise preference, prototypes are introduced and
pursued by maximizing the multi-class margin to help universal training. Taking
AutomaticCheck-out (ACO) as the typical scenario, extensive experiments
including white-box and black-box settings in both digital-world(RPC, the
largest ACO related dataset) and physical-world scenario(Taobao and JD, the
world' s largest online shopping platforms) are conducted. Experimental results
demonstrate that our proposed framework outperforms state-of-the-art
adversarial patch attack methods.


http://arxiv.org/abs/2005.09296
SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation. (54%)
Sungmin School of Computing KAIST Kang; Robert Chalmers University Feldt; Shin School of Computing KAIST Yoo
  The testing of Deep Neural Networks (DNNs) has become increasingly important
as DNNs are widely adopted by safety critical systems. While many test adequacy
criteria have been suggested, automated test input generation for many types of
DNNs remains a challenge because the raw input space is too large to randomly
sample or to navigate and search for plausible inputs. Consequently, current
testing techniques for DNNs depend on small local perturbations to existing
inputs, based on the metamorphic testing principle. We propose new ways to
search not over the entire image space, but rather over a plausible input space
that resembles the true training distribution. This space is constructed using
Variational Autoencoders (VAEs), and navigated through their latent vector
space. We show that this space helps efficiently produce test inputs that can
reveal information about the robustness of DNNs when dealing with realistic
tests, opening the field to meaningful exploration through the space of highly
structured images.


http://arxiv.org/abs/2005.09619
Identifying Statistical Bias in Dataset Replication. (50%)
Logan Engstrom; Andrew Ilyas; Shibani Santurkar; Dimitris Tsipras; Jacob Steinhardt; Aleksander Madry
  Dataset replication is a useful tool for assessing whether improvements in
test accuracy on a specific benchmark correspond to improvements in models'
ability to generalize reliably. In this work, we present unintuitive yet
significant ways in which standard approaches to dataset replication introduce
statistical bias, skewing the resulting observations. We study ImageNet-v2, a
replication of the ImageNet dataset on which models exhibit a significant
(11-14%) drop in accuracy, even after controlling for a standard
human-in-the-loop measure of data quality. We show that after correcting for
the identified statistical bias, only an estimated $3.6\% \pm 1.5\%$ of the
original $11.7\% \pm 1.0\%$ accuracy drop remains unaccounted for. We conclude
with concrete recommendations for recognizing and avoiding bias in dataset
replication. Code for our study is publicly available at
http://github.com/MadryLab/dataset-replication-analysis .


http://arxiv.org/abs/2005.09800
Fingerprinting Encrypted Voice Traffic on Smart Speakers with Deep Learning. (9%)
Chenggang Wang; Sean Kennedy; Haipeng Li; King Hudson; Gowtham Atluri; Xuetao Wei; Wenhai Sun; Boyang Wang
  This paper investigates the privacy leakage of smart speakers under an
encrypted traffic analysis attack, referred to as voice command fingerprinting.
In this attack, an adversary can eavesdrop both outgoing and incoming encrypted
voice traffic of a smart speaker, and infers which voice command a user says
over encrypted traffic. We first built an automatic voice traffic collection
tool and collected two large-scale datasets on two smart speakers, Amazon Echo
and Google Home. Then, we implemented proof-of-concept attacks by leveraging
deep learning. Our experimental results over the two datasets indicate
disturbing privacy concerns. Specifically, compared to 1% accuracy with random
guess, our attacks can correctly infer voice commands over encrypted traffic
with 92.89\% accuracy on Amazon Echo. Despite variances that human voices may
cause on outgoing traffic, our proof-of-concept attacks remain effective even
only leveraging incoming traffic (i.e., the traffic from the server). This is
because the AI-based voice services running on the server side response
commands in the same voice and with a deterministic or predictable manner in
text, which leaves distinguishable pattern over encrypted traffic. We also
built a proof-of-concept defense to obfuscate encrypted traffic. Our results
show that the defense can effectively mitigate attack accuracy on Amazon Echo
to 32.18%.


http://arxiv.org/abs/2005.09363
Enhancing Certified Robustness of Smoothed Classifiers via Weighted Model Ensembling. (5%)
Chizhou Liu; Yunzhen Feng; Ranran Wang; Bin Dong
  Randomized smoothing has achieved state-of-the-art certified robustness
against $l_2$-norm adversarial attacks. However, it also leads to accuracy drop
compared to the normally trained models. In this work, we employ a Smoothed
WEighted ENsembling (SWEEN) scheme to improve the performance of randomized
smoothed classifiers. We characterize the optimal certified robustness
attainable by SWEEN models. We show the accessibility of SWEEN models attaining
the lowest risk w.r.t. a surrogate loss function. We also develop an adaptive
prediction algorithm to reduce the prediction and certification cost of SWEEN
models. Extensive experiments show that SWEEN models outperform the upper
envelope of their corresponding base models by a large margin. Moreover, SWEEN
models constructed using a few small models are able to achieve comparable
performance to a single large model with notably reduced training time.


http://arxiv.org/abs/2005.09147
Increasing-Margin Adversarial (IMA) Training to Improve Adversarial Robustness of Neural Networks. (99%)
Linhai Ma; Liang Liang
  Convolutional neural network (CNN) has surpassed traditional methods for
med-ical image classification. However, CNN is vulnerable to adversarial
attacks which may lead to disastrous consequences in medical applications.
Although adversarial noises are usually generated by attack algorithms,
white-noise-induced adversarial samples can exist, and therefore the threats
are real. In this study, we propose a novel training method, named IMA, to
improve the robust-ness of CNN against adversarial noises. During training, the
IMA method in-creases the margins of training samples in the input space, i.e.,
moving CNN de-cision boundaries far away from the training samples to improve
robustness. The IMA method is evaluated on four publicly available datasets
under strong 100-PGD white-box adversarial attacks, and the results show that
the proposed meth-od significantly improved CNN classification accuracy on
noisy data while keep-ing a relatively high accuracy on clean data. We hope our
approach may facilitate the development of robust applications in medical
field.


http://arxiv.org/abs/2005.09161
Adversarial Attacks for Embodied Agents. (99%)
Aishan Liu; Tairan Huang; Xianglong Liu; Yitao Xu; Yuqing Ma; Xinyun Chen; Stephen J. Maybank; Dacheng Tao
  Adversarial attacks are valuable for providing insights into the blind-spots
of deep learning models and help improve their robustness. Existing work on
adversarial attacks have mainly focused on static scenes; however, it remains
unclear whether such attacks are effective against embodied agents, which could
navigate and interact with a dynamic environment. In this work, we take the
first step to study adversarial attacks for embodied agents. In particular, we
generate spatiotemporal perturbations to form 3D adversarial examples, which
exploit the interaction history in both the temporal and spatial dimensions.
Regarding the temporal dimension, since agents make predictions based on
historical observations, we develop a trajectory attention module to explore
scene view contributions, which further help localize 3D objects appeared with
the highest stimuli. By conciliating with clues from the temporal dimension,
along the spatial dimension, we adversarially perturb the physical properties
(e.g., texture and 3D shape) of the contextual objects that appeared in the
most important scene views. Extensive experiments on the EQA-v1 dataset for
several embodied tasks in both the white-box and black-box settings have been
conducted, which demonstrate that our perturbations have strong attack and
generalization abilities.


http://arxiv.org/abs/2005.08781
Defending Your Voice: Adversarial Attack on Voice Conversion. (99%)
Chien-yu Huang; Yist Y. Lin; Hung-yi Lee; Lin-shan Lee
  Substantial improvements have been achieved in recent years in voice
conversion, which converts the speaker characteristics of an utterance into
those of another speaker without changing the linguistic content of the
utterance. Nonetheless, the improved conversion technologies also led to
concerns about privacy and authentication. It thus becomes highly desired to be
able to prevent one's voice from being improperly utilized with such voice
conversion technologies. This is why we report in this paper the first known
attempt to try to perform adversarial attack on voice conversion. We introduce
human imperceptible noise into the utterances of a speaker whose voice is to be
defended. Given these adversarial examples, voice conversion models cannot
convert other utterances so as to sound like being produced by the defended
speaker. Preliminary experiments were conducted on two currently
state-of-the-art zero-shot voice conversion models. Objective and subjective
evaluation results in both white-box and black-box scenarios are reported. It
was shown that the speaker characteristics of the converted utterances were
made obviously different from those of the defended speaker, while the
adversarial examples of the defended speaker are not distinguishable from the
authentic utterances.


http://arxiv.org/abs/2005.09134
Improve robustness of DNN for ECG signal classification:a noise-to-signal ratio perspective. (99%)
Linhai Ma; Liang Liang
  Electrocardiogram (ECG) is the most widely used diagnostic tool to monitor
the condition of the cardiovascular system. Deep neural networks (DNNs), have
been developed in many research labs for automatic interpretation of ECG
signals to identify potential abnormalities in patient hearts. Studies have
shown that given a sufficiently large amount of data, the classification
accuracy of DNNs could reach human-expert cardiologist level. A DNN-based
automated ECG diagnostic system would be an affordable solution for patients in
developing countries where human-expert cardiologist are lacking. However,
despite of the excellent performance in classification accuracy, it has been
shown that DNNs are highly vulnerable to adversarial attacks: subtle changes in
input of a DNN can lead to a wrong classification output with high confidence.
Thus, it is challenging and essential to improve adversarial robustness of DNNs
for ECG signal classification, a life-critical application. In this work, we
proposed to improve DNN robustness from the perspective of noise-to-signal
ratio (NSR) and developed two methods to minimize NSR during training process.
We evaluated the proposed methods on PhysionNets MIT-BIH dataset, and the
results show that our proposed methods lead to an enhancement in robustness
against PGD adversarial attack and SPSA attack, with a minimal change in
accuracy on clean data.


http://arxiv.org/abs/2005.08632
Universalization of any adversarial attack using very few test examples. (99%)
Sandesh Kamath; Amit Deshpande; K V Subrahmanyam
  Deep learning models are known to be vulnerable not only to input-dependent
adversarial attacks but also to input-agnostic or universal adversarial
attacks. Dezfooli et al. \cite{Dezfooli17,Dezfooli17anal} construct universal
adversarial attack on a given model by looking at a large number of training
data points and the geometry of the decision boundary near them. Subsequent
work \cite{Khrulkov18} constructs universal attack by looking only at test
examples and intermediate layers of the given model. In this paper, we propose
a simple universalization technique to take any input-dependent adversarial
attack and construct a universal attack by only looking at very few adversarial
test examples. We do not require details of the given model and have negligible
computational overhead for universalization. We theoretically justify our
universalization technique by a spectral property common to many
input-dependent adversarial perturbations, e.g., gradients, Fast Gradient Sign
Method (FGSM) and DeepFool. Using matrix concentration inequalities and
spectral perturbation bounds, we show that the top singular vector of
input-dependent adversarial directions on a small test sample gives an
effective and simple universal adversarial attack. For VGG16 and VGG19 models
trained on ImageNet, our simple universalization of Gradient, FGSM, and
DeepFool perturbations using a test sample of 64 images gives fooling rates
comparable to state-of-the-art universal attacks \cite{Dezfooli17,Khrulkov18}
for reasonable norms of perturbation.


http://arxiv.org/abs/2005.09170
On Intrinsic Dataset Properties for Adversarial Machine Learning. (99%)
Jeffrey Z. Pan; Nicholas Zufelt
  Deep neural networks (DNNs) have played a key role in a wide range of machine
learning applications. However, DNN classifiers are vulnerable to
human-imperceptible adversarial perturbations, which can cause them to
misclassify inputs with high confidence. Thus, creating robust DNNs which can
defend against malicious examples is critical in applications where security
plays a major role. In this paper, we study the effect of intrinsic dataset
properties on the performance of adversarial attack and defense methods,
testing on five popular image classification datasets - MNIST, Fashion-MNIST,
CIFAR10/CIFAR100, and ImageNet. We find that input size and image contrast play
key roles in attack and defense success. Our discoveries highlight that dataset
design and data preprocessing steps are important to boost the adversarial
robustness of DNNs. To our best knowledge, this is the first comprehensive work
that studies the effect of intrinsic dataset properties on adversarial machine
learning.


http://arxiv.org/abs/2005.08454
An Evasion Attack against ML-based Phishing URL Detectors. (93%)
Bushra University of Adelaide, CREST - The Centre for Research on Engineering Software Technologies, CSIROs Data61 Sabir; M. Ali University of Adelaide, CREST - The Centre for Research on Engineering Software Technologies Babar; Raj CSIROs Data61 Gaire
  Background: Over the year, Machine Learning Phishing URL classification
(MLPU) systems have gained tremendous popularity to detect phishing URLs
proactively. Despite this vogue, the security vulnerabilities of MLPUs remain
mostly unknown. Aim: To address this concern, we conduct a study to understand
the test time security vulnerabilities of the state-of-the-art MLPU systems,
aiming at providing guidelines for the future development of these systems.
Method: In this paper, we propose an evasion attack framework against MLPU
systems. To achieve this, we first develop an algorithm to generate adversarial
phishing URLs. We then reproduce 41 MLPU systems and record their baseline
performance. Finally, we simulate an evasion attack to evaluate these MLPU
systems against our generated adversarial URLs. Results: In comparison to
previous works, our attack is: (i) effective as it evades all the models with
an average success rate of 66% and 85% for famous (such as Netflix, Google) and
less popular phishing targets (e.g., Wish, JBHIFI, Officeworks) respectively;
(ii) realistic as it requires only 23ms to produce a new adversarial URL
variant that is available for registration with a median cost of only
$11.99/year. We also found that popular online services such as Google
SafeBrowsing and VirusTotal are unable to detect these URLs. (iii) We find that
Adversarial training (successful defence against evasion attack) does not
significantly improve the robustness of these systems as it decreases the
success rate of our attack by only 6% on average for all the models. (iv)
Further, we identify the security vulnerabilities of the considered MLPU
systems. Our findings lead to promising directions for future research.
Conclusion: Our study not only illustrate vulnerabilities in MLPU systems but
also highlights implications for future study towards assessing and improving
these systems.


http://arxiv.org/abs/2005.08679
An Overview of Privacy in Machine Learning. (10%)
Cristofaro Emiliano De
  Over the past few years, providers such as Google, Microsoft, and Amazon have
started to provide customers with access to software interfaces allowing them
to easily embed machine learning tasks into their applications. Overall,
organizations can now use Machine Learning as a Service (MLaaS) engines to
outsource complex tasks, e.g., training classifiers, performing predictions,
clustering, etc. They can also let others query models trained on their data.
Naturally, this approach can also be used (and is often advocated) in other
contexts, including government collaborations, citizen science projects, and
business-to-business partnerships. However, if malicious users were able to
recover data used to train these models, the resulting information leakage
would create serious issues. Likewise, if the inner parameters of the model are
considered proprietary information, then access to the model should not allow
an adversary to learn such parameters. In this document, we set to review
privacy challenges in this space, providing a systematic review of the relevant
research literature, also exploring possible countermeasures. More
specifically, we provide ample background information on relevant concepts
around machine learning and privacy. Then, we discuss possible adversarial
models and settings, cover a wide range of attacks that relate to private
and/or sensitive information leakage, and review recent results attempting to
defend against such attacks. Finally, we conclude with a list of open problems
that require more work, including the need for better evaluations, more
targeted defenses, and the study of the relation to policy and data protection
efforts.


http://arxiv.org/abs/2005.08787
Spotr: GPS Spoofing Detection via Device Fingerprinting. (1%)
Mahsa Foruhandeh; Abdullah Z. Mohammed; Gregor Kildow; Paul Berges; Ryan Gerdes
  As the worlds predominant navigation system GPS is critical to modern life,
finding applications in diverse areas like information security, healthcare,
marketing, and power and water grid management. Unfortunately this
diversification has only served to underscore the insecurity of GPS and the
critical need to harden this system against manipulation and exploitation. A
wide variety of attacks against GPS have already been documented, both in
academia and industry. Several defenses have been proposed to combat these
attacks, but they are ultimately insufficient due to scope, expense,
complexity, or robustness. With this in mind, we present our own solution:
fingerprinting of GPS satellites. We assert that it is possible to create
signatures, or fingerprints, of the satellites (more specifically their
transmissions) that allow one to determine nearly instantly whether a received
GPS transmission is authentic or not. Furthermore, in this paper we demonstrate
that this solution detects all known spoofing attacks, that it does so while
being fast, cheap, and simpler than previous solutions, and that it is highly
robust with respect to environmental factors.


http://arxiv.org/abs/2005.08321
Toward Adversarial Robustness by Diversity in an Ensemble of Specialized Deep Neural Networks. (99%)
Mahdieh Abbasi; Arezoo Rajabi; Christian Gagne; Rakesh B. Bobba
  We aim at demonstrating the influence of diversity in the ensemble of CNNs on
the detection of black-box adversarial instances and hardening the generation
of white-box adversarial attacks. To this end, we propose an ensemble of
diverse specialized CNNs along with a simple voting mechanism. The diversity in
this ensemble creates a gap between the predictive confidences of adversaries
and those of clean samples, making adversaries detectable. We then analyze how
diversity in such an ensemble of specialists may mitigate the risk of the
black-box and white-box adversarial examples. Using MNIST and CIFAR-10, we
empirically verify the ability of our ensemble to detect a large portion of
well-known black-box adversarial examples, which leads to a significant
reduction in the risk rate of adversaries, at the expense of a small increase
in the risk rate of clean samples. Moreover, we show that the success rate of
generating white-box attacks by our ensemble is remarkably decreased compared
to a vanilla CNN and an ensemble of vanilla CNNs, highlighting the beneficial
role of diversity in the ensemble for developing more robust models.


http://arxiv.org/abs/2005.08164
Attacking Recommender Systems with Augmented User Profiles. (88%)
Chen Lin; Si Chen; Hui Li; Yanghua Xiao; Lianyun Li; Qian Yang
  Recommendation Systems (RS) have become an essential part of many online
services. Due to its pivotal role of guiding customers towards purchasing,
there is a natural motivation for unscrupulous parties to spoof RS for profits.
In this paper we study the shilling attack: a subsistent and profitable attack
where an adversarial party injects a number of user profiles to promote or
demote a target item. Conventional shilling attack models are based on simple
heuristics that can be easily detected, or directly adopt adversarial attack
methods without a special design for RS. Moreover, the study on the attack
impact on deep learning based RS is missing in the literature, making the
effects of shilling attack against real RS doubtful. We present a novel
Augmented Shilling Attack framework (AUSH) and implement it with the idea of
Generative Adversarial Network. AUSH is capable of tailoring attacks against RS
according to budget and complex attack goals such as targeting on a specific
user group. We experimentally show that the attack impact of AUSH is noticeable
on a wide range of RS including both classic and modern deep learning based RS,
while it is virtually undetectable by the state-of-the-art attack detection
model.


http://arxiv.org/abs/2005.07998
Encryption Inspired Adversarial Defense for Visual Classification. (99%)
MaungMaung AprilPyone; Hitoshi Kiya
  Conventional adversarial defenses reduce classification accuracy whether or
not a model is under attacks. Moreover, most of image processing based defenses
are defeated due to the problem of obfuscated gradients. In this paper, we
propose a new adversarial defense which is a defensive transform for both
training and test images inspired by perceptual image encryption methods. The
proposed method utilizes a block-wise pixel shuffling method with a secret key.
The experiments are carried out on both adaptive and non-adaptive maximum-norm
bounded white-box attacks while considering obfuscated gradients. The results
show that the proposed defense achieves high accuracy (91.55 %) on clean images
and (89.66 %) on adversarial examples with noise distance of 8/255 on CIFAR-10
dataset. Thus, the proposed defense outperforms state-of-the-art adversarial
defenses including latent adversarial training, adversarial training and
thermometer encoding.


http://arxiv.org/abs/2005.10884
PatchGuard: Provable Defense against Adversarial Patches Using Masks on Small Receptive Fields. (99%)
Chong Xiang; Arjun Nitin Bhagoji; Vikash Sehwag; Prateek Mittal
  Localized adversarial patches aim to induce misclassification in machine
learning models by arbitrarily modifying pixels within a restricted region of
an image. Such attacks can be realized in the physical world by attaching the
adversarial patch to the object to be misclassified. In this paper, we propose
a general defense framework that can achieve both high clean accuracy and
provable robustness against localized adversarial patches. The cornerstone of
our defense framework is to use a convolutional network with small receptive
fields that impose a bound on the number of features corrupted by an
adversarial patch. We further present the robust masking defense that robustly
detects and masks corrupted features for a secure feature aggregation. We
evaluate our defense against the most powerful white-box untargeted adaptive
attacker and achieve a 92.3% clean accuracy and an 85.2% provable robust
accuracy on a 10-class subset of ImageNet against a 31x31 adversarial patch (2%
pixels), a 57.4% clean accuracy and a 14.4% provable robust accuracy on
1000-class ImageNet against a 31x31 patch (2% pixels), and an 80.3% clean
accuracy and a 61.3% provable accuracy on CIFAR-10 against a 5x5 patch (2.4%
pixels). Notably, our provable defenses achieve state-of-the-art provable
robust accuracy on ImageNet and CIFAR-10.


http://arxiv.org/abs/2005.08087
Universal Adversarial Perturbations: A Survey. (98%)
Ashutosh Chaubey; Nikhil Agrawal; Kavya Barnwal; Keerat K. Guliani; Pramod Mehta
  Over the past decade, Deep Learning has emerged as a useful and efficient
tool to solve a wide variety of complex learning problems ranging from image
classification to human pose estimation, which is challenging to solve using
statistical machine learning algorithms. However, despite their superior
performance, deep neural networks are susceptible to adversarial perturbations,
which can cause the network's prediction to change without making perceptible
changes to the input image, thus creating severe security issues at the time of
deployment of such systems. Recent works have shown the existence of Universal
Adversarial Perturbations, which, when added to any image in a dataset,
misclassifies it when passed through a target model. Such perturbations are
more practical to deploy since there is minimal computation done during the
actual attack. Several techniques have also been proposed to defend the neural
networks against these perturbations. In this paper, we attempt to provide a
detailed discussion on the various data-driven and data-independent methods for
generating universal perturbations, along with measures to defend against such
perturbations. We also cover the applications of such universal perturbations
in various deep learning tasks.


http://arxiv.org/abs/2005.08147
Attacking Black-box Recommendations via Copying Cross-domain User Profiles. (62%)
Wenqi Fan; Tyler Derr; Xiangyu Zhao; Yao Ma; Hui Liu; Jianping Wang; Jiliang Tang; Qing Li
  Recently, recommender systems that aim to suggest personalized lists of items
for users to interact with online have drawn a lot of attention. In fact, many
of these state-of-the-art techniques have been deep learning based. Recent
studies have shown that these deep learning models (in particular for
recommendation systems) are vulnerable to attacks, such as data poisoning,
which generates users to promote a selected set of items. However, more
recently, defense strategies have been developed to detect these generated
users with fake profiles. Thus, advanced injection attacks of creating more
`realistic' user profiles to promote a set of items is still a key challenge in
the domain of deep learning based recommender systems. In this work, we present
our framework CopyAttack, which is a reinforcement learning based black-box
attack method that harnesses real users from a source domain by copying their
profiles into the target domain with the goal of promoting a subset of items.
CopyAttack is constructed to both efficiently and effectively learn policy
gradient networks that first select, and then further refine/craft, user
profiles from the source domain to ultimately copy into the target domain.
CopyAttack's goal is to maximize the hit ratio of the targeted items in the
Top-$k$ recommendation list of the users in the target domain. We have
conducted experiments on two real-world datasets and have empirically verified
the effectiveness of our proposed framework and furthermore performed a
thorough model analysis.


http://arxiv.org/abs/2005.08041
NeuroAttack: Undermining Spiking Neural Networks Security through Externally Triggered Bit-Flips. (31%)
Valerio Venceslai; Alberto Marchisio; Ihsen Alouani; Maurizio Martina; Muhammad Shafique
  Due to their proven efficiency, machine-learning systems are deployed in a
wide range of complex real-life problems. More specifically, Spiking Neural
Networks (SNNs) emerged as a promising solution to the accuracy,
resource-utilization, and energy-efficiency challenges in machine-learning
systems. While these systems are going mainstream, they have inherent security
and reliability issues. In this paper, we propose NeuroAttack, a cross-layer
attack that threatens the SNNs integrity by exploiting low-level reliability
issues through a high-level attack. Particularly, we trigger a fault-injection
based sneaky hardware backdoor through a carefully crafted adversarial input
noise. Our results on Deep Neural Networks (DNNs) and SNNs show a serious
integrity threat to state-of-the art machine-learning techniques.


http://arxiv.org/abs/2005.08016
DAMIA: Leveraging Domain Adaptation as a Defense against Membership Inference Attacks. (2%)
Hongwei Huang; Weiqi Luo; Guoqiang Zeng; Jian Weng; Yue Zhang; Anjia Yang
  Deep Learning (DL) techniques allow ones to train models from a dataset to
solve tasks. DL has attracted much interest given its fancy performance and
potential market value, while security issues are amongst the most colossal
concerns. However, the DL models may be prone to the membership inference
attack, where an attacker determines whether a given sample is from the
training dataset. Efforts have been made to hinder the attack but
unfortunately, they may lead to a major overhead or impaired usability. In this
paper, we propose and implement DAMIA, leveraging Domain Adaptation (DA) as a
defense aginist membership inference attacks. Our observation is that during
the training process, DA obfuscates the dataset to be protected using another
related dataset, and derives a model that underlyingly extracts the features
from both datasets. Seeing that the model is obfuscated, membership inference
fails, while the extracted features provide supports for usability. Extensive
experiments have been conducted to validates our intuition. The model trained
by DAMIA has a negligible footprint to the usability. Our experiment also
excludes factors that may hinder the performance of DAMIA, providing a
potential guideline to vendors and researchers to benefit from our solution in
a timely manner.


http://arxiv.org/abs/2005.07675
How to Make 5G Communications "Invisible": Adversarial Machine Learning for Wireless Privacy. (99%)
Brian Kim; Yalin E. Sagduyu; Kemal Davaslioglu; Tugba Erpek; Sennur Ulukus
  We consider the problem of hiding wireless communications from an
eavesdropper that employs a deep learning (DL) classifier to detect whether any
transmission of interest is present or not. There exists one transmitter that
transmits to its receiver in the presence of an eavesdropper, while a
cooperative jammer (CJ) transmits carefully crafted adversarial perturbations
over the air to fool the eavesdropper into classifying the received
superposition of signals as noise. The CJ puts an upper bound on the strength
of perturbation signal to limit its impact on the bit error rate (BER) at the
receiver. We show that this adversarial perturbation causes the eavesdropper to
misclassify the received signals as noise with high probability while
increasing the BER only slightly. On the other hand, the CJ cannot fool the
eavesdropper by simply transmitting Gaussian noise as in conventional jamming
and instead needs to craft perturbation signals built by adversarial machine
learning to enable covert communications. Our results show that signals with
different modulation types and eventually 5G communications can be effectively
hidden from an eavesdropper even if it is equipped with a DL classifier to
detect transmissions.


http://arxiv.org/abs/2005.07519
Practical Traffic-space Adversarial Attacks on Learning-based NIDSs. (99%)
Dongqi Han; Zhiliang Wang; Ying Zhong; Wenqi Chen; Jiahai Yang; Shuqiang Lu; Xingang Shi; Xia Yin
  Machine learning (ML) techniques have been increasingly used in anomaly-based
network intrusion detection systems (NIDS) to detect unknown attacks. However,
ML has shown to be extremely vulnerable to adversarial attacks, aggravating the
potential risk of evasion attacks against learning-based NIDSs. In this
situation, prior studies on evading traditional anomaly-based or
signature-based NIDSs are no longer valid. Existing attacks on learning-based
NIDSs mostly focused on feature-space and/or white-box attacks, leaving the
study on practical gray/black-box attacks largely unexplored. To bridge this
gap, we conduct the first systematic study of the practical traffic-space
evasion attack on learning-based NIDSs. We outperform the previous work in the
following aspects: (1) practical---instead of directly modifying features, we
provide a novel framework to automatically mutate malicious traffic with
extremely limited knowledge while preserving its functionality; (2)
generic---the proposed attack is effective for any ML classifiers (i.e.,
model-agnostic) and most non-payload-based features; (3) explainable---we
propose a feature-based interpretation method to measure the robustness of
targeted systems against such attacks. We extensively evaluate our attack and
defense scheme on Kitsune, a state-of-the-art learning-based NIDS, as well as
measuring the robustness of various NIDSs using diverse features and ML
classifiers. Experimental results show promising results and intriguing
findings.


http://arxiv.org/abs/2005.07606
Initializing Perturbations in Multiple Directions for Fast Adversarial Training. (99%)
Xunguang Wang; Ship Peng Xu; Eric Ke Wang
  Recent developments in the filed of Deep Learning have demonstrated that Deep
Neural Networks(DNNs) are vulnerable to adversarial examples. Specifically, in
image classification, an adversarial example can fool the well trained deep
neural networks by adding barely imperceptible perturbations to clean images.
Adversarial Training, one of the most direct and effective methods, minimizes
the losses of perturbed-data to learn robust deep networks against adversarial
attacks. It has been proven that using the fast gradient sign method (FGSM) can
achieve Fast Adversarial Training. However, FGSM-based adversarial training may
finally obtain a failed model because of overfitting to FGSM samples. In this
paper, we proposed the Diversified Initialized Perturbations Adversarial
Training (DIP-FAT) which involves seeking the initialization of the
perturbation via enlarging the output distances of the target model in a random
directions. Due to the diversity of random directions, the embedded fast
adversarial training using FGSM increases the information from the adversary
and reduces the possibility of overfitting. In addition to preventing
overfitting, the extensive results show that our proposed DIP-FAT technique can
also improve the accuracy of the clean data. The biggest advantage of DIP-FAT
method: achieving the best banlance among clean-data, perturbed-data and
efficiency.


http://arxiv.org/abs/2005.07821
Memoryless Cumulative Sign Detector for Stealthy CPS Sensor Attacks. (1%)
Paul J. University of Virginia Bonczek; Nicola University of Virginia Bezzo
  Stealthy false data injection attacks on cyber-physical systems introduce
erroneous measurements onto sensors with the intent to degrade system
performance. An intelligent attacker can design stealthy attacks with knowledge
of the system model and noise characteristics to evade detection from
state-of-the-art fault detectors by remaining within detection thresholds.
However, during these hidden attacks, an attacker with the intention of
hijacking a system will leave traces of non-random behavior that contradict
with the expectation of the system model. Given these premises, in this paper
we propose a run-time monitor called Cumulative Sign (CUSIGN) detector, for
identifying stealthy falsified measurements by identifying if measurements are
no longer behaving in a random manner. Specifically, our proposed CUSIGN
monitor considers the changes in sign of the measurement residuals and their
expected occurrence in order to detect if a sensor could be compromised.
Moreover, our detector is designed to be a memoryless procedure, eliminating
the need to store large sequences of data for attack detection. We characterize
the detection capabilities of the proposed CUSIGN technique following the
well-known $\chi^2$ failure detection scheme. Additionally, we show the
advantage of augmenting CUSIGN to the model-based Cumulative Sum (CUSUM)
detector, which provides magnitude bounds on attacks, for enhanced detection of
sensor spoofing attacks. Our approach is validated with simulations on an
unmanned ground vehicle (UGV) during a navigation case study.


http://arxiv.org/abs/2005.07502
Enhancing Perceptual Loss with Adversarial Feature Matching for Super-Resolution. (1%)
Akella Ravi Tej; Shirsendu Sukanta Halder; Arunav Pratap Shandeelya; Vinod Pankajakshan
  Single image super-resolution (SISR) is an ill-posed problem with an
indeterminate number of valid solutions. Solving this problem with neural
networks would require access to extensive experience, either presented as a
large training set over natural images or a condensed representation from
another pre-trained network. Perceptual loss functions, which belong to the
latter category, have achieved breakthrough success in SISR and several other
computer vision tasks. While perceptual loss plays a central role in the
generation of photo-realistic images, it also produces undesired pattern
artifacts in the super-resolved outputs. In this paper, we show that the root
cause of these pattern artifacts can be traced back to a mismatch between the
pre-training objective of perceptual loss and the super-resolution objective.
To address this issue, we propose to augment the existing perceptual loss
formulation with a novel content loss function that uses the latent features of
a discriminator network to filter the unwanted artifacts across several levels
of adversarial similarity. Further, our modification has a stabilizing effect
on non-convex optimization in adversarial training. The proposed approach
offers notable gains in perceptual quality based on an extensive human
evaluation study and a competent reconstruction fidelity when tested on
objective evaluation metrics.


http://arxiv.org/abs/2005.07099
Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning.
Jianwen Sun; Tianwei Zhang; Xiaofei Xie; Lei Ma; Yan Zheng; Kangjie Chen; Yang Liu
  Adversarial attacks against conventional Deep Learning (DL) systems and
algorithms have been widely studied, and various defenses were proposed.
However, the possibility and feasibility of such attacks against Deep
Reinforcement Learning (DRL) are less explored. As DRL has achieved great
success in various complex tasks, designing effective adversarial attacks is an
indispensable prerequisite towards building robust DRL algorithms. In this
paper, we introduce two novel adversarial attack techniques to
\emph{stealthily} and \emph{efficiently} attack the DRL agents. These two
techniques enable an adversary to inject adversarial samples in a minimal set
of critical moments while causing the most severe damage to the agent. The
first technique is the \emph{critical point attack}: the adversary builds a
model to predict the future environmental states and agent's actions, assesses
the damage of each possible attack strategy, and selects the optimal one. The
second technique is the \emph{antagonist attack}: the adversary automatically
learns a domain-agnostic model to discover the critical moments of attacking
the agent in an episode. Experimental results demonstrate the effectiveness of
our techniques. Specifically, to successfully attack the DRL agent, our
critical point technique only requires 1 (TORCS) or 2 (Atari Pong and Breakout)
steps, and the antagonist technique needs fewer than 5 steps (4 Mujoco tasks),
which are significant improvements over state-of-the-art methods.


http://arxiv.org/abs/2005.07145
A Deep Learning-based Fine-grained Hierarchical Learning Approach for Robust Malware Classification.
Ahmed Abusnaina; Mohammed Abuhamad; Hisham Alasmary; Afsah Anwar; Rhongho Jang; Saeed Salem; DaeHun Nyang; David Mohaisen
  The wide acceptance of Internet of Things (IoT) for both household and
industrial applications is accompanied by several security concerns. A major
security concern is their probable abuse by adversaries towards their malicious
intent. Understanding and analyzing IoT malicious behaviors is crucial,
especially with their rapid growth and adoption in wide-range of applications.
However, recent studies have shown that machine learning-based approaches are
susceptible to adversarial attacks by adding junk codes to the binaries, for
example, with an intention to fool those machine learning or deep
learning-based detection systems. Realizing the importance of addressing this
challenge, this study proposes a malware detection system that is robust to
adversarial attacks. To do so, examine the performance of the state-of-the-art
methods against adversarial IoT software crafted using the graph embedding and
augmentation techniques. In particular, we study the robustness of such methods
against two black-box adversarial methods, GEA and SGEA, to generate
Adversarial Examples (AEs) with reduced overhead, and keeping their
practicality intact. Our comprehensive experimentation with GEA-based AEs show
the relation between misclassification and the graph size of the injected
sample. Upon optimization and with small perturbation, by use of SGEA, all the
IoT malware samples are misclassified as benign. This highlights the
vulnerability of current detection systems under adversarial settings. With the
landscape of possible adversarial attacks, we then propose DL-FHMC, a
fine-grained hierarchical learning approach for malware detection and
classification, that is robust to AEs with a capability to detect 88.52% of the
malicious AEs.


http://arxiv.org/abs/2005.07347
Towards Assessment of Randomized Mechanisms for Certifying Adversarial Robustness. (22%)
Tianhang Zheng; Di Wang; Baochun Li; Jinhui Xu
  As a certified defensive technique, randomized smoothing has received
considerable attention due to its scalability to large datasets and neural
networks. However, several important questions remain unanswered, such as (i)
whether the Gaussian mechanism is an appropriate option for certifying
$\ell_2$-norm robustness, and (ii) whether there is an appropriate randomized
mechanism to certify $\ell_\infty$-norm robustness on high-dimensional
datasets. To shed light on these questions, we introduce a generic framework
that connects the existing frameworks to assess randomized mechanisms. Under
our framework, we define the magnitude of the noise required by a mechanism to
certify a certain extent of robustness as the metric for assessing the
appropriateness of the mechanism. We also derive lower bounds on the metric as
the criteria for assessment. Assessment of Gaussian and Exponential mechanisms
is achieved by comparing the magnitude of noise needed by these mechanisms and
the criteria, and we conclude that the Gaussian mechanism is an appropriate
option to certify both $\ell_2$-norm and $\ell_\infty$-norm robustness. The
veracity of our framework is verified by evaluations on CIFAR10 and ImageNet.


http://arxiv.org/abs/2005.07287
VirAAL: Virtual Adversarial Active Learning. (12%)
Gregory Senay; Badr Youbi Idrissi; Marine Haziza
  This paper presents VirAAL, an Active Learning framework based on Adversarial
Training. VirAAL aims to reduce the effort of annotation in Natural Language
Understanding (NLU). VirAAL is based on Virtual Adversarial Training (VAT), a
semi-supervised approach that regularizes the model through Local
Distributional Smoothness. With that, adversarial perturbations are added to
the inputs making the posterior distribution more consistent. Therefore,
entropy-based Active Learning becomes robust by querying more informative
samples without requiring additional components. The first set of experiments
studies the impact of VAT on NLU tasks (joint or not) within low labeled data
regimes. The second set shows the effect of VirAAL in an Active Learning (AL)
process. Results demonstrate that VAT is robust even on multitask training
where the adversarial noise is computed from multiple loss functions.
Substantial improvements are observed with entropy-based AL with VirAAL for
querying data to annotate. VirAAL is an inexpensive method in terms of AL
computation with a positive impact on data sampling. Furthermore, VirAAL
decreases annotations in AL up to 80%.


http://arxiv.org/abs/2005.06149
DeepRobust: A PyTorch Library for Adversarial Attacks and Defenses.
Yaxin Li; Wei Jin; Han Xu; Jiliang Tang
  DeepRobust is a PyTorch adversarial learning library which aims to build a
comprehensive and easy-to-use platform to foster this research field. It
currently contains more than 10 attack algorithms and 8 defense algorithms in
image domain and 9 attack algorithms and 4 defense algorithms in graph domain,
under a variety of deep learning architectures. In this manual, we introduce
the main contents of DeepRobust with detailed instructions. The library is kept
updated and can be found at https://github.com/DSE-MSU/DeepRobust.


http://arxiv.org/abs/2005.05552
Effective and Robust Detection of Adversarial Examples via Benford-Fourier Coefficients.
Chengcheng Ma; Baoyuan Wu; Shibiao Xu; Yanbo Fan; Yong Zhang; Xiaopeng Zhang; Zhifeng Li
  Adversarial examples have been well known as a serious threat to deep neural
networks (DNNs). In this work, we study the detection of adversarial examples,
based on the assumption that the output and internal responses of one DNN model
for both adversarial and benign examples follow the generalized Gaussian
distribution (GGD), but with different parameters (i.e., shape factor, mean,
and variance). GGD is a general distribution family to cover many popular
distributions (e.g., Laplacian, Gaussian, or uniform). It is more likely to
approximate the intrinsic distributions of internal responses than any specific
distribution. Besides, since the shape factor is more robust to different
databases rather than the other two parameters, we propose to construct
discriminative features via the shape factor for adversarial detection,
employing the magnitude of Benford-Fourier coefficients (MBF), which can be
easily estimated using responses. Finally, a support vector machine is trained
as the adversarial detector through leveraging the MBF features. Extensive
experiments in terms of image classification demonstrate that the proposed
detector is much more effective and robust on detecting adversarial examples of
different crafting methods and different sources, compared to state-of-the-art
adversarial detection methods.


http://arxiv.org/abs/2005.05750
Evaluating Ensemble Robustness Against Adversarial Attacks.
George Adam; Romain Speciel
  Adversarial examples, which are slightly perturbed inputs generated with the
aim of fooling a neural network, are known to transfer between models;
adversaries which are effective on one model will often fool another. This
concept of transferability poses grave security concerns as it leads to the
possibility of attacking models in a black box setting, during which the
internal parameters of the target model are unknown. In this paper, we seek to
analyze and minimize the transferability of adversaries between models within
an ensemble. To this end, we introduce a gradient based measure of how
effectively an ensemble's constituent models collaborate to reduce the space of
adversarial examples targeting the ensemble itself. Furthermore, we demonstrate
that this measure can be utilized during training as to increase an ensemble's
robustness to adversarial examples.


http://arxiv.org/abs/2005.06023
Increased-confidence adversarial examples for improved transferability of Counter-Forensic attacks.
Wenjie Li; Benedetta Tondi; Rongrong Ni; Mauro Barni
  Transferability of adversarial examples is a key issue to study the security
of multimedia forensics (MMF) techniques relying on Deep Learning (DL). The
transferability of the attacks, in fact, would open the way to the deployment
of successful counter forensics attacks also in cases where the attacker does
not have a full knowledge of the to-be-attacked system. Some preliminary works
have shown that adversarial examples against CNN-based image forensics
detectors are in general non-transferrable, at least when the basic versions of
the attacks implemented in the most popular attack packages are adopted. In
this paper, we introduce a general strategy to increase the strength of the
attacks and evaluate the transferability of the adversarial examples when such
a strength varies. We experimentally show that, in this way, attack
transferability can be improved to a large extent, at the expense of a larger
distortion. Our research confirms the security threats posed by the existence
of adversarial examples even in multimedia forensics scenarios, thus calling
for new defense strategies to improve the security of DL-based MMF techniques.


http://arxiv.org/abs/2005.06107
Adversarial examples are useful too!
Ali Borji
  Deep learning has come a long way and has enjoyed an unprecedented success.
Despite high accuracy, however, deep models are brittle and are easily fooled
by imperceptible adversarial perturbations. In contrast to common
inference-time attacks, Backdoor (\aka Trojan) attacks target the training
phase of model construction, and are extremely difficult to combat since a) the
model behaves normally on a pristine testing set and b) the augmented
perturbations can be minute and may only affect few training samples. Here, I
propose a new method to tell whether a model has been subject to a backdoor
attack. The idea is to generate adversarial examples, targeted or untargeted,
using conventional attacks such as FGSM and then feed them back to the
classifier. By computing the statistics (here simply mean maps) of the images
in different categories and comparing them with the statistics of a reference
model, it is possible to visually locate the perturbed regions and unveil the
attack.


http://arxiv.org/abs/2005.04871
Spanning Attack: Reinforce Black-box Attacks with Unlabeled Data.
Lu Wang; Huan Zhang; Jinfeng Yi; Cho-Jui Hsieh; Yuan Jiang
  Adversarial black-box attacks aim to craft adversarial perturbations by
querying input-output pairs of machine learning models. They are widely used to
evaluate the robustness of pre-trained models. However, black-box attacks often
suffer from the issue of query inefficiency due to the high dimensionality of
the input space, and therefore incur a false sense of model robustness. In this
paper, we relax the conditions of the black-box threat model, and propose a
novel technique called the spanning attack. By constraining adversarial
perturbations in a low-dimensional subspace via spanning an auxiliary unlabeled
dataset, the spanning attack significantly improves the query efficiency of
black-box attacks. Extensive experiments show that the proposed method works
favorably in both soft-label and hard-label black-box attacks. Our code is
available at https://github.com/wangwllu/spanning_attack.


http://arxiv.org/abs/2005.05321
Channel-Aware Adversarial Attacks Against Deep Learning-Based Wireless Signal Classifiers.
Brian Kim; Yalin E. Sagduyu; Kemal Davaslioglu; Tugba Erpek; Sennur Ulukus
  This paper presents channel-aware adversarial attacks against deep
learning-based wireless signal classifiers. There is a transmitter that
transmits signals with different modulation types. A deep neural network is
used at each receiver to classify its over-the-air received signals to
modulation types. In the meantime, an adversary transmits an adversarial
perturbation (subject to a power budget) to fool receivers into making errors
in classifying signals that are received as superpositions of transmitted
signals and adversarial perturbations. First, these evasion attacks are shown
to fail when channels are not considered in designing adversarial
perturbations. Then realistic attacks are presented by considering channel
effects from the adversary to each receiver. After showing that a channel-aware
attack is selective (i.e., it affects only the receiver whose channel is
considered in the perturbation design), a broadcast adversarial attack is
presented by crafting a common adversarial perturbation to simultaneously fool
classifiers at different receivers. The major vulnerability of modulation
classifiers to over-the-air adversarial attacks is shown by accounting for
different levels of information available about channel, transmitter input, and
classifier model. Finally, a certified defense based on randomized smoothing
that augments training data with noise is introduced to make modulation
classifier robust to adversarial perturbations.


http://arxiv.org/abs/2005.04364
It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations.
Samson Tan; Shafiq Joty; Min-Yen Kan; Richard Socher
  Training on only perfect Standard English corpora predisposes pre-trained
neural networks to discriminate against minorities from non-standard linguistic
backgrounds (e.g., African American Vernacular English, Colloquial Singapore
English, etc.). We perturb the inflectional morphology of words to craft
plausible and semantically similar adversarial examples that expose these
biases in popular NLP models, e.g., BERT and Transformer, and show that
adversarially fine-tuning them for a single epoch significantly improves
robustness without sacrificing performance on clean data.


http://arxiv.org/abs/2005.04564
Class-Aware Domain Adaptation for Improving Adversarial Robustness.
Xianxu Hou; Jingxin Liu; Bolei Xu; Xiaolong Wang; Bozhi Liu; Guoping Qiu
  Recent works have demonstrated convolutional neural networks are vulnerable
to adversarial examples, i.e., inputs to machine learning models that an
attacker has intentionally designed to cause the models to make a mistake. To
improve the adversarial robustness of neural networks, adversarial training has
been proposed to train networks by injecting adversarial examples into the
training data. However, adversarial training could overfit to a specific type
of adversarial attack and also lead to standard accuracy drop on clean images.
To this end, we propose a novel Class-Aware Domain Adaptation (CADA) method for
adversarial defense without directly applying adversarial training.
Specifically, we propose to learn domain-invariant features for adversarial
examples and clean images via a domain discriminator. Furthermore, we introduce
a class-aware component into the discriminator to increase the discriminative
power of the network for adversarial examples. We evaluate our newly proposed
approach using multiple benchmark datasets. The results demonstrate that our
method can significantly improve the state-of-the-art of adversarial robustness
for various attacks and maintain high performances on clean images.


http://arxiv.org/abs/2005.04272
Towards Robustness against Unsuspicious Adversarial Examples.
Liang Tong; Minzhe Guo; Atul Prakash; Yevgeniy Vorobeychik
  Despite the remarkable success of deep neural networks, significant concerns
have emerged about their robustness to adversarial perturbations to inputs.
While most attacks aim to ensure that these are imperceptible, physical
perturbation attacks typically aim for being unsuspicious, even if perceptible.
However, there is no universal notion of what it means for adversarial examples
to be unsuspicious. We propose an approach for modeling suspiciousness by
leveraging cognitive salience. Specifically, we split an image into foreground
(salient region) and background (the rest), and allow significantly larger
adversarial perturbations in the background. We describe how to compute the
resulting dual-perturbation attacks on both deterministic and stochastic
classifiers. We then experimentally demonstrate that our attacks do not
significantly change perceptual salience of the background, but are highly
effective against classifiers robust to conventional attacks. Furthermore, we
show that adversarial training with dual-perturbation attacks yields
classifiers that are more robust to these than state-of-the-art robust learning
approaches, and comparable in terms of robustness to conventional attacks.


http://arxiv.org/abs/2005.03597
Efficient Exact Verification of Binarized Neural Networks.
Kai Jia; Martin Rinard
  We present a new system, EEV, for verifying binarized neural networks (BNNs).
We formulate BNN verification as a Boolean satisfiability problem (SAT) with
reified cardinality constraints of the form $y = (x_1 + \cdots + x_n \le b)$,
where $x_i$ and $y$ are Boolean variables possibly with negation and $b$ is an
integer constant. We also identify two properties, specifically balanced weight
sparsity and lower cardinality bounds, that reduce the verification complexity
of BNNs. EEV contains both a SAT solver enhanced to handle reified cardinality
constraints natively and novel training strategies designed to reduce
verification complexity by delivering networks with improved sparsity
properties and cardinality bounds. We demonstrate the effectiveness of EEV by
presenting the first exact verification results for $\ell_{\infty}$-bounded
adversarial robustness of nontrivial convolutional BNNs on the MNIST and
CIFAR10 datasets. Our results also show that, depending on the dataset and
network architecture, our techniques verify BNNs between a factor of ten to ten
thousand times faster than the best previous exact verification techniques for
either binarized or real-valued networks.


http://arxiv.org/abs/2005.03644
Defending Hardware-based Malware Detectors against Adversarial Attacks.
Abraham Peedikayil Kuruvila; Shamik Kundu; Kanad Basu
  In the era of Internet of Things (IoT), Malware has been proliferating
exponentially over the past decade. Traditional anti-virus software are
ineffective against modern complex Malware. In order to address this challenge,
researchers have proposed Hardware-assisted Malware Detection (HMD) using
Hardware Performance Counters (HPCs). The HPCs are used to train a set of
Machine learning (ML) classifiers, which in turn, are used to distinguish
benign programs from Malware. Recently, adversarial attacks have been designed
by introducing perturbations in the HPC traces using an adversarial sample
predictor to misclassify a program for specific HPCs. These attacks are
designed with the basic assumption that the attacker is aware of the HPCs being
used to detect Malware. Since modern processors consist of hundreds of HPCs,
restricting to only a few of them for Malware detection aids the attacker. In
this paper, we propose a Moving target defense (MTD) for this adversarial
attack by designing multiple ML classifiers trained on different sets of HPCs.
The MTD randomly selects a classifier; thus, confusing the attacker about the
HPCs or the number of classifiers applied. We have developed an analytical
model which proves that the probability of an attacker to guess the perfect
HPC-classifier combination for MTD is extremely low (in the range of
$10^{-1864}$ for a system with 20 HPCs). Our experimental results prove that
the proposed defense is able to improve the classification accuracy of HPC
traces that have been modified through an adversarial sample generator by up to
31.5%, for a near perfect (99.4%) restoration of the original accuracy.


http://arxiv.org/abs/2005.03837
Projection & Probability-Driven Black-Box Attack.
Jie Li; Rongrong Ji; Hong Liu; Jianzhuang Liu; Bineng Zhong; Cheng Deng; Qi Tian
  Generating adversarial examples in a black-box setting retains a significant
challenge with vast practical application prospects. In particular, existing
black-box attacks suffer from the need for excessive queries, as it is
non-trivial to find an appropriate direction to optimize in the
high-dimensional space. In this paper, we propose Projection &
Probability-driven Black-box Attack (PPBA) to tackle this problem by reducing
the solution space and providing better optimization. For reducing the solution
space, we first model the adversarial perturbation optimization problem as a
process of recovering frequency-sparse perturbations with compressed sensing,
under the setting that random noise in the low-frequency space is more likely
to be adversarial. We then propose a simple method to construct a low-frequency
constrained sensing matrix, which works as a plug-and-play projection matrix to
reduce the dimensionality. Such a sensing matrix is shown to be flexible enough
to be integrated into existing methods like NES and Bandits$_{TD}$. For better
optimization, we perform a random walk with a probability-driven strategy,
which utilizes all queries over the whole progress to make full use of the
sensing matrix for a less query budget. Extensive experiments show that our
method requires at most 24% fewer queries with a higher attack success rate
compared with state-of-the-art approaches. Finally, the attack method is
evaluated on the real-world online service, i.e., Google Cloud Vision API,
which further demonstrates our practical potentials.


http://arxiv.org/abs/2005.02929
Training robust neural networks using Lipschitz bounds.
Patricia Pauli; Anne Koch; Julian Berberich; Frank Allgöwer
  Due to their susceptibility to adversarial perturbations, neural networks
(NNs) are hardly used in safety-critical applications. One measure of
robustness to such perturbations in the input is the Lipschitz constant of the
input-output map defined by an NN. In this work, we propose a framework to
train NNs while at the same time encouraging robustness by keeping their
Lipschitz constant small, thus addressing the robustness issue. More
specifically, we design an optimization scheme based on the Alternating
Direction Method of Multipliers that minimizes not only the training loss of an
NN but also its Lipschitz constant resulting in a semidefinite programming
based training procedure that promotes robustness. We design two versions of
this training procedure. The first one includes a regularizer that penalizes an
accurate upper bound on the Lipschitz constant. The second one allows to
enforce a desired Lipschitz bound on the NN at all times during training.
Finally, we provide two examples to show that the proposed framework
successfully increases the robustness of NNs.


http://arxiv.org/abs/2005.02936
GraCIAS: Grassmannian of Corrupted Images for Adversarial Security.
Ankita Shukla; Pavan Turaga; Saket Anand
  Input transformation based defense strategies fall short in defending against
strong adversarial attacks. Some successful defenses adopt approaches that
either increase the randomness within the applied transformations, or make the
defense computationally intensive, making it substantially more challenging for
the attacker. However, it limits the applicability of such defenses as a
pre-processing step, similar to computationally heavy approaches that use
retraining and network modifications to achieve robustness to perturbations. In
this work, we propose a defense strategy that applies random image corruptions
to the input image alone, constructs a self-correlation based subspace followed
by a projection operation to suppress the adversarial perturbation. Due to its
simplicity, the proposed defense is computationally efficient as compared to
the state-of-the-art, and yet can withstand huge perturbations. Further, we
develop proximity relationships between the projection operator of a clean
image and of its adversarially perturbed version, via bounds relating geodesic
distance on the Grassmannian to matrix Frobenius norms. We empirically show
that our strategy is complementary to other weak defenses like JPEG compression
and can be seamlessly integrated with them to create a stronger defense. We
present extensive experiments on the ImageNet dataset across four different
models namely InceptionV3, ResNet50, VGG16 and MobileNet models with
perturbation magnitude set to {\epsilon} = 16. Unlike state-of-the-art
approaches, even without any retraining, the proposed strategy achieves an
absolute improvement of ~ 4.5% in defense accuracy on ImageNet.


http://arxiv.org/abs/2005.02552
Enhancing Intrinsic Adversarial Robustness via Feature Pyramid Decoder.
Guanlin Li; Shuya Ding; Jun Luo; Chang Liu
  Whereas adversarial training is employed as the main defence strategy against
specific adversarial samples, it has limited generalization capability and
incurs excessive time complexity. In this paper, we propose an attack-agnostic
defence framework to enhance the intrinsic robustness of neural networks,
without jeopardizing the ability of generalizing clean samples. Our Feature
Pyramid Decoder (FPD) framework applies to all block-based convolutional neural
networks (CNNs). It implants denoising and image restoration modules into a
targeted CNN, and it also constraints the Lipschitz constant of the
classification layer. Moreover, we propose a two-phase strategy to train the
FPD-enhanced CNN, utilizing $\epsilon$-neighbourhood noisy images with
multi-task and self-supervised learning. Evaluated against a variety of
white-box and black-box attacks, we demonstrate that FPD-enhanced CNNs gain
sufficient robustness against general adversarial samples on MNIST, SVHN and
CALTECH. In addition, if we further conduct adversarial training, the
FPD-enhanced CNNs perform better than their non-enhanced versions.


http://arxiv.org/abs/2005.02540
Proper measure for adversarial robustness.
Hyeongji Kim; Ketil Malde
  This paper analyzes the problems of standard adversarial accuracy and
standard adversarial training method. We argue that standard adversarial
accuracy fails to properly measure the robustness of classifiers. The
definition allows overlaps in regions for clean samples and adversarial
examples. Thus, there is a trade-off between accuracy and standard adversarial
accuracy. Hence, using standard adversarial training can result in lowered
accuracy. Also, standard adversarial accuracy can favor classifiers with more
invariance-based adversarial examples, samples whose predicted classes are
unchanged even if the perceptual classes are changed.
  In this paper, we introduce a new measure for the robustness of classifiers
called genuine adversarial accuracy in order to handle the problems of the
standard adversarial accuracy. It can measure adversarial robustness of
classifiers without the trade-off between accuracy on clean data and
adversarially perturbed samples. In addition, it doesn't favor a model with
invariance-based adversarial examples.
  We show that a single nearest neighbor (1-NN) classifier is the most robust
classifier according to genuine adversarial accuracy for given data and a
metric when exclusive belongingness assumption is used. This result provides a
fundamental step to train adversarially robust classifiers.


http://arxiv.org/abs/2005.02270
Hacking the Waveform: Generalized Wireless Adversarial Deep Learning.
Francesco Restuccia; Salvatore D'Oro; Amani Al-Shawabka; Bruno Costa Rendon; Kaushik Chowdhury; Stratis Ioannidis; Tommaso Melodia
  This paper advances the state of the art by proposing the first comprehensive
analysis and experimental evaluation of adversarial learning attacks to
wireless deep learning systems. We postulate a series of adversarial attacks,
and formulate a Generalized Wireless Adversarial Machine Learning Problem
(GWAP) where we analyze the combined effect of the wireless channel and the
adversarial waveform on the efficacy of the attacks. We propose a new neural
network architecture called FIRNet, which can be trained to "hack" a classifier
based only on its output. We extensively evaluate the performance on (i) a
1,000-device radio fingerprinting dataset, and (ii) a 24-class modulation
dataset. Results obtained with several channel conditions show that our
algorithms can decrease the classifier accuracy up to 3x. We also
experimentally evaluate FIRNet on a radio testbed, and show that our
data-driven blackbox approach can confuse the classifier up to 97% while
keeping the waveform distortion to a minimum.


http://arxiv.org/abs/2005.02313
Adversarial Training against Location-Optimized Adversarial Patches.
Sukrut Rao; David Stutz; Bernt Schiele
  Deep neural networks have been shown to be susceptible to adversarial
examples -- small, imperceptible changes constructed to cause
mis-classification in otherwise highly accurate image classifiers. As a
practical alternative, recent work proposed so-called adversarial patches:
clearly visible, but adversarially crafted rectangular patches in images. These
patches can easily be printed and applied in the physical world. While defenses
against imperceptible adversarial examples have been studied extensively,
robustness against adversarial patches is poorly understood. In this work, we
first devise a practical approach to obtain adversarial patches while actively
optimizing their location within the image. Then, we apply adversarial training
on these location-optimized adversarial patches and demonstrate significantly
improved robustness on CIFAR10 and GTSRB. Additionally, in contrast to
adversarial training on imperceptible adversarial examples, our adversarial
patch training does not reduce accuracy.


http://arxiv.org/abs/2005.01499
On the Benefits of Models with Perceptually-Aligned Gradients.
Gunjan Aggarwal; Abhishek Sinha; Nupur Kumari; Mayank Singh
  Adversarial robust models have been shown to learn more robust and
interpretable features than standard trained models. As shown in
[\cite{tsipras2018robustness}], such robust models inherit useful interpretable
properties where the gradient aligns perceptually well with images, and adding
a large targeted adversarial perturbation leads to an image resembling the
target class. We perform experiments to show that interpretable and
perceptually aligned gradients are present even in models that do not show high
robustness to adversarial attacks. Specifically, we perform adversarial
training with attack for different max-perturbation bound. Adversarial training
with low max-perturbation bound results in models that have interpretable
features with only slight drop in performance over clean samples. In this
paper, we leverage models with interpretable perceptually-aligned features and
show that adversarial training with low max-perturbation bound can improve the
performance of models for zero-shot and weakly supervised localization tasks.


http://arxiv.org/abs/2005.01452
Do Gradient-based Explanations Tell Anything About Adversarial Robustness to Android Malware?
Marco Melis; Michele Scalas; Ambra Demontis; Davide Maiorca; Battista Biggio; Giorgio Giacinto; Fabio Roli
  Machine-learning algorithms trained on features extracted from static code
analysis can successfully detect Android malware. However, these approaches can
be evaded by sparse evasion attacks that produce adversarial malware samples in
which only few features are modified. This can be achieved, e.g., by injecting
a small set of fake permissions and system calls into the malicious
application, without compromising its intrusive functionality. To improve
adversarial robustness against such sparse attacks, learning algorithms should
avoid providing decisions which only rely upon a small subset of discriminant
features; otherwise, even manipulating some of them may easily allow evading
detection. Previous work showed that classifiers which avoid overemphasizing
few discriminant features tend to be more robust against sparse attacks, and
have developed simple metrics to help identify and select more robust
algorithms. In this work, we aim to investigate whether gradient-based
attribution methods used to explain classifiers' decisions by identifying the
most relevant features can also be used to this end. Our intuition is that a
classifier providing more uniform, evener attributions should rely upon a
larger set of features, instead of overemphasizing few of them, thus being more
robust against sparse attacks. We empirically investigate the connection
between gradient-based explanations and adversarial robustness on a case study
conducted on Android malware detection, and show that, in some cases, there is
a strong correlation between the distribution of such explanations and
adversarial robustness. We conclude the paper by discussing how our findings
may thus enable the development of more efficient mechanisms both to evaluate
and to improve adversarial robustness.


http://arxiv.org/abs/2005.01229
Robust Encodings: A Framework for Combating Adversarial Typos.
Erik Jones; Robin Jia; Aditi Raghunathan; Percy Liang
  Despite excellent performance on many tasks, NLP systems are easily fooled by
small adversarial perturbations of inputs. Existing procedures to defend
against such perturbations are either (i) heuristic in nature and susceptible
to stronger attacks or (ii) provide guaranteed robustness to worst-case
attacks, but are incompatible with state-of-the-art models like BERT. In this
work, we introduce robust encodings (RobEn): a simple framework that confers
guaranteed robustness, without making compromises on model architecture. The
core component of RobEn is an encoding function, which maps sentences to a
smaller, discrete space of encodings. Systems using these encodings as a
bottleneck confer guaranteed robustness with standard training, and the same
encodings can be used across multiple tasks. We identify two desiderata to
construct robust encoding functions: perturbations of a sentence should map to
a small set of encodings (stability), and models using encodings should still
perform well (fidelity). We instantiate RobEn to defend against a large family
of adversarial typos. Across six tasks from GLUE, our instantiation of RobEn
paired with BERT achieves an average robust accuracy of 71.3% against all
adversarial typos in the family considered, while previous work using a
typo-corrector achieves only 35.3% accuracy against a simple greedy attack.


http://arxiv.org/abs/2005.00446
Defense of Word-level Adversarial Attacks via Random Substitution Encoding.
Zhaoyang Wang; Hongtao Wang
  The adversarial attacks against deep neural networks on computer version
tasks has spawned many new technologies that help protect models avoiding false
prediction. Recently, word-level adversarial attacks on deep models of Natural
Language Processing (NLP) tasks have also demonstrated strong power, e.g.,
fooling a sentiment classification neural network to make wrong decision.
Unfortunately, few previous literatures have discussed the defense of such
word-level synonym substitution based attacks since they are hard to be
perceived and detected. In this paper, we shed light on this problem and
propose a novel defense framework called Random Substitution Encoding (RSE),
which introduces a random substitution encoder into the training process of
original neural networks. Extensive experiments on text classification tasks
demonstrate the effectiveness of our framework on defense of word-level
adversarial attacks, under various base and attack models.


http://arxiv.org/abs/2005.00656
Jacks of All Trades, Masters Of None: Addressing Distributional Shift and Obtrusiveness via Transparent Patch Attacks.
Neil Fendley; Max Lennon; I-Jeng Wang; Philippe Burlina; Nathan Drenkow
  We focus on the development of effective adversarial patch attacks and -- for
the first time -- jointly address the antagonistic objectives of attack success
and obtrusiveness via the design of novel semi-transparent patches. This work
is motivated by our pursuit of a systematic performance analysis of patch
attack robustness with regard to geometric transformations. Specifically, we
first elucidate a) key factors underpinning patch attack success and b) the
impact of distributional shift between training and testing/deployment when
cast under the Expectation over Transformation (EoT) formalism. By focusing our
analysis on three principal classes of transformations (rotation, scale, and
location), our findings provide quantifiable insights into the design of
effective patch attacks and demonstrate that scale, among all factors,
significantly impacts patch attack success. Working from these findings, we
then focus on addressing how to overcome the principal limitations of scale for
the deployment of attacks in real physical settings: namely the obtrusiveness
of large patches. Our strategy is to turn to the novel design of
irregularly-shaped, semi-transparent partial patches which we construct via a
new optimization process that jointly addresses the antagonistic goals of
mitigating obtrusiveness and maximizing effectiveness. Our study -- we hope --
will help encourage more focus in the community on the issues of obtrusiveness,
scale, and success in patch attacks.


http://arxiv.org/abs/2005.00683
Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models.
Bill Yuchen Lin; Seyeon Lee; Rahul Khanna; Xiang Ren
  Recent works show that pre-trained masked language models, such as BERT,
possess certain linguistic and commonsense knowledge. However, it remains to be
seen what types of commonsense knowledge these models have access to. In this
vein, we propose to study whether numerical commonsense knowledge --
commonsense knowledge that provides an understanding of the numeric relation
between entities -- can be induced from pre-trained masked language models and
to what extent is this access to knowledge robust against adversarial examples?
To study this, we introduce a probing task with a diagnostic dataset,
NumerSense, containing 3,145 masked-word-prediction probes. Surprisingly, our
experiments and analysis reveal that: (1) BERT and its stronger variant RoBERTa
perform poorly on our dataset prior to any fine-tuning; (2) fine-tuning with
distant supervision does improve performance; (3) the best distantly supervised
model still performs poorly when compared to humans (47.8% vs 96.3%).


http://arxiv.org/abs/2005.00616
Robust Deep Learning as Optimal Control: Insights and Convergence Guarantees.
Jacob H. Seidman; Mahyar Fazlyab; Victor M. Preciado; George J. Pappas
  The fragility of deep neural networks to adversarially-chosen inputs has
motivated the need to revisit deep learning algorithms. Including adversarial
examples during training is a popular defense mechanism against adversarial
attacks. This mechanism can be formulated as a min-max optimization problem,
where the adversary seeks to maximize the loss function using an iterative
first-order algorithm while the learner attempts to minimize it. However,
finding adversarial examples in this way causes excessive computational
overhead during training. By interpreting the min-max problem as an optimal
control problem, it has recently been shown that one can exploit the
compositional structure of neural networks in the optimization problem to
improve the training time significantly. In this paper, we provide the first
convergence analysis of this adversarial training algorithm by combining
techniques from robust optimal control and inexact oracle methods in
optimization. Our analysis sheds light on how the hyperparameters of the
algorithm affect the its stability and convergence. We support our insights
with experiments on a robust classification problem.


http://arxiv.org/abs/2005.00190
Evaluating Neural Machine Comprehension Model Robustness to Noisy Inputs and Adversarial Attacks.
Winston Wu; Dustin Arendt; Svitlana Volkova
  We evaluate machine comprehension models' robustness to noise and adversarial
attacks by performing novel perturbations at the character, word, and sentence
level. We experiment with different amounts of perturbations to examine model
confidence and misclassification rate, and contrast model performance in
adversarial training with different embedding types on two benchmark datasets.
We demonstrate improving model performance with ensembling. Finally, we analyze
factors that effect model behavior under adversarial training and develop a
model to predict model errors during adversarial attacks.


http://arxiv.org/abs/2005.00174
Universal Adversarial Attacks with Natural Triggers for Text Classification.
Liwei Song; Xinwei Yu; Hsuan-Tung Peng; Karthik Narasimhan
  Recent work has demonstrated the vulnerability of modern text classifiers to
universal adversarial attacks, which are input-agnostic sequence of words added
to any input instance. Despite being highly successful, the word sequences
produced in these attacks are often unnatural, do not carry much semantic
meaning, and can be easily distinguished from natural text. In this paper, we
develop adversarial attacks that appear closer to natural English phrases and
yet confuse classification systems when added to benign inputs. To achieve
this, we leverage an adversarially regularized autoencoder (ARAE) to generate
triggers and propose a gradient-based search method to output natural text that
fools a target classifier. Experiments on two different classification tasks
demonstrate the effectiveness of our attacks while also being less identifiable
than previous approaches on three simple detection metrics.


http://arxiv.org/abs/2005.00060
Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness.
Pu Zhao; Pin-Yu Chen; Payel Das; Karthikeyan Natesan Ramamurthy; Xue Lin
  Mode connectivity provides novel geometric insights on analyzing loss
landscapes and enables building high-accuracy pathways between well-trained
neural networks. In this work, we propose to employ mode connectivity in loss
landscapes to study the adversarial robustness of deep neural networks, and
provide novel methods for improving this robustness. Our experiments cover
various types of adversarial attacks applied to different network architectures
and datasets. When network models are tampered with backdoor or error-injection
attacks, our results demonstrate that the path connection learned using limited
amount of bonafide data can effectively mitigate adversarial effects while
maintaining the original accuracy on clean data. Therefore, mode connectivity
provides users with the power to repair backdoored or error-injected models. We
also use mode connectivity to investigate the loss landscapes of regular and
robust models against evasion attacks. Experiments show that there exists a
barrier in adversarial robustness loss on the path connecting regular and
adversarially-trained models. A high correlation is observed between the
adversarial robustness loss and the largest eigenvalue of the input Hessian
matrix, for which theoretical justifications are provided. Our results suggest
that mode connectivity offers a holistic tool and practical means for
evaluating and improving adversarial robustness.


http://arxiv.org/abs/2004.15015
Imitation Attacks and Defenses for Black-box Machine Translation Systems.
Eric Wallace; Mitchell Stern; Dawn Song
  We consider an adversary looking to steal or attack a black-box machine
translation (MT) system, either for financial gain or to exploit model errors.
We first show that black-box MT systems can be stolen by querying them with
monolingual sentences and training models to imitate their outputs. Using
simulated experiments, we demonstrate that MT model stealing is possible even
when imitation models have different input data or architectures than their
victims. Applying these ideas, we train imitation models that reach within 0.6
BLEU of three production MT systems on both high-resource and low-resource
language pairs. We then leverage the similarity of our imitation models to
transfer adversarial examples to the production systems. We use gradient-based
attacks that expose inputs which lead to semantically-incorrect translations,
dropped content, and vulgar model outputs. To mitigate these vulnerabilities,
we propose a defense that modifies translation outputs in order to misdirect
the optimization of imitation models. This defense degrades imitation model
BLEU and attack transfer rates at some cost in BLEU and inference speed.


http://arxiv.org/abs/2004.14861
Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability.
Nathan Inkawhich; Kevin J Liang; Binghui Wang; Matthew Inkawhich; Lawrence Carin; Yiran Chen
  We consider the blackbox transfer-based targeted adversarial attack threat
model in the realm of deep neural network (DNN) image classifiers. Rather than
focusing on crossing decision boundaries at the output layer of the source
model, our method perturbs representations throughout the extracted feature
hierarchy to resemble other classes. We design a flexible attack framework that
allows for multi-layer perturbations and demonstrates state-of-the-art targeted
transfer performance between ImageNet DNNs. We also show the superiority of our
feature space methods under a relaxation of the common assumption that the
source and target models are trained on the same dataset and label space, in
some instances achieving a $10\times$ increase in targeted success rate
relative to other blackbox transfer methods. Finally, we analyze why the
proposed methods outperform existing attack strategies and show an extension of
the method in the case when limited queries to the blackbox model are allowed.


http://arxiv.org/abs/2004.13617
Adversarial Learning Guarantees for Linear Hypotheses and Neural Networks.
Pranjal Awasthi; Natalie Frank; Mehryar Mohri
  Adversarial or test time robustness measures the susceptibility of a
classifier to perturbations to the test input. While there has been a flurry of
recent work on designing defenses against such perturbations, the theory of
adversarial robustness is not well understood. In order to make progress on
this, we focus on the problem of understanding generalization in adversarial
settings, via the lens of Rademacher complexity. We give upper and lower bounds
for the adversarial empirical Rademacher complexity of linear hypotheses with
adversarial perturbations measured in $l_r$-norm for an arbitrary $r \geq 1$.
This generalizes the recent result of [Yin et al.'19] that studies the case of
$r = \infty$, and provides a finer analysis of the dependence on the input
dimensionality as compared to the recent work of [Khim and Loh'19] on linear
hypothesis classes.
  We then extend our analysis to provide Rademacher complexity lower and upper
bounds for a single ReLU unit. Finally, we give adversarial Rademacher
complexity bounds for feed-forward neural networks with one hidden layer.
Unlike previous works we directly provide bounds on the adversarial Rademacher
complexity of the given network, as opposed to a bound on a surrogate. A
by-product of our analysis also leads to tighter bounds for the Rademacher
complexity of linear hypotheses, for which we give a detailed analysis and
present a comparison with existing bounds.


http://arxiv.org/abs/2004.13799
Minority Reports Defense: Defending Against Adversarial Patches.
Michael McCoyd; Won Park; Steven Chen; Neil Shah; Ryan Roggenkemper; Minjune Hwang; Jason Xinyu Liu; David Wagner
  Deep learning image classification is vulnerable to adversarial attack, even
if the attacker changes just a small patch of the image. We propose a defense
against patch attacks based on partially occluding the image around each
candidate patch location, so that a few occlusions each completely hide the
patch. We demonstrate on CIFAR-10, Fashion MNIST, and MNIST that our defense
provides certified security against patch attacks of a certain size.


http://arxiv.org/abs/2004.12864
DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking.
Christopher Hidey; Tuhin Chakrabarty; Tariq Alhindi; Siddharth Varia; Kriste Krstovski; Mona Diab; Smaranda Muresan
  The increased focus on misinformation has spurred development of data and
systems for detecting the veracity of a claim as well as retrieving
authoritative evidence. The Fact Extraction and VERification (FEVER) dataset
provides such a resource for evaluating end-to-end fact-checking, requiring
retrieval of evidence from Wikipedia to validate a veracity prediction. We show
that current systems for FEVER are vulnerable to three categories of realistic
challenges for fact-checking -- multiple propositions, temporal reasoning, and
ambiguity and lexical variation -- and introduce a resource with these types of
claims. Then we present a system designed to be resilient to these "attacks"
using multiple pointer networks for document selection and jointly modeling a
sequence of evidence sentences and veracity relation predictions. We find that
in handling these attacks we obtain state-of-the-art results on FEVER, largely
due to improved evidence retrieval.


http://arxiv.org/abs/2004.12771
Adversarial Fooling Beyond "Flipping the Label".
Konda Reddy Mopuri; Vaisakh Shaj; R. Venkatesh Babu
  Recent advancements in CNNs have shown remarkable achievements in various
CV/AI applications. Though CNNs show near human or better than human
performance in many critical tasks, they are quite vulnerable to adversarial
attacks. These attacks are potentially dangerous in real-life deployments.
Though there have been many adversarial attacks proposed in recent years, there
is no proper way of quantifying the effectiveness of these attacks. As of
today, mere fooling rate is used for measuring the susceptibility of the
models, or the effectiveness of adversarial attacks. Fooling rate just
considers label flipping and does not consider the cost of such flipping, for
instance, in some deployments, flipping between two species of dogs may not be
as severe as confusing a dog category with that of a vehicle. Therefore, the
metric to quantify the vulnerability of the models should capture the severity
of the flipping as well. In this work we first bring out the drawbacks of the
existing evaluation and propose novel metrics to capture various aspects of the
fooling. Further, for the first time, we present a comprehensive analysis of
several important adversarial attacks over a set of distinct CNN architectures.
We believe that the presented analysis brings valuable insights about the
current adversarial attacks and the CNN models.


http://arxiv.org/abs/2004.12478
Improved Image Wasserstein Attacks and Defenses.
J. Edward Hu; Adith Swaminathan; Hadi Salman; Greg Yang
  Robustness against image perturbations bounded by a $\ell_p$ ball have been
well-studied in recent literature. Perturbations in the real-world, however,
rarely exhibit the pixel independence that $\ell_p$ threat models assume. A
recently proposed Wasserstein distance-bounded threat model is a promising
alternative that limits the perturbation to pixel mass movements. We point out
and rectify flaws in previous definition of the Wasserstein threat model and
explore stronger attacks and defenses under our better-defined framework.
Lastly, we discuss the inability of current Wasserstein-robust models in
defending against perturbations seen in the real world. Our code and trained
models are available at https://github.com/edwardjhu/improved_wasserstein .


http://arxiv.org/abs/2004.12385
Towards Feature Space Adversarial Attack.
Qiuling Xu; Guanhong Tao; Siyuan Cheng; Lin Tan; Xiangyu Zhang
  We propose a new type of adversarial attack to Deep Neural Networks (DNNs)
for image classification. Different from most existing attacks that directly
perturb input pixels. Our attack focuses on perturbing abstract features, more
specifically, features that denote styles, including interpretable styles such
as vivid colors and sharp outlines, and uninterpretable ones. It induces model
misclassfication by injecting style changes insensitive for humans, through an
optimization procedure. We show that state-of-the-art adversarial attack
detection and defense techniques are ineffective in guarding against feature
space attacks.


http://arxiv.org/abs/2004.12519
Transferable Perturbations of Deep Feature Distributions.
Nathan Inkawhich; Kevin J Liang; Lawrence Carin; Yiran Chen
  Almost all current adversarial attacks of CNN classifiers rely on information
derived from the output layer of the network. This work presents a new
adversarial attack based on the modeling and exploitation of class-wise and
layer-wise deep feature distributions. We achieve state-of-the-art targeted
blackbox transfer-based attack results for undefended ImageNet models. Further,
we place a priority on explainability and interpretability of the attacking
process. Our methodology affords an analysis of how adversarial attacks change
the intermediate feature distributions of CNNs, as well as a measure of
layer-wise and class-wise feature distributional separability/entanglement. We
also conceptualize a transition from task/data-specific to model-specific
features within a CNN architecture that directly impacts the transferability of
adversarial examples.


http://arxiv.org/abs/2005.02160
Printing and Scanning Attack for Image Counter Forensics.
Hailey James; Otkrist Gupta; Dan Raviv
  Examining the authenticity of images has become increasingly important as
manipulation tools become more accessible and advanced. Recent work has shown
that while CNN-based image manipulation detectors can successfully identify
manipulations, they are also vulnerable to adversarial attacks, ranging from
simple double JPEG compression to advanced pixel-based perturbation.
Rephrase(hailey): In this paper we explore another method of highly plausible
attack: printing and scanning. We demonstrate the vulnerability of two
state-of-the-art models to this type of attack. We also propose a new machine
learning model that performs comparably to these state-of-the-art models when
trained and validated on printed and scanned images. Of the three models, our
proposed model performs the best when trained and validated on images from a
single printer. To facilitate this exploration, we create a dataset of over
6,000 printed and scanned image blocks. Further analysis suggests that
variation between images produced from different printers is significant, large
enough that good validation accuracy on images from one printer does not imply
similar validation accuracy on images from a different printer.


http://arxiv.org/abs/2004.12227
Improved Adversarial Training via Learned Optimizer.
Yuanhao Xiong; Cho-Jui Hsieh
  Adversarial attack has recently become a tremendous threat to deep learning
models. To improve the robustness of machine learning models, adversarial
training, formulated as a minimax optimization problem, has been recognized as
one of the most effective defense mechanisms. However, the non-convex and
non-concave property poses a great challenge to the minimax training. In this
paper, we empirically demonstrate that the commonly used PGD attack may not be
optimal for inner maximization, and improved inner optimizer can lead to a more
robust model. Then we leverage a learning-to-learn (L2L) framework to train an
optimizer with recurrent neural networks, providing update directions and steps
adaptively for the inner problem. By co-training optimizer's parameters and
model's weights, the proposed framework consistently improves the model
robustness over PGD-based adversarial training and TRADES.


http://arxiv.org/abs/2004.12261
Enabling Fast and Universal Audio Adversarial Attack Using Generative Model.
Yi Xie; Zhuohang Li; Cong Shi; Jian Liu; Yingying Chen; Bo Yuan
  Recently, the vulnerability of DNN-based audio systems to adversarial attacks
has obtained the increasing attention. However, the existing audio adversarial
attacks allow the adversary to possess the entire user's audio input as well as
granting sufficient time budget to generate the adversarial perturbations.
These idealized assumptions, however, makes the existing audio adversarial
attacks mostly impossible to be launched in a timely fashion in practice (e.g.,
playing unnoticeable adversarial perturbations along with user's streaming
input). To overcome these limitations, in this paper we propose fast audio
adversarial perturbation generator (FAPG), which uses generative model to
generate adversarial perturbations for the audio input in a single forward
pass, thereby drastically improving the perturbation generation speed. Built on
the top of FAPG, we further propose universal audio adversarial perturbation
generator (UAPG), a scheme crafting universal adversarial perturbation that can
be imposed on arbitrary benign audio input to cause misclassification.
Extensive experiments show that our proposed FAPG can achieve up to 167X
speedup over the state-of-the-art audio adversarial attack methods. Also our
proposed UAPG can generate universal adversarial perturbation that achieves
much better attack performance than the state-of-the-art solutions.


http://arxiv.org/abs/2004.13013
Harnessing adversarial examples with a surprisingly simple defense.
Ali Borji
  I introduce a very simple method to defend against adversarial examples. The
basic idea is to raise the slope of the ReLU function at the test time.
Experiments over MNIST and CIFAR-10 datasets demonstrate the effectiveness of
the proposed defense against a number of strong attacks in both untargeted and
targeted settings. While perhaps not as effective as the state of the art
adversarial defenses, this approach can provide insights to understand and
mitigate adversarial attacks. It can also be used in conjunction with other
defenses.


http://arxiv.org/abs/2004.11573
Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty.
Xiyue Zhang; Xiaofei Xie; Lei Ma; Xiaoning Du; Qiang Hu; Yang Liu; Jianjun Zhao; Meng Sun
  Over the past decade, deep learning (DL) has been successfully applied to
many industrial domain-specific tasks. However, the current state-of-the-art DL
software still suffers from quality issues, which raises great concern
especially in the context of safety- and security-critical scenarios.
Adversarial examples (AEs) represent a typical and important type of defects
needed to be urgently addressed, on which a DL software makes incorrect
decisions. Such defects occur through either intentional attack or
physical-world noise perceived by input sensors, potentially hindering further
industry deployment. The intrinsic uncertainty nature of deep learning
decisions can be a fundamental reason for its incorrect behavior. Although some
testing, adversarial attack and defense techniques have been recently proposed,
it still lacks a systematic study to uncover the relationship between AEs and
DL uncertainty. In this paper, we conduct a large-scale study towards bridging
this gap. We first investigate the capability of multiple uncertainty metrics
in differentiating benign examples (BEs) and AEs, which enables to characterize
the uncertainty patterns of input data. Then, we identify and categorize the
uncertainty patterns of BEs and AEs, and find that while BEs and AEs generated
by existing methods do follow common uncertainty patterns, some other
uncertainty patterns are largely missed. Based on this, we propose an automated
testing technique to generate multiple types of uncommon AEs and BEs that are
largely missed by existing techniques. Our further evaluation reveals that the
uncommon data generated by our method is hard to be defended by the existing
defense techniques with the average defense success rate reduced by 35\%. Our
results call for attention and necessity to generate more diverse data for
evaluating quality assurance solutions of DL software.


http://arxiv.org/abs/2004.14174
Reevaluating Adversarial Examples in Natural Language.
John X. Morris; Eli Lifland; Jack Lanchantin; Yangfeng Ji; Yanjun Qi
  State-of-the-art attacks on NLP models have different definitions of what
constitutes a successful attack. These differences make the attacks difficult
to compare. We propose to standardize definitions of natural language
adversarial examples based on a set of linguistic constraints: semantics,
grammaticality, edit distance, and non-suspicion. We categorize previous
attacks based on these constraints. For each constraint, we suggest options for
human and automatic evaluation methods. We use these methods to evaluate two
state-of-the-art synonym substitution attacks. We find that perturbations often
do not preserve semantics, and 45\% introduce grammatical errors. Next, we
conduct human studies to find a threshold for each evaluation method that
aligns with human judgment. Human surveys reveal that to truly preserve
semantics, we need to significantly increase the minimum cosine similarity
between the embeddings of swapped words and sentence encodings of original and
perturbed inputs. After tightening these constraints to agree with the judgment
of our human annotators, the attacks produce valid, successful adversarial
examples. But quality comes at a cost: attack success rate drops by over 70
percentage points. Finally, we introduce TextAttack, a library for adversarial
attacks in NLP.


http://arxiv.org/abs/2004.13002
One Sparse Perturbation to Fool them All, almost Always!
Arka Ghosh; Sankha Subhra Mullick; Shounak Datta; Swagatam Das; Rammohan Mallipeddi; Asit Kr. Das
  Constructing adversarial perturbations for deep neural networks is an
important direction of research. Crafting image-dependent adversarial
perturbations using white-box feedback has hitherto been the norm for such
adversarial attacks. However, black-box attacks are much more practical for
real-world applications. Universal perturbations applicable across multiple
images are gaining popularity due to their innate generalizability. There have
also been efforts to restrict the perturbations to a few pixels in the image.
This helps to retain visual similarity with the original images making such
attacks hard to detect. This paper marks an important step which combines all
these directions of research. We propose the DEceit algorithm for constructing
effective universal pixel-restricted perturbations using only black-box
feedback from the target network. We conduct empirical investigations using the
ImageNet validation set on the state-of-the-art deep neural classifiers by
varying the number of pixels to be perturbed from a meagre 10 pixels to as high
as all pixels in the image. We find that perturbing only about 10% of the
pixels in an image using DEceit achieves a commendable and highly transferable
Fooling Rate while retaining the visual quality. We further demonstrate that
DEceit can be successfully applied to image dependent attacks as well. In both
sets of experiments, we outperformed several state-of-the-art methods.


http://arxiv.org/abs/2004.11898
Adversarial Machine Learning in Network Intrusion Detection Systems.
Elie Alhajjar; Paul Maxwell; Nathaniel D. Bastian
  Adversarial examples are inputs to a machine learning system intentionally
crafted by an attacker to fool the model into producing an incorrect output.
These examples have achieved a great deal of success in several domains such as
image recognition, speech recognition and spam detection. In this paper, we
study the nature of the adversarial problem in Network Intrusion Detection
Systems (NIDS). We focus on the attack perspective, which includes techniques
to generate adversarial examples capable of evading a variety of machine
learning models. More specifically, we explore the use of evolutionary
computation (particle swarm optimization and genetic algorithm) and deep
learning (generative adversarial networks) as tools for adversarial example
generation. To assess the performance of these algorithms in evading a NIDS, we
apply them to two publicly available data sets, namely the NSL-KDD and
UNSW-NB15, and we contrast them to a baseline perturbation method: Monte Carlo
simulation. The results show that our adversarial example generation techniques
cause high misclassification rates in eleven different machine learning models,
along with a voting classifier. Our work highlights the vulnerability of
machine learning based NIDS in the face of adversarial perturbation.


http://arxiv.org/abs/2004.11488
Adversarial Machine Learning: An Interpretation Perspective.
Ninghao Liu; Mengnan Du; Xia Hu
  Recent years have witnessed the significant advances of machine learning in a
wide spectrum of applications. However, machine learning models, especially
deep neural networks, have been recently found to be vulnerable to
carefully-crafted input called adversarial samples. The difference between
normal and adversarial samples is almost imperceptible to human. Many work have
been proposed to study adversarial attack and defense in different scenarios.
An intriguing and crucial aspect among those work is to understand the
essential cause of model vulnerability, which requires in-depth exploration of
another concept in machine learning models, i.e., interpretability.
Interpretable machine learning tries to extract human-understandable terms for
the working mechanism of models, which also receives a lot of attention from
both academia and industry. Recently, an increasing number of work start to
incorporate interpretation into the exploration of adversarial robustness.
Furthermore, we observe that many previous work of adversarial attacking,
although did not mention it explicitly, can be regarded as natural extension of
interpretation. In this paper, we review recent work on adversarial attack and
defense, particularly, from the perspective of machine learning interpretation.
We categorize interpretation into two types, according to whether it focuses on
raw features or model components. For each type of interpretation, we elaborate
on how it could be used in attacks, or defense against adversaries. After that,
we briefly illustrate other possible correlations between the two domains.
Finally, we discuss the challenges and future directions along tackling
adversary issues with interpretation.


http://arxiv.org/abs/2004.14798
RAIN: Robust and Accurate Classification Networks with Randomization and Enhancement.
Jiawei Du; Hanshu Yan; Joey Tianyi Zhou; Rick Siow Mong Goh; Jiashi Feng
  Along with the extensive applications of CNN models for classification, there
has been a growing requirement for their robustness against adversarial
examples. In recent years, many adversarial defense methods have been
introduced, but most of them have to sacrifice classification accuracy on clean
samples to achieve better robustness of CNNs. In this paper, we propose a novel
framework to improve robustness and meanwhile retain the accuracy of given
classification CNN models, termed as RAIN, which consists of two conjugate
modules: structured randomization (SRd) and detail generation (DG).
Specifically, the SRd module randomly downsamples and shifts the input, which
can destroy the structure of adversarial perturbations so as to improve the
model robustness. However, such operations also incur accuracy drop inevitably.
Through our empirical study, the resultant image of the SRd module suffers loss
of high-frequency details that are crucial for model accuracy. To remedy the
accuracy drop, RAIN couples a deep super-resolution model as the DG module for
recovering rich details in the resultant image. We evaluate RAIN on STL10 and
the ImageNet datasets, and experiment results well demonstrate its great
robustness against adversarial examples as well as comparable classification
accuracy to non-robustified counterparts on clean samples. Our framework is
simple, effective and substantially extends the application of adversarial
defense techniques to realistic scenarios where clean and adversarial samples
are mixed.


http://arxiv.org/abs/2004.11114
Evaluating Adversarial Robustness for Deep Neural Network Interpretability using fMRI Decoding.
Patrick McClure; Dustin Moraczewski; Ka Chun Lam; Adam Thomas; Francisco Pereira
  While deep neural networks (DNNs) are being increasingly used to make
predictions from high-dimensional, complex data, they are widely seen as
uninterpretable "black boxes", since it can be difficult to discover what input
information is used to make predictions. This ability is particularly important
for applications in cognitive neuroscience and neuroinformatics. A saliency map
is a common approach for producing interpretable visualizations of the relative
importance of input features for a prediction. However, many methods for
creating these maps fail due to focusing too much on the input or being
extremely sensitive to small input noise. It is also challenging to
quantitatively evaluate how well saliency maps correspond to the truly relevant
input information. In this paper, we develop two quantitative evaluation
procedures for saliency methods, using the fact that the Human Connectome
Project (HCP) dataset contains functional magnetic resonance imaging(fMRI) data
from multiple tasks per subject to create ground truth saliency maps.We then
introduce an adversarial training method that makes DNNs robust to small input
noise, and use these evaluations to demonstrate that it greatly improves
interpretability.


http://arxiv.org/abs/2004.11157
On Adversarial Examples for Biomedical NLP Tasks.
Vladimir Araujo; Andres Carvallo; Carlos Aspillaga; Denis Parra
  The success of pre-trained word embeddings has motivated its use in tasks in
the biomedical domain. The BERT language model has shown remarkable results on
standard performance metrics in tasks such as Named Entity Recognition (NER)
and Semantic Textual Similarity (STS), which has brought significant progress
in the field of NLP. However, it is unclear whether these systems work
seemingly well in critical domains, such as legal or medical. For that reason,
in this work, we propose an adversarial evaluation scheme on two well-known
datasets for medical NER and STS. We propose two types of attacks inspired by
natural spelling errors and typos made by humans. We also propose another type
of attack that uses synonyms of medical terms. Under these adversarial
settings, the accuracy of the models drops significantly, and we quantify the
extent of this performance loss. We also show that we can significantly improve
the robustness of the models by training them with adversarial examples. We
hope our work will motivate the use of adversarial examples to evaluate and
develop models with increased robustness for medical tasks.


http://arxiv.org/abs/2004.11273
Ensemble Generative Cleaning with Feedback Loops for Defending Adversarial Attacks.
Jianhe Yuan; Zhihai He
  Effective defense of deep neural networks against adversarial attacks remains
a challenging problem, especially under powerful white-box attacks. In this
paper, we develop a new method called ensemble generative cleaning with
feedback loops (EGC-FL) for effective defense of deep neural networks. The
proposed EGC-FL method is based on two central ideas. First, we introduce a
transformed deadzone layer into the defense network, which consists of an
orthonormal transform and a deadzone-based activation function, to destroy the
sophisticated noise pattern of adversarial attacks. Second, by constructing a
generative cleaning network with a feedback loop, we are able to generate an
ensemble of diverse estimations of the original clean image. We then learn a
network to fuse this set of diverse estimations together to restore the
original image. Our extensive experimental results demonstrate that our
approach improves the state-of-art by large margins in both white-box and
black-box attacks. It significantly improves the classification accuracy for
white-box PGD attacks upon the second best method by more than 29% on the SVHN
dataset and more than 39% on the challenging CIFAR-10 dataset.


http://arxiv.org/abs/2004.11072
Improved Noise and Attack Robustness for Semantic Segmentation by Using Multi-Task Training with Self-Supervised Depth Estimation.
Marvin Klingner; Andreas Bär; Tim Fingscheidt
  While current approaches for neural network training often aim at improving
performance, less focus is put on training methods aiming at robustness towards
varying noise conditions or directed attacks by adversarial examples. In this
paper, we propose to improve robustness by a multi-task training, which extends
supervised semantic segmentation by a self-supervised monocular depth
estimation on unlabeled videos. This additional task is only performed during
training to improve the semantic segmentation model's robustness at test time
under several input perturbations. Moreover, we even find that our joint
training approach also improves the performance of the model on the original
(supervised) semantic segmentation task. Our evaluation exhibits a particular
novelty in that it allows to mutually compare the effect of input noises and
adversarial attacks on the robustness of the semantic segmentation. We show the
effectiveness of our method on the Cityscapes dataset, where our multi-task
training approach consistently outperforms the single-task semantic
segmentation baseline in terms of both robustness vs. noise and in terms of
adversarial attacks, without the need for depth labels in training.


http://arxiv.org/abs/2004.10700
CodNN -- Robust Neural Networks From Coded Classification.
Netanel Andrew Raviv; Siddharth Andrew Jain; Pulakesh Andrew Upadhyaya; Jehoshua Andrew Bruck; Andrew Anxiao; Jiang
  Deep Neural Networks (DNNs) are a revolutionary force in the ongoing
information revolution, and yet their intrinsic properties remain a mystery. In
particular, it is widely known that DNNs are highly sensitive to noise, whether
adversarial or random. This poses a fundamental challenge for hardware
implementations of DNNs, and for their deployment in critical applications such
as autonomous driving. In this paper we construct robust DNNs via error
correcting codes. By our approach, either the data or internal layers of the
DNN are coded with error correcting codes, and successful computation under
noise is guaranteed. Since DNNs can be seen as a layered concatenation of
classification tasks, our research begins with the core task of classifying
noisy coded inputs, and progresses towards robust DNNs. We focus on binary data
and linear codes. Our main result is that the prevalent parity code can
guarantee robustness for a large family of DNNs, which includes the recently
popularized binarized neural networks. Further, we show that the coded
classification problem has a deep connection to Fourier analysis of Boolean
functions. In contrast to existing solutions in the literature, our results do
not rely on altering the training process of the DNN, and provide
mathematically rigorous guarantees rather than experimental evidence.


http://arxiv.org/abs/2004.10608
Provably robust deep generative models.
Filipe Condessa; Zico Kolter
  Recent work in adversarial attacks has developed provably robust methods for
training deep neural network classifiers. However, although they are often
mentioned in the context of robustness, deep generative models themselves have
received relatively little attention in terms of formally analyzing their
robustness properties. In this paper, we propose a method for training provably
robust generative models, specifically a provably robust version of the
variational auto-encoder (VAE). To do so, we first formally define a
(certifiably) robust lower bound on the variational lower bound of the
likelihood, and then show how this bound can be optimized during training to
produce a robust VAE. We evaluate the method on simple examples, and show that
it is able to produce generative models that are substantially more robust to
adversarial attacks (i.e., an adversary trying to perturb inputs so as to
drastically lower their likelihood under the model).


http://arxiv.org/abs/2004.11233
QUANOS- Adversarial Noise Sensitivity Driven Hybrid Quantization of Neural Networks.
Priyadarshini Panda
  Deep Neural Networks (DNNs) have been shown to be vulnerable to adversarial
attacks, wherein, a model gets fooled by applying slight perturbations on the
input. With the advent of Internet-of-Things and the necessity to enable
intelligence in embedded devices, low-power and secure hardware implementation
of DNNs is vital. In this paper, we investigate the use of quantization to
potentially resist adversarial attacks. Several recent studies have reported
remarkable results in reducing the energy requirement of a DNN through
quantization. However, no prior work has considered the relationship between
adversarial sensitivity of a DNN and its effect on quantization. We propose
QUANOS- a framework that performs layer-specific hybrid quantization based on
Adversarial Noise Sensitivity (ANS). We identify a novel noise stability metric
(ANS) for DNNs, i.e., the sensitivity of each layer's computation to
adversarial noise. ANS allows for a principled way of determining optimal
bit-width per layer that incurs adversarial robustness as well as
energy-efficiency with minimal loss in accuracy. Essentially, QUANOS assigns
layer significance based on its contribution to adversarial perturbation and
accordingly scales the precision of the layers. A key advantage of QUANOS is
that it does not rely on a pre-trained model and can be applied in the initial
stages of training. We evaluate the benefits of QUANOS on precision scalable
Multiply and Accumulate (MAC) hardware architectures with data gating and
subword parallelism capabilities. Our experiments on CIFAR10, CIFAR100 datasets
show that QUANOS outperforms homogenously quantized 8-bit precision baseline in
terms of adversarial robustness (3%-4% higher) while yielding improved
compression (>5x) and energy savings (>2x) at iso-accuracy.


http://arxiv.org/abs/2004.10882
Adversarial examples and where to find them.
Niklas Risse; Christina Göpfert; Jan Philip Göpfert
  Adversarial robustness of trained models has attracted considerable attention
over recent years, within and beyond the scientific community. This is not only
because of a straight-forward desire to deploy reliable systems, but also
because of how adversarial attacks challenge our beliefs about deep neural
networks. Demanding more robust models seems to be the obvious solution --
however, this requires a rigorous understanding of how one should judge
adversarial robustness as a property of a given model. In this work, we analyze
where adversarial examples occur, in which ways they are peculiar, and how they
are processed by robust models. We use robustness curves to show that
$\ell_\infty$ threat models are surprisingly effective in improving robustness
for other $\ell_p$ norms; we introduce perturbation cost trajectories to
provide a broad perspective on how robust and non-robust networks perceive
adversarial perturbations as opposed to random perturbations; and we explicitly
examine the scale of certain common data sets, showing that robustness
thresholds must be adapted to the data set they pertain to. This allows us to
provide concrete recommendations for anyone looking to train a robust model or
to estimate how much robustness they should require for their operation. The
code for all our experiments is available at
www.github.com/niklasrisse/adversarial-examples-and-where-to-find-them .


http://arxiv.org/abs/2004.13825
Scalable Attack on Graph Data by Injecting Vicious Nodes.
Jihong Wang; Minnan Luo; Fnu Suya; Jundong Li; Zijiang Yang; Qinghua Zheng
  Recent studies have shown that graph convolution networks (GCNs) are
vulnerable to carefully designed attacks, which aim to cause misclassification
of a specific node on the graph with unnoticeable perturbations. However, a
vast majority of existing works cannot handle large-scale graphs because of
their high time complexity. Additionally, existing works mainly focus on
manipulating existing nodes on the graph, while in practice, attackers usually
do not have the privilege to modify information of existing nodes. In this
paper, we develop a more scalable framework named Approximate Fast Gradient
Sign Method (AFGSM) which considers a more practical attack scenario where
adversaries can only inject new vicious nodes to the graph while having no
control over the original graph. Methodologically, we provide an approximation
strategy to linearize the model we attack and then derive an approximate
closed-from solution with a lower time cost. To have a fair comparison with
existing attack methods that manipulate the original graph, we adapt them to
the new attack scenario by injecting vicious nodes. Empirical experimental
results show that our proposed attack method can significantly reduce the
classification accuracy of GCNs and is much faster than existing methods
without jeopardizing the attack performance.


http://arxiv.org/abs/2004.10281
Probabilistic Safety for Bayesian Neural Networks.
Matthew Wicker; Luca Laurenti; Andrea Patane; Marta Kwiatkowska
  We study probabilistic safety for Bayesian Neural Networks (BNNs) under
adversarial input perturbations. Given a compact set of input points, T
\subseteq R^m, we study the probability w.r.t. the BNN posterior that all the
points in T are mapped to the same, given region S in the output space. In
particular, this can be used to evaluate the probability that a network sampled
from the BNN is vulnerable to adversarial attacks. We rely on relaxation
techniques from non-convex optimization to develop a method for computing a
lower bound on probabilistic safety for BNNs, deriving explicit procedures for
the case of interval and linear function propagation techniques. We apply our
methods to BNNs trained on a regression task, airborne collision avoidance, and
MNIST, empirically showing that our approach allows one to certify
probabilistic safety of BNNs with thousands of neurons.


http://arxiv.org/abs/2004.10250
Certifying Joint Adversarial Robustness for Model Ensembles.
Mainuddin Ahmad Jonas; David Evans
  Deep Neural Networks (DNNs) are often vulnerable to adversarial
examples.Several proposed defenses deploy an ensemble of models with the hope
that, although the individual models may be vulnerable, an adversary will not
be able to find an adversarial example that succeeds against the ensemble.
Depending on how the ensemble is used, an attacker may need to find a single
adversarial example that succeeds against all, or a majority, of the models in
the ensemble. The effectiveness of ensemble defenses against strong adversaries
depends on the vulnerability spaces of models in the ensemble being disjoint.
We consider the joint vulnerability of an ensemble of models, and propose a
novel technique for certifying the joint robustness of ensembles, building upon
prior works on single-model robustness certification. We evaluate the
robustness of various models ensembles, including models trained using
cost-sensitive robustness to be diverse, to improve understanding of the
potential effectiveness of ensemble models as a defense against adversarial
examples.


http://arxiv.org/abs/2004.09984
BERT-ATTACK: Adversarial Attack Against BERT Using BERT.
Linyang Li; Ruotian Ma; Qipeng Guo; Xiangyang Xue; Xipeng Qiu
  Adversarial attacks for discrete data (such as text) has been proved
significantly more challenging than continuous data (such as image), since it
is difficult to generate adversarial samples with gradient-based methods.
Currently, the successful attack methods for text usually adopt heuristic
replacement strategies on character or word level, which remains challenging to
find the optimal solution in the massive space of possible combination of
replacements, while preserving semantic consistency and language fluency. In
this paper, we propose \textbf{BERT-Attack}, a high-quality and effective
method to generate adversarial samples using pre-trained masked language models
exemplified by BERT. We turn BERT against its fine-tuned models and other deep
neural models for downstream tasks. Our method successfully misleads the target
models to predict incorrectly, outperforming state-of-the-art attack strategies
in both success rate and perturb percentage, while the generated adversarial
samples are fluent and semantically preserved. Also, the cost of calculation is
low, thus possible for large-scale generations.


http://arxiv.org/abs/2004.10162
EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks.
Sanchari Sen; Balaraman Ravindran; Anand Raghunathan
  Ensuring robustness of Deep Neural Networks (DNNs) is crucial to their
adoption in safety-critical applications such as self-driving cars, drones, and
healthcare. Notably, DNNs are vulnerable to adversarial attacks in which small
input perturbations can produce catastrophic misclassifications. In this work,
we propose EMPIR, ensembles of quantized DNN models with different numerical
precisions, as a new approach to increase robustness against adversarial
attacks. EMPIR is based on the observation that quantized neural networks often
demonstrate much higher robustness to adversarial attacks than full precision
networks, but at the cost of a substantial loss in accuracy on the original
(unperturbed) inputs. EMPIR overcomes this limitation to achieve the 'best of
both worlds', i.e., the higher unperturbed accuracies of the full precision
models combined with the higher robustness of the low precision models, by
composing them in an ensemble. Further, as low precision DNN models have
significantly lower computational and storage requirements than full precision
models, EMPIR models only incur modest compute and memory overheads compared to
a single full-precision model (<25% in our evaluations). We evaluate EMPIR
across a suite of DNNs for 3 different image recognition tasks (MNIST, CIFAR-10
and ImageNet) and under 4 different adversarial attacks. Our results indicate
that EMPIR boosts the average adversarial accuracies by 42.6%, 15.2% and 10.5%
for the DNN models trained on the MNIST, CIFAR-10 and ImageNet datasets
respectively, when compared to single full-precision models, without
sacrificing accuracy on the unperturbed inputs.


http://arxiv.org/abs/2004.09179
GraN: An Efficient Gradient-Norm Based Detector for Adversarial and Misclassified Examples.
Julia Lust; Alexandru Paul Condurache
  Deep neural networks (DNNs) are vulnerable to adversarial examples and other
data perturbations. Especially in safety critical applications of DNNs, it is
therefore crucial to detect misclassified samples. The current state-of-the-art
detection methods require either significantly more runtime or more parameters
than the original network itself. This paper therefore proposes GraN, a time-
and parameter-efficient method that is easily adaptable to any DNN.
  GraN is based on the layer-wise norm of the DNN's gradient regarding the loss
of the current input-output combination, which can be computed via
backpropagation. GraN achieves state-of-the-art performance on numerous problem
set-ups.


http://arxiv.org/abs/2004.08833
Dynamic Knowledge Graph-based Dialogue Generation with Improved Adversarial Meta-Learning.
Hongcai Xu; Junpeng Bao; Gaojie Zhang
  Knowledge graph-based dialogue systems are capable of generating more
informative responses and can implement sophisticated reasoning mechanisms.
However, these models do not take into account the sparseness and
incompleteness of knowledge graph (KG)and current dialogue models cannot be
applied to dynamic KG. This paper proposes a dynamic Knowledge graph-based
dialogue generation method with improved adversarial Meta-Learning (KDAD). KDAD
formulates dynamic knowledge triples as a problem of adversarial attack and
incorporates the objective of quickly adapting to dynamic knowledge-aware
dialogue generation. We train a knowledge graph-based dialog model with
improved ADML using minimal training samples. The model can initialize the
parameters and adapt to previous unseen knowledge so that training can be
quickly completed based on only a few knowledge triples. We show that our model
significantly outperforms other baselines. We evaluate and demonstrate that our
method adapts extremely fast and well to dynamic knowledge graph-based dialogue
generation.


http://arxiv.org/abs/2004.08994
Adversarial Training for Large Neural Language Models.
Xiaodong Liu; Hao Cheng; Pengcheng He; Weizhu Chen; Yu Wang; Hoifung Poon; Jianfeng Gao
  Generalization and robustness are both key desiderata for designing machine
learning methods. Adversarial training can enhance robustness, but past work
often finds it hurts generalization. In natural language processing (NLP),
pre-training large neural language models such as BERT have demonstrated
impressive gain in generalization for a variety of tasks, with further
improvement from adversarial fine-tuning. However, these models are still
vulnerable to adversarial attacks. In this paper, we show that adversarial
pre-training can improve both generalization and robustness. We propose a
general algorithm ALUM (Adversarial training for large neural LangUage Models),
which regularizes the training objective by applying perturbations in the
embedding space that maximizes the adversarial loss. We present the first
comprehensive study of adversarial training in all stages, including
pre-training from scratch, continual pre-training on a well-trained model, and
task-specific fine-tuning. ALUM obtains substantial gains over BERT on a wide
range of NLP tasks, in both regular and adversarial scenarios. Even for models
that have been well trained on extremely large text corpora, such as RoBERTa,
ALUM can still produce significant gains from continual pre-training, whereas
conventional non-adversarial methods can not. ALUM can be further combined with
task-specific fine-tuning to attain additional gains. The ALUM code and
pre-trained models will be made publicly available at
https://github.com/namisan/mt-dnn.


http://arxiv.org/abs/2004.09007
Headless Horseman: Adversarial Attacks on Transfer Learning Models.
Ahmed Abdelkader; Michael J. Curry; Liam Fowl; Tom Goldstein; Avi Schwarzschild; Manli Shu; Christoph Studer; Chen Zhu
  Transfer learning facilitates the training of task-specific classifiers using
pre-trained models as feature extractors. We present a family of transferable
adversarial attacks against such classifiers, generated without access to the
classification head; we call these \emph{headless attacks}. We first
demonstrate successful transfer attacks against a victim network using
\textit{only} its feature extractor. This motivates the introduction of a
label-blind adversarial attack. This transfer attack method does not require
any information about the class-label space of the victim. Our attack lowers
the accuracy of a ResNet18 trained on CIFAR10 by over 40\%.


http://arxiv.org/abs/2004.08705
Protecting Classifiers From Attacks. A Bayesian Approach.
Victor Gallego; Roi Naveiro; Alberto Redondo; David Rios Insua; Fabrizio Ruggeri
  Classification problems in security settings are usually modeled as
confrontations in which an adversary tries to fool a classifier manipulating
the covariates of instances to obtain a benefit. Most approaches to such
problems have focused on game-theoretic ideas with strong underlying common
knowledge assumptions, which are not realistic in the security realm. We
provide an alternative Bayesian framework that accounts for the lack of precise
knowledge about the attacker's behavior using adversarial risk analysis. A key
ingredient required by our framework is the ability to sample from the
distribution of originating instances given the possibly attacked observed one.
We propose a sampling procedure based on approximate Bayesian computation, in
which we simulate the attacker's problem taking into account our uncertainty
about his elements. For large scale problems, we propose an alternative,
scalable approach that could be used when dealing with differentiable
classifiers. Within it, we move the computational load to the training phase,
simulating attacks from an adversary, adapting the framework to obtain a
classifier robustified against attacks.


http://arxiv.org/abs/2004.08628
Single-step Adversarial training with Dropout Scheduling.
Vivek B. S.; R. Venkatesh Babu
  Deep learning models have shown impressive performance across a spectrum of
computer vision applications including medical diagnosis and autonomous
driving. One of the major concerns that these models face is their
susceptibility to adversarial attacks. Realizing the importance of this issue,
more researchers are working towards developing robust models that are less
affected by adversarial attacks. Adversarial training method shows promising
results in this direction. In adversarial training regime, models are trained
with mini-batches augmented with adversarial samples. Fast and simple methods
(e.g., single-step gradient ascent) are used for generating adversarial
samples, in order to reduce computational complexity. It is shown that models
trained using single-step adversarial training method (adversarial samples are
generated using non-iterative method) are pseudo robust. Further, this pseudo
robustness of models is attributed to the gradient masking effect. However,
existing works fail to explain when and why gradient masking effect occurs
during single-step adversarial training. In this work, (i) we show that models
trained using single-step adversarial training method learn to prevent the
generation of single-step adversaries, and this is due to over-fitting of the
model during the initial stages of training, and (ii) to mitigate this effect,
we propose a single-step adversarial training method with dropout scheduling.
Unlike models trained using existing single-step adversarial training methods,
models trained using the proposed single-step adversarial training method are
robust against both single-step and multi-step adversarial attacks, and the
performance is on par with models trained using computationally expensive
multi-step adversarial training methods, in white-box and black-box settings.


http://arxiv.org/abs/2004.08443
Adversarial Attack on Deep Learning-Based Splice Localization.
Andras Rozsa; Zheng Zhong; Terrance E. Boult
  Regarding image forensics, researchers have proposed various approaches to
detect and/or localize manipulations, such as splices. Recent best performing
image-forensics algorithms greatly benefit from the application of deep
learning, but such tools can be vulnerable to adversarial attacks. Due to the
fact that most of the proposed adversarial example generation techniques can be
used only on end-to-end classifiers, the adversarial robustness of
image-forensics methods that utilize deep learning only for feature extraction
has not been studied yet. Using a novel algorithm capable of directly adjusting
the underlying representations of patches we demonstrate on three non
end-to-end deep learning-based splice localization tools that hiding
manipulations of images is feasible via adversarial attacks. While the tested
image-forensics methods, EXIF-SC, SpliceRadar, and Noiseprint, rely on feature
extractors that were trained on different surrogate tasks, we find that the
formed adversarial perturbations can be transferable among them regarding the
deterioration of their localization performance.


http://arxiv.org/abs/2004.07780
Shortcut Learning in Deep Neural Networks.
Robert Geirhos; Jörn-Henrik Jacobsen; Claudio Michaelis; Richard Zemel; Wieland Brendel; Matthias Bethge; Felix A. Wichmann
  Deep learning has triggered the current rise of artificial intelligence and
is the workhorse of today's machine intelligence. Numerous success stories have
rapidly spread all over science, industry and society, but its limitations have
only recently come into focus. In this perspective we seek to distil how many
of deep learning's problem can be seen as different symptoms of the same
underlying problem: shortcut learning. Shortcuts are decision rules that
perform well on standard benchmarks but fail to transfer to more challenging
testing conditions, such as real-world scenarios. Related issues are known in
Comparative Psychology, Education and Linguistics, suggesting that shortcut
learning may be a common characteristic of learning systems, biological and
artificial alike. Based on these observations, we develop a set of
recommendations for model interpretation and benchmarking, highlighting recent
advances in machine learning to improve robustness and transferability from the
lab to real-world applications.


http://arxiv.org/abs/2004.07919
Enhancing Deep Neural Networks Against Adversarial Malware Examples.
Deqiang Li; Qianmu Li; Yanfang Ye; Shouhuai Xu
  Machine learning based malware detection is known to be vulnerable to
adversarial evasion attacks. The state-of-the-art is that there are no
effective countermeasures against these attacks. Inspired by the AICS'2019
Challenge organized by the MIT Lincoln Lab, we systematize a number of
principles for enhancing the robustness of neural networks against adversarial
malware evasion attacks. Some of these principles have been scattered in the
literature, but others are proposed in this paper for the first time. Under the
guidance of these principles, we propose a framework for defending against
adversarial malware evasion attacks. We validated the framework using the
Drebin dataset of Android malware. We applied the defense framework to the
AICS'2019 Challenge and won, without knowing how the organizers generated the
adversarial examples. However, we see a ~22\% difference between the accuracy
in the experiment with the Drebin dataset (for binary classification) and the
accuracy in the experiment with respect to the AICS'2019 Challenge (for
multiclass classification). We attribute this gap to a fundamental barrier that
without knowing the attacker's manipulation set, the defender cannot do
effective Adversarial Training.


http://arxiv.org/abs/2004.07955
Targeted Attack for Deep Hashing based Retrieval.
Jiawang Bai; Bin Chen; Yiming Li; Dongxian Wu; Weiwei Guo; Shu-tao Xia; En-hui Yang
  The deep hashing based retrieval method is widely adopted in large-scale
image and video retrieval. However, there is little investigation on its
security. In this paper, we propose a novel method, dubbed deep hashing
targeted attack (DHTA), to study the targeted attack on such retrieval.
Specifically, we first formulate the targeted attack as a point-to-set
optimization, which minimizes the average distance between the hash code of an
adversarial example and those of a set of objects with the target label. Then
we design a novel component-voting scheme to obtain an anchor code as the
representative of the set of hash codes of objects with the target label, whose
optimality guarantee is also theoretically derived. To balance the performance
and perceptibility, we propose to minimize the Hamming distance between the
hash code of the adversarial example and the anchor code under the
$\ell^\infty$ restriction on the perturbation. Extensive experiments verify
that DHTA is effective in attacking both deep hashing based image retrieval and
video retrieval.


http://arxiv.org/abs/2004.06954
Advanced Evasion Attacks and Mitigations on Practical ML-Based Phishing Website Classifiers.
Yusi Lei; Sen Chen; Lingling Fan; Fu Song; Yang Liu
  Machine learning (ML) based approaches have been the mainstream solution for
anti-phishing detection. When they are deployed on the client-side, ML-based
classifiers are vulnerable to evasion attacks. However, such potential threats
have received relatively little attention because existing attacks destruct the
functionalities or appearance of webpages and are conducted in the white-box
scenario, making it less practical. Consequently, it becomes imperative to
understand whether it is possible to launch evasion attacks with limited
knowledge of the classifier, while preserving the functionalities and
appearance.
  In this work, we show that even in the grey-, and black-box scenarios,
evasion attacks are not only effective on practical ML-based classifiers, but
can also be efficiently launched without destructing the functionalities and
appearance. For this purpose, we propose three mutation-based attacks,
differing in the knowledge of the target classifier, addressing a key technical
challenge: automatically crafting an adversarial sample from a known phishing
website in a way that can mislead classifiers. To launch attacks in the white-
and grey-box scenarios, we also propose a sample-based collision attack to gain
the knowledge of the target classifier. We demonstrate the effectiveness and
efficiency of our evasion attacks on the state-of-the-art, Google's phishing
page filter, achieved 100% attack success rate in less than one second per
website. Moreover, the transferability attack on BitDefender's industrial
phishing page classifier, TrafficLight, achieved up to 81.25% attack success
rate. We further propose a similarity-based method to mitigate such evasion
attacks, Pelican. We demonstrate that Pelican can effectively detect evasion
attacks. Our findings contribute to design more robust phishing website
classifiers in practice.


http://arxiv.org/abs/2004.06562
On the Optimal Interaction Range for Multi-Agent Systems Under Adversarial Attack.
Saad J Saleh
  Consider a consensus-driven multi-agent dynamic system. The interaction
range, which defines the set of neighbors for each agent, plays a key role in
influencing connectivity of the underlying network. In this paper, we assume
the system is under attack by a predator and explore the question of finding
the optimal interaction range that facilitates the most-efficient escape
trajectories for the group of agents. We find that for many cases of interest
the optimal interaction range is one that forces the network to break up into a
handful of disconnected graphs, each containing a subset of agents, thus
outperforming the two extreme cases corresponding to fully-connected and
fully-disconnected networks. In other words, the results indicate that some
connectivity among the agents is helpful because information is effectively
transmitted from the agents closest to the predator to others slightly farther
away, but also that too much connectivity can be detrimental to the agility of
the group, thus hampering efficient and rapid escape.


http://arxiv.org/abs/2004.06383
Extending Adversarial Attacks to Produce Adversarial Class Probability Distributions.
Jon Vadillo; Roberto Santana; Jose A. Lozano
  Despite the remarkable performance and generalization levels of deep learning
models in a wide range of artificial intelligence tasks, it has been
demonstrated that these models can be easily fooled by the addition of
imperceptible but malicious perturbations to natural inputs. These altered
inputs are known in the literature as adversarial examples. In this paper we
propose a novel probabilistic framework to generalize and extend adversarial
attacks in order to produce a desired probability distribution for the classes
when we apply the attack method to a large number of inputs. This novel attack
strategy provides the attacker with greater control over the target model, and
increases the complexity of detecting that the model is being attacked. We
introduce three different strategies to efficiently generate such attacks, and
illustrate our approach extending DeepFool, a state-of-the-art attack algorithm
to generate adversarial examples. We also experimentally validate our approach
for the spoken command classification task, an exemplary machine learning
problem in the audio domain. Our results demonstrate that we can closely
approximate any probability distribution for the classes while maintaining a
high fooling rate and by injecting imperceptible perturbations to the inputs.


http://arxiv.org/abs/2004.05923
Adversarial robustness guarantees for random deep neural networks.
Palma Giacomo De; Bobak T. Kiani; Seth Lloyd
  The reliability of most deep learning algorithms is fundamentally challenged
by the existence of adversarial examples, which are incorrectly classified
inputs that are extremely close to a correctly classified input. We study
adversarial examples for deep neural networks with random weights and biases
and prove that the $\ell^1$ distance of any given input from the classification
boundary scales at least as $\sqrt{n}$, where $n$ is the dimension of the
input. We also extend our proof to cover all the $\ell^p$ norms. Our results
constitute a fundamental advance in the study of adversarial examples, and
encompass a wide variety of architectures, which include any combination of
convolutional or fully connected layers with skipped connections and pooling.
We validate our results with experiments on both random deep neural networks
and deep neural networks trained on the MNIST and CIFAR10 datasets. Given the
results of our experiments on MNIST and CIFAR10, we conjecture that the proof
of our adversarial robustness guarantee can be extended to trained deep neural
networks. This extension will open the way to a thorough theoretical study of
neural network robustness by classifying the relation between network
architecture and adversarial distance.


http://arxiv.org/abs/2004.05884
Revisiting Loss Landscape for Adversarial Robustness.
Dongxian Wu; Yisen Wang; Shu-tao Xia
  The study on improving the robustness of deep neural networks against
adversarial examples grows rapidly in recent years. Among them, adversarial
training is the most promising one, based on which, a lot of improvements have
been developed, such as adding regularizations or leveraging unlabeled data.
However, these improvements seem to come from isolated perspectives, so that we
are curious about if there is something in common behind them. In this paper,
we investigate the surface geometry of several well-recognized adversarial
training variants, and reveal that their adversarial loss landscape is closely
related to the adversarially robust generalization, i.e., the flatter the
adversarial loss landscape, the smaller the adversarially robust generalization
gap. Based on this finding, we then propose a simple yet effective module,
Adversarial Weight Perturbation (AWP), to directly regularize the flatness of
the adversarial loss landscape in the adversarial training framework. Extensive
experiments demonstrate that AWP indeed owns flatter landscape and can be
easily incorporated into various adversarial training variants to enhance their
adversarial robustness further.


http://arxiv.org/abs/2004.05887
Frequency-Guided Word Substitutions for Detecting Textual Adversarial Examples.
Maximilian Mozes; Pontus Stenetorp; Bennett Kleinberg; Lewis D. Griffin
  While recent efforts have shown that neural text processing models are
vulnerable to adversarial examples, comparatively little attention has been
paid to explicitly characterize their effectiveness. To overcome this, we
present analytical insights into the word frequency characteristics of
word-level adversarial examples for neural text classification models. We show
that adversarial attacks against CNN-, LSTM- and Transformer-based
classification models perform token substitutions that are identifiable through
word frequency differences between replaced words and their substitutions.
Based on these findings, we propose frequency-guided word substitutions (FGWS)
as a simple algorithm for the automatic detection of adversarially perturbed
textual sequences. FGWS exploits the word frequency properties of adversarial
word substitutions, and we assess its suitability for the automatic detection
of adversarial examples generated from the SST-2 and IMDb sentiment datasets.
Our method provides promising results by accurately detecting adversarial
examples, with $F_1$ detection scores of up to 93.7% on adversarial examples
against BERT-based classification models. We compare our approach against
baseline detection approaches as well as a recently proposed perturbation
discrimination framework, and show that we outperform existing approaches by up
to 15.1% $F_1$ in our experiments.


http://arxiv.org/abs/2004.05790
Towards Transferable Adversarial Attack against Deep Face Recognition.
Yaoyao Zhong; Weihong Deng
  Face recognition has achieved great success in the last five years due to the
development of deep learning methods. However, deep convolutional neural
networks (DCNNs) have been found to be vulnerable to adversarial examples. In
particular, the existence of transferable adversarial examples could severely
hinder the robustness of DCNNs since this type of attacks could be applied in a
fully black-box manner without queries on the target system. In this work, we
first investigate the characteristics of transferable adversarial attacks in
face recognition by showing the superiority of feature-level methods over
label-level methods. Then, to further improve transferability of feature-level
adversarial examples, we propose DFANet, a dropout-based method used in
convolutional layers, which could increase the diversity of surrogate models
and obtain ensemble-like effects. Extensive experiments on state-of-the-art
face models with various training databases, loss functions and network
architectures show that the proposed method can significantly enhance the
transferability of existing attack methods. Finally, by applying DFANet to the
LFW database, we generate a new set of adversarial face pairs that can
successfully attack four commercial APIs without any queries. This TALFW
database is available to facilitate research on the robustness and defense of
deep face recognition.


http://arxiv.org/abs/2004.06076
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension.
Adyasha Maharana; Mohit Bansal
  Reading comprehension models often overfit to nuances of training datasets
and fail at adversarial evaluation. Training with adversarially augmented
dataset improves robustness against those adversarial attacks but hurts
generalization of the models. In this work, we present several effective
adversaries and automated data augmentation policy search methods with the goal
of making reading comprehension models more robust to adversarial evaluation,
but also improving generalization to the source domain as well as new domains
and languages. We first propose three new methods for generating QA
adversaries, that introduce multiple points of confusion within the context,
show dependence on insertion location of the distractor, and reveal the
compounding effect of mixing adversarial strategies with syntactic and semantic
paraphrasing methods. Next, we find that augmenting the training datasets with
uniformly sampled adversaries improves robustness to the adversarial attacks
but leads to decline in performance on the original unaugmented dataset. We
address this issue via RL and more efficient Bayesian policy search methods for
automatically learning the best augmentation policy combinations of the
transformation probability for each adversary in a large search space. Using
these learned policies, we show that adversarial training can lead to
significant improvements in in-domain, out-of-domain, and cross-lingual
(German, Russian, Turkish) generalization without any use of training data from
the target domain or language.


http://arxiv.org/abs/2004.06288
Towards Robust Classification with Image Quality Assessment.
Yeli Feng; Yiyu Cai
  Recent studies have shown that deep convolutional neural networks (DCNN) are
vulnerable to adversarial examples and sensitive to perceptual quality as well
as the acquisition condition of images. These findings raise a big concern for
the adoption of DCNN-based applications for critical tasks. In the literature,
various defense strategies have been introduced to increase the robustness of
DCNN, including re-training an entire model with benign noise injection,
adversarial examples, or adding extra layers. In this paper, we investigate the
connection between adversarial manipulation and image quality, subsequently
propose a protective mechanism that doesnt require re-training a DCNN. Our
method combines image quality assessment with knowledge distillation to detect
input images that would trigger a DCCN to produce egregiously wrong results.
Using the ResNet model trained on ImageNet as an example, we demonstrate that
the detector can effectively identify poor quality and adversarial images.


http://arxiv.org/abs/2004.05682
PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning.
Chenglin Yang; Adam Kortylewski; Cihang Xie; Yinzhi Cao; Alan Yuille
  Patch-based attacks introduce a perceptible but localized change to the input
that induces misclassification. A limitation of current patch-based black-box
attacks is that they perform poorly for targeted attacks, and even for the less
challenging non-targeted scenarios, they require a large number of queries. Our
proposed PatchAttack is query efficient and can break models for both targeted
and non-targeted attacks. PatchAttack induces misclassifications by
superimposing small textured patches on the input image. We parametrize the
appearance of these patches by a dictionary of class-specific textures. This
texture dictionary is learned by clustering Gram matrices of feature
activations from a VGG backbone. PatchAttack optimizes the position and texture
parameters of each patch using reinforcement learning. Our experiments show
that PatchAttack achieves > 99% success rate on ImageNet for a wide range of
architectures, while only manipulating 3% of the image for non-targeted attacks
and 10% on average for targeted attacks. Furthermore, we show that PatchAttack
circumvents state-of-the-art adversarial defense methods successfully.


http://arxiv.org/abs/2004.05465
Robust Large-Margin Learning in Hyperbolic Space.
Melanie Weber; Manzil Zaheer; Ankit Singh Rawat; Aditya Menon; Sanjiv Kumar
  Recently, there has been a surge of interest in representation learning in
hyperbolic spaces, driven by their ability to represent hierarchical data with
significantly fewer dimensions than standard Euclidean spaces. However, the
viability and benefits of hyperbolic spaces for downstream machine learning
tasks have received less attention. In this paper, we present, to our
knowledge, the first theoretical guarantees for learning a classifier in
hyperbolic rather than Euclidean space. Specifically, we consider the problem
of learning a large-margin classifier for data possessing a hierarchical
structure. Our first contribution is a hyperbolic perceptron algorithm, which
provably converges to a separating hyperplane. We then provide an algorithm to
efficiently learn a large-margin hyperplane, relying on the careful injection
of adversarial examples. Finally, we prove that for hierarchical data that
embeds well into hyperbolic space, the low embedding dimension ensures superior
guarantees when learning the classifier directly in hyperbolic space.


http://arxiv.org/abs/2004.11819
Domain Adaptive Transfer Attack (DATA)-based Segmentation Networks for Building Extraction from Aerial Images.
Younghwan Na; Jun Hee Kim; Kyungsu Lee; Juhum Park; Jae Youn Hwang; Jihwan P. Choi
  Semantic segmentation models based on convolutional neural networks (CNNs)
have gained much attention in relation to remote sensing and have achieved
remarkable performance for the extraction of buildings from high-resolution
aerial images. However, the issue of limited generalization for unseen images
remains. When there is a domain gap between the training and test datasets,
CNN-based segmentation models trained by a training dataset fail to segment
buildings for the test dataset. In this paper, we propose segmentation networks
based on a domain adaptive transfer attack (DATA) scheme for building
extraction from aerial images. The proposed system combines the domain transfer
and adversarial attack concepts. Based on the DATA scheme, the distribution of
the input images can be shifted to that of the target images while turning
images into adversarial examples against a target network. Defending
adversarial examples adapted to the target domain can overcome the performance
degradation due to the domain gap and increase the robustness of the
segmentation model. Cross-dataset experiments and the ablation study are
conducted for the three different datasets: the Inria aerial image labeling
dataset, the Massachusetts building dataset, and the WHU East Asia dataset.
Compared to the performance of the segmentation network without the DATA
scheme, the proposed method shows improvements in the overall IoU. Moreover, it
is verified that the proposed method outperforms even when compared to feature
adaptation (FA) and output space adaptation (OSA).


http://arxiv.org/abs/2004.06496
Certified Adversarial Robustness for Deep Reinforcement Learning.
Michael Everett; Bjorn Lutjens; Jonathan P. How
  Deep Neural Network-based systems are now the state-of-the-art in many
robotics tasks, but their application in safety-critical domains remains
dangerous without formal guarantees on network robustness. Small perturbations
to sensor inputs (from noise or adversarial examples) are often enough to
change network-based decisions, which was recently shown to cause an autonomous
vehicle to swerve into another lane. In light of these dangers, numerous
algorithms have been developed as defensive mechanisms from these adversarial
inputs, some of which provide formal robustness guarantees or certificates.
This work leverages research on certified adversarial robustness to develop an
online certified defense for deep reinforcement learning algorithms. The
proposed defense computes guaranteed lower bounds on state-action values during
execution to identify and choose a robust action under a worst-case deviation
in input space due to possible adversaries or noise. The approach is
demonstrated on a Deep Q-Network policy and is shown to increase robustness to
noise and adversaries in pedestrian collision avoidance scenarios and a classic
control task. This work extends our previous paper with new performance
guarantees, expanded results aggregated across more scenarios, an extension
into scenarios with adversarial behavior, comparisons with a more
computationally expensive method, and visualizations that provide intuition
about the robustness algorithm.


http://arxiv.org/abs/2004.05511
Verification of Deep Convolutional Neural Networks Using ImageStars.
Hoang-Dung Tran; Stanley Bak; Weiming Xiang; Taylor T. Johnson
  Convolutional Neural Networks (CNN) have redefined the state-of-the-art in
many real-world applications, such as facial recognition, image classification,
human pose estimation, and semantic segmentation. Despite their success, CNNs
are vulnerable to adversarial attacks, where slight changes to their inputs may
lead to sharp changes in their output in even well-trained networks. Set-based
analysis methods can detect or prove the absence of bounded adversarial
attacks, which can then be used to evaluate the effectiveness of neural network
training methodology. Unfortunately, existing verification approaches have
limited scalability in terms of the size of networks that can be analyzed.
  In this paper, we describe a set-based framework that successfully deals with
real-world CNNs, such as VGG16 and VGG19, that have high accuracy on ImageNet.
Our approach is based on a new set representation called the ImageStar, which
enables efficient exact and over-approximative analysis of CNNs. ImageStars
perform efficient set-based analysis by combining operations on concrete images
with linear programming (LP). Our approach is implemented in a tool called NNV,
and can verify the robustness of VGG networks with respect to a small set of
input states, derived from adversarial attacks, such as the DeepFool attack.
The experimental results show that our approach is less conservative and faster
than existing zonotope methods, such as those used in DeepZ, and the polytope
method used in DeepPoly.


http://arxiv.org/abs/2004.05005
Adversarial Attacks on Machine Learning Cybersecurity Defences in Industrial Control Systems.
Eirini Anthi; Lowri Williams; Matilda Rhode; Pete Burnap; Adam Wedgbury
  The proliferation and application of machine learning based Intrusion
Detection Systems (IDS) have allowed for more flexibility and efficiency in the
automated detection of cyber attacks in Industrial Control Systems (ICS).
However, the introduction of such IDSs has also created an additional attack
vector; the learning models may also be subject to cyber attacks, otherwise
referred to as Adversarial Machine Learning (AML). Such attacks may have severe
consequences in ICS systems, as adversaries could potentially bypass the IDS.
This could lead to delayed attack detection which may result in infrastructure
damages, financial loss, and even loss of life. This paper explores how
adversarial learning can be used to target supervised models by generating
adversarial samples using the Jacobian-based Saliency Map attack and exploring
classification behaviours. The analysis also includes the exploration of how
such samples can support the robustness of supervised models using adversarial
training. An authentic power system dataset was used to support the experiments
presented herein. Overall, the classification performance of two widely used
classifiers, Random Forest and J48, decreased by 16 and 20 percentage points
when adversarial samples were present. Their performances improved following
adversarial training, demonstrating their robustness towards such attacks.


http://arxiv.org/abs/2004.04919
Luring of Adversarial Perturbations.
Rémi Bernhard; Pierre-Alain Moellic; Jean-Max Dutertre
  The growing interest for adversarial examples, i.e. maliciously modified
examples which fool a classifier, has resulted in many defenses intended to
detect them, render them inoffensive or make the model more robust against
them. In this paper, we pave the way towards a new approach to defend a distant
system against adversarial examples, which we name the luring of adversarial
perturbations. A component is included in the target model to form an augmented
and equally accurate version of it. This additional component is designed to be
removable and to give false indications on the way to fool the target model
alone: the adversary is tricked into fooling the augmented version of the
target model, and not the target model. We explain the intuition of our defense
with the principle of the luring effect, inspired by the notion of robust and
non-robust features, and experimentally justify its validity. Eventually, we
propose a simple prediction strategy which takes advantage of this effect, and
show that our defense scheme on MNIST, SVHN and CIFAR10 can efficiently thwart
an adversary using state-of-the-art attacks and allowed to perform large
perturbations.


http://arxiv.org/abs/2004.05914
Blind Adversarial Training: Balance Accuracy and Robustness.
Haidong Xie; Xueshuang Xiang; Naijin Liu; Bin Dong
  Adversarial training (AT) aims to improve the robustness of deep learning
models by mixing clean data and adversarial examples (AEs). Most existing AT
approaches can be grouped into restricted and unrestricted approaches.
Restricted AT requires a prescribed uniform budget to constrain the magnitude
of the AE perturbations during training, with the obtained results showing high
sensitivity to the budget. On the other hand, unrestricted AT uses
unconstrained AEs, resulting in the use of AEs located beyond the decision
boundary; these overestimated AEs significantly lower the accuracy on clean
data. These limitations mean that the existing AT approaches have difficulty in
obtaining a comprehensively robust model with high accuracy and robustness when
confronting attacks with varying strengths. Considering this problem, this
paper proposes a novel AT approach named blind adversarial training (BAT) to
better balance the accuracy and robustness. The main idea of this approach is
to use a cutoff-scale strategy to adaptively estimate a nonuniform budget to
modify the AEs used in the training, ensuring that the strengths of the AEs are
dynamically located in a reasonable range and ultimately improving the overall
robustness of the AT model. The experimental results obtained using BAT for
training classification models on several benchmarks demonstrate the
competitive performance of this method.


http://arxiv.org/abs/2004.05913
Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness.
Haidong Xie; Lixin Qian; Xueshuang Xiang; Naijin Liu
  With the growth of interest in the attack and defense of deep neural
networks, researchers are focusing more on the robustness of applying them to
devices with limited memory. Thus, unlike adversarial training, which only
considers the balance between accuracy and robustness, we come to a more
meaningful and critical issue, i.e., the balance among accuracy, efficiency and
robustness (AER). Recently, some related works focused on this issue, but with
different observations, and the relations among AER remain unclear. This paper
first investigates the robustness of pruned models with different compression
ratios under the gradual pruning process and concludes that the robustness of
the pruned model drastically varies with different pruning processes,
especially in response to attacks with large strength. Second, we test the
performance of mixing the clean data and adversarial examples (generated with a
prescribed uniform budget) into the gradual pruning process, called adversarial
pruning, and find the following: the pruned model's robustness exhibits high
sensitivity to the budget. Furthermore, to better balance the AER, we propose
an approach called blind adversarial pruning (BAP), which introduces the idea
of blind adversarial training into the gradual pruning process. The main idea
is to use a cutoff-scale strategy to adaptively estimate a nonuniform budget to
modify the AEs used during pruning, thus ensuring that the strengths of AEs are
dynamically located within a reasonable range at each pruning step and
ultimately improving the overall AER of the pruned model. The experimental
results obtained using BAP for pruning classification models based on several
benchmarks demonstrate the competitive performance of this method: the
robustness of the model pruned by BAP is more stable among varying pruning
processes, and BAP exhibits better overall AER than adversarial pruning.


http://arxiv.org/abs/2004.04479
On Adversarial Examples and Stealth Attacks in Artificial Intelligence Systems.
Ivan Y. Tyukin; Desmond J. Higham; Alexander N. Gorban
  In this work we present a formal theoretical framework for assessing and
analyzing two classes of malevolent action towards generic Artificial
Intelligence (AI) systems. Our results apply to general multi-class classifiers
that map from an input space into a decision space, including artificial neural
networks used in deep learning applications. Two classes of attacks are
considered. The first class involves adversarial examples and concerns the
introduction of small perturbations of the input data that cause
misclassification. The second class, introduced here for the first time and
named stealth attacks, involves small perturbations to the AI system itself.
Here the perturbed system produces whatever output is desired by the attacker
on a specific small data set, perhaps even a single input, but performs as
normal on a validation set (which is unknown to the attacker). We show that in
both cases, i.e., in the case of an attack based on adversarial examples and in
the case of a stealth attack, the dimensionality of the AI's decision-making
space is a major contributor to the AI's susceptibility. For attacks based on
adversarial examples, a second crucial parameter is the absence of local
concentrations in the data probability distribution, a property known as
Smeared Absolute Continuity. According to our findings, robustness to
adversarial examples requires either (a) the data distributions in the AI's
feature space to have concentrated probability density functions or (b) the
dimensionality of the AI's decision variables to be sufficiently small. We also
show how to construct stealth attacks on high-dimensional AI systems that are
hard to spot unless the validation set is made exponentially large.


http://arxiv.org/abs/2004.04199
Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking.
Hongjun Wang; Guangrun Wang; Ya Li; Dongyu Zhang; Liang Lin
  The success of DNNs has driven the extensive applications of person
re-identification (ReID) into a new era. However, whether ReID inherits the
vulnerability of DNNs remains unexplored. To examine the robustness of ReID
systems is rather important because the insecurity of ReID systems may cause
severe losses, e.g., the criminals may use the adversarial perturbations to
cheat the CCTV systems. In this work, we examine the insecurity of current
best-performing ReID models by proposing a learning-to-mis-rank formulation to
perturb the ranking of the system output. As the cross-dataset transferability
is crucial in the ReID domain, we also perform a back-box attack by developing
a novel multi-stage network architecture that pyramids the features of
different levels to extract general and transferable features for the
adversarial perturbations. Our method can control the number of malicious
pixels by using differentiable multi-shot sampling. To guarantee the
inconspicuousness of the attack, we also propose a new perception loss to
achieve better visual quality. Extensive experiments on four of the largest
ReID benchmarks (i.e., Market1501 [45], CUHK03 [18], DukeMTMC [33], and MSMT17
[40]) not only show the effectiveness of our method, but also provides
directions of the future improvement in the robustness of ReID systems. For
example, the accuracy of one of the best-performing ReID systems drops sharply
from 91.8% to 1.4% after being attacked by our method. Some attack results are
shown in Fig. 1. The code is available at
https://github.com/whj363636/Adversarial-attack-on-Person-ReID-With-Deep-Mis-Ranking.


http://arxiv.org/abs/2004.03742
Towards Evaluating the Robustness of Chinese BERT Classifiers.
Boxin Wang; Boyuan Pan; Xin Li; Bo Li
  Recent advances in large-scale language representation models such as BERT
have improved the state-of-the-art performances in many NLP tasks. Meanwhile,
character-level Chinese NLP models, including BERT for Chinese, have also
demonstrated that they can outperform the existing models. In this paper, we
show that, however, such BERT-based models are vulnerable under character-level
adversarial attacks. We propose a novel Chinese char-level attack method
against BERT-based classifiers. Essentially, we generate "small" perturbation
on the character level in the embedding space and guide the character
substitution procedure. Extensive experiments show that the classification
accuracy on a Chinese news dataset drops from 91.8% to 0% by manipulating less
than 2 characters on average based on the proposed attack. Human evaluations
also confirm that our generated Chinese adversarial examples barely affect
human performance on these NLP tasks.


http://arxiv.org/abs/2004.03295
Feature Partitioning for Robust Tree Ensembles and their Certification in Adversarial Scenarios.
Stefano Calzavara; Claudio Lucchese; Federico Marcuzzi; Salvatore Orlando
  Machine learning algorithms, however effective, are known to be vulnerable in
adversarial scenarios where a malicious user may inject manipulated instances.
In this work we focus on evasion attacks, where a model is trained in a safe
environment and exposed to attacks at test time. The attacker aims at finding a
minimal perturbation of a test instance that changes the model outcome.
  We propose a model-agnostic strategy that builds a robust ensemble by
training its basic models on feature-based partitions of the given dataset. Our
algorithm guarantees that the majority of the models in the ensemble cannot be
affected by the attacker. We experimented the proposed strategy on decision
tree ensembles, and we also propose an approximate certification method for
tree ensembles that efficiently assess the minimal accuracy of a forest on a
given dataset avoiding the costly computation of evasion attacks.
  Experimental evaluation on publicly available datasets shows that proposed
strategy outperforms state-of-the-art adversarial learning algorithms against
evasion attacks.


http://arxiv.org/abs/2004.03434
Learning to fool the speaker recognition.
Jiguo Li; Xinfeng Zhang; Jizheng Xu; Li Zhang; Yue Wang; Siwei Ma; Wen Gao
  Due to the widespread deployment of fingerprint/face/speaker recognition
systems, attacking deep learning based biometric systems has drawn more and
more attention. Previous research mainly studied the attack to the vision-based
system, such as fingerprint and face recognition. While the attack for speaker
recognition has not been investigated yet, although it has been widely used in
our daily life. In this paper, we attempt to fool the state-of-the-art speaker
recognition model and present \textit{speaker recognition attacker}, a
lightweight model to fool the deep speaker recognition model by adding
imperceptible perturbations onto the raw speech waveform. We find that the
speaker recognition system is also vulnerable to the attack, and we achieve a
high success rate on the non-targeted attack. Besides, we also present an
effective method to optimize the speaker recognition attacker to obtain a
trade-off between the attack success rate with the perceptual quality.
Experiments on the TIMIT dataset show that we can achieve a sentence error rate
of $99.2\%$ with an average SNR $57.2\text{dB}$ and PESQ 4.2 with speed rather
faster than real-time.


http://arxiv.org/abs/2004.03428
Universal Adversarial Perturbations Generative Network for Speaker Recognition.
Jiguo Li; Xinfeng Zhang; Chuanmin Jia; Jizheng Xu; Li Zhang; Yue Wang; Siwei Ma; Wen Gao
  Attacking deep learning based biometric systems has drawn more and more
attention with the wide deployment of fingerprint/face/speaker recognition
systems, given the fact that the neural networks are vulnerable to the
adversarial examples, which have been intentionally perturbed to remain almost
imperceptible for human. In this paper, we demonstrated the existence of the
universal adversarial perturbations~(UAPs) for the speaker recognition systems.
We proposed a generative network to learn the mapping from the low-dimensional
normal distribution to the UAPs subspace, then synthesize the UAPs to perturbe
any input signals to spoof the well-trained speaker recognition model with high
probability. Experimental results on TIMIT and LibriSpeech datasets demonstrate
the effectiveness of our model.


http://arxiv.org/abs/2004.02183
Approximate Manifold Defense Against Multiple Adversarial Perturbations.
Jay Nandy; Wynne Hsu; Mong Li Lee
  Existing defenses against adversarial attacks are typically tailored to a
specific perturbation type. Using adversarial training to defend against
multiple types of perturbation requires expensive adversarial examples from
different perturbation types at each training step. In contrast, manifold-based
defense incorporates a generative network to project an input sample onto the
clean data manifold. This approach eliminates the need to generate expensive
adversarial examples while achieving robustness against multiple perturbation
types. However, the success of this approach relies on whether the generative
network can capture the complete clean data manifold, which remains an open
problem for complex input domain. In this work, we devise an approximate
manifold defense mechanism, called RBF-CNN, for image classification. Instead
of capturing the complete data manifold, we use an RBF layer to learn the
density of small image patches. RBF-CNN also utilizes a reconstruction layer
that mitigates any minor adversarial perturbations. Further, incorporating our
proposed reconstruction process for training improves the adversarial
robustness of our RBF-CNN models. Experiment results on MNIST and CIFAR-10
datasets indicate that RBF-CNN offers robustness for multiple perturbations
without the need for expensive adversarial training.


http://arxiv.org/abs/2004.01903
Understanding (Non-)Robust Feature Disentanglement and the Relationship Between Low- and High-Dimensional Adversarial Attacks.
Zuowen Wang; Leo Horne
  Recent work has put forth the hypothesis that adversarial vulnerabilities in
neural networks are due to them overusing "non-robust features" inherent in the
training data. We show empirically that for PGD-attacks, there is a training
stage where neural networks start heavily relying on non-robust features to
boost natural accuracy. We also propose a mechanism reducing vulnerability to
PGD-style attacks consisting of mixing in a certain amount of images
contain-ing mostly "robust features" into each training batch, and then show
that robust accuracy is improved, while natural accuracy is not substantially
hurt. We show that training on "robust features" provides boosts in robust
accuracy across various architectures and for different attacks. Finally, we
demonstrate empirically that these "robust features" do not induce spatial
invariance.


http://arxiv.org/abs/2004.01970
BAE: BERT-based Adversarial Examples for Text Classification.
Siddhant Garg; Goutham Ramakrishnan
  Modern text classification models are susceptible to adversarial examples,
perturbed versions of the original text indiscernible by humans but which get
misclassified by the model. We present BAE, a powerful black box attack for
generating grammatically correct and semantically coherent adversarial
examples. BAE replaces and inserts tokens in the original text by masking a
portion of the text and leveraging a language model to generate alternatives
for the masked tokens. Compared to prior work, we show that BAE performs a
stronger attack on three widely used models for seven text classification
datasets.


http://arxiv.org/abs/2004.01832
Adversarial Robustness through Regularization: A Second-Order Approach.
Avery Ma; Fartash Faghri; Amir-massoud Farahmand
  Adversarial training is a common approach to improving the robustness of deep
neural networks against adversarial examples. In this work, we propose a novel
regularization approach as an alternative. To derive the regularizer, we
formulate the adversarial robustness problem under the robust optimization
framework and approximate the loss function using a second-order Taylor series
expansion. Our proposed second-order adversarial regularizer (SOAR) is an upper
bound based on the Taylor approximation of the inner-max in the robust
optimization objective. We empirically show that the proposed method improves
the robustness of networks on the CIFAR-10 dataset.


http://arxiv.org/abs/2004.00622
Evading Deepfake-Image Detectors with White- and Black-Box Attacks.
Nicholas Carlini; Hany Farid
  It is now possible to synthesize highly realistic images of people who don't
exist. Such content has, for example, been implicated in the creation of
fraudulent social-media profiles responsible for dis-information campaigns.
Significant efforts are, therefore, being deployed to detect
synthetically-generated content. One popular forensic approach trains a neural
network to distinguish real from synthetic content.
  We show that such forensic classifiers are vulnerable to a range of attacks
that reduce the classifier to near-0% accuracy. We develop five attack case
studies on a state-of-the-art classifier that achieves an area under the ROC
curve (AUC) of 0.95 on almost all existing image generators, when only trained
on one generator. With full access to the classifier, we can flip the lowest
bit of each pixel in an image to reduce the classifier's AUC to 0.0005; perturb
1% of the image area to reduce the classifier's AUC to 0.08; or add a single
noise pattern in the synthesizer's latent space to reduce the classifier's AUC
to 0.17. We also develop a black-box attack that, with no access to the target
classifier, reduces the AUC to 0.22. These attacks reveal significant
vulnerabilities of certain image-forensic classifiers.


http://arxiv.org/abs/2004.00306
Towards Achieving Adversarial Robustness by Enforcing Feature Consistency Across Bit Planes.
Sravanti Addepalli; Vivek B. S.; Arya Baburaj; Gaurang Sriramanan; R. Venkatesh Babu
  As humans, we inherently perceive images based on their predominant features,
and ignore noise embedded within lower bit planes. On the contrary, Deep Neural
Networks are known to confidently misclassify images corrupted with
meticulously crafted perturbations that are nearly imperceptible to the human
eye. In this work, we attempt to address this problem by training networks to
form coarse impressions based on the information in higher bit planes, and use
the lower bit planes only to refine their prediction. We demonstrate that, by
imposing consistency on the representations learned across differently
quantized images, the adversarial robustness of networks improves significantly
when compared to a normally trained model. Present state-of-the-art defenses
against adversarial attacks require the networks to be explicitly trained using
adversarial samples that are computationally expensive to generate. While such
methods that use adversarial training continue to achieve the best results,
this work paves the way towards achieving robustness without having to
explicitly train on adversarial samples. The proposed approach is therefore
faster, and also closer to the natural learning process in humans.


http://arxiv.org/abs/2004.00543
Physically Realizable Adversarial Examples for LiDAR Object Detection.
James Tu; Mengye Ren; Siva Manivasagam; Ming Liang; Bin Yang; Richard Du; Frank Cheng; Raquel Urtasun
  Modern autonomous driving systems rely heavily on deep learning models to
process point cloud sensory data; meanwhile, deep models have been shown to be
susceptible to adversarial attacks with visually imperceptible perturbations.
Despite the fact that this poses a security concern for the self-driving
industry, there has been very little exploration in terms of 3D perception, as
most adversarial attacks have only been applied to 2D flat images. In this
paper, we address this issue and present a method to generate universal 3D
adversarial objects to fool LiDAR detectors. In particular, we demonstrate that
placing an adversarial object on the rooftop of any target vehicle to hide the
vehicle entirely from LiDAR detectors with a success rate of 80%. We report
attack results on a suite of detectors using various input representation of
point clouds. We also conduct a pilot study on adversarial defense using data
augmentation. This is one step closer towards safer self-driving under unseen
conditions from limited training data.


http://arxiv.org/abs/2003.13969
A Thorough Comparison Study on Adversarial Attacks and Defenses for Common Thorax Disease Classification in Chest X-rays.
Chendi Rao; Jiezhang Cao; Runhao Zeng; Qi Chen; Huazhu Fu; Yanwu Xu; Mingkui Tan
  Recently, deep neural networks (DNNs) have made great progress on automated
diagnosis with chest X-rays images. However, DNNs are vulnerable to adversarial
examples, which may cause misdiagnoses to patients when applying the DNN based
methods in disease detection. Recently, there is few comprehensive studies
exploring the influence of attack and defense methods on disease detection,
especially for the multi-label classification problem. In this paper, we aim to
review various adversarial attack and defense methods on chest X-rays. First,
the motivations and the mathematical representations of attack and defense
methods are introduced in details. Second, we evaluate the influence of several
state-of-the-art attack and defense methods for common thorax disease
classification in chest X-rays. We found that the attack and defense methods
have poor performance with excessive iterations and large perturbations. To
address this, we propose a new defense method that is robust to different
degrees of perturbations. This study could provide new insights into
methodological development for the community.


http://arxiv.org/abs/2003.13917
Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement.
Chao-Han Huck Yang; Jun Qi; Pin-Yu Chen; Xiaoli Ma; Chin-Hui Lee
  Recent studies have highlighted adversarial examples as ubiquitous threats to
the deep neural network (DNN) based speech recognition systems. In this work,
we present a U-Net based attention model, U-Net$_{At}$, to enhance adversarial
speech signals. Specifically, we evaluate the model performance by
interpretable speech recognition metrics and discuss the model performance by
the augmented adversarial training. Our experiments show that our proposed
U-Net$_{At}$ improves the perceptual evaluation of speech quality (PESQ) from
1.13 to 2.78, speech transmission index (STI) from 0.65 to 0.75, short-term
objective intelligibility (STOI) from 0.83 to 0.96 on the task of speech
enhancement with adversarial speech examples. We conduct experiments on the
automatic speech recognition (ASR) task with adversarial audio attacks. We find
that (i) temporal features learned by the attention network are capable of
enhancing the robustness of DNN based ASR models; (ii) the generalization power
of DNN based ASR model could be enhanced by applying adversarial training with
an additive adversarial data augmentation. The ASR metric on word-error-rates
(WERs) shows that there is an absolute 2.22 $\%$ decrease under gradient-based
perturbation, and an absolute 2.03 $\%$ decrease, under evolutionary-optimized
perturbation, which suggests that our enhancement models with adversarial
training can further secure a resilient ASR system.


http://arxiv.org/abs/2004.00410
Adversarial Attacks on Multivariate Time Series.
Samuel Harford; Fazle Karim; Houshang Darabi
  Classification models for the multivariate time series have gained
significant importance in the research community, but not much research has
been done on generating adversarial samples for these models. Such samples of
adversaries could become a security concern. In this paper, we propose
transforming the existing adversarial transformation network (ATN) on a
distilled model to attack various multivariate time series classification
models. The proposed attack on the classification model utilizes a distilled
model as a surrogate that mimics the behavior of the attacked classical
multivariate time series classification models. The proposed methodology is
tested onto 1-Nearest Neighbor Dynamic Time Warping (1-NN DTW) and a Fully
Convolutional Network (FCN), all of which are trained on 18 University of East
Anglia (UEA) and University of California Riverside (UCR) datasets. We show
both models were susceptible to attacks on all 18 datasets. To the best of our
knowledge, adversarial attacks have only been conducted in the domain of
univariate time series and have not been conducted on multivariate time series.
such an attack on time series classification models has never been done before.
Additionally, we recommend future researchers that develop time series
classification models to incorporating adversarial data samples into their
training data sets to improve resilience on adversarial samples and to consider
model robustness as an evaluative metric.


http://arxiv.org/abs/2003.13511
Improved Gradient based Adversarial Attacks for Quantized Networks.
Kartik Gupta; Thalaiyasingam Ajanthan
  Neural network quantization has become increasingly popular due to efficient
memory consumption and faster computation resulting from bitwise operations on
the quantized networks. Even though they exhibit excellent generalization
capabilities, their robustness properties are not well-understood. In this
work, we systematically study the robustness of quantized networks against
gradient based adversarial attacks and demonstrate that these quantized models
suffer from gradient vanishing issues and show a fake sense of security. By
attributing gradient vanishing to poor forward-backward signal propagation in
the trained network, we introduce a simple temperature scaling approach to
mitigate this issue while preserving the decision boundary. Despite being a
simple modification to existing gradient based adversarial attacks, experiments
on CIFAR-10/100 datasets with VGG-16 and ResNet-18 networks demonstrate that
our temperature scaled attacks obtain near-perfect success rate on quantized
networks while outperforming original attacks on adversarially trained models
as well as floating-point networks.


http://arxiv.org/abs/2003.13370
Towards Deep Learning Models Resistant to Large Perturbations.
Amirreza Shaeiri; Rozhin Nobahari; Mohammad Hossein Rohban
  Adversarial robustness has proven to be a required property of machine
learning algorithms. A key and often overlooked aspect of this problem is to
try to make the adversarial noise magnitude as large as possible to enhance the
benefits of the model robustness. We show that the well-established algorithm
called "adversarial training" fails to train a deep neural network given a
large, but reasonable, perturbation magnitude. In this paper, we propose a
simple yet effective initialization of the network weights that makes learning
on higher levels of noise possible. We next evaluate this idea rigorously on
MNIST ($\epsilon$ up to $\approx 0.40$) and CIFAR10 ($\epsilon$ up to $\approx
32/255$) datasets assuming the $\ell_{\infty}$ attack model. Additionally, in
order to establish the limits of $\epsilon$ in which the learning is feasible,
we study the optimal robust classifier assuming full access to the joint data
and label distribution. Then, we provide some theoretical results on the
adversarial accuracy for a simple multi-dimensional Bernoulli distribution,
which yields some insights on the range of feasible perturbations for the MNIST
dataset.


http://arxiv.org/abs/2003.13526
Efficient Black-box Optimization of Adversarial Windows Malware with Constrained Manipulations.
Luca Demetrio; Battista Biggio; Giovanni Lagorio; Fabio Roli; Alessandro Armando
  Windows malware detectors based on machine learning are vulnerable to
adversarial examples, even if the attacker is only given black-box access to
the model. The main drawback of these attacks is that they require executing
the adversarial malware sample in a sandbox at each iteration of its
optimization process, to ensure that its intrusive functionality is preserved.
In this paper, we present a novel black-box attack that leverages a set of
semantics-preserving, constrained malware manipulations to overcome this
computationally-demanding validation step. Our attack is formalized as a
constrained minimization problem which also enables optimizing the trade-off
between the probability of evading detection and the size of the injected
adversarial payload. We investigate this trade-off empirically, on two popular
static Windows malware detectors, and show that our black-box attack is able to
bypass them with only few iterations and changes. We also evaluate whether our
attack transfers to other commercial antivirus solutions, and surprisingly find
that it can increase the probability of evading some of them. We conclude by
discussing the limitations of our approach, and its possible future extensions
to target malware classifiers based on dynamic analysis.


http://arxiv.org/abs/2003.12862
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning.
Tianlong Chen; Sijia Liu; Shiyu Chang; Yu Cheng; Lisa Amini; Zhangyang Wang
  Pretrained models from self-supervision are prevalently used in fine-tuning
downstream tasks faster or for better accuracy. However, gaining robustness
from pretraining is left unexplored. We introduce adversarial training into
self-supervision, to provide general-purpose robust pre-trained models for the
first time. We find these robust pre-trained models can benefit the subsequent
fine-tuning in two ways: i) boosting final model robustness; ii) saving the
computation cost, if proceeding towards adversarial fine-tuning. We conduct
extensive experiments to demonstrate that the proposed framework achieves large
performance margins (eg, 3.83% on robust accuracy and 1.3% on standard
accuracy, on the CIFAR-10 dataset), compared with the conventional end-to-end
adversarial training baseline. Moreover, we find that different self-supervised
pre-trained models have a diverse adversarial vulnerability. It inspires us to
ensemble several pretraining tasks, which boosts robustness more. Our ensemble
strategy contributes to a further improvement of 3.59% on robust accuracy,
while maintaining a slightly higher standard accuracy on CIFAR-10. Our codes
are available at https://github.com/TAMU-VITA/Adv-SS-Pretraining.


http://arxiv.org/abs/2003.12703
DaST: Data-free Substitute Training for Adversarial Attacks.
Mingyi Zhou; Jing Wu; Yipeng Liu; Shuaicheng Liu; Ce Zhu
  Machine learning models are vulnerable to adversarial examples. For the
black-box setting, current substitute attacks need pre-trained models to
generate adversarial examples. However, pre-trained models are hard to obtain
in real-world tasks. In this paper, we propose a data-free substitute training
method (DaST) to obtain substitute models for adversarial black-box attacks
without the requirement of any real data. To achieve this, DaST utilizes
specially designed generative adversarial networks (GANs) to train the
substitute models. In particular, we design a multi-branch architecture and
label-control loss for the generative model to deal with the uneven
distribution of synthetic samples. The substitute model is then trained by the
synthetic samples generated by the generative model, which are labeled by the
attacked model subsequently. The experiments demonstrate the substitute models
produced by DaST can achieve competitive performance compared with the baseline
models which are trained by the same train set with attacked models.
Additionally, to evaluate the practicability of the proposed method on the
real-world task, we attack an online machine learning model on the Microsoft
Azure platform. The remote model misclassifies 98.35% of the adversarial
examples crafted by our method. To the best of our knowledge, we are the first
to train a substitute model for adversarial attacks without any real data.


http://arxiv.org/abs/2003.12760
Adversarial Imitation Attack.
Mingyi Zhou; Jing Wu; Yipeng Liu; Shuaicheng Liu; Xiang Zhang; Ce Zhu
  Deep learning models are known to be vulnerable to adversarial examples. A
practical adversarial attack should require as little as possible knowledge of
attacked models. Current substitute attacks need pre-trained models to generate
adversarial examples and their attack success rates heavily rely on the
transferability of adversarial examples. Current score-based and decision-based
attacks require lots of queries for the attacked models. In this study, we
propose a novel adversarial imitation attack. First, it produces a replica of
the attacked model by a two-player game like the generative adversarial
networks (GANs). The objective of the generative model is to generate examples
that lead the imitation model returning different outputs with the attacked
model. The objective of the imitation model is to output the same labels with
the attacked model under the same inputs. Then, the adversarial examples
generated by the imitation model are utilized to fool the attacked model.
Compared with the current substitute attacks, imitation attacks can use less
training data to produce a replica of the attacked model and improve the
transferability of adversarial examples. Experiments demonstrate that our
imitation attack requires less training data than the black-box substitute
attacks, but achieves an attack success rate close to the white-box attack on
unseen data with no query.


http://arxiv.org/abs/2003.11816
Do Deep Minds Think Alike? Selective Adversarial Attacks for Fine-Grained Manipulation of Multiple Deep Neural Networks.
Zain Khan; Jirong Yi; Raghu Mudumbai; Xiaodong Wu; Weiyu Xu
  Recent works have demonstrated the existence of {\it adversarial examples}
targeting a single machine learning system. In this paper we ask a simple but
fundamental question of "selective fooling": given {\it multiple} machine
learning systems assigned to solve the same classification problem and taking
the same input signal, is it possible to construct a perturbation to the input
signal that manipulates the outputs of these {\it multiple} machine learning
systems {\it simultaneously} in arbitrary pre-defined ways? For example, is it
possible to selectively fool a set of "enemy" machine learning systems but does
not fool the other "friend" machine learning systems? The answer to this
question depends on the extent to which these different machine learning
systems "think alike". We formulate the problem of "selective fooling" as a
novel optimization problem, and report on a series of experiments on the MNIST
dataset. Our preliminary findings from these experiments show that it is in
fact very easy to selectively manipulate multiple MNIST classifiers
simultaneously, even when the classifiers are identical in their architectures,
training algorithms and training datasets except for random initialization
during training. This suggests that two nominally equivalent machine learning
systems do not in fact "think alike" at all, and opens the possibility for many
novel applications and deeper understandings of the working principles of deep
neural networks.


http://arxiv.org/abs/2003.11855
On the adversarial robustness of DNNs based on error correcting output codes.
Bowen Zhang; Benedetta Tondi; Mauro Barni
  Adversarial examples represent a great security threat for deep learning
systems, pushing researchers to develop suitable defense mechanisms. The use of
networks adopting error-correcting output codes (ECOC) has recently been
proposed to deal with white-box attacks. In this paper, we carry out an
in-depth investigation of the security achieved by the ECOC approach. In
contrast to previous findings, our analysis reveals that, when the attack in
the white-box framework is carried out properly, the ECOC scheme can be
attacked by introducing a rather small perturbation. We do so by considering
both the popular adversarial attack proposed by Carlini and Wagner (C&W) and a
new variant of C&W attack specifically designed for multi-label classification
architectures, like the ECOC-based structure. Experimental results regarding
different classification tasks demonstrate that ECOC networks can be
successfully attacked by both the original C&W attack and the new attack.


http://arxiv.org/abs/2003.11323
Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples.
Alejandro Barredo-Arrieta; Ser Javier Del
  The last decade has witnessed the proliferation of Deep Learning models in
many applications, achieving unrivaled levels of predictive performance.
Unfortunately, the black-box nature of Deep Learning models has posed
unanswered questions about what they learn from data. Certain application
scenarios have highlighted the importance of assessing the bounds under which
Deep Learning models operate, a problem addressed by using assorted approaches
aimed at audiences from different domains. However, as the focus of the
application is placed more on non-expert users, it results mandatory to provide
the means for him/her to trust the model, just like a human gets familiar with
a system or process: by understanding the hypothetical circumstances under
which it fails. This is indeed the angular stone for this research work: to
undertake an adversarial analysis of a Deep Learning model. The proposed
framework constructs counterfactual examples by ensuring their plausibility,
e.g. there is a reasonable probability that a human could generate them without
resorting to a computer program. Therefore, this work must be regarded as
valuable auditing exercise of the usable bounds a certain model is constrained
within, thereby allowing for a much greater understanding of the capabilities
and pitfalls of a model used in a real application. To this end, a Generative
Adversarial Network (GAN) and multi-objective heuristics are used to furnish a
plausible attack to the audited model, efficiently trading between the
confusion of this model, the intensity and plausibility of the generated
counterfactual. Its utility is showcased within a human face classification
task, unveiling the enormous potential of the proposed framework.


http://arxiv.org/abs/2003.11145
Adversarial Light Projection Attacks on Face Recognition Systems: A Feasibility Study.
Luan Nguyen; Sunpreet S. Arora; Yuhang Wu; Hao Yang
  Deep learning-based systems have been shown to be vulnerable to adversarial
attacks in both digital and physical domains. While feasible, digital attacks
have limited applicability in attacking deployed systems, including face
recognition systems, where an adversary typically has access to the input and
not the transmission channel. In such setting, physical attacks that directly
provide a malicious input through the input channel pose a bigger threat. We
investigate the feasibility of conducting real-time physical attacks on face
recognition systems using adversarial light projections. A setup comprising a
commercially available web camera and a projector is used to conduct the
attack. The adversary uses a transformation-invariant adversarial pattern
generation method to generate a digital adversarial pattern using one or more
images of the target available to the adversary. The digital adversarial
pattern is then projected onto the adversary's face in the physical domain to
either impersonate a target (impersonation) or evade recognition (obfuscation).
We conduct preliminary experiments using two open-source and one commercial
face recognition system on a pool of 50 subjects. Our experimental results
demonstrate the vulnerability of face recognition systems to light projection
attacks in both white-box and black-box attack settings.


http://arxiv.org/abs/2003.10602
Defense Through Diverse Directions.
Christopher M. Bender; Yang Li; Yifeng Shi; Michael K. Reiter; Junier B. Oliva
  In this work we develop a novel Bayesian neural network methodology to
achieve strong adversarial robustness without the need for online adversarial
training. Unlike previous efforts in this direction, we do not rely solely on
the stochasticity of network weights by minimizing the divergence between the
learned parameter distribution and a prior. Instead, we additionally require
that the model maintain some expected uncertainty with respect to all input
covariates. We demonstrate that by encouraging the network to distribute evenly
across inputs, the network becomes less susceptible to localized, brittle
features which imparts a natural robustness to targeted perturbations. We show
empirical robustness on several benchmark datasets.


http://arxiv.org/abs/2003.10399
Inherent Adversarial Robustness of Deep Spiking Neural Networks: Effects of Discrete Input Encoding and Non-Linear Activations.
Saima Sharmin; Nitin Rathi; Priyadarshini Panda; Kaushik Roy
  In the recent quest for trustworthy neural networks, we present Spiking
Neural Network (SNN) as a potential candidate for inherent robustness against
adversarial attacks. In this work, we demonstrate that accuracy degradation is
less severe in SNNs than in their non-spiking counterparts for CIFAR10 and
CIFAR100 datasets on deep VGG architectures. We attribute this robustness to
two fundamental characteristics of SNNs and analyze their effects. First, we
exhibit that input discretization introduced by the Poisson encoder improves
adversarial robustness with reduced number of timesteps. Second, we quantify
the amount of adversarial accuracy with increased leak rate in
Leaky-Integrate-Fire (LIF) neurons. Our results suggest that SNNs trained with
LIF neurons and smaller number of timesteps are more robust than the ones with
IF (Integrate-Fire) neurons and larger number of timesteps. We overcome the
bottleneck of creating gradient-based adversarial inputs in temporal domain by
proposing a technique for crafting attacks from SNN.


http://arxiv.org/abs/2003.10315
Adversarial Attacks on Monocular Depth Estimation.
Ziqi Zhang; Xinge Zhu; Yingwei Li; Xiangqun Chen; Yao Guo
  Recent advances of deep learning have brought exceptional performance on many
computer vision tasks such as semantic segmentation and depth estimation.
However, the vulnerability of deep neural networks towards adversarial examples
have caused grave concerns for real-world deployment. In this paper, we present
to the best of our knowledge the first systematic study of adversarial attacks
on monocular depth estimation, an important task of 3D scene understanding in
scenarios such as autonomous driving and robot navigation. In order to
understand the impact of adversarial attacks on depth estimation, we first
define a taxonomy of different attack scenarios for depth estimation, including
non-targeted attacks, targeted attacks and universal attacks. We then adapt
several state-of-the-art attack methods for classification on the field of
depth estimation. Besides, multi-task attacks are introduced to further improve
the attack performance for universal attacks. Experimental results show that it
is possible to generate significant errors on depth estimation. In particular,
we demonstrate that our methods can conduct targeted attacks on given objects
(such as a car), resulting in depth estimation 3-4x away from the ground truth
(e.g., from 20m to 80m).


http://arxiv.org/abs/2003.10596
Adversarial Perturbations Fool Deepfake Detectors.
Apurva Gandhi; Shomik Jain
  This work uses adversarial perturbations to enhance deepfake images and fool
common deepfake detectors. We created adversarial perturbations using the Fast
Gradient Sign Method and the Carlini and Wagner L2 norm attack in both blackbox
and whitebox settings. Detectors achieved over 95% accuracy on unperturbed
deepfakes, but less than 27% accuracy on perturbed deepfakes. We also explore
two improvements to deepfake detectors: (i) Lipschitz regularization, and (ii)
Deep Image Prior (DIP). Lipschitz regularization constrains the gradient of the
detector with respect to the input in order to increase robustness to input
perturbations. The DIP defense removes perturbations using generative
convolutional neural networks in an unsupervised manner. Regularization
improved the detection of perturbed deepfakes on average, including a 10%
accuracy boost in the blackbox case. The DIP defense achieved 95% accuracy on
perturbed deepfakes that fooled the original detector, while retaining 98%
accuracy in other cases on a 100 image subsample.


http://arxiv.org/abs/2003.10041
Understanding the robustness of deep neural network classifiers for breast cancer screening.
Witold Oleszkiewicz; Taro Makino; Stanisław Jastrzębski; Tomasz Trzciński; Linda Moy; Kyunghyun Cho; Laura Heacock; Krzysztof J. Geras
  Deep neural networks (DNNs) show promise in breast cancer screening, but
their robustness to input perturbations must be better understood before they
can be clinically implemented. There exists extensive literature on this
subject in the context of natural images that can potentially be built upon.
However, it cannot be assumed that conclusions about robustness will transfer
from natural images to mammogram images, due to significant differences between
the two image modalities. In order to determine whether conclusions will
transfer, we measure the sensitivity of a radiologist-level screening mammogram
image classifier to four commonly studied input perturbations that natural
image classifiers are sensitive to. We find that mammogram image classifiers
are also sensitive to these perturbations, which suggests that we can build on
the existing literature. We also perform a detailed analysis on the effects of
low-pass filtering, and find that it degrades the visibility of clinically
meaningful features called microcalcifications. Since low-pass filtering
removes semantically meaningful information that is predictive of breast
cancer, we argue that it is undesirable for mammogram image classifiers to be
invariant to it. This is in contrast to natural images, where we do not want
DNNs to be sensitive to low-pass filtering due to its tendency to remove
information that is human-incomprehensible.


http://arxiv.org/abs/2003.10045
Architectural Resilience to Foreground-and-Background Adversarial Noise.
Carl Cheng; Evan Hu
  Adversarial attacks in the form of imperceptible perturbations of normal
images have been extensively studied, and for every new defense methodology
created, multiple adversarial attacks are found to counteract it. In
particular, a popular style of attack, exemplified in recent years by DeepFool
and Carlini-Wagner, relies solely on white-box scenarios in which full access
to the predictive model and its weights are required. In this work, we instead
propose distinct model-agnostic benchmark perturbations of images in order to
investigate the resilience and robustness of different network architectures.
Results empirically determine that increasing depth within most types of
Convolutional Neural Networks typically improves model resilience towards
general attacks, with improvement steadily decreasing as the model becomes
deeper. Additionally, we find that a notable difference in adversarial
robustness exists between residual architectures with skip connections and
non-residual architectures of similar complexity. Our findings provide
direction for future understanding of residual connections and depth on network
robustness.


http://arxiv.org/abs/2003.10804
Detecting Adversarial Examples in Learning-Enabled Cyber-Physical Systems using Variational Autoencoder for Regression.
Feiyang Cai; Jiani Li; Xenofon Koutsoukos
  Learning-enabled components (LECs) are widely used in cyber-physical systems
(CPS) since they can handle the uncertainty and variability of the environment
and increase the level of autonomy. However, it has been shown that LECs such
as deep neural networks (DNN) are not robust and adversarial examples can cause
the model to make a false prediction. The paper considers the problem of
efficiently detecting adversarial examples in LECs used for regression in CPS.
The proposed approach is based on inductive conformal prediction and uses a
regression model based on variational autoencoder. The architecture allows to
take into consideration both the input and the neural network prediction for
detecting adversarial, and more generally, out-of-distribution examples. We
demonstrate the method using an advanced emergency braking system implemented
in an open source simulator for self-driving cars where a DNN is used to
estimate the distance to an obstacle. The simulation results show that the
method can effectively detect adversarial examples with a short detection
delay.


http://arxiv.org/abs/2003.09711
Robust Out-of-distribution Detection in Neural Networks.
Jiefeng Chen; Yixuan Li; Xi Wu; Yingyu Liang; Somesh Jha
  Detecting anomalous inputs is critical for safely deploying deep learning
models in the real world. Existing approaches for detecting out-of-distribution
(OOD) examples work well when evaluated on natural samples drawn from a
sufficiently different distribution than the training data distribution.
However, in this paper, we show that existing detection mechanisms can be
extremely brittle when evaluating on inputs with minimal adversarial
perturbations which don't change their semantics. Formally, we introduce a
novel and challenging problem, Robust Out-of-Distribution Detection, and
propose an algorithm that can fool existing OOD detectors by adding small
perturbations to the inputs while preserving their semantics and thus the
distributional membership. We take a first step to solve this challenge, and
propose an effective algorithm called ALOE, which performs robust training by
exposing the model to both adversarially crafted inlier and outlier examples.
Our method can be flexibly combined with, and render existing methods robust.
On common benchmark datasets, we show that ALOE substantially improves the
robustness of state-of-the-art OOD detection, with 58.4% AUROC improvement on
CIFAR-10 and 46.59% improvement on CIFAR-100. Finally, we provide theoretical
analysis for our method, underpinning the empirical results above.


http://arxiv.org/abs/2003.09595
Cooling-Shrinking Attack: Blinding the Tracker with Imperceptible Noises.
Bin Yan; Dong Wang; Huchuan Lu; Xiaoyun Yang
  Adversarial attack of CNN aims at deceiving models to misbehave by adding
imperceptible perturbations to images. This feature facilitates to understand
neural networks deeply and to improve the robustness of deep learning models.
Although several works have focused on attacking image classifiers and object
detectors, an effective and efficient method for attacking single object
trackers of any target in a model-free way remains lacking. In this paper, a
cooling-shrinking attack method is proposed to deceive state-of-the-art
SiameseRPN-based trackers. An effective and efficient perturbation generator is
trained with a carefully designed adversarial loss, which can simultaneously
cool hot regions where the target exists on the heatmaps and force the
predicted bounding box to shrink, making the tracked target invisible to
trackers. Numerous experiments on OTB100, VOT2018, and LaSOT datasets show that
our method can effectively fool the state-of-the-art SiameseRPN++ tracker by
adding small perturbations to the template or the search regions. Besides, our
method has good transferability and is able to deceive other top-performance
trackers such as DaSiamRPN, DaSiamRPN-UpdateNet, and DiMP. The source codes are
available at https://github.com/MasterBin-IIAU/CSA.


http://arxiv.org/abs/2003.11917
Adversarial Examples and the Deeper Riddle of Induction: The Need for a Theory of Artifacts in Deep Learning.
Cameron Buckner
  Deep learning is currently the most widespread and successful technology in
artificial intelligence. It promises to push the frontier of scientific
discovery beyond current limits. However, skeptics have worried that deep
neural networks are black boxes, and have called into question whether these
advances can really be deemed scientific progress if humans cannot understand
them. Relatedly, these systems also possess bewildering new vulnerabilities:
most notably a susceptibility to "adversarial examples". In this paper, I argue
that adversarial examples will become a flashpoint of debate in philosophy and
diverse sciences. Specifically, new findings concerning adversarial examples
have challenged the consensus view that the networks' verdicts on these cases
are caused by overfitting idiosyncratic noise in the training set, and may
instead be the result of detecting predictively useful "intrinsic features of
the data geometry" that humans cannot perceive (Ilyas et al., 2019). These
results should cause us to re-examine responses to one of the deepest puzzles
at the intersection of philosophy and science: Nelson Goodman's "new riddle" of
induction. Specifically, they raise the possibility that progress in a number
of sciences will depend upon the detection and manipulation of useful features
that humans find inscrutable. Before we can evaluate this possibility, however,
we must decide which (if any) of these inscrutable features are real but
available only to "alien" perception and cognition, and which are distinctive
artifacts of deep learning-for artifacts like lens flares or Gibbs phenomena
can be similarly useful for prediction, but are usually seen as obstacles to
scientific theorizing. Thus, machine learning researchers urgently need to
develop a theory of artifacts for deep neural networks, and I conclude by
sketching some initial directions for this area of research.


http://arxiv.org/abs/2004.02756
Investigating Image Applications Based on Spatial-Frequency Transform and Deep Learning Techniques.
Qinkai Zheng; Han Qiu; Gerard Memmi; Isabelle Bloch
  This is the report for the PRIM project in Telecom Paris. This report is
about applications based on spatial-frequency transform and deep learning
techniques. In this report, there are two main works. The first work is about
the enhanced JPEG compression method based on deep learning. we propose a novel
method to highly enhance the JPEG compression by transmitting fewer image data
at the sender's end. At the receiver's end, we propose a DC recovery algorithm
together with the deep residual learning framework to recover images with high
quality. The second work is about adversarial examples defenses based on signal
processing. We propose the wavelet extension method to extend image data
features, which makes it more difficult to generate adversarial examples. We
further adopt wavelet denoising to reduce the influence of the adversarial
perturbations. With intensive experiments, we demonstrate that both works are
effective in their application scenarios.


http://arxiv.org/abs/2003.09416
Quantum noise protects quantum classifiers against adversaries.
Yuxuan Du; Min-Hsiu Hsieh; Tongliang Liu; Dacheng Tao; Nana Liu
  Noise in quantum information processing is often viewed as a disruptive and
difficult-to-avoid feature, especially in near-term quantum technologies.
However, noise has often played beneficial roles, from enhancing weak signals
in stochastic resonance to protecting the privacy of data in differential
privacy. It is then natural to ask, can we harness the power of quantum noise
that is beneficial to quantum computing? An important current direction for
quantum computing is its application to machine learning, such as
classification problems. One outstanding problem in machine learning for
classification is its sensitivity to adversarial examples. These are small,
undetectable perturbations from the original data where the perturbed data is
completely misclassified in otherwise extremely accurate classifiers. They can
also be considered as `worst-case' perturbations by unknown noise sources. We
show that by taking advantage of depolarisation noise in quantum circuits for
classification, a robustness bound against adversaries can be derived where the
robustness improves with increasing noise. This robustness property is
intimately connected with an important security concept called differential
privacy which can be extended to quantum differential privacy. For the
protection of quantum data, this is the first quantum protocol that can be used
against the most general adversaries. Furthermore, we show how the robustness
in the classical case can be sensitive to the details of the classification
model, but in the quantum case the details of classification model are absent,
thus also providing a potential quantum advantage for classical data that is
independent of quantum speedups. This opens the opportunity to explore other
ways in which quantum noise can be used in our favour, as well as identifying
other ways quantum algorithms can be helpful that is independent of quantum
speedups.


http://arxiv.org/abs/2003.09372
One Neuron to Fool Them All.
Anshuman Suri; David Evans
  Despite vast research in adversarial examples, the root causes of model
susceptibility are not well understood. Instead of looking at attack-specific
robustness, we propose a notion that evaluates the sensitivity of individual
neurons in terms of how robust the model's output is to direct perturbations of
that neuron's output. Analyzing models from this perspective reveals
distinctive characteristics of standard as well as adversarially-trained robust
models, and leads to several curious results. In our experiments on CIFAR-10
and ImageNet, we find that attacks using a loss function that targets just a
single sensitive neuron find adversarial examples nearly as effectively as ones
that target the full model. We analyze the properties of these sensitive
neurons to propose a regularization term that can help a model achieve
robustness to a variety of different perturbation constraints while maintaining
accuracy on natural data distributions. Code for all our experiments is
available at https://github.com/iamgroot42/sauron .


http://arxiv.org/abs/2003.09461
Adversarial Robustness on In- and Out-Distribution Improves Explainability.
Maximilian Augustin; Alexander Meinke; Matthias Hein
  Neural networks have led to major improvements in image classification but
suffer from being non-robust to adversarial changes, unreliable uncertainty
estimates on out-distribution samples and their inscrutable black-box
decisions. In this work we propose RATIO, a training procedure for Robustness
via Adversarial Training on In- and Out-distribution, which leads to robust
models with reliable and robust confidence estimates on the out-distribution.
RATIO has similar generative properties to adversarial training so that visual
counterfactuals produce class specific features. While adversarial training
comes at the price of lower clean accuracy, RATIO achieves state-of-the-art
$l_2$-adversarial robustness on CIFAR10 and maintains better clean accuracy.


http://arxiv.org/abs/2003.08861
Face-Off: Adversarial Face Obfuscation.
Chuhan Gao; Varun Chandrasekaran; Kassem Fawaz; Somesh Jha
  Advances in deep learning have made face recognition increasingly feasible
and pervasive. While useful to social media platforms and users, this
technology carries significant privacy threats. Coupled with the abundant
information they have about users, service providers can associate users with
social interactions, visited places, activities, and preferences - some of
which the user may not want to share. Additionally, facial recognition models
used by various agencies are trained by data scraped from social media
platforms. Existing approaches to mitigate these privacy risks from unwanted
face recognition result in an imbalanced privacy-utility trade-off to the
users. In this paper, we address this trade-off by proposing Face-Off, a
privacy-preserving framework that introduces minor perturbations to the user's
face to prevent it from being correctly recognized. To realize Face-Off, we
overcome a set of challenges related to the black box nature of commercial face
recognition services, and the lack literature for adversarial attacks on metric
networks. We implement and evaluate Face-Off to find that it deceives three
commercial face recognition services from Microsoft, Amazon, and Face++. Our
user study with 330 participants further shows that the perturbations come at
an acceptable cost for the users.


http://arxiv.org/abs/2003.08938
Robust Deep Reinforcement Learning against Adversarial Perturbations on Observations.
Huan Zhang; Hongge Chen; Chaowei Xiao; Bo Li; Duane Boning; Cho-Jui Hsieh
  Deep Reinforcement Learning (DRL) is vulnerable to small adversarial
perturbations on state observations. These perturbations do not alter the
environment directly but can mislead the agent into making suboptimal
decisions. We analyze the Markov Decision Process (MDP) under this threat model
and utilize tools from the neural net-work verification literature to enable
robust train-ing for DRL under observational perturbations. Our techniques are
general and can be applied to both Deep Q Networks (DQN) and Deep Deterministic
Policy Gradient (DDPG) algorithms for discrete and continuous action control
problems. We demonstrate that our proposed training procedure significantly
improves the robustness of DQN and DDPG agents under a suite of strong
white-box attacks on observations, including a few novel attacks we
specifically craft. Additionally, our training procedure can produce provable
certificates for the robustness of a Deep RL agent.


http://arxiv.org/abs/2003.08937
Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates.
Amin Ghiasi; Ali Shafahi; Tom Goldstein
  To deflect adversarial attacks, a range of "certified" classifiers have been
proposed. In addition to labeling an image, certified classifiers produce (when
possible) a certificate guaranteeing that the input image is not an
$\ell_p$-bounded adversarial example. We present a new attack that exploits not
only the labelling function of a classifier, but also the certificate
generator. The proposed method applies large perturbations that place images
far from a class boundary while maintaining the imperceptibility property of
adversarial examples. The proposed "Shadow Attack" causes certifiably robust
networks to mislabel an image and simultaneously produce a "spoofed"
certificate of robustness.


http://arxiv.org/abs/2003.08837
Vulnerabilities of Connectionist AI Applications: Evaluation and Defence.
Christian Berghoff; Matthias Neu; Twickel Arndt von
  This article deals with the IT security of connectionist artificial
intelligence (AI) applications, focusing on threats to integrity, one of the
three IT security goals. Such threats are for instance most relevant in
prominent AI computer vision applications. In order to present a holistic view
on the IT security goal integrity, many additional aspects such as
interpretability, robustness and documentation are taken into account. A
comprehensive list of threats and possible mitigations is presented by
reviewing the state-of-the-art literature. AI-specific vulnerabilities such as
adversarial attacks and poisoning attacks as well as their AI-specific root
causes are discussed in detail. Additionally and in contrast to former reviews,
the whole AI supply chain is analysed with respect to vulnerabilities,
including the planning, data acquisition, training, evaluation and operation
phases. The discussion of mitigations is likewise not restricted to the level
of the AI system itself but rather advocates viewing AI systems in the context
of their supply chains and their embeddings in larger IT infrastructures and
hardware devices. Based on this and the observation that adaptive attackers may
circumvent any single published AI-specific defence to date, the article
concludes that single protective measures are not sufficient but rather
multiple measures on different levels have to be combined to achieve a minimum
level of IT security for AI applications.


http://arxiv.org/abs/2003.09347
Improving Adversarial Robustness Through Progressive Hardening.
Chawin Sitawarin; Supriyo Chakraborty; David Wagner
  Adversarial training (AT) has become a popular choice for training robust
networks. However, by virtue of its formulation, AT tends to sacrifice clean
accuracy heavily in favor of robustness. Furthermore, AT with a large
perturbation budget can cause models to get stuck at poor local minima and
behave like a constant function, always predicting the same class. To address
the above concerns we propose Adversarial Training with Early Stopping (ATES).
The design of ATES is guided by principles from curriculum learning that
emphasizes on starting "easy" and gradually ramping up on the "difficulty" of
training. We do so by early stopping the adversarial example generation step in
AT, progressively increasing difficulty of the samples the network trains on.
This stabilizes network training even for large perturbation budgets and allows
the network to operate at a better clean accuracy versus robustness trade-off
curve compared to AT. Functionally, this leads to a significant improvement in
both clean accuracy and robustness for ATES models.


http://arxiv.org/abs/2003.08034
Generating Socially Acceptable Perturbations for Efficient Evaluation of Autonomous Vehicles.
Songan Zhang; Huei Peng; Subramanya Nageshrao; H. Eric Tseng
  Deep reinforcement learning methods have been widely used in recent years for
autonomous vehicle's decision-making. A key issue is that deep neural networks
can be fragile to adversarial attacks or other unseen inputs. In this paper, we
address the latter issue: we focus on generating socially acceptable
perturbations (SAP), so that the autonomous vehicle (AV agent), instead of the
challenging vehicle (attacker), is primarily responsible for the crash. In our
process, one attacker is added to the environment and trained by deep
reinforcement learning to generate the desired perturbation. The reward is
designed so that the attacker aims to fail the AV agent in a socially
acceptable way. After training the attacker, the agent policy is evaluated in
both the original naturalistic environment and the environment with one
attacker. The results show that the agent policy which is safe in the
naturalistic environment has many crashes in the perturbed environment.


http://arxiv.org/abs/2003.08093
Solving Non-Convex Non-Differentiable Min-Max Games using Proximal Gradient Method.
Babak Barazandeh; Meisam Razaviyayn
  Min-max saddle point games appear in a wide range of applications in machine
leaning and signal processing. Despite their wide applicability, theoretical
studies are mostly limited to the special convex-concave structure. While some
recent works generalized these results to special smooth non-convex cases, our
understanding of non-smooth scenarios is still limited. In this work, we study
special form of non-smooth min-max games when the objective function is
(strongly) convex with respect to one of the player's decision variable. We
show that a simple multi-step proximal gradient descent-ascent algorithm
converges to $\epsilon$-first-order Nash equilibrium of the min-max game with
the number of gradient evaluations being polynomial in $1/\epsilon$. We will
also show that our notion of stationarity is stronger than existing ones in the
literature. Finally, we evaluate the performance of the proposed algorithm
through adversarial attack on a LASSO estimator.


http://arxiv.org/abs/2003.07637
Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior.
Hu Zhang; Linchao Zhu; Yi Zhu; Yi Yang
  Deep neural networks are known to be susceptible to adversarial noise, which
are tiny and imperceptible perturbations. Most of previous work on adversarial
attack mainly focus on image models, while the vulnerability of video models is
less explored. In this paper, we aim to attack video models by utilizing
intrinsic movement pattern and regional relative motion among video frames. We
propose an effective motion-excited sampler to obtain motion-aware noise prior,
which we term as sparked prior. Our sparked prior underlines frame correlations
and utilizes video dynamics via relative motion. By using the sparked prior in
gradient estimation, we can successfully attack a variety of video
classification models with fewer number of queries. Extensive experimental
results on four benchmark datasets validate the efficacy of our proposed
method.


http://arxiv.org/abs/2003.07573
Heat and Blur: An Effective and Fast Defense Against Adversarial Examples.
Haya Brama; Tal Grinshpoun
  The growing incorporation of artificial neural networks (NNs) into many
fields, and especially into life-critical systems, is restrained by their
vulnerability to adversarial examples (AEs). Some existing defense methods can
increase NNs' robustness, but they often require special architecture or
training procedures and are irrelevant to already trained models. In this
paper, we propose a simple defense that combines feature visualization with
input modification, and can, therefore, be applicable to various pre-trained
networks. By reviewing several interpretability methods, we gain new insights
regarding the influence of AEs on NNs' computation. Based on that, we
hypothesize that information about the "true" object is preserved within the
NN's activity, even when the input is adversarial, and present a feature
visualization version that can extract that information in the form of
relevance heatmaps. We then use these heatmaps as a basis for our defense, in
which the adversarial effects are corrupted by massive blurring. We also
provide a new evaluation metric that can capture the effects of both attacks
and defenses more thoroughly and descriptively, and demonstrate the
effectiveness of the defense and the utility of the suggested evaluation
measurement with VGG19 results on the ImageNet dataset.


http://arxiv.org/abs/2003.07982
Adversarial Transferability in Wearable Sensor Systems.
Ramesh Kumar Sah; Hassan Ghasemzadeh
  Machine learning has increasingly become the most used approach for inference
and decision making in wearable sensor systems. However, recent studies have
found that machine learning systems are easily fooled by the addition of
adversarial perturbation to their inputs. What is more interesting is that the
adversarial examples generated for one machine learning system can also degrade
the performance of another. This property of adversarial examples is called
transferability. In this work, we take the first strides in studying
adversarial transferability in wearable sensor systems, from the following
perspectives: 1) Transferability between machine learning models, 2)
Transferability across subjects, 3) Transferability across sensor locations,
and 4) Transferability across datasets. With Human Activity Recognition (HAR)
as an example sensor system, we found strong untargeted transferability in all
cases of transferability. Specifically, gradient-based attacks were able to
achieve higher misclassification rates compared to non-gradient attacks. The
misclassification rate of untargeted adversarial examples ranged from 20% to
98%. For targeted transferability between machine learning models, the success
rate of adversarial examples was 100% for iterative attack methods. However,
the success rate for other types of targeted transferability ranged from 20% to
0%. Our findings strongly suggest that adversarial transferability has serious
consequences not only in sensor systems but also across the broad spectrum of
ubiquitous computing.


http://arxiv.org/abs/2003.06814
Towards Privacy Protection by Generating Adversarial Identity Masks.
Xiao Yang; Yinpeng Dong; Tianyu Pang; Jun Zhu; Hang Su
  As billions of personal data such as photos are shared through social media
and network, the privacy and security of data have drawn an increasing
attention. Several attempts have been made to alleviate the leakage of identity
information with the aid of image obfuscation techniques. However, most of the
present results are either perceptually unsatisfactory or ineffective against
real-world recognition systems. In this paper, we argue that an algorithm for
privacy protection must block the ability of automatic inference of the
identity and at the same time, make the resultant image natural from the users'
point of view. To achieve this, we propose a targeted identity-protection
iterative method (TIP-IM), which can generate natural face images by adding
adversarial identity masks to conceal ones' identity against a recognition
system. Extensive experiments on various state-of-the-art face recognition
models demonstrate the effectiveness of our proposed method on alleviating the
identity leakage of face images, without sacrificing? the visual quality of the
protected images.


http://arxiv.org/abs/2003.06979
Anomalous Instance Detection in Deep Learning: A Survey.
Saikiran Bulusu; Bhavya Kailkhura; Bo Li; Pramod K. Varshney; Dawn Song
  Deep Learning (DL) is vulnerable to out-of-distribution and adversarial
examples resulting in incorrect outputs. To make DL more robust, several
posthoc anomaly detection techniques to detect (and discard) these anomalous
samples have been proposed in the recent past. This survey tries to provide a
structured and comprehensive overview of the research on anomaly detection for
DL based applications. We provide a taxonomy for existing techniques based on
their underlying assumptions and adopted approaches. We discuss various
techniques in each of the categories and provide the relative strengths and
weaknesses of the approaches. Our goal in this survey is to provide an easier
yet better understanding of the techniques belonging to different categories in
which research has been done on this topic. Finally, we highlight the unsolved
research challenges while applying anomaly detection techniques in DL systems
and present some high-impact future research directions.


http://arxiv.org/abs/2003.06878
Output Diversified Initialization for Adversarial Attacks.
Yusuke Tashiro; Yang Song; Stefano Ermon
  Adversarial examples are often constructed by iteratively refining a randomly
perturbed input. To improve diversity and thus also the success rates of
attacks, we propose Output Diversified Initialization (ODI), a novel random
initialization strategy that can be combined with most existing white-box
adversarial attacks. Instead of using uniform perturbations in the input space,
we seek diversity in the output logits space of the target model. Empirically,
we demonstrate that existing $\ell_\infty$ and $\ell_2$ adversarial attacks
with ODI become much more efficient on several datasets including MNIST,
CIFAR-10 and ImageNet, reducing the accuracy of recently proposed defense
models by 1--17\%. Moreover, PGD attack with ODI outperforms current
state-of-the-art attacks against robust models, while also being roughly 50
times faster on CIFAR-10. The code is available on
https://github.com/ermongroup/ODI/.


http://arxiv.org/abs/2003.06974
Toward Adversarial Robustness via Semi-supervised Robust Training.
Yiming Li; Baoyuan Wu; Yan Feng; Yanbo Fan; Yong Jiang; Zhifeng Li; Shutao Xia
  Adversarial examples have been shown to be the severe threat to deep neural
networks (DNNs). One of the most effective adversarial defense methods is
adversarial training (AT) through minimizing the adversarial risk $R_{adv}$,
which encourages both the benign example $x$ and its adversarially perturbed
neighborhoods within the $\ell_{p}$-ball to be predicted as the ground-truth
label. In this work, we propose a novel defense method, the robust training
(RT), by jointly minimizing two separated risks ($R_{stand}$ and $R_{rob}$),
which is with respect to the benign example and its neighborhoods respectively.
The motivation is to explicitly and jointly enhance the accuracy and the
adversarial robustness. We prove that $R_{adv}$ is upper-bounded by $R_{stand}
+ R_{rob}$, which implies that RT has similar effect as AT. Intuitively,
minimizing the standard risk enforces the benign example to be correctly
predicted, and the robust risk minimization encourages the predictions of the
neighbor examples to be consistent with the prediction of the benign example.
Besides, since $R_{rob}$ is independent of the ground-truth label, RT is
naturally extended to the semi-supervised mode ($i.e.$, SRT), to further
enhance the adversarial robustness. Moreover, we extend the $\ell_{p}$-bounded
neighborhood to a general case, which covers different types of perturbations,
such as the pixel-wise ($i.e.$, $x + \delta$) or the spatial perturbation
($i.e.$, $ AX + b$). Extensive experiments on benchmark datasets not only
verify the superiority of the proposed SRT method to state-of-the-art methods
for defensing pixel-wise or spatial perturbations separately, but also
demonstrate its robustness to both perturbations simultaneously. The code for
reproducing main results is available at
\url{https://github.com/THUYimingLi/Semi-supervised_Robust_Training}.


http://arxiv.org/abs/2003.06555
Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation.
Xiaogang Xu; Hengshuang Zhao; Jiaya Jia
  Adversarial training is promising for improving robustness of deep neural
networks towards adversarial perturbation, especially on classification tasks.
Effect of this type of training on semantic segmentation, contrarily, just
commences. We make the initial attempt to explore the defense strategy on
semantic segmentation by formulating a general adversarial training procedure
that can perform decently on both adversarial and clean samples. We propose a
dynamic divide-and-conquer adversarial training (DDC-AT) strategy to enhance
the defense effect, by setting additional branches in the target model during
training, and dealing with pixels with diverse properties towards adversarial
perturbation. Our dynamical division mechanism divides pixels into multiple
branches automatically, achieved by unsupervised learning. Note all these
additional branches can be abandoned during inference and thus leave no extra
parameter and computation cost. Extensive experiments with various segmentation
models are conducted on PASCAL VOC 2012 and Cityscapes datasets, in which
DDC-AT yields satisfying performance under both white- and black-box attack.


http://arxiv.org/abs/2003.06693
Certified Defenses for Adversarial Patches.
Ping-Yeh Chiang; Renkun Ni; Ahmed Abdelkader; Chen Zhu; Christoph Studor; Tom Goldstein
  Adversarial patch attacks are among one of the most practical threat models
against real-world computer vision systems. This paper studies certified and
empirical defenses against patch attacks. We begin with a set of experiments
showing that most existing defenses, which work by pre-processing input images
to mitigate adversarial patches, are easily broken by simple white-box
adversaries. Motivated by this finding, we propose the first certified defense
against patch attacks, and propose faster methods for its training.
Furthermore, we experiment with different patch shapes for testing, obtaining
surprisingly good robustness transfer across shapes, and present preliminary
results on certified defense against sparse attacks. Our complete
implementation can be found on:
https://github.com/Ping-C/certifiedpatchdefense.


http://arxiv.org/abs/2003.06559
Minimum-Norm Adversarial Examples on KNN and KNN-Based Models.
Chawin Sitawarin; David Wagner
  We study the robustness against adversarial examples of kNN classifiers and
classifiers that combine kNN with neural networks. The main difficulty lies in
the fact that finding an optimal attack on kNN is intractable for typical
datasets. In this work, we propose a gradient-based attack on kNN and kNN-based
defenses, inspired by the previous work by Sitawarin & Wagner [1]. We
demonstrate that our attack outperforms their method on all of the models we
tested with only a minimal increase in the computation time. The attack also
beats the state-of-the-art attack [2] on kNN when k > 1 using less than 1% of
its running time. We hope that this attack can be used as a new baseline for
evaluating the robustness of kNN and its variants.


http://arxiv.org/abs/2003.06566
VarMixup: Exploiting the Latent Space for Robust Training and Inference.
Puneet Mangla; Vedant Singh; Shreyas Jayant Havaldar; Vineeth N Balasubramanian
  The vulnerability of Deep Neural Networks (DNNs) to adversarial attacks has
led to the development of many defense approaches. Among them, Adversarial
Training (AT) is a popular and widely used approach for training adversarially
robust models. Mixup Training (MT), a recent popular training algorithm,
improves the generalization performance of models by introducing globally
linear behavior in between training examples. Although still in its early
phase, we observe a shift in trend of exploiting Mixup from perspectives of
generalisation to that of adversarial robustness. It has been shown that the
Mixup trained models improves the robustness of models but only passively. A
recent approach, Mixup Inference (MI), proposes an inference principle for
Mixup trained models to counter adversarial examples at inference time by
mixing the input with other random clean samples. In this work, we propose a
new approach - \textit{VarMixup (Variational Mixup)} - to better sample mixup
images by using the latent manifold underlying the data. Our experiments on
CIFAR-10, CIFAR-100, SVHN and Tiny-Imagenet demonstrate that \textit{VarMixup}
beats state-of-the-art AT techniques without training the model adversarially.
Additionally, we also conduct ablations that show that models trained on
\textit{VarMixup} samples are also robust to various input
corruptions/perturbations, have low calibration error and are transferable.


http://arxiv.org/abs/2003.06428
Towards a Resilient Machine Learning Classifier -- a Case Study of Ransomware Detection.
Chih-Yuan Yang; Ravi Sahita
  The damage caused by crypto-ransomware, due to encryption, is difficult to
revert and cause data losses. In this paper, a machine learning (ML) classifier
was built to early detect ransomware (called crypto-ransomware) that uses
cryptography by program behavior. If a signature-based detection was missed, a
behavior-based detector can be the last line of defense to detect and contain
the damages. We find that input/output activities of ransomware and the
file-content entropy are unique traits to detect crypto-ransomware. A
deep-learning (DL) classifier can detect ransomware with a high accuracy and a
low false positive rate. We conduct an adversarial research against the models
generated. We use simulated ransomware programs to launch a gray-box analysis
to probe the weakness of ML classifiers and to improve model robustness. In
addition to accuracy and resiliency, trustworthiness is the other key criteria
for a quality detector. Making sure that the correct information was used for
inference is important for a security application. The Integrated Gradient
method was used to explain the deep learning model and also to reveal why false
negatives evade the detection. The approaches to build and to evaluate a
real-world detector were demonstrated and discussed.


http://arxiv.org/abs/2003.06468
GeoDA: a geometric framework for black-box adversarial attacks.
Ali Rahmati; Seyed-Mohsen Moosavi-Dezfooli; Pascal Frossard; Huaiyu Dai
  Adversarial examples are known as carefully perturbed images fooling image
classifiers. We propose a geometric framework to generate adversarial examples
in one of the most challenging black-box settings where the adversary can only
generate a small number of queries, each of them returning the top-$1$ label of
the classifier. Our framework is based on the observation that the decision
boundary of deep networks usually has a small mean curvature in the vicinity of
data samples. We propose an effective iterative algorithm to generate
query-efficient black-box perturbations with small $\ell_p$ norms for $p \ge
1$, which is confirmed via experimental evaluations on state-of-the-art natural
image classifiers. Moreover, for $p=2$, we theoretically show that our
algorithm actually converges to the minimal $\ell_2$-perturbation when the
curvature of the decision boundary is bounded. We also obtain the optimal
distribution of the queries over the iterations of the algorithm. Finally,
experimental results confirm that our principled black-box attack algorithm
performs better than state-of-the-art algorithms as it generates smaller
perturbations with a reduced number of queries.


http://arxiv.org/abs/2003.06121
When are Non-Parametric Methods Robust?
Robi Bhattacharjee; Kamalika Chaudhuri
  A growing body of research has shown that many classifiers are susceptible to
{\em{adversarial examples}} -- small strategic modifications to test inputs
that lead to misclassification. In this work, we study general non-parametric
methods, with a view towards understanding when they are robust to these
modifications. We establish general conditions under which non-parametric
methods are r-consistent -- in the sense that they converge to optimally robust
and accurate classifiers in the large sample limit.
  Concretely, our results show that when data is well-separated, nearest
neighbors and kernel classifiers are r-consistent, while histograms are not.
For general data distributions, we prove that preprocessing by Adversarial
Pruning (Yang et. al., 2019) -- that makes data well-separated -- followed by
nearest neighbors or kernel classifiers also leads to r-consistency.


http://arxiv.org/abs/2003.05669
ARAE: Adversarially Robust Training of Autoencoders Improves Novelty Detection.
Mohammadreza Salehi; Atrin Arya; Barbod Pajoum; Mohammad Otoofi; Amirreza Shaeiri; Mohammad Hossein Rohban; Hamid R. Rabiee
  Autoencoders (AE) have recently been widely employed to approach the novelty
detection problem. Trained only on the normal data, the AE is expected to
reconstruct the normal data effectively while fail to regenerate the anomalous
data, which could be utilized for novelty detection. However, in this paper, it
is demonstrated that this does not always hold. AE often generalizes so
perfectly that it can also reconstruct the anomalous data well. To address this
problem, we propose a novel AE that can learn more semantically meaningful
features. Specifically, we exploit the fact that adversarial robustness
promotes learning of meaningful features. Therefore, we force the AE to learn
such features by penalizing networks with a bottleneck layer that is unstable
against adversarial perturbations. We show that despite using a much simpler
architecture in comparison to the prior methods, the proposed AE outperforms or
is competitive to state-of-the-art on three benchmark datasets.


http://arxiv.org/abs/2003.05822
Topological Effects on Attacks Against Vertex Classification.
Benjamin A. Miller; Mustafa Çamurcu; Alexander J. Gomez; Kevin Chan; Tina Eliassi-Rad
  Vertex classification is vulnerable to perturbations of both graph topology
and vertex attributes, as shown in recent research. As in other machine
learning domains, concerns about robustness to adversarial manipulation can
prevent potential users from adopting proposed methods when the consequence of
action is very high. This paper considers two topological characteristics of
graphs and explores the way these features affect the amount the adversary must
perturb the graph in order to be successful. We show that, if certain vertices
are included in the training set, it is possible to substantially an
adversary's required perturbation budget. On four citation datasets, we
demonstrate that if the training set includes high degree vertices or vertices
that ensure all unlabeled nodes have neighbors in the training set, we show
that the adversary's budget often increases by a substantial factor---often a
factor of 2 or more---over random training for the Nettack poisoning attack.
Even for especially easy targets (those that are misclassified after just one
or two perturbations), the degradation of performance is much slower, assigning
much lower probabilities to the incorrect classes. In addition, we demonstrate
that this robustness either persists when recently proposed defenses are
applied, or is competitive with the resulting performance improvement for the
defender.


http://arxiv.org/abs/2003.05631
ConAML: Constrained Adversarial Machine Learning for Cyber-Physical Systems.
Jiangnan Li; Jin Young Lee; Yingyuan Yang; Jinyuan Stella Sun; Kevin Tomsovic
  Recent research demonstrated that the superficially well-trained machine
learning (ML) models are highly vulnerable to adversarial examples. As ML
techniques are rapidly employed in cyber-physical systems (CPSs), the security
of these applications is of concern. However, current studies on adversarial
machine learning (AML) mainly focus on computer vision and related fields. The
risks the adversarial examples can bring to the CPS applications have not been
well investigated. In particular, due to the distributed property of data
sources and the inherent physical constraints imposed by CPSs, the widely-used
threat models in previous research and the state-of-the-art AML algorithms are
no longer practical when applied to CPS applications.
  We study the vulnerabilities of ML applied in CPSs by proposing Constrained
Adversarial Machine Learning (ConAML), which generates adversarial examples
used as ML model input that meet the intrinsic constraints of the physical
systems. We first summarize the difference between AML in CPSs and AML in
existing cyber systems and propose a general threat model for ConAML. We then
design a best-effort search algorithm to iteratively generate adversarial
examples with linear physical constraints. As proofs of concept, we evaluate
the vulnerabilities of ML models used in the electric power grid and water
treatment systems. The results show that our ConAML algorithms can effectively
generate adversarial examples which significantly decrease the performance of
the ML models even under practical physical constraints.


http://arxiv.org/abs/2003.05703
Inline Detection of DGA Domains Using Side Information.
Raaghavi Sivaguru; Jonathan Peck; Femi Olumofin; Anderson Nascimento; Cock Martine De
  Malware applications typically use a command and control (C&C) server to
manage bots to perform malicious activities. Domain Generation Algorithms
(DGAs) are popular methods for generating pseudo-random domain names that can
be used to establish a communication between an infected bot and the C&C
server. In recent years, machine learning based systems have been widely used
to detect DGAs. There are several well known state-of-the-art classifiers in
the literature that can detect DGA domain names in real-time applications with
high predictive performance. However, these DGA classifiers are highly
vulnerable to adversarial attacks in which adversaries purposely craft domain
names to evade DGA detection classifiers. In our work, we focus on hardening
DGA classifiers against adversarial attacks. To this end, we train and evaluate
state-of-the-art deep learning and random forest (RF) classifiers for DGA
detection using side information that is harder for adversaries to manipulate
than the domain name itself. Additionally, the side information features are
selected such that they are easily obtainable in practice to perform inline DGA
detection. The performance and robustness of these models is assessed by
exposing them to one day of real-traffic data as well as domains generated by
adversarial attack algorithms. We found that the DGA classifiers that rely on
both the domain name and side information have high performance and are more
robust against adversaries.


http://arxiv.org/abs/2003.05549
Frequency-Tuned Universal Adversarial Attacks.
Yingpeng Deng; Lina J. Karam
  Researchers have shown that the predictions of a convolutional neural network
(CNN) for an image set can be severely distorted by one single image-agnostic
perturbation, or universal perturbation, usually with an empirically fixed
threshold in the spatial domain to restrict its perceivability. However, by
considering the human perception, we propose to adopt JND thresholds to guide
the perceivability of universal adversarial perturbations. Based on this, we
propose a frequency-tuned universal attack method to compute universal
perturbations and show that our method can realize a good balance between
perceivability and effectiveness in terms of fooling rate by adapting the
perturbations to the local frequency content. Compared with existing universal
adversarial attack techniques, our frequency-tuned attack method can achieve
cutting-edge quantitative results. We demonstrate that our approach can
significantly improve the performance of the baseline on both white-box and
black-box attacks.


http://arxiv.org/abs/2003.04884
Cryptanalytic Extraction of Neural Network Models.
Nicholas Carlini; Matthew Jagielski; Ilya Mironov
  We argue that the machine learning problem of model extraction is actually a
cryptanalytic problem in disguise, and should be studied as such. Given oracle
access to a neural network, we introduce a differential attack that can
efficiently steal the parameters of the remote model up to floating point
precision. Our attack relies on the fact that ReLU neural networks are
piecewise linear functions, and that queries at the critical points reveal
information about the model parameters.
  We evaluate our attack on multiple neural network models and extract models
that are 2^20 times more precise and require 100x fewer queries than prior
work. For example, we extract a 100,000 parameter neural network trained on the
MNIST digit recognition task with 2^21.5 queries in under an hour, such that
the extracted model agrees with the oracle on all inputs up to a worst-case
error of 2^-25, or a model with 4,000 parameters in 2^18.5 queries with
worst-case error of 2^-40.4.


http://arxiv.org/abs/2003.04820
SAD: Saliency-based Defenses Against Adversarial Examples.
Richard Tran; David Patrick; Michael Geyer; Amanda Fernandez
  With the rise in popularity of machine and deep learning models, there is an
increased focus on their vulnerability to malicious inputs. These adversarial
examples drift model predictions away from the original intent of the network
and are a growing concern in practical security. In order to combat these
attacks, neural networks can leverage traditional image processing approaches
or state-of-the-art defensive models to reduce perturbations in the data.
Defensive approaches that take a global approach to noise reduction are
effective against adversarial attacks, however their lossy approach often
distorts important data within the image. In this work, we propose a visual
saliency based approach to cleaning data affected by an adversarial attack. Our
model leverages the salient regions of an adversarial image in order to provide
a targeted countermeasure while comparatively reducing loss within the cleaned
images. We measure the accuracy of our model by evaluating the effectiveness of
state-of-the-art saliency methods prior to attack, under attack, and after
application of cleaning methods. We demonstrate the effectiveness of our
proposed approach in comparison with related defenses and against established
adversarial attack methods, across two saliency datasets. Our targeted approach
shows significant improvements in a range of standard statistical and distance
saliency metrics, in comparison with both traditional and state-of-the-art
approaches.


http://arxiv.org/abs/2003.05005
Using an ensemble color space model to tackle adversarial examples.
Shreyank N Gowda; Chun Yuan
  Minute pixel changes in an image drastically change the prediction that the
deep learning model makes. One of the most significant problems that could
arise due to this, for instance, is autonomous driving. Many methods have been
proposed to combat this with varying amounts of success. We propose a 3 step
method for defending such attacks. First, we denoise the image using
statistical methods. Second, we show that adopting multiple color spaces in the
same model can help us to fight these adversarial attacks further as each color
space detects certain features explicit to itself. Finally, the feature maps
generated are enlarged and sent back as an input to obtain even smaller
features. We show that the proposed model does not need to be trained to defend
an particular type of attack and is inherently more robust to black-box,
white-box, and grey-box adversarial attack techniques. In particular, the model
is 56.12 percent more robust than compared models in case of white box attacks
when the models are not subject to adversarial example training.


http://arxiv.org/abs/2003.05730
A Survey of Adversarial Learning on Graphs.
Liang Chen; Jintang Li; Jiaying Peng; Tao Xie; Zengxu Cao; Kun Xu; Xiangnan He; Zibin Zheng
  Deep learning models on graphs have achieved remarkable performance in
various graph analysis tasks, e.g., node classification, link prediction and
graph clustering. However, they expose uncertainty and unreliability against
the well-designed inputs, i.e., adversarial examples. Accordingly, a line of
studies have emerged for both attack and defense addressed in different graph
analysis tasks, leading to the arms race in graph adversarial learning.
  Despite the booming works, there still lacks a unified problem definition and
a comprehensive review. To bridge this gap, we investigate and summarize the
existing works on graph adversarial learning tasks systemically. Specifically,
we survey and unify the existing works w.r.t. attack and defense in graph
analysis tasks, and give appropriate definitions and taxonomies at the same
time. Besides, we emphasize the importance of related evaluation metrics,
investigate and summarize them comprehensively. Hopefully, our works can
provide a comprehensive overview and offer insights for the relevant
researchers. More details of our works are available at
https://github.com/gitgiter/Graph-Adversarial-Learning.


http://arxiv.org/abs/2003.04475
Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift.
Remi Tachet des Combes; Han Zhao; Yu-Xiang Wang; Geoff Gordon
  Adversarial learning has demonstrated good performance in the unsupervised
domain adaptation setting, by learning domain-invariant representations that
perform well on the source domain. However, recent work has underlined
limitations of existing methods in the presence of mismatched label
distributions between the source and target domains. In this paper, we extend a
recent upper-bound on the performance of adversarial domain adaptation to
multi-class classification and more general discriminators. We then propose
generalized label shift (GLS) as a way to improve robustness against mismatched
label distributions. GLS states that, conditioned on the label, there exists a
representation of the input that is invariant between the source and target
domains. Under GLS, we provide theoretical guarantees on the transfer
performance of any classifier. We also devise necessary and sufficient
conditions for GLS to hold. The conditions are based on the estimation of the
relative class weights between domains and on an appropriate reweighting of
samples. Guided by our theoretical insights, we modify three widely used
algorithms, JAN, DANN and CDAN and evaluate their performance on standard
domain adaptation tasks where our method outperforms the base versions. We also
demonstrate significant gains on artificially created tasks with large
divergences between their source and target label distributions.


http://arxiv.org/abs/2003.04247
Towards Probabilistic Verification of Machine Unlearning.
David Marco Sommer; Liwei Song; Sameer Wagh; Prateek Mittal
  Right to be forgotten, also known as the right to erasure, is the right of
individuals to have their data erased from an entity storing it. The General
Data Protection Regulation in the European Union legally solidified the status
of this long held notion. As a consequence, there is a growing need for the
development of mechanisms whereby users can verify if service providers comply
with their deletion requests.
  In this work, we take the first step in proposing a formal framework to study
the design of such verification mechanisms for data deletion requests -- also
known as machine unlearning -- in the context of systems that provide machine
learning as a service. We propose a backdoor-based verification mechanism and
demonstrate its effectiveness in certifying data deletion with high confidence
using the above framework. Our mechanism makes a novel use of backdoor attacks
in ML as a basis for quantitatively inferring machine unlearning. In our
mechanism, each user poisons part of its training data by injecting a
user-specific backdoor trigger associated with a user-specific target label.
The prediction of target labels on test samples with the backdoor trigger is
then used as an indication of the user's data being used to train the ML model.
  We formalize the verification process as a hypothesis testing problem, and
provide theoretical guarantees on the statistical power of the hypothesis test.
We experimentally demonstrate that our approach has minimal effect on the
machine learning service but provides high confidence verification of
unlearning. We show that with a $30\%$ poison ratio and merely $20$ test
queries, our verification mechanism has both false positive and false negative
ratios below $10^{-5}$. Furthermore, we also show the effectiveness of our
approach by testing it against an adaptive adversary that uses a
state-of-the-art backdoor defense method.


http://arxiv.org/abs/2003.04286
Manifold Regularization for Adversarial Robustness.
Charles Jin; Martin Rinard
  Manifold regularization is a technique that penalizes the complexity of
learned functions over the intrinsic geometry of input data. We develop a
connection to learning functions which are "locally stable", and propose new
regularization terms for training deep neural networks that are stable against
a class of local perturbations. These regularizers enable us to train a network
to state-of-the-art robust accuracy of 70% on CIFAR-10 against a PGD adversary
using $\ell_\infty$ perturbations of size $\epsilon = 8/255$. Furthermore, our
techniques do not rely on the construction of any adversarial examples, thus
running orders of magnitude faster than standard algorithms for adversarial
training.


http://arxiv.org/abs/2003.04173
Gradient-based adversarial attacks on categorical sequence models via traversing an embedded world.
Ivan Fursov; Alexey Zaytsev; Nikita Kluchnikov; Andrey Kravchenko; Evgeny Burnaev
  An adversarial attack paradigm explores various scenarios for vulnerability
of machine and especially deep learning models: we can apply minor changes to
the model input to force a classifier's failure for a particular example. Most
of the state of the art frameworks focus on adversarial attacks for images and
other structured model inputs. The adversarial attacks for categorical
sequences can also be harmful if they are successful. However, successful
attacks for inputs based on categorical sequences should address the following
challenges: (1) non-differentiability of the target function, (2) constraints
on transformations of initial sequences, and (3) diversity of possible
problems. We handle these challenges using two approaches. The first approach
adopts Monte-Carlo methods and allows usage in any scenario, the second
approach uses a continuous relaxation of models and target metrics, and thus
allows using general state of the art methods on adversarial attacks with
little additional effort. Results for money transactions, medical fraud, and
NLP datasets suggest the proposed methods generate reasonable adversarial
sequences that are close to original ones, but fool machine learning models
even for blackbox adversarial attacks.


http://arxiv.org/abs/2003.10388
Generating Natural Language Adversarial Examples on a Large Scale with Generative Models.
Yankun Ren; Jianbin Lin; Siliang Tang; Jun Zhou; Shuang Yang; Yuan Qi; Xiang Ren
  Today text classification models have been widely used. However, these
classifiers are found to be easily fooled by adversarial examples. Fortunately,
standard attacking methods generate adversarial texts in a pair-wise way, that
is, an adversarial text can only be created from a real-world text by replacing
a few words. In many applications, these texts are limited in numbers,
therefore their corresponding adversarial examples are often not diverse enough
and sometimes hard to read, thus can be easily detected by humans and cannot
create chaos at a large scale. In this paper, we propose an end to end solution
to efficiently generate adversarial texts from scratch using generative models,
which are not restricted to perturbing the given texts. We call it unrestricted
adversarial text generation. Specifically, we train a conditional variational
autoencoder (VAE) with an additional adversarial loss to guide the generation
of adversarial examples. Moreover, to improve the validity of adversarial
texts, we utilize discrimators and the training framework of generative
adversarial networks (GANs) to make adversarial texts consistent with real
data. Experimental results on sentiment analysis demonstrate the scalability
and efficiency of our method. It can attack text classification models with a
higher success rate than existing methods, and provide acceptable quality for
humans in the meantime.


http://arxiv.org/abs/2003.04735
Security of Distributed Machine Learning: A Game-Theoretic Approach to Design Secure DSVM.
Rui Zhang; Quanyan Zhu
  Distributed machine learning algorithms play a significant role in processing
massive data sets over large networks. However, the increasing reliance on
machine learning on information and communication technologies (ICTs) makes it
inherently vulnerable to cyber threats. This work aims to develop secure
distributed algorithms to protect the learning from data poisoning and network
attacks. We establish a game-theoretic framework to capture the conflicting
goals of a learner who uses distributed support vector machines (SVMs) and an
attacker who is capable of modifying training data and labels. We develop a
fully distributed and iterative algorithm to capture real-time reactions of the
learner at each node to adversarial behaviors. The numerical results show that
distributed SVM is prone to fail in different types of attacks, and their
impact has a strong dependence on the network structure and attack
capabilities.


http://arxiv.org/abs/2003.03879
An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods.
Sanghyuk Chun; Seong Joon Oh; Sangdoo Yun; Dongyoon Han; Junsuk Choe; Youngjoon Yoo
  Despite apparent human-level performances of deep neural networks (DNN), they
behave fundamentally differently from humans. They easily change predictions
when small corruptions such as blur and noise are applied on the input (lack of
robustness), and they often produce confident predictions on
out-of-distribution samples (improper uncertainty measure). While a number of
researches have aimed to address those issues, proposed solutions are typically
expensive and complicated (e.g. Bayesian inference and adversarial training).
Meanwhile, many simple and cheap regularization methods have been developed to
enhance the generalization of classifiers. Such regularization methods have
largely been overlooked as baselines for addressing the robustness and
uncertainty issues, as they are not specifically designed for that. In this
paper, we provide extensive empirical evaluations on the robustness and
uncertainty estimates of image classifiers (CIFAR-100 and ImageNet) trained
with state-of-the-art regularization methods. Furthermore, experimental results
show that certain regularization methods can serve as strong baseline methods
for robustness and uncertainty estimation of DNNs.


http://arxiv.org/abs/2003.03722
On the Robustness of Cooperative Multi-Agent Reinforcement Learning.
Jieyu Lin; Kristina Dzeparoska; Sai Qian Zhang; Alberto Leon-Garcia; Nicolas Papernot
  In cooperative multi-agent reinforcement learning (c-MARL), agents learn to
cooperatively take actions as a team to maximize a total team reward. We
analyze the robustness of c-MARL to adversaries capable of attacking one of the
agents on a team. Through the ability to manipulate this agent's observations,
the adversary seeks to decrease the total team reward.
  Attacking c-MARL is challenging for three reasons: first, it is difficult to
estimate team rewards or how they are impacted by an agent mispredicting;
second, models are non-differentiable; and third, the feature space is
low-dimensional. Thus, we introduce a novel attack. The attacker first trains a
policy network with reinforcement learning to find a wrong action it should
encourage the victim agent to take. Then, the adversary uses targeted
adversarial examples to force the victim to take this action.
  Our results on the StartCraft II multi-agent benchmark demonstrate that
c-MARL teams are highly vulnerable to perturbations applied to one of their
agent's observations. By attacking a single agent, our attack method has highly
negative impact on the overall team reward, reducing it from 20 to 9.4. This
results in the team's winning rate to go down from 98.9% to 0%.


http://arxiv.org/abs/2003.03778
Adversarial Attacks on Probabilistic Autoregressive Forecasting Models.
Raphaël Dang-Nhu; Gagandeep Singh; Pavol Bielik; Martin Vechev
  We develop an effective generation of adversarial attacks on neural models
that output a sequence of probability distributions rather than a sequence of
single values. This setting includes the recently proposed deep probabilistic
autoregressive forecasting models that estimate the probability distribution of
a time series given its past and achieve state-of-the-art results in a diverse
set of application domains. The key technical challenge we address is
effectively differentiating through the Monte-Carlo estimation of statistics of
the joint distribution of the output sequence. Additionally, we extend prior
work on probabilistic forecasting to the Bayesian setting which allows
conditioning on future observations, instead of only on past observations. We
demonstrate that our approach can successfully generate attacks with small
input perturbations in two challenging tasks where robust decision making is
crucial: stock market trading and prediction of electricity consumption.


http://arxiv.org/abs/2003.03824
No Surprises: Training Robust Lung Nodule Detection for Low-Dose CT Scans by Augmenting with Adversarial Attacks.
Siqi Liu; Arnaud Arindra Adiyoso Setio; Florin C. Ghesu; Eli Gibson; Sasa Grbic; Bogdan Georgescu; Dorin Comaniciu
  Detecting malignant pulmonary nodules at an early stage can allow medical
interventions which increases the survival rate of lung cancer patients. Using
computer vision techniques to detect nodules can improve the sensitivity and
the speed of interpreting chest CT for lung cancer screening. Many studies have
used CNNs to detect nodule candidates. Though such approaches have been shown
to outperform the conventional image processing based methods regarding the
detection accuracy, CNNs are also known to be limited to generalize on
under-represented samples in the training set and prone to imperceptible noise
perturbations. Such limitations can not be easily addressed by scaling up the
dataset or the models. In this work, we propose to add adversarial synthetic
nodules and adversarial attack samples to the training data to improve the
generalization and the robustness of the lung nodule detection systems. In
order to generate hard examples of nodules from a differentiable nodule
synthesizer, we use projected gradient descent (PGD) to search the latent code
within a bounded neighbourhood that would generate nodules to decrease the
detector response. To make the network more robust to unanticipated noise
perturbations, we use PGD to search for noise patterns that can trigger the
network to give over-confident mistakes. By evaluating on two different
benchmark datasets containing consensus annotations from three radiologists, we
show that the proposed techniques can improve the detection performance on real
CT data. To understand the limitations of both the conventional networks and
the proposed augmented networks, we also perform stress-tests on the false
positive reduction networks by feeding different types of artificially produced
patches. We show that the augmented networks are more robust to both
under-represented nodules as well as resistant to noise perturbations.


http://arxiv.org/abs/2003.08757
Adversarial Camouflage: Hiding Physical-World Attacks with Natural Styles.
Ranjie Duan; Xingjun Ma; Yisen Wang; James Bailey; A. K. Qin; Yun Yang
  Deep neural networks (DNNs) are known to be vulnerable to adversarial
examples. Existing works have mostly focused on either digital adversarial
examples created via small and imperceptible perturbations, or physical-world
adversarial examples created with large and less realistic distortions that are
easily identified by human observers. In this paper, we propose a novel
approach, called Adversarial Camouflage (\emph{AdvCam}), to craft and
camouflage physical-world adversarial examples into natural styles that appear
legitimate to human observers. Specifically, \emph{AdvCam} transfers large
adversarial perturbations into customized styles, which are then "hidden"
on-target object or off-target background. Experimental evaluation shows that,
in both digital and physical-world scenarios, adversarial examples crafted by
\emph{AdvCam} are well camouflaged and highly stealthy, while remaining
effective in fooling state-of-the-art DNN image classifiers. Hence,
\emph{AdvCam} is a flexible approach that can help craft stealthy attacks to
evaluate the robustness of DNNs. \emph{AdvCam} can also be used to protect
private information from being detected by deep learning systems.


http://arxiv.org/abs/2003.03675
Dynamic Backdoor Attacks Against Machine Learning Models.
Ahmed Salem; Rui Wen; Michael Backes; Shiqing Ma; Yang Zhang
  Machine learning (ML) has made tremendous progress during the past decade and
is being adopted in various critical real-world applications. However, recent
research has shown that ML models are vulnerable to multiple security and
privacy attacks. In particular, backdoor attacks against ML models that have
recently raised a lot of awareness. A successful backdoor attack can cause
severe consequences, such as allowing an adversary to bypass critical
authentication systems.
  Current backdooring techniques rely on adding static triggers (with fixed
patterns and locations) on ML model inputs. In this paper, we propose the first
class of dynamic backdooring techniques: Random Backdoor, Backdoor Generating
Network (BaN), and conditional Backdoor Generating Network (c-BaN). Triggers
generated by our techniques can have random patterns and locations, which
reduce the efficacy of the current backdoor detection mechanisms. In
particular, BaN and c-BaN are the first two schemes that algorithmically
generate triggers, which rely on a novel generative network. Moreover, c-BaN is
the first conditional backdooring technique, that given a target label, it can
generate a target-specific trigger. Both BaN and c-BaN are essentially a
general framework which renders the adversary the flexibility for further
customizing backdoor attacks.
  We extensively evaluate our techniques on three benchmark datasets: MNIST,
CelebA, and CIFAR-10. Our techniques achieve almost perfect attack performance
on backdoored data with a negligible utility loss. We further show that our
techniques can bypass current state-of-the-art defense mechanisms against
backdoor attacks, including Neural Cleanse, ABS, and STRIP.


http://arxiv.org/abs/2003.03143
Triple Memory Networks: a Brain-Inspired Method for Continual Learning.
Liyuan Wang; Bo Lei; Qian Li; Hang Su; Jun Zhu; Yi Zhong
  Continual acquisition of novel experience without interfering previously
learned knowledge, i.e. continual learning, is critical for artificial neural
networks, but limited by catastrophic forgetting. A neural network adjusts its
parameters when learning a new task, but then fails to conduct the old tasks
well. By contrast, the brain has a powerful ability to continually learn new
experience without catastrophic interference. The underlying neural mechanisms
possibly attribute to the interplay of hippocampus-dependent memory system and
neocortex-dependent memory system, mediated by prefrontal cortex. Specifically,
the two memory systems develop specialized mechanisms to consolidate
information as more specific forms and more generalized forms, respectively,
and complement the two forms of information in the interplay. Inspired by such
brain strategy, we propose a novel approach named triple memory networks (TMNs)
for continual learning. TMNs model the interplay of hippocampus, prefrontal
cortex and sensory cortex (a neocortex region) as a triple-network architecture
of generative adversarial networks (GAN). The input information is encoded as
specific representation of the data distributions in a generator, or
generalized knowledge of solving tasks in a discriminator and a classifier,
with implementing appropriate brain-inspired algorithms to alleviate
catastrophic forgetting in each module. Particularly, the generator replays
generated data of the learned tasks to the discriminator and the classifier,
both of which are implemented with a weight consolidation regularizer to
complement the lost information in generation process. TMNs achieve new
state-of-the-art performance on a variety of class-incremental learning
benchmarks on MNIST, SVHN, CIFAR-10 and ImageNet-50, comparing with strong
baseline methods.


http://arxiv.org/abs/2003.03065
Defense against adversarial attacks on spoofing countermeasures of ASV.
Haibin Wu; Songxiang Liu; Helen Meng; Hung-yi Lee
  Various forefront countermeasure methods for automatic speaker verification
(ASV) with considerable performance in anti-spoofing are proposed in the
ASVspoof 2019 challenge. However, previous work has shown that countermeasure
models are vulnerable to adversarial examples indistinguishable from natural
data. A good countermeasure model should not only be robust against spoofing
audio, including synthetic, converted, and replayed audios; but counteract
deliberately generated examples by malicious adversaries. In this work, we
introduce a passive defense method, spatial smoothing, and a proactive defense
method, adversarial training, to mitigate the vulnerability of ASV spoofing
countermeasure models against adversarial examples. This paper is among the
first to use defense methods to improve the robustness of ASV spoofing
countermeasure models under adversarial attacks. The experimental results show
that these two defense methods positively help spoofing countermeasure models
counter adversarial examples.


http://arxiv.org/abs/2003.03100
Automatic Generation of Adversarial Examples for Interpreting Malware Classifiers.
Wei Song; Xuezixiang Li; Sadia Afroz; Deepali Garg; Dmitry Kuznetsov; Heng Yin
  Recent advances in adversarial attacks have shown that machine learning
classifiers based on static analysis are vulnerable to adversarial attacks.
However, real-world antivirus systems do not rely only on static classifiers,
thus many of these static evasions get detected by dynamic analysis whenever
the malware runs. The real question is to what extent these adversarial attacks
are actually harmful to the real users? In this paper, we propose a systematic
framework to create and evaluate realistic adversarial malware to evade
real-world systems. We propose new adversarial attacks against real-world
antivirus systems based on code randomization and binary manipulation, and use
our framework to perform the attacks on 1000 malware samples and test four
commercial antivirus software and two open-source classifiers. We demonstrate
that the static detectors of real-world antivirus can be evaded 24.3%-41.9% of
the cases and often by changing only one byte. We also find that the
adversarial attacks are transferable between different antivirus up to 16% of
the cases. We also tested the efficacy of the complete (i.e. static + dynamic)
classifiers in protecting users. While most of the commercial antivirus use
their dynamic engines to protect the users' device when the static classifiers
are evaded, we are the first to demonstrate that for one commercial antivirus,
static evasions can also evade the offline dynamic detectors and infect users'
machines. Our framework can also help explain which features are responsible
for evasion and thus can help improve the robustness of malware detectors.


http://arxiv.org/abs/2003.05733
Towards Practical Lottery Ticket Hypothesis for Adversarial Training.
Bai Li; Shiqi Wang; Yunhan Jia; Yantao Lu; Zhenyu Zhong; Lawrence Carin; Suman Jana
  Recent research has proposed the lottery ticket hypothesis, suggesting that
for a deep neural network, there exist trainable sub-networks performing
equally or better than the original model with commensurate training steps.
While this discovery is insightful, finding proper sub-networks requires
iterative training and pruning. The high cost incurred limits the applications
of the lottery ticket hypothesis. We show there exists a subset of the
aforementioned sub-networks that converge significantly faster during the
training process and thus can mitigate the cost issue. We conduct extensive
experiments to show such sub-networks consistently exist across various model
structures for a restrictive setting of hyperparameters ($e.g.$, carefully
selected learning rate, pruning ratio, and model capacity). As a practical
application of our findings, we demonstrate that such sub-networks can help in
cutting down the total time of adversarial training, a standard approach to
improve robustness, by up to 49\% on CIFAR-10 to achieve the state-of-the-art
robustness.


http://arxiv.org/abs/2003.03021
Exploiting Verified Neural Networks via Floating Point Numerical Error.
Kai Jia; Martin Rinard
  We show how to construct adversarial examples for neural networks with
exactly verified robustness against $\ell_{\infty}$-bounded input perturbations
by exploiting floating point error. We argue that any exact verification of
real-valued neural networks must accurately model the implementation details of
any floating point arithmetic used during inference or verification.


http://arxiv.org/abs/2003.02732
Detection and Recovery of Adversarial Attacks with Injected Attractors.
Jiyi Zhang; Ee-Chien Chang; Hwee Kuan Lee
  Many machine learning adversarial attacks find adversarial samples of a
victim model ${\mathcal M}$ by following the gradient of some functions, either
explicitly or implicitly. To detect and recover from such attacks, we take the
proactive approach that modifies those functions with the goal of misleading
the attacks to some local minimals, or to some designated regions that can be
easily picked up by a forensic analyzer. To achieve the goal, we propose adding
a large number of artifacts, which we called $attractors$, onto the otherwise
smooth function. An attractor is a point in the input space, which has a
neighborhood of samples with gradients pointing toward it. We observe that
decoders of watermarking schemes exhibit properties of attractors, and give a
generic method that injects attractors from a watermark decoder into the victim
model ${\mathcal M}$. This principled approach allows us to leverage on known
watermarking schemes for scalability and robustness. Experimental studies show
that our method has competitive performance. For instance, for un-targeted
attacks on CIFAR-10 dataset, we can reduce the overall attack success rate of
DeepFool to 1.9%, whereas known defence LID, FS and MagNet can reduce the rate
to 90.8%, 98.5% and 78.5% respectively.


http://arxiv.org/abs/2003.02460
Adversarial Robustness Through Local Lipschitzness.
Yao-Yuan Yang; Cyrus Rashtchian; Hongyang Zhang; Ruslan Salakhutdinov; Kamalika Chaudhuri
  A standard method for improving the robustness of neural networks is
adversarial training, where the network is trained on adversarial examples that
are close to the training inputs. This produces classifiers that are robust,
but it often decreases clean accuracy. Prior work even posits that the tradeoff
between robustness and accuracy may be inevitable. We investigate this tradeoff
in more depth through the lens of local Lipschitzness. In many image datasets,
the classes are separated in the sense that images with different labels are
not extremely close in $\ell_\infty$ distance. Using this separation as a
starting point, we argue that it is possible to achieve both accuracy and
robustness by encouraging the classifier to be locally smooth around the data.
More precisely, we consider classifiers that are obtained by rounding locally
Lipschitz functions. Theoretically, we show that such classifiers exist for any
dataset such that there is a positive distance between the support of different
classes. Empirically, we compare the local Lipschitzness of classifiers trained
by several methods. Our results show that having a small Lipschitz constant
correlates with achieving high clean and robust accuracy, and therefore, the
smoothness of the classifier is an important property to consider in the
context of adversarial examples. Code available at
https://github.com/yangarbiter/robust-local-lipschitz .


http://arxiv.org/abs/2003.02484
Adversarial Vertex Mixup: Toward Better Adversarially Robust Generalization.
Saehyung Lee; Hyungyu Lee; Sungroh Yoon
  Adversarial examples cause neural networks to produce incorrect outputs with
high confidence. Although adversarial training is one of the most effective
forms of defense against adversarial examples, unfortunately, a large gap
exists between test accuracy and training accuracy in adversarial training. In
this paper, we identify Adversarial Feature Overfitting (AFO), which may cause
poor adversarially robust generalization, and we show that adversarial training
can overshoot the optimal point in terms of robust generalization, leading to
AFO in our simple Gaussian model. Considering these theoretical results, we
present soft labeling as a solution to the AFO problem. Furthermore, we propose
Adversarial Vertex mixup (AVmixup), a soft-labeled data augmentation approach
for improving adversarially robust generalization. We complement our
theoretical analysis with experiments on CIFAR10, CIFAR100, SVHN, and Tiny
ImageNet, and show that AVmixup significantly improves the robust
generalization performance and that it reduces the trade-off between standard
accuracy and adversarial robustness.


http://arxiv.org/abs/2003.02750
Search Space of Adversarial Perturbations against Image Filters.
Dang Duy Thang; Toshihiro Matsui
  The superiority of deep learning performance is threatened by safety issues
for itself. Recent findings have shown that deep learning systems are very weak
to adversarial examples, an attack form that was altered by the attacker's
intent to deceive the deep learning system. There are many proposed defensive
methods to protect deep learning systems against adversarial examples. However,
there is still a lack of principal strategies to deceive those defensive
methods. Any time a particular countermeasure is proposed, a new powerful
adversarial attack will be invented to deceive that countermeasure. In this
study, we focus on investigating the ability to create adversarial patterns in
search space against defensive methods that use image filters. Experimental
results conducted on the ImageNet dataset with image classification tasks
showed the correlation between the search space of adversarial perturbation and
filters. These findings open a new direction for building stronger offensive
methods towards deep learning systems.


http://arxiv.org/abs/2003.02301
Real-time, Universal, and Robust Adversarial Attacks Against Speaker Recognition Systems.
Yi Xie; Cong Shi; Zhuohang Li; Jian Liu; Yingying Chen; Bo Yuan
  As the popularity of voice user interface (VUI) exploded in recent years,
speaker recognition system has emerged as an important medium of identifying a
speaker in many security-required applications and services. In this paper, we
propose the first real-time, universal, and robust adversarial attack against
the state-of-the-art deep neural network (DNN) based speaker recognition
system. Through adding an audio-agnostic universal perturbation on arbitrary
enrolled speaker's voice input, the DNN-based speaker recognition system would
identify the speaker as any target (i.e., adversary-desired) speaker label. In
addition, we improve the robustness of our attack by modeling the sound
distortions caused by the physical over-the-air propagation through estimating
room impulse response (RIR). Experiment using a public dataset of $109$ English
speakers demonstrates the effectiveness and robustness of our proposed attack
with a high attack success rate of over 90%. The attack launching time also
achieves a 100X speedup over contemporary non-universal attacks.


http://arxiv.org/abs/2003.02188
Colored Noise Injection for Training Adversarially Robust Neural Networks.
Evgenii Zheltonozhskii; Chaim Baskin; Yaniv Nemcovsky; Brian Chmiel; Avi Mendelson; Alex M. Bronstein
  Even though deep learning have shown unmatched performance on various tasks,
neural networks has been shown to be vulnerable to small adversarial
perturbation of the input which lead to significant performance degradation. In
this work we extend the idea of adding independent Gaussian noise to weights
and activation during adversarial training (PNI) to injection of colored noise
for defense against common white-box and black-box attacks. We show that our
approach outperforms PNI and various previous approaches in terms of
adversarial accuracy on CIFAR-10 dataset. In addition, we provide an extensive
ablation study of the proposed method justifying the chosen configurations.


http://arxiv.org/abs/2003.01895
Double Backpropagation for Training Autoencoders against Adversarial Attack.
Chengjin Sun; Sizhe Chen; Xiaolin Huang
  Deep learning, as widely known, is vulnerable to adversarial samples. This
paper focuses on the adversarial attack on autoencoders. Safety of the
autoencoders (AEs) is important because they are widely used as a compression
scheme for data storage and transmission, however, the current autoencoders are
easily attacked, i.e., one can slightly modify an input but has totally
different codes. The vulnerability is rooted the sensitivity of the
autoencoders and to enhance the robustness, we propose to adopt double
backpropagation (DBP) to secure autoencoder such as VAE and DRAW. We restrict
the gradient from the reconstruction image to the original one so that the
autoencoder is not sensitive to trivial perturbation produced by the
adversarial attack. After smoothing the gradient by DBP, we further smooth the
label by Gaussian Mixture Model (GMM), aiming for accurate and robust
classification. We demonstrate in MNIST, CelebA, SVHN that our method leads to
a robust autoencoder resistant to attack and a robust classifier able for image
transition and immune to adversarial attack if combined with GMM.


http://arxiv.org/abs/2003.01908
Black-box Smoothing: A Provable Defense for Pretrained Classifiers.
Hadi Salman; Mingjie Sun; Greg Yang; Ashish Kapoor; J. Zico Kolter
  We present a method for provably defending any pretrained image classifier
against $\ell_p$ adversarial attacks. By prepending a custom-trained denoiser
to any off-the-shelf image classifier and using randomized smoothing, we
effectively create a new classifier that is guaranteed to be $\ell_p$-robust to
adversarial examples, without modifying the pretrained classifier. The approach
applies both to the case where we have full access to the pretrained classifier
as well as the case where we only have query access. We refer to this defense
as black-box smoothing, and we demonstrate its effectiveness through extensive
experimentation on ImageNet and CIFAR-10. Finally, we use our method to
provably defend the Azure, Google, AWS, and ClarifAI image classification APIs.
Our code replicating all the experiments in the paper can be found at
https://github.com/microsoft/blackbox-smoothing .


http://arxiv.org/abs/2003.01993
Metrics and methods for robustness evaluation of neural networks with generative models.
Igor Buzhinsky; Arseny Nerinovsky; Stavros Tripakis
  Recent studies have shown that modern deep neural network classifiers are
easy to fool, assuming that an adversary is able to slightly modify their
inputs. Many papers have proposed adversarial attacks, defenses and methods to
measure robustness to such adversarial perturbations. However, most commonly
considered adversarial examples are based on $\ell_p$-bounded perturbations in
the input space of the neural network, which are unlikely to arise naturally.
Recently, especially in computer vision, researchers discovered "natural" or
"semantic" perturbations, such as rotations, changes of brightness, or more
high-level changes, but these perturbations have not yet been systematically
utilized to measure the performance of classifiers. In this paper, we propose
several metrics to measure robustness of classifiers to natural adversarial
examples, and methods to evaluate them. These metrics, called latent space
performance metrics, are based on the ability of generative models to capture
probability distributions, and are defined in their latent spaces. On three
image classification case studies, we evaluate the proposed metrics for several
classifiers, including ones trained in conventional and robust ways. We find
that the latent counterparts of adversarial robustness are associated with the
accuracy of the classifier rather than its conventional adversarial robustness,
but the latter is still reflected on the properties of found latent
perturbations. In addition, our novel method of finding latent adversarial
perturbations demonstrates that these perturbations are often perceptually
small.


http://arxiv.org/abs/2003.01690
Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.
Francesco Croce; Matthias Hein
  The field of defense strategies against adversarial attacks has significantly
grown over the last years, but progress is hampered as the evaluation of
adversarial defenses is often insufficient and thus gives a wrong impression of
robustness. Many promising defenses could be broken later on, making it
difficult to identify the state-of-the-art. Frequent pitfalls in the evaluation
are improper tuning of hyperparameters of the attacks, gradient obfuscation or
masking. In this paper we first propose two extensions of the PGD-attack
overcoming failures due to suboptimal step size and problems of the objective
function. We then combine our novel attacks with two complementary existing
ones to form a parameter-free, computationally affordable and user-independent
ensemble of attacks to test adversarial robustness. We apply our ensemble to
over 40 models from papers published at recent top machine learning and
computer vision venues. In all except one of the cases we achieve lower robust
test accuracy than reported in these papers, often by more than $10\%$,
identifying several broken defenses.


http://arxiv.org/abs/2003.01595
Analyzing Accuracy Loss in Randomized Smoothing Defenses.
Yue Gao; Harrison Rosenberg; Kassem Fawaz; Somesh Jha; Justin Hsu
  Recent advances in machine learning (ML) algorithms, especially deep neural
networks (DNNs), have demonstrated remarkable success (sometimes exceeding
human-level performance) on several tasks, including face and speech
recognition. However, ML algorithms are vulnerable to \emph{adversarial
attacks}, such test-time, training-time, and backdoor attacks. In test-time
attacks an adversary crafts adversarial examples, which are specially crafted
perturbations imperceptible to humans which, when added to an input example,
force a machine learning model to misclassify the given input example.
Adversarial examples are a concern when deploying ML algorithms in critical
contexts, such as information security and autonomous driving. Researchers have
responded with a plethora of defenses. One promising defense is
\emph{randomized smoothing} in which a classifier's prediction is smoothed by
adding random noise to the input example we wish to classify. In this paper, we
theoretically and empirically explore randomized smoothing. We investigate the
effect of randomized smoothing on the feasible hypotheses space, and show that
for some noise levels the set of hypotheses which are feasible shrinks due to
smoothing, giving one reason why the natural accuracy drops after smoothing. To
perform our analysis, we introduce a model for randomized smoothing which
abstracts away specifics, such as the exact distribution of the noise. We
complement our theoretical results with extensive experiments.


http://arxiv.org/abs/2003.01782
Security of Deep Learning based Lane Keeping System under Physical-World Adversarial Attack.
Takami Sato; Junjie Shen; Ningfei Wang; Yunhan Jack Jia; Xue Lin; Qi Alfred Chen
  Lane-Keeping Assistance System (LKAS) is convenient and widely available
today, but also extremely security and safety critical. In this work, we design
and implement the first systematic approach to attack real-world DNN-based
LKASes. We identify dirty road patches as a novel and domain-specific threat
model for practicality and stealthiness. We formulate the attack as an
optimization problem, and address the challenge from the inter-dependencies
among attacks on consecutive camera frames. We evaluate our approach on a
state-of-the-art LKAS and our preliminary results show that our attack can
successfully cause it to drive off lane boundaries within as short as 1.3
seconds.


http://arxiv.org/abs/2003.01872
Type I Attack for Generative Models.
Chengjin Sun; Sizhe Chen; Jia Cai; Xiaolin Huang
  Generative models are popular tools with a wide range of applications.
Nevertheless, it is as vulnerable to adversarial samples as classifiers. The
existing attack methods mainly focus on generating adversarial examples by
adding imperceptible perturbations to input, which leads to wrong result.
However, we focus on another aspect of attack, i.e., cheating models by
significant changes. The former induces Type II error and the latter causes
Type I error. In this paper, we propose Type I attack to generative models such
as VAE and GAN. One example given in VAE is that we can change an original
image significantly to a meaningless one but their reconstruction results are
similar. To implement the Type I attack, we destroy the original one by
increasing the distance in input space while keeping the output similar because
different inputs may correspond to similar features for the property of deep
neural network. Experimental results show that our attack method is effective
to generate Type I adversarial examples for generative models on large-scale
image datasets.


http://arxiv.org/abs/2003.01249
Rethinking Randomized Smoothing for Adversarial Robustness.
Jeet Lily Mohapatra; Ching-Yun Lily Ko; Lily Tsui-Wei; Weng; Sijia Liu; Pin-Yu Chen; Luca Daniel
  The fragility of modern machine learning models has drawn a considerable
amount of attention from both academia and the public. While immense interests
were in either crafting adversarial attacks as a way to measure the robustness
of neural networks or devising worst-case analytical robustness verification
with guarantees, few methods could enjoy both scalability and robustness
guarantees at the same time. As an alternative to these attempts, randomized
smoothing adopts a different prediction rule that enables statistical
robustness arguments and can scale to large networks. However, in this paper,
we point out for the first time the side effects of current randomized
smoothing workflows. Specifically, we articulate and prove two major points: 1)
the decision boundaries shrink with the adoption of randomized smoothing
prediction rule; 2) noise augmentation does not necessarily resolve the
shrinking issue and can even create additional issues.


http://arxiv.org/abs/2003.01295
Data-Free Adversarial Perturbations for Practical Black-Box Attack.
ZhaoXin Huan; Yulong Wang; Xiaolu Zhang; Lin Shang; Chilin Fu; Jun Zhou
  Neural networks are vulnerable to adversarial examples, which are malicious
inputs crafted to fool pre-trained models. Adversarial examples often exhibit
black-box attacking transferability, which allows that adversarial examples
crafted for one model can fool another model. However, existing black-box
attack methods require samples from the training data distribution to improve
the transferability of adversarial examples across different models. Because of
the data dependence, the fooling ability of adversarial perturbations is only
applicable when training data are accessible. In this paper, we present a
data-free method for crafting adversarial perturbations that can fool a target
model without any knowledge about the training data distribution. In the
practical setting of a black-box attack scenario where attackers do not have
access to target models and training data, our method achieves high fooling
rates on target models and outperforms other universal adversarial perturbation
methods. Our method empirically shows that current deep learning models are
still at risk even when the attackers do not have access to training data.


http://arxiv.org/abs/2003.01090
Learn2Perturb: an End-to-end Feature Perturbation Learning to Improve Adversarial Robustness.
Ahmadreza Jeddi; Mohammad Javad Shafiee; Michelle Karg; Christian Scharfenberger; Alexander Wong
  While deep neural networks have been achieving state-of-the-art performance
across a wide variety of applications, their vulnerability to adversarial
attacks limits their widespread deployment for safety-critical applications.
Alongside other adversarial defense approaches being investigated, there has
been a very recent interest in improving adversarial robustness in deep neural
networks through the introduction of perturbations during the training process.
However, such methods leverage fixed, pre-defined perturbations and require
significant hyper-parameter tuning that makes them very difficult to leverage
in a general fashion. In this study, we introduce Learn2Perturb, an end-to-end
feature perturbation learning approach for improving the adversarial robustness
of deep neural networks. More specifically, we introduce novel
perturbation-injection modules that are incorporated at each layer to perturb
the feature space and increase uncertainty in the network. This feature
perturbation is performed at both the training and the inference stages.
Furthermore, inspired by the Expectation-Maximization, an alternating
back-propagation training algorithm is introduced to train the network and
noise parameters consecutively. Experimental results on CIFAR-10 and CIFAR-100
datasets show that the proposed Learn2Perturb method can result in deep neural
networks which are $4-7\%$ more robust on $l_{\infty}$ FGSM and PDG adversarial
attacks and significantly outperforms the state-of-the-art against $l_2$ $C\&W$
attack and a wide range of well-known black-box attacks.


http://arxiv.org/abs/2003.01279
Disrupting Deepfakes: Adversarial Attacks Against Conditional Image Translation Networks and Facial Manipulation Systems.
Nataniel Ruiz; Sarah Adel Bargal; Stan Sclaroff
  Face modification systems using deep learning have become increasingly
powerful and accessible. Given images of a person's face, such systems can
generate new images of that same person under different expressions and poses.
Some systems can also modify targeted attributes such as hair color or age.
This type of manipulated images and video have been coined Deepfakes. In order
to prevent a malicious user from generating modified images of a person without
their consent we tackle the new problem of generating adversarial attacks
against such image translation systems, which disrupt the resulting output
image. We call this problem disrupting deepfakes. Most image translation
architectures are generative models conditioned on an attribute (e.g. put a
smile on this person's face). We are first to propose and successfully apply
(1) class transferable adversarial attacks that generalize to different
classes, which means that the attacker does not need to have knowledge about
the conditioning class, and (2) adversarial training for generative adversarial
networks (GANs) as a first step towards robust image translation networks.
Finally, in gray-box scenarios, blurring can mount a successful defense against
disruption. We present a spread-spectrum adversarial attack, which evades blur
defenses.


http://arxiv.org/abs/2003.01261
Adversarial Network Traffic: Towards Evaluating the Robustness of Deep Learning-Based Network Traffic Classification.
Amir Mahdi Sadeghzadeh; Rasool Jalili; Saeed Shiravi
  Network traffic classification is used in various applications such as
network traffic management, policy enforcement, and intrusion detection
systems. Although most applications encrypt their network traffic and some of
them dynamically change their port numbers, Machine Learning (ML) and
especially Deep Learning (DL)-based classifiers have shown impressive
performance in network traffic classification. In this paper, we evaluate the
robustness of DL-based network traffic classifiers against Adversarial Network
Traffic (ANT). ANT causes DL-based network traffic classifiers to predict
incorrectly using Universal Adversarial Perturbation (UAP) generating methods.
Since there is no need to buffer network traffic before sending ANT, it is
generated live. Also, ANT is independent of applications that generate network
traffic. We partition the input space of the DL-based network traffic
classification into three categories: packet classification, flow content
classification, and flow time series classification. To generate ANT, we
propose three new attacks injecting UAP into network traffic. AdvPad attack
injects a UAP into the content of packets to evaluate the robustness of packet
classifiers. AdvPay attack injects a UAP into the payload of a dummy packet to
evaluate the robustness of flow content classifiers. AdvBurst attack injects a
specific number of dummy packets with crafted statistical features based on a
UAP into a selected burst of a flow to evaluate the robustness of flow time
series classifiers. The results indicate injecting a little UAP into network
traffic, highly decreases the performance of DL-based network traffic
classifiers in all categories.


http://arxiv.org/abs/2003.00653
Adversarial Attacks and Defenses on Graphs: A Review and Empirical Study.
Wei Jin; Yaxin Li; Han Xu; Yiqi Wang; Jiliang Tang
  Deep neural networks (DNNs) have achieved significant performance in various
tasks. However, recent studies have shown that DNNs can be easily fooled by
small perturbation on the input, called adversarial attacks. As the extensions
of DNNs to graphs, Graph Neural Networks (GNNs) have been demonstrated to
inherit this vulnerability. Adversary can mislead GNNs to give wrong
predictions by modifying the graph structure such as manipulating a few edges.
This vulnerability has arisen tremendous concerns for adapting GNNs in
safety-critical applications and has attracted increasing research attention in
recent years. Thus, it is necessary and timely to provide a comprehensive
overview of existing graph adversarial attacks and the countermeasures. In this
survey, we categorize existing attacks and defenses, and review the
corresponding state-of-the-art methods. Furthermore, we have developed a
repository with representative algorithms
(https://github.com/DSE-MSU/DeepRobust/tree/master/deeprobust/graph). The
repository enables us to conduct empirical studies to deepen our understandings
on attacks and defenses on graphs.


http://arxiv.org/abs/2003.00378
Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Models.
Xiao Zhang; Jinghui Chen; Quanquan Gu; David Evans
  Starting with Gilmer et al. (2018), several works have demonstrated the
inevitability of adversarial examples based on different assumptions about the
underlying input probability space. It remains unclear, however, whether these
results apply to natural image distributions. In this work, we assume the
underlying data distribution is captured by some conditional generative model,
and prove intrinsic robustness bounds for a general class of classifiers, which
solves an open problem in Fawzi et al. (2018). Building upon the
state-of-the-art conditional generative models, we study the intrinsic
robustness of two common image benchmarks under $\ell_2$ perturbations, and
show the existence of a large gap between the robustness limits implied by our
theory and the adversarial robustness achieved by current state-of-the-art
robust models. Code for all our experiments is available at
https://github.com/xiaozhanguva/Intrinsic-Rob.


http://arxiv.org/abs/2003.00402
Why is the Mahalanobis Distance Effective for Anomaly Detection?
Ryo Kamoi; Kei Kobayashi
  The Mahalanobis distance-based confidence score, a recently proposed anomaly
detection method for pre-trained neural classifiers, achieves state-of-the-art
performance on both out-of-distribution (OoD) and adversarial examples
detection. This work analyzes why this method exhibits such strong performance
in practical settings while imposing an implausible assumption; namely, that
class conditional distributions of pre-trained features have tied covariance.
Although the Mahalanobis distance-based method is claimed to be motivated by
classification prediction confidence, we find that its superior performance
stems from information not useful for classification. This suggests that the
reason the Mahalanobis confidence score works so well is mistaken, and makes
use of different information from ODIN, another popular OoD detection method
based on prediction confidence. This perspective motivates us to combine these
two methods, and the combined detector exhibits improved performance and
robustness. These findings provide insight into the behavior of neural
classifiers in response to anomalous inputs.


http://arxiv.org/abs/2003.00120
End-to-end Robustness for Sensing-Reasoning Machine Learning Pipelines.
Zhuolin Yang; Zhikuan Zhao; Hengzhi Pei; Boxin Wang; Bojan Karlas; Ji Liu; Heng Guo; Bo Li; Ce Zhang
  As machine learning (ML) being applied to many mission-critical scenarios,
certifying ML model robustness becomes increasingly important. Many previous
works focuses on the robustness of independent ML and ensemble models, and can
only certify a very small magnitude of the adversarial perturbation. In this
paper, we take a different viewpoint and improve learning robustness by going
beyond independent ML and ensemble models. We aim at promoting the generic
Sensing-Reasoning machine learning pipeline which contains both the sensing
(e.g. deep neural networks) and reasoning (e.g. Markov logic networks (MLN))
components enriched with domain knowledge. Can domain knowledge help improve
learning robustness? Can we formally certify the end-to-end robustness of such
an ML pipeline?
  We first theoretically analyze the computational complexity of checking the
provable robustness in the reasoning component. We then derive the provable
robustness bound for several concrete reasoning components. We show that for
reasoning components such as MLN and a specific family of Bayesian networks it
is possible to certify the robustness of the whole pipeline even with a large
magnitude of perturbation which cannot be certified by existing work. Finally,
we conduct extensive real-world experiments on large scale datasets to evaluate
the certified robustness for Sensing-Reasoning ML pipelines.


http://arxiv.org/abs/2002.12913
Applying Tensor Decomposition to image for Robustness against Adversarial Attack.
Seungju Cho; Tae Joon Jun; Mingu Kang; Daeyoung Kim
  Nowadays the deep learning technology is growing faster and shows dramatic
performance in computer vision areas. However, it turns out a deep learning
based model is highly vulnerable to some small perturbation called an
adversarial attack. It can easily fool the deep learning model by adding small
perturbations. On the other hand, tensor decomposition method widely uses for
compressing the tensor data, including data matrix, image, etc. In this paper,
we suggest combining tensor decomposition for defending the model against
adversarial example. We verify this idea is simple and effective to resist
adversarial attack. In addition, this method rarely degrades the original
performance of clean data. We experiment on MNIST, CIFAR10 and ImageNet data
and show our method robust on state-of-the-art attack methods.


http://arxiv.org/abs/2003.04985
Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT.
Lichao Sun; Kazuma Hashimoto; Wenpeng Yin; Akari Asai; Jia Li; Philip Yu; Caiming Xiong
  There is an increasing amount of literature that claims the brittleness of
deep neural networks in dealing with adversarial examples that are created
maliciously. It is unclear, however, how the models will perform in realistic
scenarios where \textit{natural rather than malicious} adversarial instances
often exist. This work systematically explores the robustness of BERT, the
state-of-the-art Transformer-style model in NLP, in dealing with noisy data,
particularly mistakes in typing the keyboard, that occur inadvertently.
Intensive experiments on sentiment analysis and question answering benchmarks
indicate that: (i) Typos in various words of a sentence do not influence
equally. The typos in informative words make severer damages; (ii) Mistype is
the most damaging factor, compared with inserting, deleting, etc.; (iii) Humans
and machines have different focuses on recognizing adversarial attacks.


http://arxiv.org/abs/2002.12520
Utilizing Network Properties to Detect Erroneous Inputs.
Matt Gorbett; Nathaniel Blanchard
  Neural networks are vulnerable to a wide range of erroneous inputs such as
adversarial, corrupted, out-of-distribution, and misclassified examples. In
this work, we train a linear SVM classifier to detect these four types of
erroneous data using hidden and softmax feature vectors of pre-trained neural
networks. Our results indicate that these faulty data types generally exhibit
linearly separable activation properties from correct examples, giving us the
ability to reject bad inputs with no extra training or overhead. We
experimentally validate our findings across a diverse range of datasets,
domains, pre-trained models, and adversarial attacks.


http://arxiv.org/abs/2002.12504
Detecting Patch Adversarial Attacks with Image Residuals.
Marius Arvinte; Ahmed Tewfik; Sriram Vishwanath
  We introduce an adversarial sample detection algorithm based on image
residuals, specifically designed to guard against patch-based attacks. The
image residual is obtained as the difference between an input image and a
denoised version of it, and a discriminator is trained to distinguish between
clean and adversarial samples. More precisely, we use a wavelet domain
algorithm for denoising images and demonstrate that the obtained residuals act
as a digital fingerprint for adversarial attacks. To emulate the limitations of
a physical adversary, we evaluate the performance of our approach against
localized (patch-based) adversarial attacks, including in settings where the
adversary has complete knowledge about the detection scheme. Our results show
that the proposed detection method generalizes to previously unseen, stronger
attacks and that it is able to reduce the success rate (conversely, increase
the computational effort) of an adaptive attacker.


http://arxiv.org/abs/2002.12527
Detecting and Recovering Adversarial Examples: An Input Sensitivity Guided Method.
Mingxuan Li; Jingyuan Wang; Yufan Wu; Shuchang Zhou
  Deep neural networks undergo rapid development and achieve notable success in
various tasks, including many security concerned scenarios. However, a
considerable amount of works have proved its vulnerability in adversaries. To
address this problem, we propose a Guided Robust and Efficient Defensive Model
GRED integrating detection and recovery processes together. From the lens of
the properties of gradient distribution of adversarial examples, our model
detects malicious inputs effectively, as well as recovering the ground-truth
label with high accuracy. Compared with commonly used adversarial training
methods, our model is more efficient and outperforms state-of-the-art
adversarial trained models by a large margin up to 99% on MNIST, 89 % on
CIFAR-10 and 87% on ImageNet subsets. When exclusively compared with previous
adversarial detection methods, the detector of GRED is robust under all threat
settings with a detection rate of over 95% against most of the attacks. It is
also demonstrated by empirical assessment that our model could increase
attacking cost significantly resulting in either unacceptable time consuming or
human perceptible image distortions.


http://arxiv.org/abs/2002.12398
Provable Robust Learning Based on Transformation-Specific Smoothing.
Linyi Li; Maurice Weber; Xiaojun Xu; Luka Rimanic; Tao Xie; Ce Zhang; Bo Li
  As machine learning systems become pervasive, safeguarding their security is
critical. Recent work has demonstrated that motivated adversaries could
manipulate the test data to mislead ML systems to make arbitrary mistakes. So
far, most research has focused on providing provable robustness guarantees for
a specific $\ell_p$ norm bounded adversarial perturbation. However, in practice
there are more adversarial transformations that are realistic and of semantic
meaning, requiring to be analyzed and ideally certified. In this paper we aim
to provide a unified framework for certifying ML model robustness against
general adversarial transformations. First, we leverage the function smoothing
strategy to certify robustness against a series of adversarial transformations
such as rotation, translation, Gaussian blur, etc. We then provide sufficient
conditions and strategies for certifying certain transformations. For instance,
we propose a novel sampling based interpolation approach with the estimated
Lipschitz upper bound to certify the robustness against rotation
transformation. In addition, we theoretically optimize the smoothing strategies
for certifying the robustness of ML models against different transformations.
For instance, we show that smoothing by sampling from exponential distribution
provides tighter robustness bound than Gaussian. We also prove two
generalization gaps for the proposed framework to understand its theoretic
barrier. Extensive experiments show that our proposed unified framework
significantly outperforms the state-of-the-art certified robustness approaches
on several datasets including ImageNet.


http://arxiv.org/abs/2002.12222
On Isometry Robustness of Deep 3D Point Cloud Models under Adversarial Attacks.
Yue Zhao; Yuwei Wu; Caihua Chen; Andrew Lim
  While deep learning in 3D domain has achieved revolutionary performance in
many tasks, the robustness of these models has not been sufficiently studied or
explored. Regarding the 3D adversarial samples, most existing works focus on
manipulation of local points, which may fail to invoke the global geometry
properties, like robustness under linear projection that preserves the
Euclidean distance, i.e., isometry. In this work, we show that existing
state-of-the-art deep 3D models are extremely vulnerable to isometry
transformations. Armed with the Thompson Sampling, we develop a black-box
attack with success rate over 95\% on ModelNet40 data set. Incorporating with
the Restricted Isometry Property, we propose a novel framework of white-box
attack on top of spectral norm based perturbation. In contrast to previous
works, our adversarial samples are experimentally shown to be strongly
transferable. Evaluated on a sequence of prevailing 3D models, our white-box
attack achieves success rates from 98.88\% to 100\%. It maintains a successful
attack rate over 95\% even within an imperceptible rotation range $[\pm
2.81^{\circ}]$.


http://arxiv.org/abs/2002.11798
Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization.
Sicheng Zhu; Xiao Zhang; David Evans
  Training machine learning models to be robust against adversarial inputs
poses seemingly insurmountable challenges. To better understand model
robustness, we consider the underlying problem of learning robust
representations. We develop a general definition of representation
vulnerability that captures the maximum change of mutual information between
the input and output distributions, under the worst-case input distribution
perturbation. We prove a theorem that establishes a lower bound on the minimum
adversarial risk that can be achieved for any downstream classifier based on
this definition. We then propose an unsupervised learning method for obtaining
intrinsically robust representations by maximizing the worst-case mutual
information between input and output distributions. Experiments on downstream
classification tasks and analyses of saliency maps support the robustness of
the representations found using unsupervised learning with our training
principle.


http://arxiv.org/abs/2002.11572
Revisiting Ensembles in an Adversarial Context: Improving Natural Accuracy.
Aditya Saligrama; Guillaume Leclerc
  A necessary characteristic for the deployment of deep learning models in real
world applications is resistance to small adversarial perturbations while
maintaining accuracy on non-malicious inputs. While robust training provides
models that exhibit better adversarial accuracy than standard models, there is
still a significant gap in natural accuracy between robust and non-robust
models which we aim to bridge. We consider a number of ensemble methods
designed to mitigate this performance difference. Our key insight is that model
trained to withstand small attacks, when ensembled, can often withstand
significantly larger attacks, and this concept can in turn be leveraged to
optimize natural accuracy. We consider two schemes, one that combines
predictions from several randomly initialized robust models, and the other that
fuses features from robust and standard models.


http://arxiv.org/abs/2002.11565
Randomization matters. How to defend against strong adversarial attacks.
Rafael Pinot; Raphael Ettedgui; Geovani Rizk; Yann Chevaleyre; Jamal Atif
  Is there a classifier that ensures optimal robustness against all adversarial
attacks? This paper answers this question by adopting a game-theoretic point of
view. We show that adversarial attacks and defenses form an infinite zero-sum
game where classical results (e.g. Sion theorem) do not apply. We demonstrate
the non-existence of a Nash equilibrium in our game when the classifier and the
Adversary are both deterministic, hence giving a negative answer to the above
question in the deterministic regime. Nonetheless, the question remains open in
the randomized regime. We tackle this problem by showing that, undermild
conditions on the dataset distribution, any deterministic classifier can be
outperformed by a randomized one. This gives arguments for using randomization,
and leads us to a new algorithm for building randomized classifiers that are
robust to strong adversarial attacks. Empirical results validate our
theoretical analysis, and show that our defense method considerably outperforms
Adversarial Training against state-of-the-art attacks.


http://arxiv.org/abs/2002.11318
Invariance vs. Robustness of Neural Networks.
Sandesh Kamath; Amit Deshpande; K V Subrahmanyam
  We study the performance of neural network models on random geometric
transformations and adversarial perturbations. Invariance means that the
model's prediction remains unchanged when a geometric transformation is applied
to an input. Adversarial robustness means that the model's prediction remains
unchanged after small adversarial perturbations of an input. In this paper, we
show a quantitative trade-off between rotation invariance and robustness. We
empirically study the following two cases: (a) change in adversarial robustness
as we improve only the invariance of equivariant models via training
augmentation, (b) change in invariance as we improve only the adversarial
robustness using adversarial training. We observe that the rotation invariance
of equivariant models (StdCNNs and GCNNs) improves by training augmentation
with progressively larger random rotations but while doing so, their
adversarial robustness drops progressively, and very significantly on MNIST. We
take adversarially trained LeNet and ResNet models which have good $L_\infty$
adversarial robustness on MNIST and CIFAR-10, respectively, and observe that
adversarial training with progressively larger perturbations results in a
progressive drop in their rotation invariance profiles. Similar to the
trade-off between accuracy and robustness known in previous work, we give a
theoretical justification for the invariance vs. robustness trade-off observed
in our experiments.


http://arxiv.org/abs/2002.11569
Overfitting in adversarially robust deep learning.
Leslie Rice; Eric Wong; J. Zico Kolter
  It is common practice in deep learning to use overparameterized networks and
train for as long as possible; there are numerous studies that show, both
theoretically and empirically, that such practices surprisingly do not unduly
harm the generalization performance of the classifier. In this paper, we
empirically study this phenomenon in the setting of adversarially trained deep
networks, which are trained to minimize the loss under worst-case adversarial
perturbations. We find that overfitting to the training set does in fact harm
robust performance to a very large degree in adversarially robust training
across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and ImageNet) and
perturbation models ($\ell_\infty$ and $\ell_2$). Based upon this observed
effect, we show that the performance gains of virtually all recent algorithmic
improvements upon adversarial training can be matched by simply using early
stopping. We also show that effects such as the double descent curve do still
occur in adversarially trained models, yet fail to explain the observed
overfitting. Finally, we study several classical and modern deep learning
remedies for overfitting, including regularization and data augmentation, and
find that no approach in isolation improves significantly upon the gains
achieved by early stopping. All code for reproducing the experiments as well as
pretrained model weights and training logs can be found at
https://github.com/locuslab/robust_overfitting.


http://arxiv.org/abs/2002.11320
MGA: Momentum Gradient Attack on Network.
Jinyin Chen; Yixian Chen; Haibin Zheng; Shijing Shen; Shanqing Yu; Dan Zhang; Qi Xuan
  The adversarial attack methods based on gradient information can adequately
find the perturbations, that is, the combinations of rewired links, thereby
reducing the effectiveness of the deep learning model based graph embedding
algorithms, but it is also easy to fall into a local optimum. Therefore, this
paper proposes a Momentum Gradient Attack (MGA) against the GCN model, which
can achieve more aggressive attacks with fewer rewiring links. Compared with
directly updating the original network using gradient information, integrating
the momentum term into the iterative process can stabilize the updating
direction, which makes the model jump out of poor local optimum and enhance the
method with stronger transferability. Experiments on node classification and
community detection methods based on three well-known network embedding
algorithms show that MGA has a better attack effect and transferability.


http://arxiv.org/abs/2002.11821
Improving Robustness of Deep-Learning-Based Image Reconstruction.
Ankit Raj; Yoram Bresler; Bo Li
  Deep-learning-based methods for different applications have been shown
vulnerable to adversarial examples. These examples make deployment of such
models in safety-critical tasks questionable. Use of deep neural networks as
inverse problem solvers has generated much excitement for medical imaging
including CT and MRI, but recently a similar vulnerability has also been
demonstrated for these tasks. We show that for such inverse problem solvers,
one should analyze and study the effect of adversaries in the
measurement-space, instead of the signal-space as in previous work. In this
paper, we propose to modify the training strategy of end-to-end
deep-learning-based inverse problem solvers to improve robustness. We introduce
an auxiliary network to generate adversarial examples, which is used in a
min-max formulation to build robust image reconstruction networks.
Theoretically, we show for a linear reconstruction scheme the min-max
formulation results in a singular-value(s) filter regularized solution, which
suppresses the effect of adversarial examples occurring because of
ill-conditioning in the measurement matrix. We find that a linear network using
the proposed min-max learning scheme indeed converges to the same solution. In
addition, for non-linear Compressed Sensing (CS) reconstruction using deep
networks, we show significant improvement in robustness using the proposed
approach over other methods. We complement the theory by experiments for CS on
two different datasets and evaluate the effect of increasing perturbations on
trained networks. We find the behavior for ill-conditioned and well-conditioned
measurement matrices to be qualitatively different.


http://arxiv.org/abs/2002.11881
Defense-PointNet: Protecting PointNet Against Adversarial Attacks.
Yu Zhang; Gongbo Liang; Tawfiq Salem; Nathan Jacobs
  Despite remarkable performance across a broad range of tasks, neural networks
have been shown to be vulnerable to adversarial attacks. Many works focus on
adversarial attacks and defenses on 2D images, but few focus on 3D point
clouds. In this paper, our goal is to enhance the adversarial robustness of
PointNet, which is one of the most widely used models for 3D point clouds. We
apply the fast gradient sign attack method (FGSM) on 3D point clouds and find
that FGSM can be used to generate not only adversarial images but also
adversarial point clouds. To minimize the vulnerability of PointNet to
adversarial attacks, we propose Defense-PointNet. We compare our model with two
baseline approaches and show that Defense-PointNet significantly improves the
robustness of the network against adversarial samples.


http://arxiv.org/abs/2002.11374
Adversarial Attack on Deep Product Quantization Network for Image Retrieval.
Yan Feng; Bin Chen; Tao Dai; Shutao Xia
  Deep product quantization network (DPQN) has recently received much attention
in fast image retrieval tasks due to its efficiency of encoding
high-dimensional visual features especially when dealing with large-scale
datasets. Recent studies show that deep neural networks (DNNs) are vulnerable
to input with small and maliciously designed perturbations (a.k.a., adversarial
examples). This phenomenon raises the concern of security issues for DPQN in
the testing/deploying stage as well. However, little effort has been devoted to
investigating how adversarial examples affect DPQN. To this end, we propose
product quantization adversarial generation (PQ-AG), a simple yet effective
method to generate adversarial examples for product quantization based
retrieval systems. PQ-AG aims to generate imperceptible adversarial
perturbations for query images to form adversarial queries, whose nearest
neighbors from a targeted product quantizaiton model are not semantically
related to those from the original queries. Extensive experiments show that our
PQ-AQ successfully creates adversarial examples to mislead targeted product
quantization retrieval models. Besides, we found that our PQ-AG significantly
degrades retrieval performance in both white-box and black-box settings.


http://arxiv.org/abs/2002.10716
Understanding and Mitigating the Tradeoff Between Robustness and Accuracy.
Aditi Raghunathan; Sang Michael Xie; Fanny Yang; John Duchi; Percy Liang
  Adversarial training augments the training set with perturbations to improve
the robust error (over worst-case perturbations), but it often leads to an
increase in the standard error (on unperturbed test inputs). Previous
explanations for this tradeoff rely on the assumption that no predictor in the
hypothesis class has low standard and robust error. In this work, we precisely
characterize the effect of augmentation on the standard error in linear
regression when the optimal linear predictor has zero standard and robust
error. In particular, we show that the standard error could increase even when
the augmented perturbations have noiseless observations from the optimal linear
predictor. We then prove that the recently proposed robust self-training (RST)
estimator improves robust error without sacrificing standard error for
noiseless linear regression. Empirically, for neural networks, we find that RST
with different adversarial training methods improves both standard and robust
error for random and adversarial rotations and adversarial $\ell_\infty$
perturbations in CIFAR-10.


http://arxiv.org/abs/2002.11080
The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization.
Yifei Min; Lin Chen; Amin Karbasi
  Despite remarkable success, deep neural networks are sensitive to
human-imperceptible small perturbations on the data and could be adversarially
misled to produce incorrect or even dangerous predictions. To circumvent these
issues, practitioners introduced adversarial training to produce adversarially
robust models whose predictions are robust to small perturbations to the data.
It is widely believed that more training data will help adversarially robust
models generalize better on the test data. In this paper, however, we challenge
this conventional belief and show that more training data could hurt the
generalization of adversarially robust models for the linear classification
problem. We identify three regimes based on the strength of the adversary. In
the weak adversary regime, more data improves the generalization of
adversarially robust models. In the medium adversary regime, with more training
data, the generalization loss exhibits a double descent curve. This implies
that in this regime, there is an intermediate stage where more training data
hurts their generalization. In the strong adversary regime, more data almost
immediately causes the generalization error to increase.


http://arxiv.org/abs/2002.10703
G\"odel's Sentence Is An Adversarial Example But Unsolvable.
Xiaodong Qi; Lansheng Han
  In recent years, different types of adversarial examples from different
fields have emerged endlessly, including purely natural ones without
perturbations. A variety of defenses are proposed and then broken quickly. Two
fundamental questions need to be asked: What's the reason for the existence of
adversarial examples and are adversarial examples unsolvable? In this paper, we
will show the reason for the existence of adversarial examples is there are
non-isomorphic natural explanations that can all explain data set.
Specifically, for two natural explanations of being true and provable,
G\"odel's sentence is an adversarial example but ineliminable. It can't be
solved by the re-accumulation of data set or the re-improvement of learning
algorithm. Finally, from the perspective of computability, we will prove the
incomputability for adversarial examples, which are unrecognizable.


http://arxiv.org/abs/2002.10733
(De)Randomized Smoothing for Certifiable Defense against Patch Attacks.
Alexander Levine; Soheil Feizi
  Patch adversarial attacks on images, in which the attacker can distort pixels
within a region of bounded size, are an important threat model since they
provide a quantitative model for physical adversarial attacks. In this paper,
we introduce a certifiable defense against patch attacks that guarantees for a
given image and patch attack size, no patch adversarial examples exist. Our
method is related to the broad class of randomized smoothing robustness schemes
which provide high-confidence probabilistic robustness certificates. By
exploiting the fact that patch attacks are more constrained than general sparse
attacks, we derive meaningfully large robustness certificates. Additionally,
the algorithm we propose is de-randomized, providing deterministic
certificates. To the best of our knowledge, there exists only one prior method
for certifiable defense against patch attacks, which relies on interval bound
propagation. While this sole existing method performs well on MNIST, it has
several limitations: it requires computationally expensive training, does not
scale to ImageNet, and performs poorly on CIFAR-10. In contrast, our proposed
method effectively addresses all of these issues: our classifier can be trained
quickly, achieves high clean and certified robust accuracy on CIFAR-10, and
provides certificates at the ImageNet scale. For example, for a 5*5 patch
attack on CIFAR-10, our method achieves up to around 57.8% certified accuracy
(with a classifier around 83.9% clean accuracy), compared to at most 30.3%
certified accuracy for the existing method (with a classifier with around 47.8%
clean accuracy), effectively establishing a new state-of-the-art. Code is
available at https://github.com/alevine0/patchSmoothing.


http://arxiv.org/abs/2002.10947
Towards an Efficient and General Framework of Robust Training for Graph Neural Networks.
Kaidi Xu; Sijia Liu; Pin-Yu Chen; Mengshu Sun; Caiwen Ding; Bhavya Kailkhura; Xue Lin
  Graph Neural Networks (GNNs) have made significant advances on several
fundamental inference tasks. As a result, there is a surge of interest in using
these models for making potentially important decisions in high-regret
applications. However, despite GNNs' impressive performance, it has been
observed that carefully crafted perturbations on graph structures (or nodes
attributes) lead them to make wrong predictions. Presence of these adversarial
examples raises serious security concerns. Most of the existing robust GNN
design/training methods are only applicable to white-box settings where model
parameters are known and gradient based methods can be used by performing
convex relaxation of the discrete graph domain. More importantly, these methods
are not efficient and scalable which make them infeasible in time sensitive
tasks and massive graph datasets. To overcome these limitations, we propose a
general framework which leverages the greedy search algorithms and zeroth-order
methods to obtain robust GNNs in a generic and an efficient manner. On several
applications, we show that the proposed techniques are significantly less
computationally expensive and, in some cases, more robust than the
state-of-the-art methods making them suitable to large-scale problems which
were out of the reach of traditional robust training methods.


http://arxiv.org/abs/2002.11242
Attacks Which Do Not Kill Training Make Adversarial Learning Stronger.
Jingfeng Zhang; Xilie Xu; Bo Han; Gang Niu; Lizhen Cui; Masashi Sugiyama; Mohan Kankanhalli
  Adversarial training based on the minimax formulation is necessary for
obtaining adversarial robustness of trained models. However, it is conservative
or even pessimistic so that it sometimes hurts the natural generalization. In
this paper, we raise a fundamental question---do we have to trade off natural
generalization for adversarial robustness? We argue that adversarial training
is to employ confident adversarial data for updating the current model. We
propose a novel approach of friendly adversarial training (FAT): rather than
employing most adversarial data maximizing the loss, we search for least
adversarial (i.e., friendly adversarial) data minimizing the loss, among the
adversarial data that are confidently misclassified. Our novel formulation is
easy to implement by just stopping the most adversarial data searching
algorithms such as PGD (projected gradient descent) early, which we call
early-stopped PGD. Theoretically, FAT is justified by an upper bound of the
adversarial risk. Empirically, early-stopped PGD allows us to answer the
earlier question negatively---adversarial robustness can indeed be achieved
without compromising the natural generalization.


http://arxiv.org/abs/2002.11293
Adversarial Ranking Attack and Defense.
Mo Zhou; Zhenxing Niu; Le Wang; Qilin Zhang; Gang Hua
  Deep Neural Network (DNN) classifiers are vulnerable to adversarial attack,
where an imperceptible perturbation could result in misclassification. However,
the vulnerability of DNN-based image ranking systems remains under-explored. In
this paper, we propose two attacks against deep ranking systems, i.e.,
Candidate Attack and Query Attack, that can raise or lower the rank of chosen
candidates by adversarial perturbations. Specifically, the expected ranking
order is first represented as a set of inequalities, and then a triplet-like
objective function is designed to obtain the optimal perturbation. Conversely,
a defense method is also proposed to improve the ranking system robustness,
which can mitigate all the proposed attacks simultaneously. Our adversarial
ranking attacks and defense are evaluated on datasets including MNIST,
Fashion-MNIST, and Stanford-Online-Products. Experimental results demonstrate
that a typical deep ranking system can be effectively compromised by our
attacks. Meanwhile, the system robustness can be moderately improved with our
defense. Furthermore, the transferable and universal properties of our
adversary illustrate the possibility of realistic black-box attack.


http://arxiv.org/abs/2002.10349
A Model-Based Derivative-Free Approach to Black-Box Adversarial Examples: BOBYQA.
Giuseppe Ughi; Vinayak Abrol; Jared Tanner
  We demonstrate that model-based derivative free optimisation algorithms can
generate adversarial targeted misclassification of deep networks using fewer
network queries than non-model-based methods. Specifically, we consider the
black-box setting, and show that the number of networks queries is less
impacted by making the task more challenging either through reducing the
allowed $\ell^{\infty}$ perturbation energy or training the network with
defences against adversarial misclassification. We illustrate this by
contrasting the BOBYQA algorithm with the state-of-the-art model-free
adversarial targeted misclassification approaches based on genetic,
combinatorial, and direct-search algorithms. We observe that for high
$\ell^{\infty}$ energy perturbations on networks, the aforementioned simpler
model-free methods require the fewest queries. In contrast, the proposed BOBYQA
based method achieves state-of-the-art results when the perturbation energy
decreases, or if the network is trained against adversarial perturbations.


http://arxiv.org/abs/2002.10084
Utilizing a null class to restrict decision spaces and defend against neural network adversarial attacks.
Matthew J. Roos
  Despite recent progress, deep neural networks generally continue to be
vulnerable to so-called adversarial examples--input images with small
perturbations that can result in changes in the output classifications, despite
no such change in the semantic meaning to human viewers. This is true even for
seemingly simple challenges such as the MNIST digit classification task. In
part, this suggests that these networks are not relying on the same set of
object features as humans use to make these classifications. In this paper we
examine an additional, and largely unexplored, cause behind this
phenomenon--namely, the use of the conventional training paradigm in which the
entire input space is parcellated among the training classes. Owing to this
paradigm, learned decision spaces for individual classes span excessively large
regions of the input space and include images that have no semantic similarity
to images in the training set. In this study, we train models that include a
null class. That is, models may "opt-out" of classifying an input image as one
of the digit classes. During training, null images are created through a
variety of methods, in an attempt to create tighter and more semantically
meaningful decision spaces for the digit classes. The best performing models
classify nearly all adversarial examples as nulls, rather than mistaking them
as a member of an incorrect digit class, while simultaneously maintaining high
accuracy on the unperturbed test set. The use of a null class and the training
paradigm presented herein may provide an effective defense against adversarial
attacks for some applications. Code for replicating this study will be made
available at https://github.com/mattroos/null_class_adversarial_defense .


http://arxiv.org/abs/2003.00883
Adversarial Perturbations Prevail in the Y-Channel of the YCbCr Color Space.
Camilo Pestana; Naveed Akhtar; Wei Liu; David Glance; Ajmal Mian
  Deep learning offers state of the art solutions for image recognition.
However, deep models are vulnerable to adversarial perturbations in images that
are subtle but significantly change the model's prediction. In a white-box
attack, these perturbations are generally learned for deep models that operate
on RGB images and, hence, the perturbations are equally distributed in the RGB
color space. In this paper, we show that the adversarial perturbations prevail
in the Y-channel of the YCbCr space. Our finding is motivated from the fact
that the human vision and deep models are more responsive to shape and texture
rather than color. Based on our finding, we propose a defense against
adversarial images. Our defence, coined ResUpNet, removes perturbations only
from the Y-channel by exploiting ResNet features in an upsampling framework
without the need for a bottleneck. At the final stage, the untouched
CbCr-channels are combined with the refined Y-channel to restore the clean
image. Note that ResUpNet is model agnostic as it does not modify the DNN
structure. ResUpNet is trained end-to-end in Pytorch and the results are
compared to existing defence techniques in the input transformation category.
Our results show that our approach achieves the best balance between defence
against adversarial attacks such as FGSM, PGD and DDN and maintaining the
original accuracies of VGG-16, ResNet50 and DenseNet121 on clean images. We
perform another experiment to show that learning adversarial perturbations only
for the Y-channel results in higher fooling rates for the same perturbation
magnitude.


http://arxiv.org/abs/2002.10097
Towards Rapid and Robust Adversarial Training with One-Step Attacks.
Leo Schwinn; René Raab; Björn Eskofier
  Adversarial training is the most successful empirical method for increasing
the robustness of neural networks against adversarial attacks. However, the
most effective approaches, like training with Projected Gradient Descent (PGD)
are accompanied by high computational complexity. In this paper, we present two
ideas that, in combination, enable adversarial training with the
computationally less expensive Fast Gradient Sign Method (FGSM). First, we add
uniform noise to the initial data point of the FGSM attack, which creates a
wider variety of adversaries, thus prohibiting overfitting to one particular
perturbation bound. Further, we add a learnable regularization step prior to
the neural network, which we call Pixelwise Noise Injection Layer (PNIL).
Inputs propagated trough the PNIL are resampled from a learned Gaussian
distribution. The regularization induced by the PNIL prevents the model form
learning to obfuscate its gradients, a factor that hindered prior approaches
from successfully applying one-step methods for adversarial training. We show
that noise injection in conjunction with FGSM-based adversarial training
achieves comparable results to adversarial training with PGD while being
considerably faster. Moreover, we outperform PGD-based adversarial training by
combining noise injection and PNIL.


http://arxiv.org/abs/2002.10477
Precise Tradeoffs in Adversarial Training for Linear Regression.
Adel Javanmard; Mahdi Soltanolkotabi; Hamed Hassani
  Despite breakthrough performance, modern learning models are known to be
highly vulnerable to small adversarial perturbations in their inputs. While a
wide variety of recent \emph{adversarial training} methods have been effective
at improving robustness to perturbed inputs (robust accuracy), often this
benefit is accompanied by a decrease in accuracy on benign inputs (standard
accuracy), leading to a tradeoff between often competing objectives.
Complicating matters further, recent empirical evidence suggest that a variety
of other factors (size and quality of training data, model size, etc.) affect
this tradeoff in somewhat surprising ways. In this paper we provide a precise
and comprehensive understanding of the role of adversarial training in the
context of linear regression with Gaussian features. In particular, we
characterize the fundamental tradeoff between the accuracies achievable by any
algorithm regardless of computational power or size of the training data.
Furthermore, we precisely characterize the standard/robust accuracy and the
corresponding tradeoff achieved by a contemporary mini-max adversarial training
approach in a high-dimensional regime where the number of data points and the
parameters of the model grow in proportion to each other. Our theory for
adversarial training algorithms also facilitates the rigorous study of how a
variety of factors (size and quality of training data, model
overparametrization etc.) affect the tradeoff between these two competing
accuracies.


http://arxiv.org/abs/2002.10509
On Pruning Adversarially Robust Neural Networks.
Vikash Sehwag; Shiqi Wang; Prateek Mittal; Suman Jana
  In safety-critical but computationally resource-constrained applications,
deep learning faces two key challenges: lack of robustness against adversarial
attacks and large neural network size (often millions of parameters). While the
research community has extensively explored the use of robust training and
network pruning \emph{independently} to address one of these challenges, we
show that integrating existing pruning techniques with multiple types of robust
training techniques, including verifiably robust training, leads to poor robust
accuracy even though such techniques can preserve high regular accuracy. We
further demonstrate that making pruning techniques aware of the robust learning
objective can lead to a large improvement in performance. We realize this
insight by formulating the pruning objective as an empirical risk minimization
problem which is then solved using SGD. We demonstrate the success of the
proposed pruning technique across CIFAR-10, SVHN, and ImageNet dataset with
four different robust training techniques: iterative adversarial training,
randomized smoothing, MixTrain, and CROWN-IBP. Specifically, at 99\% connection
pruning ratio, we achieve gains up to 3.2, 10.0, and 17.8 percentage points in
robust accuracy under state-of-the-art adversarial attacks for ImageNet,
CIFAR-10, and SVHN dataset, respectively. Our code and compressed networks are
publicly available at https://github.com/inspire-group/compactness-robustness


http://arxiv.org/abs/2002.09896
Adversarial Attack on DL-based Massive MIMO CSI Feedback.
Qing Liu; Jiajia Guo; Chao-Kai Wen; Shi Jin
  With the increasing application of deep learning (DL) algorithms in wireless
communications, the physical layer faces new challenges caused by adversarial
attack. Such attack has significantly affected the neural network in computer
vision. We chose DL-based analog channel state information (CSI) to show the
effect of adversarial attack on DL-based communication system. We present a
practical method to craft white-box adversarial attack on DL-based CSI feedback
process. Our simulation results showed the destructive effect adversarial
attack caused on DL-based CSI feedback by analyzing the performance of
normalized mean square error. We also launched a jamming attack for comparison
and found that the jamming attack could be prevented with certain precautions.
As DL algorithm becomes the trend in developing wireless communication, this
work raises concerns regarding the security in the use of DL-based algorithms.


http://arxiv.org/abs/2002.10025
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference.
Ting-Kuei Hu; Tianlong Chen; Haotao Wang; Zhangyang Wang
  Deep networks were recently suggested to face the odds between accuracy (on
clean natural images) and robustness (on adversarially perturbed images)
(Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently
higher sample complexity (Schmidt et al., 2018) and/or model capacity
(Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view
of that, give a classification task, growing the model capacity appears to help
draw a win-win between accuracy and robustness, yet at the expense of model
size and latency, therefore posing challenges for resource-constrained
applications. Is it possible to co-design model accuracy, robustness and
efficiency to achieve their triple wins? This paper studies multi-exit networks
associated with input-adaptive efficient inference, showing their strong
promise in achieving a "sweet point" in cooptimizing model accuracy, robustness
and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks
(RDI-Nets), allows for each input (either clean or adversarial) to adaptively
choose one of the multiple output layers (early branches or the final one) to
output its prediction. That multi-loss adaptivity adds new variations and
flexibility to adversarial attacks and defenses, on which we present a
systematical investigation. We show experimentally that by equipping existing
backbones with such robust adaptive inference, the resulting RDI-Nets can
achieve better accuracy and robustness, yet with over 30% computational
savings, compared to the defended original models.


http://arxiv.org/abs/2002.09792
VisionGuard: Runtime Detection of Adversarial Inputs to Perception Systems.
Yiannis Kantaros; Taylor Carpenter; Sangdon Park; Radoslav Ivanov; Sooyong Jang; Insup Lee; James Weimer
  Deep neural network (DNN) models have proven to be vulnerable to adversarial
attacks. In this paper, we propose VisionGuard, a novel attack- and
dataset-agnostic and computationally-light defense mechanism for adversarial
inputs to DNN-based perception systems. In particular, VisionGuard relies on
the observation that adversarial images are sensitive to lossy compression
transformations. Specifically, to determine if an image is adversarial,
VisionGuard checks if the output of the target classifier on a given input
image changes significantly after feeding it a transformed version of the image
under investigation. Moreover, we show that VisionGuard is
computationally-light both at runtime and design-time which makes it suitable
for real-time applications that may also involve large-scale image domains. To
highlight this, we demonstrate the efficiency of VisionGuard on ImageNet, a
task that is computationally challenging for the majority of relevant defenses.
Finally, we include extensive comparative experiments on the MNIST, CIFAR10,
and ImageNet datasets that show that VisionGuard outperforms existing defenses
in terms of scalability and detection performance.


http://arxiv.org/abs/2002.09772
Non-Intrusive Detection of Adversarial Deep Learning Attacks via Observer Networks.
Kirthi Shankar Sivamani; Rajeev Sahay; Aly El Gamal
  Recent studies have shown that deep learning models are vulnerable to
specifically crafted adversarial inputs that are quasi-imperceptible to humans.
In this letter, we propose a novel method to detect adversarial inputs, by
augmenting the main classification network with multiple binary detectors
(observer networks) which take inputs from the hidden layers of the original
network (convolutional kernel outputs) and classify the input as clean or
adversarial. During inference, the detectors are treated as a part of an
ensemble network and the input is deemed adversarial if at least half of the
detectors classify it as so. The proposed method addresses the trade-off
between accuracy of classification on clean and adversarial samples, as the
original classification network is not modified during the detection process.
The use of multiple observer networks makes attacking the detection mechanism
non-trivial even when the attacker is aware of the victim classifier. We
achieve a 99.5% detection accuracy on the MNIST dataset and 97.5% on the
CIFAR-10 dataset using the Fast Gradient Sign Attack in a semi-white box setup.
The number of false positive detections is a mere 0.12% in the worst case
scenario.


http://arxiv.org/abs/2002.09674
Temporal Sparse Adversarial Attack on Gait Recognition.
Ziwen He; Wei Wang; Jing Dong; Tieniu Tan
  Gait recognition has a broad application in social security due to its
advantages in long-distance human identification. Despite the high accuracy of
gait recognition systems, their adversarial robustness has not been explored.
In this paper, we demonstrate that the state-of-the-art gait recognition model
is vulnerable to adversarial attacks. A novel temporal sparse adversarial
attack under a new defined distortion measurement is proposed. GAN-based
architecture is employed to semantically generate adversarial high-quality gait
silhouette. By sparsely substituting or inserting a few adversarial gait
silhouettes, our proposed method can achieve a high attack success rate. The
imperceptibility and the attacking success rate of the adversarial examples are
well balanced. Experimental results show even only one-fortieth frames are
attacked, the attack success rate still reaches 76.8%.


http://arxiv.org/abs/2002.09632
Using Single-Step Adversarial Training to Defend Iterative Adversarial Examples.
Guanxiong Liu; Issa Khalil; Abdallah Khreishah
  Adversarial examples have become one of the largest challenges that machine
learning models, especially neural network classifiers, face. These adversarial
examples break the assumption of attack-free scenario and fool state-of-the-art
(SOTA) classifiers with insignificant perturbations to human. So far,
researchers achieved great progress in utilizing adversarial training as a
defense. However, the overwhelming computational cost degrades its
applicability and little has been done to overcome this issue. Single-Step
adversarial training methods have been proposed as computationally viable
solutions, however they still fail to defend against iterative adversarial
examples. In this work, we first experimentally analyze several different SOTA
defense methods against adversarial examples. Then, based on observations from
experiments, we propose a novel single-step adversarial training method which
can defend against both single-step and iterative adversarial examples. Lastly,
through extensive evaluations, we demonstrate that our proposed method
outperforms the SOTA single-step and iterative adversarial training defense.
Compared with ATDA (single-step method) on CIFAR10 dataset, our proposed method
achieves 35.67% enhancement in test accuracy and 19.14% reduction in training
time. When compared with methods that use BIM or Madry examples (iterative
methods) on CIFAR10 dataset, it saves up to 76.03% in training time with less
than 3.78% degeneration in test accuracy.


http://arxiv.org/abs/2002.09580
Polarizing Front Ends for Robust CNNs.
Can Bakiskan; Soorya Gopalakrishnan; Metehan Cekic; Upamanyu Madhow; Ramtin Pedarsani
  The vulnerability of deep neural networks to small, adversarially designed
perturbations can be attributed to their "excessive linearity." In this paper,
we propose a bottom-up strategy for attenuating adversarial perturbations using
a nonlinear front end which polarizes and quantizes the data. We observe that
ideal polarization can be utilized to completely eliminate perturbations,
develop algorithms to learn approximately polarizing bases for data, and
investigate the effectiveness of the proposed strategy on the MNIST and Fashion
MNIST datasets.


http://arxiv.org/abs/2002.09579
Robustness to Programmable String Transformations via Augmented Abstract Training.
Yuhao Zhang; Aws Albarghouthi; Loris D'Antoni
  Deep neural networks for natural language processing tasks are vulnerable to
adversarial input perturbations. In this paper, we present a versatile language
for programmatically specifying string transformations -- e.g., insertions,
deletions, substitutions, swaps, etc. -- that are relevant to the task at hand.
We then present an approach to adversarially training models that are robust to
such user-defined string transformations. Our approach combines the advantages
of search-based techniques for adversarial training with abstraction-based
techniques. Specifically, we show how to decompose a set of user-defined string
transformations into two component specifications, one that benefits from
search and another from abstraction. We use our technique to train models on
the AG and SST2 datasets and show that the resulting models are robust to
combinations of user-defined transformations mimicking spelling mistakes and
other meaning-preserving transformations.


http://arxiv.org/abs/2002.09576
UnMask: Adversarial Detection and Defense Through Robust Feature Alignment.
Scott Freitas; Shang-Tse Chen; Zijie Wang; Duen Horng Chau
  Deep learning models are being integrated into a wide range of high-impact,
security-critical systems, from self-driving cars to medical diagnosis.
However, recent research has demonstrated that many of these deep learning
architectures are vulnerable to adversarial attacks--highlighting the vital
need for defensive techniques to detect and mitigate these attacks before they
occur. To combat these adversarial attacks, we developed UnMask, an adversarial
detection and defense framework based on robust feature alignment. The core
idea behind UnMask is to protect these models by verifying that an image's
predicted class ("bird") contains the expected robust features (e.g., beak,
wings, eyes). For example, if an image is classified as "bird", but the
extracted features are wheel, saddle and frame, the model may be under attack.
UnMask detects such attacks and defends the model by rectifying the
misclassification, re-classifying the image based on its robust features. Our
extensive evaluation shows that UnMask (1) detects up to 96.75% of attacks,
with a false positive rate of 9.66% and (2) defends the model by correctly
classifying up to 93% of adversarial images produced by the current strongest
attack, Projected Gradient Descent, in the gray-box setting. UnMask provides
significantly better protection than adversarial training across 8 attack
vectors, averaging 31.18% higher accuracy. Our proposed method is architecture
agnostic and fast. We open source the code repository and data with this paper:
https://github.com/unmaskd/unmask.


http://arxiv.org/abs/2002.09169
Black-Box Certification with Randomized Smoothing: A Functional Optimization Based Framework.
Dinghuai Zhang; Mao Ye; Chengyue Gong; Zhanxing Zhu; Qiang Liu
  Randomized classifiers have been shown to provide a promising approach for
achieving certified robustness against adversarial attacks in deep learning.
However, most existing methods only leverage Gaussian smoothing noise and only
work for $\ell_2$ perturbation. We propose a general framework of adversarial
certification with non-Gaussian noise and for more general types of attacks,
from a unified functional optimization perspective. Our new framework allows us
to identify a key trade-off between accuracy and robustness via designing
smoothing distributions, helping to design new families of non-Gaussian
smoothing distributions that work more efficiently for different $\ell_p$
settings, including $\ell_1$, $\ell_2$ and $\ell_\infty$ attacks. Our proposed
methods achieve better certification results than previous works and provide a
new perspective on randomized smoothing certification.


http://arxiv.org/abs/2002.09422
Robustness from Simple Classifiers.
Sharon Qian; Dimitris Kalimeris; Gal Kaplun; Yaron Singer
  Despite the vast success of Deep Neural Networks in numerous application
domains, it has been shown that such models are not robust i.e., they are
vulnerable to small adversarial perturbations of the input. While extensive
work has been done on why such perturbations occur or how to successfully
defend against them, we still do not have a complete understanding of
robustness. In this work, we investigate the connection between robustness and
simplicity. We find that simpler classifiers, formed by reducing the number of
output classes, are less susceptible to adversarial perturbations.
Consequently, we demonstrate that decomposing a complex multiclass model into
an aggregation of binary models enhances robustness. This behavior is
consistent across different datasets and model architectures and can be
combined with known defense techniques such as adversarial training. Moreover,
we provide further evidence of a disconnect between standard and robust
learning regimes. In particular, we show that elaborate label information can
help standard accuracy but harm robustness.


http://arxiv.org/abs/2002.09364
Adversarial Detection and Correction by Matching Prediction Distributions.
Giovanni Vacanti; Looveren Arnaud Van
  We present a novel adversarial detection and correction method for machine
learning classifiers.The detector consists of an autoencoder trained with a
custom loss function based on the Kullback-Leibler divergence between the
classifier predictions on the original and reconstructed instances.The method
is unsupervised, easy to train and does not require any knowledge about the
underlying attack. The detector almost completely neutralises powerful attacks
like Carlini-Wagner or SLIDE on MNIST and Fashion-MNIST, and remains very
effective on CIFAR-10 when the attack is granted full access to the
classification model but not the defence. We show that our method is still able
to detect the adversarial examples in the case of a white-box attack where the
attacker has full knowledge of both the model and the defence and investigate
the robustness of the attack. The method is very flexible and can also be used
to detect common data corruptions and perturbations which negatively impact the
model performance. We illustrate this capability on the CIFAR-10-C dataset.


http://arxiv.org/abs/2002.09565
Adversarial Attacks on Machine Learning Systems for High-Frequency Trading.
Micah Goldblum; Avi Schwarzschild; Ankit B. Patel; Tom Goldstein
  Algorithmic trading systems are often completely automated, and deep learning
is increasingly receiving attention in this domain. Nonetheless, little is
known about the robustness properties of these models. We study valuation
models for algorithmic trading from the perspective of adversarial machine
learning. We introduce new attacks specific to this domain with size
constraints that minimize attack costs. We further discuss how these attacks
can be used as an analysis tool to study and evaluate the robustness properties
of financial models. Finally, we investigate the feasibility of realistic
adversarial attacks in which an adversarial trader fools automated trading
systems into making inaccurate predictions.


http://arxiv.org/abs/2002.09027
Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning.
Chao-Han Huck Yang; Jun Qi; Pin-Yu Chen; Yi Ouyang; I-Te Danny Hung; Chin-Hui Lee; Xiaoli Ma
  Recent deep neural networks based techniques, especially those equipped with
the ability of self-adaptation in the system level such as deep reinforcement
learning (DRL), are shown to possess many advantages of optimizing robot
learning systems (e.g., autonomous navigation and continuous robot arm
control.) However, the learning-based systems and the associated models may be
threatened by the risks of intentionally adaptive (e.g., noisy sensor
confusion) and adversarial perturbations from real-world scenarios. In this
paper, we introduce timing-based adversarial strategies against a DRL-based
navigation system by jamming in physical noise patterns on the selected time
frames. To study the vulnerability of learning-based navigation systems, we
propose two adversarial agent models: one refers to online learning; another
one is based on evolutionary learning. Besides, three open-source robot
learning and navigation control environments are employed to study the
vulnerability under adversarial timing attacks. Our experimental results show
that the adversarial timing attacks can lead to a significant performance drop,
and also suggest the necessity of enhancing the robustness of robot learning
systems.


http://arxiv.org/abs/2002.08838
On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective.
Motasem Alfarra; Adel Bibi; Hasan Hammoud; Mohamed Gaafar; Bernard Ghanem
  This work tackles the problem of characterizing and understanding the
decision boundaries of neural networks with piecewise linear non-linearity
activations. We use tropical geometry, a new development in the area of
algebraic geometry, to characterize the decision boundaries of a simple neural
network of the form (Affine, ReLU, Affine). Our main finding is that the
decision boundaries are a subset of a tropical hypersurface, which is
intimately related to a polytope formed by the convex hull of two zonotopes.
The generators of these zonotopes are functions of the neural network
parameters. This geometric characterization provides new perspective to three
tasks. Specifically, we propose a new tropical perspective to the lottery
ticket hypothesis, where we see the effect of different initializations on the
tropical geometric representation of a network's decision boundaries. Moreover,
we use this characterization to propose a new set of tropical regularizers,
which directly deal with the decision boundaries of a network. We investigate
the use of these regularizers in neural network pruning (by removing network
parameters that do not contribute to the tropical geometric representation of
the decision boundaries) and in generating adversarial input attacks (by
producing input perturbations that explicitly perturb the decision boundaries'
geometry and ultimately change the network's prediction).


http://arxiv.org/abs/2002.08619
Boosting Adversarial Training with Hypersphere Embedding.
Tianyu Pang; Xiao Yang; Yinpeng Dong; Kun Xu; Hang Su; Jun Zhu
  Adversarial training (AT) is one of the most effective defenses to improve
the adversarial robustness of deep learning models. In order to promote the
reliability of the adversarially trained models, we propose to boost AT via
incorporating hypersphere embedding (HE), which can regularize the adversarial
features onto compact hypersphere manifolds. We formally demonstrate that AT
and HE are well coupled, which tunes up the learning dynamics of AT from
several aspects. We comprehensively validate the effectiveness and universality
of HE by embedding it into the popular AT frameworks including PGD-AT, ALP, and
TRADES, as well as the FreeAT and FastAT strategies. In experiments, we
evaluate our methods on the CIFAR-10 and ImageNet datasets, and verify that
integrating HE can consistently enhance the performance of the models trained
by each AT framework with little extra computation.


http://arxiv.org/abs/2002.08859
A Bayes-Optimal View on Adversarial Examples.
Eitan Richardson; Yair Weiss
  The ability to fool modern CNN classifiers with tiny perturbations of the
input has lead to the development of a large number of candidate defenses and
often conflicting explanations. In this paper, we argue for examining
adversarial examples from the perspective of Bayes-Optimal classification. We
construct realistic image datasets for which the Bayes-Optimal classifier can
be efficiently computed and derive analytic conditions on the distributions so
that the optimal classifier is either robust or vulnerable. By training
different classifiers on these datasets (for which the "gold standard" optimal
classifiers are known), we can disentangle the possible sources of
vulnerability and avoid the accuracy-robustness tradeoff that may occur in
commonly used datasets. Our results show that even when the optimal classifier
is robust, standard CNN training consistently learns a vulnerable classifier.
At the same time, for exactly the same training data, RBF SVMs consistently
learn a robust classifier. The same trend is observed in experiments with real
images.


http://arxiv.org/abs/2002.08740
Towards Certifiable Adversarial Sample Detection.
Ilia Shumailov; Yiren Zhao; Robert Mullins; Ross Anderson
  Convolutional Neural Networks (CNNs) are deployed in more and more
classification systems, but adversarial samples can be maliciously crafted to
trick them, and are becoming a real threat. There have been various proposals
to improve CNNs' adversarial robustness but these all suffer performance
penalties or other limitations. In this paper, we provide a new approach in the
form of a certifiable adversarial detection scheme, the Certifiable Taboo Trap
(CTT). The system can provide certifiable guarantees of detection of
adversarial inputs for certain $l_{\infty}$ sizes on a reasonable assumption,
namely that the training data have the same distribution as the test data. We
develop and evaluate several versions of CTT with a range of defense
capabilities, training overheads and certifiability on adversarial samples.
Against adversaries with various $l_p$ norms, CTT outperforms existing defense
methods that focus purely on improving network robustness. We show that CTT has
small false positive rates on clean test data, minimal compute overheads when
deployed, and can support complex security policies.


http://arxiv.org/abs/2002.10248
Bayes-Probe: Distribution-Guided Sampling for Prediction Level Sets.
Serena Booth; Yilun Zhou; Ankit Shah; Julie Shah
  Building machine learning models requires a suite of tools for
interpretation, understanding, and debugging. Many existing methods have been
proposed, but it can still be difficult to probe for examples which communicate
model behaviour. We introduce Bayes-Probe, a model inspection method for
analyzing neural networks by generating distribution-conforming examples of
known prediction confidence. By selecting appropriate distributions and
confidence prediction values, Bayes-Probe can be used to synthesize ambivalent
predictions, uncover in-distribution adversarial examples, and understand
novel-class extrapolation and domain adaptation behaviours. Bayes-Probe is
model agnostic, requiring only a data generator and classifier prediction. We
use Bayes-Probe to analyze models trained on both procedurally-generated data
(CLEVR) and organic data (MNIST and Fashion-MNIST). Code is available at
https://github.com/serenabooth/Bayes-Probe.


http://arxiv.org/abs/2002.08439
AdvMS: A Multi-source Multi-cost Defense Against Adversarial Attacks.
Xiao Wang; Siyue Wang; Pin-Yu Chen; Xue Lin; Peter Chin
  Designing effective defense against adversarial attacks is a crucial topic as
deep neural networks have been proliferated rapidly in many security-critical
domains such as malware detection and self-driving cars. Conventional defense
methods, although shown to be promising, are largely limited by their
single-source single-cost nature: The robustness promotion tends to plateau
when the defenses are made increasingly stronger while the cost tends to
amplify. In this paper, we study principles of designing multi-source and
multi-cost schemes where defense performance is boosted from multiple defending
components. Based on this motivation, we propose a multi-source and multi-cost
defense scheme, Adversarially Trained Model Switching (AdvMS), that inherits
advantages from two leading schemes: adversarial training and random model
switching. We show that the multi-source nature of AdvMS mitigates the
performance plateauing issue and the multi-cost nature enables improving
robustness at a flexible and adjustable combination of costs over different
factors which can better suit specific restrictions and needs in practice.


http://arxiv.org/abs/2002.08527
NAttack! Adversarial Attacks to bypass a GAN based classifier trained to detect Network intrusion.
Aritran Piplai; Sai Sree Laya Chukkapalli; Anupam Joshi
  With the recent developments in artificial intelligence and machine learning,
anomalies in network traffic can be detected using machine learning approaches.
Before the rise of machine learning, network anomalies which could imply an
attack, were detected using well-crafted rules. An attacker who has knowledge
in the field of cyber-defence could make educated guesses to sometimes
accurately predict which particular features of network traffic data the
cyber-defence mechanism is looking at. With this information, the attacker can
circumvent a rule-based cyber-defense system. However, after the advancements
of machine learning for network anomaly, it is not easy for a human to
understand how to bypass a cyber-defence system. Recently, adversarial attacks
have become increasingly common to defeat machine learning algorithms. In this
paper, we show that even if we build a classifier and train it with adversarial
examples for network data, we can use adversarial attacks and successfully
break the system. We propose a Generative Adversarial Network(GAN)based
algorithm to generate data to train an efficient neural network based
classifier, and we subsequently break the system using adversarial attacks.


http://arxiv.org/abs/2002.08012
Indirect Adversarial Attacks via Poisoning Neighbors for Graph Convolutional Networks.
Tsubasa Takahashi
  Graph convolutional neural networks, which learn aggregations over neighbor
nodes, have achieved great performance in node classification tasks. However,
recent studies reported that such graph convolutional node classifier can be
deceived by adversarial perturbations on graphs. Abusing graph convolutions, a
node's classification result can be influenced by poisoning its neighbors.
Given an attributed graph and a node classifier, how can we evaluate robustness
against such indirect adversarial attacks? Can we generate strong adversarial
perturbations which are effective on not only one-hop neighbors, but more far
from the target? In this paper, we demonstrate that the node classifier can be
deceived with high-confidence by poisoning just a single node even two-hops or
more far from the target. Towards achieving the attack, we propose a new
approach which searches smaller perturbations on just a single node far from
the target. In our experiments, our proposed method shows 99% attack success
rate within two-hops from the target in two datasets. We also demonstrate that
m-layer graph convolutional neural networks have chance to be deceived by our
indirect attack within m-hop neighbors. The proposed attack can be used as a
benchmark in future defense attempts to develop graph convolutional neural
networks with having adversary robustness.


http://arxiv.org/abs/2002.08347
On Adaptive Attacks to Adversarial Example Defenses.
Florian Tramer; Nicholas Carlini; Wieland Brendel; Aleksander Madry
  Adaptive attacks have (rightfully) become the de facto standard for
evaluating defenses to adversarial examples. We find, however, that typical
adaptive evaluations are incomplete. We demonstrate that thirteen defenses
recently published at ICLR, ICML and NeurIPS---and chosen for illustrative and
pedagogical purposes---can be circumvented despite attempting to perform
evaluations using adaptive attacks. While prior evaluation papers focused
mainly on the end result---showing that a defense was ineffective---this paper
focuses on laying out the methodology and the approach necessary to perform an
adaptive attack. We hope that these analyses will serve as guidance on how to
properly perform adaptive attacks against defenses to adversarial examples, and
thus will allow the community to make further progress in building more robust
models.


http://arxiv.org/abs/2002.08118
Randomized Smoothing of All Shapes and Sizes.
Greg Yang; Tony Duan; J. Edward Hu; Hadi Salman; Ilya Razenshteyn; Jerry Li
  Randomized smoothing is a recently proposed defense against adversarial
attacks that has achieved state-of-the-art provable robustness against $\ell_2$
perturbations. Soon after, a number of works devised new randomized smoothing
schemes for other metrics, such as $\ell_1$ or $\ell_\infty$; however, for each
geometry, substantial effort was needed to derive new robustness guarantees.
This begs the question: can we find a general theory for randomized smoothing?
  In this work we propose a novel framework for devising and analyzing
randomized smoothing schemes, and validate its effectiveness in practice. Our
theoretical contributions are as follows: (1) We show that for an appropriate
notion of "optimal", the optimal smoothing distributions for any "nice" norm
have level sets given by the *Wulff Crystal* of that norm. (2) We propose two
novel and complementary methods for deriving provably robust radii for any
smoothing distribution. Finally, (3) we show fundamental limits to current
randomized smoothing techniques via the theory of *Banach space cotypes*. By
combining (1) and (2), we significantly improve the state-of-the-art certified
accuracy in $\ell_1$ on standard datasets. On the other hand, using (3), we
show that, without more information than label statistics under random input
perturbations, randomized smoothing cannot achieve nontrivial certified
accuracy against perturbations of $\ell_p$-norm $\Omega(\min(1,
d^{\frac{1}{p}-\frac{1}{2}}))$, when the input dimension $d$ is large. We
provide code in github.com/tonyduan/rs4a.


http://arxiv.org/abs/2002.08000
Action-Manipulation Attacks Against Stochastic Bandits: Attacks and Defense.
Guanlin Liu; Lifeng lai
  Due to the broad range of applications of stochastic multi-armed bandit
model, understanding the effects of adversarial attacks and designing bandit
algorithms robust to attacks are essential for the safe applications of this
model. In this paper, we introduce a new class of attack named
action-manipulation attack. In this attack, an adversary can change the action
signal selected by the user. We show that without knowledge of mean rewards of
arms, our proposed attack can manipulate Upper Confidence Bound (UCB)
algorithm, a widely used bandit algorithm, into pulling a target arm very
frequently by spending only logarithmic cost. To defend against this class of
attacks, we introduce a novel algorithm that is robust to action-manipulation
attacks when an upper bound for the total attack cost is given. We prove that
our algorithm has a pseudo-regret upper bounded by $\mathcal{O}(\max\{\log
T,A\})$, where $T$ is the total number of rounds and $A$ is the upper bound of
the total attack cost.


http://arxiv.org/abs/2002.07405
Deflecting Adversarial Attacks.
Yao Qin; Nicholas Frosst; Colin Raffel; Garrison Cottrell; Geoffrey Hinton
  There has been an ongoing cycle where stronger defenses against adversarial
attacks are subsequently broken by a more advanced defense-aware attack. We
present a new approach towards ending this cycle where we "deflect''
adversarial attacks by causing the attacker to produce an input that
semantically resembles the attack's target class. To this end, we first propose
a stronger defense based on Capsule Networks that combines three detection
mechanisms to achieve state-of-the-art detection performance on both standard
and defense-aware attacks. We then show that undetected attacks against our
defense often perceptually resemble the adversarial target class by performing
a human study where participants are asked to label images produced by the
attack. These attack images can no longer be called "adversarial'' because our
network classifies them the same way as humans do.


http://arxiv.org/abs/2002.07920
Block Switching: A Stochastic Approach for Deep Learning Security.
Xiao Wang; Siyue Wang; Pin-Yu Chen; Xue Lin; Peter Chin
  Recent study of adversarial attacks has revealed the vulnerability of modern
deep learning models. That is, subtly crafted perturbations of the input can
make a trained network with high accuracy produce arbitrary incorrect
predictions, while maintain imperceptible to human vision system. In this
paper, we introduce Block Switching (BS), a defense strategy against
adversarial attacks based on stochasticity. BS replaces a block of model layers
with multiple parallel channels, and the active channel is randomly assigned in
the run time hence unpredictable to the adversary. We show empirically that BS
leads to a more dispersed input gradient distribution and superior defense
effectiveness compared with other stochastic defenses such as stochastic
activation pruning (SAP). Compared to other defenses, BS is also characterized
by the following features: (i) BS causes less test accuracy drop; (ii) BS is
attack-independent and (iii) BS is compatible with other defenses and can be
used jointly with others.


http://arxiv.org/abs/2002.07891
Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent.
Pu Zhao; Pin-Yu Chen; Siyue Wang; Xue Lin
  Despite the great achievements of the modern deep neural networks (DNNs), the
vulnerability/robustness of state-of-the-art DNNs raises security concerns in
many application domains requiring high reliability. Various adversarial
attacks are proposed to sabotage the learning performance of DNN models. Among
those, the black-box adversarial attack methods have received special
attentions owing to their practicality and simplicity. Black-box attacks
usually prefer less queries in order to maintain stealthy and low costs.
However, most of the current black-box attack methods adopt the first-order
gradient descent method, which may come with certain deficiencies such as
relatively slow convergence and high sensitivity to hyper-parameter settings.
In this paper, we propose a zeroth-order natural gradient descent (ZO-NGD)
method to design the adversarial attacks, which incorporates the zeroth-order
gradient estimation technique catering to the black-box attack scenario and the
second-order natural gradient descent to achieve higher query efficiency. The
empirical evaluations on image classification datasets demonstrate that ZO-NGD
can obtain significantly lower model query complexities compared with
state-of-the-art attack methods.


http://arxiv.org/abs/2002.10252
TensorShield: Tensor-based Defense Against Adversarial Attacks on Images.
Negin Entezari; Evangelos E. Papalexakis
  Recent studies have demonstrated that machine learning approaches like deep
neural networks (DNNs) are easily fooled by adversarial attacks. Subtle and
imperceptible perturbations of the data are able to change the result of deep
neural networks. Leveraging vulnerable machine learning methods raises many
concerns especially in domains where security is an important factor.
Therefore, it is crucial to design defense mechanisms against adversarial
attacks. For the task of image classification, unnoticeable perturbations
mostly occur in the high-frequency spectrum of the image. In this paper, we
utilize tensor decomposition techniques as a preprocessing step to find a
low-rank approximation of images which can significantly discard high-frequency
perturbations. Recently a defense framework called Shield could "vaccinate"
Convolutional Neural Networks (CNN) against adversarial examples by performing
random-quality JPEG compressions on local patches of images on the ImageNet
dataset. Our tensor-based defense mechanism outperforms the SLQ method from
Shield by 14% against FastGradient Descent (FGSM) adversarial attacks, while
maintaining comparable speed.


http://arxiv.org/abs/2002.06816
On the Similarity of Deep Learning Representations Across Didactic and Adversarial Examples.
Pamela K. Douglas; Farzad Vasheghani Farahani
  The increasing use of deep neural networks (DNNs) has motivated a parallel
endeavor: the design of adversaries that profit from successful
misclassifications. However, not all adversarial examples are crafted for
malicious purposes. For example, real world systems often contain physical,
temporal, and sampling variability across instrumentation. Adversarial examples
in the wild may inadvertently prove deleterious for accurate predictive
modeling. Conversely, naturally occurring covariance of image features may
serve didactic purposes. Here, we studied the stability of deep learning
representations for neuroimaging classification across didactic and adversarial
conditions characteristic of MRI acquisition variability. We show that
representational similarity and performance vary according to the frequency of
adversarial examples in the input space.


http://arxiv.org/abs/2002.06864
Scalable Quantitative Verification For Deep Neural Networks.
Teodora Baluta; Zheng Leong Chua; Kuldeep S. Meel; Prateek Saxena
  Verifying security properties of deep neural networks (DNNs) is becoming
increasingly important. This paper introduces a new quantitative verification
framework for DNNs that can decide, with user-specified confidence, whether a
given logical property {\psi} defined over the space of inputs of the given DNN
holds for less than a user-specified threshold, {\theta}. We present new
algorithms that are scalable to large real-world models as well as proven to be
sound. Our approach requires only black-box access to the models. Further, it
certifies properties of both deterministic and non-deterministic DNNs. We
implement our approach in a tool called PROVERO. We apply PROVERO to the
problem of certifying adversarial robustness. In this context, PROVERO provides
an attack-agnostic measure of robustness for a given DNN and a test input.
First, we find that this metric has a strong statistical correlation with
perturbation bounds reported by 2 of the most prominent white-box attack
strategies today. Second, we show that PROVERO can quantitatively certify
robustness with high confidence in cases where the state-of-the-art qualitative
verification tool (ERAN) fails to produce conclusive results. Thus,
quantitative verification scales easily to large DNNs.


http://arxiv.org/abs/2002.07088
Query-Efficient Physical Hard-Label Attacks on Deep Learning Visual Classification.
Ryan Feng; Jiefeng Chen; Nelson Manohar; Earlence Fernandes; Somesh Jha; Atul Prakash
  We present Survival-OPT, a physical adversarial example algorithm in the
black-box hard-label setting where the attacker only has access to the model
prediction class label. Assuming such limited access to the model is more
relevant for settings such as proprietary cyber-physical and cloud systems than
the whitebox setting assumed by prior work. By leveraging the properties of
physical attacks, we create a novel approach based on the survivability of
perturbations corresponding to physical transformations. Through simply
querying the model for hard-label predictions, we optimize perturbations to
survive in many different physical conditions and show that adversarial
examples remain a security risk to cyber-physical systems (CPSs) even in the
hard-label threat model. We show that Survival-OPT is query-efficient and
robust: using fewer than 200K queries, we successfully attack a stop sign to be
misclassified as a speed limit 30 km/hr sign in 98.5% of video frames in a
drive-by setting. Survival-OPT also outperforms our baseline combination of
existing hard-label and physical approaches, which required over 10x more
queries for less robust results.


http://arxiv.org/abs/2002.06789
CAT: Customized Adversarial Training for Improved Robustness.
Minhao Cheng; Qi Lei; Pin-Yu Chen; Inderjit Dhillon; Cho-Jui Hsieh
  Adversarial training has become one of the most effective methods for
improving robustness of neural networks. However, it often suffers from poor
generalization on both clean and perturbed data. In this paper, we propose a
new algorithm, named Customized Adversarial Training (CAT), which adaptively
customizes the perturbation level and the corresponding label for each training
sample in adversarial training. We show that the proposed algorithm achieves
better clean and robust accuracy than previous adversarial training methods
through extensive experiments.


http://arxiv.org/abs/2002.07317
On the Matrix-Free Generation of Adversarial Perturbations for Black-Box Attacks.
Hisaichi Shibata; Shouhei Hanaoka; Yukihiro Nomura; Naoto Hayashi; Osamu Abe
  In general, adversarial perturbations superimposed on inputs are realistic
threats for a deep neural network (DNN). In this paper, we propose a practical
generation method of such adversarial perturbation to be applied to black-box
attacks that demand access to an input-output relationship only. Thus, the
attackers generate such perturbation without invoking inner functions and/or
accessing the inner states of a DNN. Unlike the earlier studies, the algorithm
to generate the perturbation presented in this study requires much fewer query
trials. Moreover, to show the effectiveness of the adversarial perturbation
extracted, we experiment with a DNN for semantic segmentation. The result shows
that the network is easily deceived with the perturbation generated than using
uniformly distributed random noise with the same magnitude.


http://arxiv.org/abs/2002.07214
Robust Stochastic Bandit Algorithms under Probabilistic Unbounded Adversarial Attack.
Ziwei Guan; Kaiyi Ji; Donald J Jr Bucci; Timothy Y Hu; Joseph Palombo; Michael Liston; Yingbin Liang
  The multi-armed bandit formalism has been extensively studied under various
attack models, in which an adversary can modify the reward revealed to the
player. Previous studies focused on scenarios where the attack value either is
bounded at each round or has a vanishing probability of occurrence. These
models do not capture powerful adversaries that can catastrophically perturb
the revealed reward. This paper investigates the attack model where an
adversary attacks with a certain probability at each round, and its attack
value can be arbitrary and unbounded if it attacks. Furthermore, the attack
value does not necessarily follow a statistical distribution. We propose a
novel sample median-based and exploration-aided UCB algorithm (called
med-E-UCB) and a median-based $\epsilon$-greedy algorithm (called
med-$\epsilon$-greedy). Both of these algorithms are provably robust to the
aforementioned attack model. More specifically we show that both algorithms
achieve $\mathcal{O}(\log T)$ pseudo-regret (i.e., the optimal regret without
attacks). We also provide a high probability guarantee of $\mathcal{O}(\log T)$
regret with respect to random rewards and random occurrence of attacks. These
bounds are achieved under arbitrary and unbounded reward perturbation as long
as the attack probability does not exceed a certain constant threshold. We
provide multiple synthetic simulations of the proposed algorithms to verify
these claims and showcase the inability of existing techniques to achieve
sublinear regret. We also provide experimental results of the algorithm
operating in a cognitive radio setting using multiple software-defined radios.


http://arxiv.org/abs/2002.07246
Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness.
Huijie Feng; Chunpeng Wu; Guoyang Chen; Weifeng Zhang; Yang Ning
  Recently smoothing deep neural network based classifiers via isotropic
Gaussian perturbation is shown to be an effective and scalable way to provide
state-of-the-art probabilistic robustness guarantee against $\ell_2$ norm
bounded adversarial perturbations. However, how to train a good base classifier
that is accurate and robust when smoothed has not been fully investigated. In
this work, we derive a new regularized risk, in which the regularizer can
adaptively encourage the accuracy and robustness of the smoothed counterpart
when training the base classifier. It is computationally efficient and can be
implemented in parallel with other empirical defense methods. We discuss how to
implement it under both standard (non-adversarial) and adversarial training
scheme. At the same time, we also design a new certification algorithm, which
can leverage the regularization effect to provide tighter robustness lower
bound that holds with high probability. Our extensive experimentation
demonstrates the effectiveness of the proposed training and certification
approaches on CIFAR-10 and ImageNet datasets.


http://arxiv.org/abs/2002.06668
Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality.
Yi Zhang; Orestis Plevrakis; Simon S. Du; Xingguo Li; Zhao Song; Sanjeev Arora
  Adversarial training is a popular method to give neural nets robustness
against adversarial perturbations. In practice adversarial training leads to
low robust training loss. However, a rigorous explanation for why this happens
under natural conditions is still missing. Recently a convergence theory for
standard (non-adversarial) supervised training was developed by various groups
for {\em very overparametrized} nets. It is unclear how to extend these results
to adversarial training because of the min-max objective. Recently, a first
step towards this direction was made by Gao et al. using tools from online
learning, but they require the width of the net to be \emph{exponential} in
input dimension $d$, and with an unnatural activation function. Our work proves
convergence to low robust training loss for \emph{polynomial} width instead of
exponential, under natural assumptions and with the ReLU activation. Key
element of our proof is showing that ReLU networks near initialization can
approximate the step function, which may be of independent interest.


http://arxiv.org/abs/2003.04808
Undersensitivity in Neural Reading Comprehension.
Johannes Welbl; Pasquale Minervini; Max Bartolo; Pontus Stenetorp; Sebastian Riedel
  Current reading comprehension models generalise well to in-distribution test
sets, yet perform poorly on adversarially selected inputs. Most prior work on
adversarial inputs studies oversensitivity: semantically invariant text
perturbations that cause a model's prediction to change when it should not. In
this work we focus on the complementary problem: excessive prediction
undersensitivity, where input text is meaningfully changed but the model's
prediction does not, even though it should. We formulate a noisy adversarial
attack which searches among semantic variations of the question for which a
model erroneously predicts the same answer, and with even higher probability.
Despite comprising unanswerable questions, both SQuAD2.0 and NewsQA models are
vulnerable to this attack. This indicates that although accurate, models tend
to rely on spurious patterns and do not fully consider the information
specified in a question. We experiment with data augmentation and adversarial
training as defences, and find that both substantially decrease vulnerability
to attacks on held out data, as well as held out attack spaces. Addressing
undersensitivity also improves results on AddSent and AddOneSent, and models
furthermore generalise better when facing train/evaluation distribution
mismatch: they are less prone to overly rely on predictive cues present only in
the training set, and outperform a conventional model by as much as 10.9% F1.


http://arxiv.org/abs/2002.06349
Hold me tight! Influence of discriminative features on deep network boundaries.
Guillermo Ortiz-Jimenez; Apostolos Modas; Seyed-Mohsen Moosavi-Dezfooli; Pascal Frossard
  Important insights towards the explainability of neural networks and their
properties reside in the formation of their decision boundaries. In this work,
we borrow tools from the field of adversarial robustness and propose a new
framework that permits to relate the features of the dataset with the distance
of data samples to the decision boundary along specific directions. We
demonstrate that the inductive bias of deep learning has the tendency to
generate classification functions that are invariant along non-discriminative
directions of the dataset. More surprisingly, we further show that training on
small perturbations of the data samples are sufficient to completely change the
decision boundary. This is actually the characteristic exploited by the
so-called adversarial training to produce robust classifiers. Our general
framework can be used to reveal the effect of specific dataset features on the
macroscopic properties of deep models and to develop a better understanding of
the successes and limitations of deep learning.


http://arxiv.org/abs/2002.06495
Blind Adversarial Network Perturbations.
Milad Nasr; Alireza Bahramali; Amir Houmansadr
  Deep Neural Networks (DNNs) are commonly used for various traffic analysis
problems, such as website fingerprinting and flow correlation, as they
outperform traditional (e.g., statistical) techniques by large margins.
However, deep neural networks are known to be vulnerable to adversarial
examples: adversarial inputs to the model that get labeled incorrectly by the
model due to small adversarial perturbations. In this paper, for the first
time, we show that an adversary can defeat DNN-based traffic analysis
techniques by applying \emph{adversarial perturbations} on the patterns of
\emph{live} network traffic.


http://arxiv.org/abs/2002.05999
Adversarial Distributional Training for Robust Deep Learning.
Zhijie Deng; Yinpeng Dong; Tianyu Pang; Hang Su; Jun Zhu
  Adversarial training (AT) is among the most effective techniques to improve
model robustness by augmenting training data with adversarial examples.
However, the adversarially trained models do not perform well enough on test
data or under other attack algorithms unseen during training, which remains to
be improved. In this paper, we introduce a novel adversarial distributional
training (ADT) framework for learning robust models. Specifically, we formulate
ADT as a minimax optimization problem, where the inner maximization aims to
learn an adversarial distribution to characterize the potential adversarial
examples around a natural one, and the outer minimization aims to train robust
classifiers by minimizing the expected loss over the worst-case adversarial
distributions. We conduct a theoretical analysis on how to solve the minimax
problem, leading to a general algorithm for ADT. We further propose three
different approaches to parameterize the adversarial distributions. Empirical
results on various benchmarks validate the effectiveness of ADT compared with
the state-of-the-art AT methods.


http://arxiv.org/abs/2002.05990
Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets.
Dongxian Wu; Yisen Wang; Shu-Tao Xia; James Bailey; Xingjun Ma
  Skip connections are an essential component of current state-of-the-art deep
neural networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt.
Despite their huge success in building deeper and more powerful DNNs, we
identify a surprising security weakness of skip connections in this paper. Use
of skip connections allows easier generation of highly transferable adversarial
examples. Specifically, in ResNet-like (with skip connections) neural networks,
gradients can backpropagate through either skip connections or residual
modules. We find that using more gradients from the skip connections rather
than the residual modules according to a decay factor, allows one to craft
adversarial examples with high transferability. Our method is termed Skip
Gradient Method(SGM). We conduct comprehensive transfer attacks against
state-of-the-art DNNs including ResNets, DenseNets, Inceptions,
Inception-ResNet, Squeeze-and-Excitation Network (SENet) and robustly trained
DNNs. We show that employing SGM on the gradient flow can greatly improve the
transferability of crafted attacks in almost all cases. Furthermore, SGM can be
easily combined with existing black-box attack techniques, and obtain high
improvements over state-of-the-art transferability methods. Our findings not
only motivate new research into the architectural vulnerability of DNNs, but
also open up further challenges for the design of secure DNN architectures.


http://arxiv.org/abs/2002.05388
Recurrent Attention Model with Log-Polar Mapping is Robust against Adversarial Attacks.
Taro Kiritani; Koji Ono
  Convolutional neural networks are vulnerable to small $\ell^p$ adversarial
attacks, while the human visual system is not. Inspired by neural networks in
the eye and the brain, we developed a novel artificial neural network model
that recurrently collects data with a log-polar field of view that is
controlled by attention. We demonstrate the effectiveness of this design as a
defense against SPSA and PGD adversarial attacks. It also has beneficial
properties observed in the animal visual system, such as reflex-like pathways
for low-latency inference, fixed amount of computation independent of image
size, and rotation and scale invariance. The code for experiments is available
at https://gitlab.com/exwzd-public/kiritani_ono_2020.


http://arxiv.org/abs/2002.05379
The Conditional Entropy Bottleneck.
Ian Fischer
  Much of the field of Machine Learning exhibits a prominent set of failure
modes, including vulnerability to adversarial examples, poor
out-of-distribution (OoD) detection, miscalibration, and willingness to
memorize random labelings of datasets. We characterize these as failures of
robust generalization, which extends the traditional measure of generalization
as accuracy or related metrics on a held-out set. We hypothesize that these
failures to robustly generalize are due to the learning systems retaining too
much information about the training data. To test this hypothesis, we propose
the Minimum Necessary Information (MNI) criterion for evaluating the quality of
a model. In order to train models that perform well with respect to the MNI
criterion, we present a new objective function, the Conditional Entropy
Bottleneck (CEB), which is closely related to the Information Bottleneck (IB).
We experimentally test our hypothesis by comparing the performance of CEB
models with deterministic models and Variational Information Bottleneck (VIB)
models on a variety of different datasets and robustness challenges. We find
strong empirical evidence supporting our hypothesis that MNI models improve on
these problems of robust generalization.


http://arxiv.org/abs/2002.05463
Identifying Audio Adversarial Examples via Anomalous Pattern Detection.
Victor Akinwande; Celia Cintas; Skyler Speakman; Srihari Sridharan
  Audio processing models based on deep neural networks are susceptible to
adversarial attacks even when the adversarial audio waveform is 99.9% similar
to a benign sample. Given the wide application of DNN-based audio recognition
systems, detecting the presence of adversarial examples is of high practical
relevance. By applying anomalous pattern detection techniques in the activation
space of these models, we show that 2 of the recent and current
state-of-the-art adversarial attacks on audio processing systems systematically
lead to higher-than-expected activation at some subset of nodes and we can
detect these with up to an AUC of 0.98 with no degradation in performance on
benign samples.


http://arxiv.org/abs/2002.05283
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization.
Xiangning Chen; Cho-Jui Hsieh
  Differentiable architecture search (DARTS) is a prevailing NAS solution to
identify architectures. Based on the continuous relaxation of the architecture
space, DARTS learns a differentiable architecture weight and largely reduces
the search cost. However, its stability and generalizability have been
challenged for yielding deteriorating architectures as the search proceeds. We
find that the precipitous validation loss landscape, which leads to a dramatic
performance drop when distilling the final architecture, is an essential factor
that causes instability. Based on this observation, we propose a
perturbation-based regularization, named SmoothDARTS (SDARTS), to smooth the
loss landscape and improve the generalizability of DARTS. In particular, our
new formulations stabilize DARTS by either random smoothing or adversarial
attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the
effectiveness of our approach and due to the improved stability, we achieve
performance gain across various search spaces on 4 datasets. Furthermore, we
mathematically show that SDARTS implicitly regularizes the Hessian norm of the
validation loss, which accounts for a smoother loss landscape and improved
performance. The code is available at
https://github.com/xiangning-chen/SmoothDARTS.


http://arxiv.org/abs/2002.05123
Patternless Adversarial Attacks on Video Recognition Networks.
Itay Naeh; Roi Pony; Shie Mannor
  Deep neural networks for classification of videos, just like image
classification networks, may be subjected to adversarial manipulation. The main
difference between image classifiers and video classifiers is that the latter
usually use temporal information contained within the video in the form of
optical flow or implicitly by various differences between adjacent frames. In
this work we present a manipulation scheme for fooling video classifiers by
introducing a spatial patternless temporal perturbation that is practically
unnoticed by human observers and undetectable by leading image adversarial
pattern detection algorithms. After demonstrating the manipulation of action
classification of single videos, we generalize the procedure to make
adversarial patterns with temporal invariance that generalizes across different
classes for both targeted and untargeted attacks.


http://arxiv.org/abs/2002.04359
Robustness of Bayesian Neural Networks to Gradient-Based Attacks.
Ginevra Carbone; Matthew Wicker; Luca Laurenti; Andrea Patane; Luca Bortolussi; Guido Sanguinetti
  Vulnerability to adversarial attacks is one of the principal hurdles to the
adoption of deep learning in safety-critical applications. Despite significant
efforts, both practical and theoretical, the problem remains open. In this
paper, we analyse the geometry of adversarial attacks in the large-data,
overparametrized limit for Bayesian Neural Networks (BNNs). We show that, in
the limit, vulnerability to gradient-based attacks arises as a result of
degeneracy in the data distribution, i.e., when the data lies on a
lower-dimensional submanifold of the ambient space. As a direct consequence, we
demonstrate that in the limit BNN posteriors are robust to gradient-based
adversarial attacks. Experimental results on the MNIST and Fashion MNIST
datasets with BNNs trained with Hamiltonian Monte Carlo and Variational
Inference support this line of argument, showing that BNNs can display both
high accuracy and robustness to gradient based adversarial attacks.


http://arxiv.org/abs/2002.04784
Graph Universal Adversarial Attacks: A Few Bad Actors Ruin Graph Learning Models.
Xiao Zang; Yi Xie; Jie Chen; Bo Yuan
  Deep neural networks, while generalize well, are known to be sensitive to
small adversarial perturbations. This phenomenon poses severe security threat
and calls for in-depth investigation of the robustness of deep learning models.
With the emergence of neural networks for graph structured data, similar
investigations are urged to understand their robustness. It has been found that
adversarially perturbing the graph structure and/or node features may result in
a significant degradation of the model performance. In this work, we show from
a different angle that such fragility similarly occurs if the graph contains a
few bad-actor nodes, which compromise a trained graph neural network through
flipping the connections to any targeted victim. Worse, the bad actors found
for one graph model severely compromise other models as well. We call the bad
actors "anchor nodes" and propose an algorithm, named GUA, to identify them.
Thorough empirical investigations suggest an interesting finding that the
anchor nodes often belong to the same class; and they also corroborate the
intuitive trade-off between the number of anchor nodes and the attack success
rate. For the data set Cora which contains 2708 nodes, as few as six anchor
nodes will result in an attack success rate higher than 80\% for GCN and other
three models.


http://arxiv.org/abs/2002.04725
More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models.
Lin Chen; Yifei Min; Mingrui Zhang; Amin Karbasi
  Despite remarkable success in practice, modern machine learning models have
been found to be susceptible to adversarial attacks that make
human-imperceptible perturbations to the data, but result in serious and
potentially dangerous prediction errors. To address this issue, practitioners
often use adversarial training to learn models that are robust against such
attacks at the cost of weaker generalization accuracy on unperturbed test sets.
The conventional wisdom is that more training data should shrink the
generalization gap between adversarially-trained models and standard models.
However, we study the training of robust classifiers for both Gaussian and
Bernoulli models under $\ell_\infty$ attacks, and we prove that more data may
actually increase this gap. Furthermore, our theoretical results identify if
and when additional data will finally begin to shrink the gap. Lastly, we
experimentally demonstrate that our results also hold for linear regression
models, which may indicate that this phenomenon occurs more broadly.


http://arxiv.org/abs/2002.04599
Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations.
Florian Tramèr; Jens Behrmann; Nicholas Carlini; Nicolas Papernot; Jörn-Henrik Jacobsen
  Adversarial examples are malicious inputs crafted to induce
misclassification. Commonly studied sensitivity-based adversarial examples
introduce semantically-small changes to an input that result in a different
model prediction. This paper studies a complementary failure mode,
invariance-based adversarial examples, that introduce minimal semantic changes
that modify an input's true label yet preserve the model's prediction. We
demonstrate fundamental tradeoffs between these two types of adversarial
examples.
  We show that defenses against sensitivity-based attacks actively harm a
model's accuracy on invariance-based attacks, and that new approaches are
needed to resist both attack types. In particular, we break state-of-the-art
adversarially-trained and certifiably-robust models by generating small
perturbations that the models are (provably) robust to, yet that change an
input's class according to human labelers. Finally, we formally show that the
existence of excessively invariant classifiers arises from the presence of
overly-robust predictive features in standard datasets.


http://arxiv.org/abs/2002.04237
Improving the affordability of robustness training for DNNs.
Sidharth Gupta; Parijat Dube; Ashish Verma
  Projected Gradient Descent (PGD) based adversarial training has become one of
the most prominent methods for building robust deep neural network models.
However, the computational complexity associated with this approach, due to the
maximization of the loss function when finding adversaries, is a longstanding
problem and may be prohibitive when using larger and more complex models. In
this paper we show that the initial phase of adversarial training is redundant
and can be replaced with natural training which significantly improves the
computational efficiency. We demonstrate that this efficiency gain can be
achieved without any loss in accuracy on natural and adversarial test samples.
We support our argument with insights on the nature of the adversaries and
their relative strength during the training process. We show that our proposed
method can reduce the training time by a factor of up to 2.5 with comparable or
better model test accuracy and generalization on various strengths of
adversarial attacks.


http://arxiv.org/abs/2002.03924
Playing to Learn Better: Repeated Games for Adversarial Learning with Multiple Classifiers.
Prithviraj Dasgupta; Joseph B. Collins; Michael McCarrick
  We consider the problem of prediction by a machine learning algorithm, called
learner, within an adversarial learning setting. The learner's task is to
correctly predict the class of data passed to it as a query. However, along
with queries containing clean data, the learner could also receive malicious or
adversarial queries from an adversary. The objective of the adversary is to
evade the learner's prediction mechanism by sending adversarial queries that
result in erroneous class prediction by the learner, while the learner's
objective is to reduce the incorrect prediction of these adversarial queries
without degrading the prediction quality of clean queries. We propose a game
theory-based technique called a Repeated Bayesian Sequential Game where the
learner interacts repeatedly with a model of the adversary using self play to
determine the distribution of adversarial versus clean queries. It then
strategically selects a classifier from a set of pre-trained classifiers that
balances the likelihood of correct prediction for the query along with reducing
the costs to use the classifier. We have evaluated our proposed technique using
clean and adversarial text data with deep neural network-based classifiers and
shown that the learner can select an appropriate classifier that is
commensurate with the query type (clean or adversarial) while remaining aware
of the cost to use the classifier.


http://arxiv.org/abs/2002.03793
Adversarial Data Encryption.
Yingdong Hu; Liang Zhang; Wei Shan; Xiaoxiao Qin; Jing Qi; Zhenzhou Wu; Yang Yuan
  In the big data era, many organizations face the dilemma of data sharing.
Regular data sharing is often necessary for human-centered discussion and
communication, especially in medical scenarios. However, unprotected data
sharing may also lead to data leakage. Inspired by adversarial attack, we
propose a method for data encryption, so that for human beings the encrypted
data look identical to the original version, but for machine learning methods
they are misleading. To show the effectiveness of our method, we collaborate
with the Beijing Tiantan Hospital, which has a world leading neurological
center. We invite $3$ doctors to manually inspect our encryption method based
on real world medical images. The results show that the encrypted images can be
used for diagnosis by the doctors, but not by machine learning methods.


http://arxiv.org/abs/2002.04197
Generalised Lipschitz Regularisation Equals Distributional Robustness.
Zac Cranko; Zhan Shi; Xinhua Zhang; Richard Nock; Simon Kornblith
  The problem of adversarial examples has highlighted the need for a theory of
regularisation that is general enough to apply to exotic function classes, such
as universal approximators. In response, we give a very general equality result
regarding the relationship between distributional robustness and
regularisation, as defined with a transportation cost uncertainty set. The
theory allows us to (tightly) certify the robustness properties of a
Lipschitz-regularised model with very mild assumptions. As a theoretical
application we show a new result explicating the connection between adversarial
learning and distributional robustness. We then give new results for how to
achieve Lipschitz regularisation of kernel classifiers, which are demonstrated
experimentally.


http://arxiv.org/abs/2002.03421
Certified Robustness of Community Detection against Adversarial Structural Perturbation via Randomized Smoothing.
Jinyuan Jia; Binghui Wang; Xiaoyu Cao; Neil Zhenqiang Gong
  Community detection plays a key role in understanding graph structure.
However, several recent studies showed that community detection is vulnerable
to adversarial structural perturbation. In particular, via adding or removing a
small number of carefully selected edges in a graph, an attacker can manipulate
the detected communities. However, to the best of our knowledge, there are no
studies on certifying robustness of community detection against such
adversarial structural perturbation. In this work, we aim to bridge this gap.
Specifically, we develop the first certified robustness guarantee of community
detection against adversarial structural perturbation. Given an arbitrary
community detection method, we build a new smoothed community detection method
via randomly perturbing the graph structure. We theoretically show that the
smoothed community detection method provably groups a given arbitrary set of
nodes into the same community (or different communities) when the number of
edges added/removed by an attacker is bounded. Moreover, we show that our
certified robustness is tight. We also empirically evaluate our method on
multiple real-world graphs with ground truth communities.


http://arxiv.org/abs/2002.03331
MDEA: Malware Detection with Evolutionary Adversarial Learning.
Xiruo Wang; Risto Miikkulainen
  Malware detection have used machine learning to detect malware in programs.
These applications take in raw or processed binary data to neural network
models to classify as benign or malicious files. Even though this approach has
proven effective against dynamic changes, such as encrypting, obfuscating and
packing techniques, it is vulnerable to specific evasion attacks where that
small changes in the input data cause misclassification at test time. This
paper proposes a new approach: MDEA, an Adversarial Malware Detection model
uses evolutionary optimization to create attack samples to make the network
robust against evasion attacks. By retraining the model with the evolved
malware samples, its performance improves a significant margin.


http://arxiv.org/abs/2002.03339
Input Validation for Neural Networks via Runtime Local Robustness Verification.
Jiangchao Liu; Liqian Chen; Antoine Mine; Ji Wang
  Local robustness verification can verify that a neural network is robust wrt.
any perturbation to a specific input within a certain distance. We call this
distance Robustness Radius. We observe that the robustness radii of correctly
classified inputs are much larger than that of misclassified inputs which
include adversarial examples, especially those from strong adversarial attacks.
Another observation is that the robustness radii of correctly classified inputs
often follow a normal distribution. Based on these two observations, we propose
to validate inputs for neural networks via runtime local robustness
verification. Experiments show that our approach can protect neural networks
from adversarial examples and improve their accuracies.


http://arxiv.org/abs/2002.03444
Robust binary classification with the 01 loss.
Yunzhe Xue; Meiyan Xie; Usman Roshan
  The 01 loss is robust to outliers and tolerant to noisy data compared to
convex loss functions. We conjecture that the 01 loss may also be more robust
to adversarial attacks. To study this empirically we have developed a
stochastic coordinate descent algorithm for a linear 01 loss classifier and a
single hidden layer 01 loss neural network. Due to the absence of the gradient
we iteratively update coordinates on random subsets of the data for fixed
epochs. We show our algorithms to be fast and comparable in accuracy to the
linear support vector machine and logistic loss single hidden layer network for
binary classification on several image benchmarks, thus establishing that our
method is on-par in test accuracy with convex losses. We then subject them to
accurately trained substitute model black box attacks on the same image
benchmarks and find them to be more robust than convex counterparts. On CIFAR10
binary classification task between classes 0 and 1 with adversarial
perturbation of 0.0625 we see that the MLP01 network loses 27\% in accuracy
whereas the MLP-logistic counterpart loses 83\%. Similarly on STL10 and
ImageNet binary classification between classes 0 and 1 the MLP01 network loses
21\% and 20\% while MLP-logistic loses 67\% and 45\% respectively. On MNIST
that is a well-separable dataset we find MLP01 comparable to MLP-logistic and
show under simulation how and why our 01 loss solver is less robust there. We
then propose adversarial training for our linear 01 loss solver that
significantly improves its robustness on MNIST and all other datasets and
retains clean test accuracy. Finally we show practical applications of our
method to deter traffic sign and facial recognition adversarial attacks. We
discuss attacks with 01 loss, substitute model accuracy, and several future
avenues like multiclass, 01 loss convolutions, and further adversarial
training.


http://arxiv.org/abs/2002.03500
ABBA: Saliency-Regularized Motion-Based Adversarial Blur Attack.
Qing Guo; Felix Juefei-Xu; Xiaofei Xie; Lei Ma; Jian Wang; Wei Feng; Yang Liu
  Deep neural networks are vulnerable to noise-based adversarial examples,
which can mislead the networks by adding random-like noise. However, such
examples are hardly found in the real world and easily perceived when thumping
noises are used to keep their high transferability across different models. In
this paper, we identify a new attacking method termed motion-based adversarial
blur attack (ABBA) that can generate visually natural motion-blurred
adversarial examples even with relatively high perturbation, allowing much
better transferability than noise-based methods. To this end, we first
formulate the kernel-prediction-based attack where an input image is convolved
with kernels in a pixel-wise way, and the misclassification capability is
achieved by tuning the kernel weights. To generate visually more natural and
plausible examples, we further propose the saliency-regularized adversarial
kernel prediction where the salient region serves as a moving object, and the
predicted kernel is regularized to achieve naturally visual effects. Besides,
the attack can be further enhanced by adaptively tuning the translations of
object and background. Extensive experimental results on the NeurIPS'17
adversarial competition dataset validate the effectiveness of ABBA by
considering various kernel sizes, translations, and regions. Furthermore, we
study the effects of state-of-the-art GAN-based deblurring mechanisms to our
methods.


http://arxiv.org/abs/2002.05517
Feature-level Malware Obfuscation in Deep Learning.
Keith Dillon
  We consider the problem of detecting malware with deep learning models, where
the malware may be combined with significant amounts of benign code. Examples
of this include piggybacking and trojan horse attacks on a system, where
malicious behavior is hidden within a useful application. Such added
flexibility in augmenting the malware enables significantly more code
obfuscation. Hence we focus on the use of static features, particularly
Intents, Permissions, and API calls, which we presume cannot be ultimately
hidden from the Android system, but only augmented with yet more such features.
We first train a deep neural network classifier for malware classification
using features of benign and malware samples. Then we demonstrate a steep
increase in false negative rate (i.e., attacks succeed), simply by randomly
adding features of a benign app to malware. Finally we test the use of data
augmentation to harden the classifier against such attacks. We find that for
API calls, it is possible to reject the vast majority of attacks, where using
Intents or Permissions is less successful.


http://arxiv.org/abs/2002.12749
Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples.
Paarth Neekhara; Shehzeen Hussain; Malhar Jere; Farinaz Koushanfar; Julian McAuley
  Recent advances in video manipulation techniques have made the generation of
fake videos more accessible than ever before. Manipulated videos can fuel
disinformation and reduce trust in media. Therefore detection of fake videos
has garnered immense interest in academia and industry. Recently developed
Deepfake detection methods rely on deep neural networks (DNNs) to distinguish
AI-generated fake videos from real videos. In this work, we demonstrate that it
is possible to bypass such detectors by adversarially modifying fake videos
synthesized using existing Deepfake generation methods. We further demonstrate
that our adversarial perturbations are robust to image and video compression
codecs, making them a real-world threat. We present pipelines in both white-box
and black-box attack scenarios that can fool DNN based Deepfake detectors into
classifying fake videos as real.


http://arxiv.org/abs/2003.04367
Category-wise Attack: Transferable Adversarial Examples for Anchor Free Object Detection.
Quanyu Liao; Xin Wang; Bin Kong; Siwei Lyu; Youbing Yin; Qi Song; Xi Wu
  Deep neural networks have been demonstrated to be vulnerable to adversarial
attacks: subtle perturbations can completely change the classification results.
Their vulnerability has led to a surge of research in this direction. However,
most works dedicated to attacking anchor-based object detection models. In this
work, we aim to present an effective and efficient algorithm to generate
adversarial examples to attack anchor-free object models based on two
approaches. First, we conduct category-wise instead of instance-wise attacks on
the object detectors. Second, we leverage the high-level semantic information
to generate the adversarial examples. Surprisingly, the generated adversarial
examples it not only able to effectively attack the targeted anchor-free object
detector but also to be transferred to attack other object detectors, even
anchor-based detectors such as Faster R-CNN.


http://arxiv.org/abs/2002.03517
Random Smoothing Might be Unable to Certify $\ell_\infty$ Robustness for High-Dimensional Images.
Avrim Blum; Travis Dick; Naren Manoj; Hongyang Zhang
  We show a hardness result for random smoothing to achieve certified
adversarial robustness against attacks in the $\ell_p$ ball of radius
$\epsilon$ when $p>2$. Although random smoothing has been well understood for
the $\ell_2$ case using the Gaussian distribution, much remains unknown
concerning the existence of a noise distribution that works for the case of
$p>2$. This has been posed as an open problem by Cohen et al. (2019) and
includes many significant paradigms such as the $\ell_\infty$ threat model. In
this work, we show that any noise distribution $\mathcal{D}$ over
$\mathbb{R}^d$ that provides $\ell_p$ robustness for all base classifiers with
$p>2$ must satisfy
$\mathbb{E}\eta_i^2=\Omega(d^{1-2/p}\epsilon^2(1-\delta)/\delta^2)$ for 99% of
the features (pixels) of vector $\eta\sim\mathcal{D}$, where $\epsilon$ is the
robust radius and $\delta$ is the score gap between the highest-scored class
and the runner-up. Therefore, for high-dimensional images with pixel values
bounded in $[0,255]$, the required noise will eventually dominate the useful
information in the images, leading to trivial smoothed classifiers.


http://arxiv.org/abs/2002.03239
Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness.
Aounon Kumar; Alexander Levine; Tom Goldstein; Soheil Feizi
  Randomized smoothing, using just a simple isotropic Gaussian distribution,
has been shown to produce good robustness guarantees against $\ell_2$-norm
bounded adversaries. In this work, we show that extending the smoothing
technique to defend against other attack models can be challenging, especially
in the high-dimensional regime. In particular, for a vast class of i.i.d.
smoothing distributions, we prove that the largest $\ell_p$-radius that can be
certified decreases as $O(1/d^{\frac{1}{2} - \frac{1}{p}})$ with dimension $d$
for $p > 2$. Notably, for $p \geq 2$, this dependence on $d$ is no better than
that of the $\ell_p$-radius that can be certified using isotropic Gaussian
smoothing, essentially putting a matching lower bound on the robustness radius.
When restricted to generalized Gaussian smoothing, these two bounds can be
shown to be within a constant factor of each other in an asymptotic sense,
establishing that Gaussian smoothing provides the best possible results, up to
a constant factor, when $p \geq 2$. We present experimental results on CIFAR to
validate our theory. For other smoothing distributions, such as, a uniform
distribution within an $\ell_1$ or an $\ell_\infty$-norm ball, we show upper
bounds of the form $O(1 / d)$ and $O(1 / d^{1 - \frac{1}{p}})$ respectively,
which have an even worse dependence on $d$.


http://arxiv.org/abs/2002.03095
Attacking Optical Character Recognition (OCR) Systems with Adversarial Watermarks.
Lu Chen; Wei Xu
  Optical character recognition (OCR) is widely applied in real applications
serving as a key preprocessing tool. The adoption of deep neural network (DNN)
in OCR results in the vulnerability against adversarial examples which are
crafted to mislead the output of the threat model. Different from vanilla
colorful images, images of printed text have clear backgrounds usually.
However, adversarial examples generated by most of the existing adversarial
attacks are unnatural and pollute the background severely. To address this
issue, we propose a watermark attack method to produce natural distortion that
is in the disguise of watermarks and evade human eyes' detection. Experimental
results show that watermark attacks can yield a set of natural adversarial
examples attached with watermarks and attain similar attack performance to the
state-of-the-art methods in different attack scenarios.


http://arxiv.org/abs/2002.02998
Improving the Adversarial Robustness of Transfer Learning via Noisy Feature Distillation.
Ting-Wu Chin; Cha Zhang; Diana Marculescu
  Fine-tuning through knowledge transfer from a pre-trained model on a
large-scale dataset is a widely spread approach to effectively build models on
small-scale datasets. However, recent literature has shown that such a
fine-tuning approach is vulnerable to adversarial examples based on the
pre-trained model, which raises security concerns for many industrial
applications. In contrast, models trained with random initialization are much
more robust to such attacks, although these models often exhibit much lower
accuracy. In this work, we propose noisy feature distillation, a new transfer
learning method that trains a network from random initialization while
achieving clean-data performance competitive with fine-tuning. In addition, the
method is shown empirically to significantly improve the robustness compared to
fine-tuning with 15x reduction in attack success rate for ResNet-50, from 66%
to 4.4% averaged across Stanford 120 Dogs, Caltech-UCSD 200 Birds, Stanford 40
Actions, MIT 67 Indoor Scenes, and Oxford 102 Flowers datasets. Code is
available at https://github.com/cmu-enyac/Renofeation.


http://arxiv.org/abs/2002.03043
Semantic Robustness of Models of Source Code.
Goutham Ramakrishnan; Jordan Henkel; Zi Wang; Aws Albarghouthi; Somesh Jha; Thomas Reps
  Deep neural networks are vulnerable to adversarial examples - small input
perturbations that result in incorrect predictions. We study this problem in
the context of models of source code, where we want the network to be robust to
source-code modifications that preserve code functionality. We define a natural
notion of robustness, $k$-transformation robustness, in which an adversary
performs up to $k$ semantics-preserving transformations to an input program. We
show how to train robust models using an adversarial training objective
inspired by that of Madry et al. (2018) for continuous domains.
  We implement an extensible framework for adversarial training over source
code, and conduct a thorough evaluation on a number of datasets and two
different architectures. Our results show (1) the increase in robustness
following adversarial training, (2) the ability of training on weak adversaries
to provide robustness to attacks by stronger adversaries, and (3) the shift in
attribution focus of adversarially trained models towards semantic vs.
syntactic features.


http://arxiv.org/abs/2002.03080
An Empirical Evaluation of Perturbation-based Defenses.
Adam Dziedzic; Sanjay Krishnan
  Recent work has extensively shown that randomized perturbations of a neural
network can improve its robustness to adversarial attacks. The literature is,
however, lacking a detailed compare-and-contrast of the latest proposals to
understand what classes of perturbations work, when they work, and why they
work. We contribute a detailed experimental evaluation that elucidates these
questions and benchmarks perturbation defenses in a consistent way. In
particular, we show five main results: (1) all input perturbation defenses,
whether random or deterministic, are essentially equivalent in their efficacy,
(2) such defenses offer almost no robustness to adaptive attacks unless these
perturbations are observed during training, (3) a tuned sequence of noise
layers across a network provides the best empirical robustness, (4) attacks
transfer between perturbation defenses so the attackers need not know the
specific type of defense only that it involves perturbations, and (5)
adversarial examples very close to original images show an elevated sensitivity
to perturbation in a first-order analysis. Based on these insights, we
demonstrate a new robust model built on noise injection and adversarial
training that achieves state-of-the-art robustness.


http://arxiv.org/abs/2002.02776
RAID: Randomized Adversarial-Input Detection for Neural Networks.
Hasan Ferit Eniser; Maria Christakis; Valentin Wüstholz
  In recent years, neural networks have become the default choice for image
classification and many other learning tasks, even though they are vulnerable
to so-called adversarial attacks. To increase their robustness against these
attacks, there have emerged numerous detection mechanisms that aim to
automatically determine if an input is adversarial. However, state-of-the-art
detection mechanisms either rely on being tuned for each type of attack, or
they do not generalize across different attack types. To alleviate these
issues, we propose a novel technique for adversarial-image detection, RAID,
that trains a secondary classifier to identify differences in neuron activation
values between benign and adversarial inputs. Our technique is both more
reliable and more effective than the state of the art when evaluated against
six popular attacks. Moreover, a straightforward extension of RAID increases
its robustness against detection-aware adversaries without affecting its
effectiveness.


http://arxiv.org/abs/2002.02842
Assessing the Adversarial Robustness of Monte Carlo and Distillation Methods for Deep Bayesian Neural Network Classification.
Meet P. Vadera; Satya Narayan Shukla; Brian Jalaian; Benjamin M. Marlin
  In this paper, we consider the problem of assessing the adversarial
robustness of deep neural network models under both Markov chain Monte Carlo
(MCMC) and Bayesian Dark Knowledge (BDK) inference approximations. We
characterize the robustness of each method to two types of adversarial attacks:
the fast gradient sign method (FGSM) and projected gradient descent (PGD). We
show that full MCMC-based inference has excellent robustness, significantly
outperforming standard point estimation-based learning. On the other hand, BDK
provides marginal improvements. As an additional contribution, we present a
storage-efficient approach to computing adversarial examples for large Monte
Carlo ensembles using both the FGSM and PGD attacks.


http://arxiv.org/abs/2002.02424
Reliability Validation of Learning Enabled Vehicle Tracking.
Youcheng Sun; Yifan Zhou; Simon Maskell; James Sharp; Xiaowei Huang
  This paper studies the reliability of a real-world learning-enabled system,
which conducts dynamic vehicle tracking based on a high-resolution wide-area
motion imagery input. The system consists of multiple neural network components
-- to process the imagery inputs -- and multiple symbolic (Kalman filter)
components -- to analyse the processed information for vehicle tracking. It is
known that neural networks suffer from adversarial examples, which make them
lack robustness. However, it is unclear if and how the adversarial examples
over learning components can affect the overall system-level reliability. By
integrating a coverage-guided neural network testing tool, DeepConcolic, with
the vehicle tracking system, we found that (1) the overall system can be
resilient to some adversarial examples thanks to the existence of other
components, and (2) the overall system presents an extra level of uncertainty
which cannot be determined by analysing the deep learning components only. This
research suggests the need for novel verification and validation methods for
learning-enabled systems.


http://arxiv.org/abs/2002.02175
An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models.
Yao Deng; Xi Zheng; Tianyi Zhang; Chen Chen; Guannan Lou; Miryung Kim
  Nowadays, autonomous driving has attracted much attention from both industry
and academia. Convolutional neural network (CNN) is a key component in
autonomous driving, which is also increasingly adopted in pervasive computing
such as smartphones, wearable devices, and IoT networks. Prior work shows
CNN-based classification models are vulnerable to adversarial attacks. However,
it is uncertain to what extent regression models such as driving models are
vulnerable to adversarial attacks, the effectiveness of existing defense
techniques, and the defense implications for system and middleware builders.
This paper presents an in-depth analysis of five adversarial attacks and four
defense methods on three driving models. Experiments show that, similar to
classification models, these models are still highly vulnerable to adversarial
attacks. This poses a big security threat to autonomous driving and thus should
be taken into account in practice. While these defense methods can effectively
defend against different attacks, none of them are able to provide adequate
protection against all five attacks. We derive several implications for system
and middleware builders: (1) when adding a defense component against
adversarial attacks, it is important to deploy multiple defense methods in
tandem to achieve a good coverage of various attacks, (2) a blackbox attack is
much less effective compared with a white-box attack, implying that it is
important to keep model details (e.g., model architecture, hyperparameters)
confidential via model obfuscation, and (3) driving models with a complex
architecture are preferred if computing resources permit as they are more
resilient to adversarial attacks than simple models.


http://arxiv.org/abs/2002.02196
AI-GAN: Attack-Inspired Generation of Adversarial Examples.
Tao Bai; Jun Zhao; Jinlin Zhu; Shoudong Han; Jiefeng Chen; Bo Li
  Adversarial examples that can fool deep models are mainly crafted by adding
small perturbations imperceptible to human eyes. There are various
optimization-based methods in the literature to generate adversarial
perturbations, most of which are time-consuming. AdvGAN, a method proposed by
Xiao~\emph{et al.}~in IJCAI~2018, employs Generative Adversarial Networks (GAN)
to generate adversarial perturbation with original images as inputs, which is
faster than optimization-based methods at inference time. AdvGAN, however,
fixes the target classes in the training and we find it difficult to train
AdvGAN when it is modified to take original images and target classes as
inputs. In this paper, we propose \mbox{Attack-Inspired} GAN (\mbox{AI-GAN})
with a different training strategy to solve this problem. \mbox{AI-GAN} is a
two-stage method, in which we use projected gradient descent (PGD) attack to
inspire the training of GAN in the first stage and apply standard training of
GAN in the second stage.
  Once trained, the Generator can approximate the conditional distribution of
adversarial instances and generate \mbox{imperceptible} adversarial
perturbations given different target classes. We conduct experiments and
evaluate the performance of \mbox{AI-GAN} on MNIST and \mbox{CIFAR-10}.
Compared with AdvGAN, \mbox{AI-GAN} achieves higher attack success rates with
similar perturbation magnitudes.


http://arxiv.org/abs/2002.02400
Over-the-Air Adversarial Attacks on Deep Learning Based Modulation Classifier over Wireless Channels.
Brian Kim; Yalin E. Sagduyu; Kemal Davaslioglu; Tugba Erpek; Sennur Ulukus
  We consider a wireless communication system that consists of a transmitter, a
receiver, and an adversary. The transmitter transmits signals with different
modulation types, while the receiver classifies its received signals to
modulation types using a deep learning-based classifier. In the meantime, the
adversary makes over-the-air transmissions that are received as superimposed
with the transmitter's signals to fool the classifier at the receiver into
making errors. While this evasion attack has received growing interest
recently, the channel effects from the adversary to the receiver have been
ignored so far such that the previous attack mechanisms cannot be applied under
realistic channel effects. In this paper, we present how to launch a realistic
evasion attack by considering channels from the adversary to the receiver. Our
results show that modulation classification is vulnerable to an adversarial
attack over a wireless channel that is modeled as Rayleigh fading with path
loss and shadowing. We present various adversarial attacks with respect to
availability of information about channel, transmitter input, and classifier
architecture. First, we present two types of adversarial attacks, namely a
targeted attack (with minimum power) and non-targeted attack that aims to
change the classification to a target label or to any other label other than
the true label, respectively. Both are white-box attacks that are transmitter
input-specific and use channel information. Then we introduce an algorithm to
generate adversarial attacks using limited channel information where the
adversary only knows the channel distribution. Finally, we present a black-box
universal adversarial perturbation (UAP) attack where the adversary has limited
knowledge about both channel and transmitter input.


http://arxiv.org/abs/2002.01810
Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study.
David Mickisch; Felix Assion; Florens Greßner; Wiebke Günther; Mariele Motta
  Despite achieving remarkable performance on many image classification tasks,
state-of-the-art machine learning (ML) classifiers remain vulnerable to small
input perturbations. Especially, the existence of adversarial examples raises
concerns about the deployment of ML models in safety- and security-critical
environments, like autonomous driving and disease detection. Over the last few
years, numerous defense methods have been published with the goal of improving
adversarial as well as corruption robustness. However, the proposed measures
succeeded only to a very limited extent. This limited progress is partly due to
the lack of understanding of the decision boundary and decision regions of deep
neural networks. Therefore, we study the minimum distance of data points to the
decision boundary and how this margin evolves over the training of a deep
neural network. By conducting experiments on MNIST, FASHION-MNIST, and
CIFAR-10, we observe that the decision boundary moves closer to natural images
over training. This phenomenon even remains intact in the late epochs of
training, where the classifier already obtains low training and test error
rates. On the other hand, adversarial training appears to have the potential to
prevent this undesired convergence of the decision boundary.


http://arxiv.org/abs/2002.01147
Adversarially Robust Frame Sampling with Bounded Irregularities.
Hanhan Li; Pin Wang
  In recent years, video analysis tools for automatically extracting meaningful
information from videos are widely studied and deployed. Because most of them
use deep neural networks which are computationally expensive, feeding only a
subset of video frames into such algorithms is desired. Sampling the frames
with fixed rate is always attractive for its simplicity, representativeness,
and interpretability. For example, a popular cloud video API generated video
and shot labels by processing only the first frame of every second in a video.
However, one can easily attack such strategies by placing chosen frames at the
sampled locations. In this paper, we present an elegant solution to this
sampling problem that is provably robust against adversarial attacks and
introduces bounded irregularities as well.


http://arxiv.org/abs/2002.01249
Adversarial Attacks to Scale-Free Networks: Testing the Robustness of Physical Criteria.
Qi Xuan; Yalu Shan; Jinhuan Wang; Zhongyuan Ruan; Guanrong Chen
  Adversarial attacks have been alerting the artificial intelligence community
recently, since many machine learning algorithms were found vulnerable to
malicious attacks. This paper studies adversarial attacks to scale-free
networks to test their robustness in terms of statistical measures. In addition
to the well-known random link rewiring (RLR) attack, two heuristic attacks are
formulated and simulated: degree-addition-based link rewiring (DALR) and
degree-interval-based link rewiring (DILR). These three strategies are applied
to attack a number of strong scale-free networks of various sizes generated
from the Barab\'asi-Albert model. It is found that both DALR and DILR are more
effective than RLR, in the sense that rewiring a smaller number of links can
succeed in the same attack. However, DILR is as concealed as RLR in the sense
that they both are constructed by introducing a relatively small number of
changes on several typical structural properties such as average shortest
path-length, average clustering coefficient, and average diagonal distance. The
results of this paper suggest that to classify a network to be scale-free has
to be very careful from the viewpoint of adversarial attack effects.


http://arxiv.org/abs/2002.01256
Minimax Defense against Gradient-based Adversarial Attacks.
Blerta Lindqvist; Rauf Izmailov
  State-of-the-art adversarial attacks are aimed at neural network classifiers.
By default, neural networks use gradient descent to minimize their loss
function. The gradient of a classifier's loss function is used by
gradient-based adversarial attacks to generate adversarially perturbed images.
We pose the question whether another type of optimization could give neural
network classifiers an edge. Here, we introduce a novel approach that uses
minimax optimization to foil gradient-based adversarial attacks. Our minimax
classifier is the discriminator of a generative adversarial network (GAN) that
plays a minimax game with the GAN generator. In addition, our GAN generator
projects all points onto a manifold that is different from the original
manifold since the original manifold might be the cause of adversarial attacks.
To measure the performance of our minimax defense, we use adversarial attacks -
Carlini Wagner (CW), DeepFool, Fast Gradient Sign Method (FGSM) - on three
datasets: MNIST, CIFAR-10 and German Traffic Sign (TRAFFIC). Against CW
attacks, our minimax defense achieves 98.07% (MNIST-default 98.93%), 73.90%
(CIFAR-10-default 83.14%) and 94.54% (TRAFFIC-default 96.97%). Against DeepFool
attacks, our minimax defense achieves 98.87% (MNIST), 76.61% (CIFAR-10) and
94.57% (TRAFFIC). Against FGSM attacks, we achieve 97.01% (MNIST), 76.79%
(CIFAR-10) and 81.41% (TRAFFIC). Our Minimax adversarial approach presents a
significant shift in defense strategy for neural network classifiers.


http://arxiv.org/abs/2002.01008
A Differentiable Color Filter for Generating Unrestricted Adversarial Images.
Zhengyu Zhao; Zhuoran Liu; Martha Larson
  We propose Adversarial Color Filtering (AdvCF), an approach that uses a
differentiable color filter to create adversarial images. The color filter
allows us to introduce large perturbations into images, while still maintaining
or enhancing their photographic quality and appeal. AdvCF is motivated by
properties that are necessary if adversarial images are to be used to protect
the content of images shared online from unethical machine learning
classifiers: First, perturbations must be imperceptible and adversarial images
must look realistic to the human eye. Second, adversarial impact must be
maintained in the face of classifiers unknown when the perturbations are
generated (transferability). The paper presents evidence that AdvCF has these
two properties, and also points out that AdvCF has the potential for further
improvement if image semantics are taken into account.


http://arxiv.org/abs/2002.00614
Regularizers for Single-step Adversarial Training.
B. S. Vivek; R. Venkatesh Babu
  The progress in the last decade has enabled machine learning models to
achieve impressive performance across a wide range of tasks in Computer Vision.
However, a plethora of works have demonstrated the susceptibility of these
models to adversarial samples. Adversarial training procedure has been proposed
to defend against such adversarial attacks. Adversarial training methods
augment mini-batches with adversarial samples, and typically single-step
(non-iterative) methods are used for generating these adversarial samples.
However, models trained using single-step adversarial training converge to
degenerative minima where the model merely appears to be robust. The pseudo
robustness of these models is due to the gradient masking effect. Although
multi-step adversarial training helps to learn robust models, they are hard to
scale due to the use of iterative methods for generating adversarial samples.
To address these issues, we propose three different types of regularizers that
help to learn robust models using single-step adversarial training methods. The
proposed regularizers mitigate the effect of gradient masking by harnessing on
properties that differentiate a robust model from that of a pseudo robust
model. Performance of models trained using the proposed regularizers is on par
with models trained using computationally expensive multi-step adversarial
training methods.


http://arxiv.org/abs/2002.02007
Defending Adversarial Attacks via Semantic Feature Manipulation.
Shuo Wang; Tianle Chen; Surya Nepal; Carsten Rudolph; Marthie Grobler; Shangyu Chen
  Machine learning models have demonstrated vulnerability to adversarial
attacks, more specifically misclassification of adversarial examples. In this
paper, we propose a one-off and attack-agnostic Feature Manipulation
(FM)-Defense to detect and purify adversarial examples in an interpretable and
efficient manner. The intuition is that the classification result of a normal
image is generally resistant to non-significant intrinsic feature changes,
e.g., varying thickness of handwritten digits. In contrast, adversarial
examples are sensitive to such changes since the perturbation lacks
transferability. To enable manipulation of features, a combo-variational
autoencoder is applied to learn disentangled latent codes that reveal semantic
features. The resistance to classification change over the morphs, derived by
varying and reconstructing latent codes, is used to detect suspicious inputs.
Further, combo-VAE is enhanced to purify the adversarial examples with good
quality by considering both class-shared and class-unique features. We
empirically demonstrate the effectiveness of detection and the quality of
purified instance. Our experiments on three datasets show that FM-Defense can
detect nearly $100\%$ of adversarial examples produced by different
state-of-the-art adversarial attacks. It achieves more than $99\%$ overall
purification accuracy on the suspicious instances that close the manifold of
normal examples.


http://arxiv.org/abs/2002.00526
Robust saliency maps with decoy-enhanced saliency score.
Yang Lu; Wenbo Guo; Xinyu Xing; William Stafford Noble
  Saliency methods help to make deep neural network predictions more
interpretable by identifying particular features, such as pixels in an image,
that contribute most strongly to the network's prediction. Unfortunately,
recent evidence suggests that many saliency methods perform poorly when
gradients are saturated or in the presence of strong inter-feature dependence
or noise injected by an adversarial attack. In this work, we propose to infer
robust saliency scores by integrating the saliency scores of a set of decoys
with a novel decoy-enhanced saliency score, in which the decoys are generated
by either solving an optimization problem or blurring the original input. We
theoretically analyze that our method compensates for gradient saturation and
considers joint activation patterns of pixels. We also apply our method to
three different CNNs---VGGNet, AlexNet, and ResNet trained on ImageNet data
set. The empirical results show both qualitatively and quantitatively that our
method outperforms raw scores produced by three existing saliency methods, even
in the presence of adversarial attacks.


http://arxiv.org/abs/2002.02372
Towards Sharper First-Order Adversary with Quantized Gradients.
Zhuanghua Liu; Ivor W. Tsang
  Despite the huge success of Deep Neural Networks (DNNs) in a wide spectrum of
machine learning and data mining tasks, recent research shows that this
powerful tool is susceptible to maliciously crafted adversarial examples. Up
until now, adversarial training has been the most successful defense against
adversarial attacks. To increase adversarial robustness, a DNN can be trained
with a combination of benign and adversarial examples generated by first-order
methods. However, in state-of-the-art first-order attacks, adversarial examples
with sign gradients retain the sign information of each gradient component but
discard the relative magnitude between components. In this work, we replace
sign gradients with quantized gradients. Gradient quantization not only
preserves the sign information, but also keeps the relative magnitude between
components. Experiments show white-box first-order attacks with quantized
gradients outperform their variants with sign gradients on multiple datasets.
Notably, our BLOB\_QG attack achieves an accuracy of $88.32\%$ on the secret
MNIST model from the MNIST Challenge and it outperforms all other methods on
the leaderboard of white-box attacks.


http://arxiv.org/abs/2002.00179
AdvJND: Generating Adversarial Examples with Just Noticeable Difference.
Zifei Zhang; Kai Qiao; Lingyun Jiang; Linyuan Wang; Bin Yan
  Compared with traditional machine learning models, deep neural networks
perform better, especially in image classification tasks. However, they are
vulnerable to adversarial examples. Adding small perturbations on examples
causes a good-performance model to misclassify the crafted examples, without
category differences in the human eyes, and fools deep models successfully.
There are two requirements for generating adversarial examples: the attack
success rate and image fidelity metrics. Generally, perturbations are increased
to ensure the adversarial examples' high attack success rate; however, the
adversarial examples obtained have poor concealment. To alleviate the tradeoff
between the attack success rate and image fidelity, we propose a method named
AdvJND, adding visual model coefficients, just noticeable difference
coefficients, in the constraint of a distortion function when generating
adversarial examples. In fact, the visual subjective feeling of the human eyes
is added as a priori information, which decides the distribution of
perturbations, to improve the image quality of adversarial examples. We tested
our method on the FashionMNIST, CIFAR10, and MiniImageNet datasets. Adversarial
examples generated by our AdvJND algorithm yield gradient distributions that
are similar to those of the original inputs. Hence, the crafted noise can be
hidden in the original inputs, thus improving the attack concealment
significantly.


http://arxiv.org/abs/2001.11905
Additive Tree Ensembles: Reasoning About Potential Instances.
Laurens Devos; Wannes Meert; Jesse Davis
  Imagine being able to ask questions to a black box model such as "Which
adversarial examples exist?", "Does a specific attribute have a
disproportionate effect on the model's prediction?" or "What kind of
predictions are possible for a partially described example?" This last question
is particularly important if your partial description does not correspond to
any observed example in your data, as it provides insight into how the model
will extrapolate to unseen data. These capabilities would be extremely helpful
as it would allow a user to better understand the model's behavior,
particularly as it relates to issues such as robustness, fairness, and bias. In
this paper, we propose such an approach for an ensemble of trees. Since, in
general, this task is intractable we present a strategy that (1) can prune part
of the input space given the question asked to simplify the problem; and (2)
follows a divide and conquer approach that is incremental and can always return
some answers and indicates which parts of the input domains are still
uncertain. The usefulness of our approach is shown on a diverse set of use
cases.


http://arxiv.org/abs/2002.05648
Politics of Adversarial Machine Learning.
Kendra Albert; Jonathon Penney; Bruce Schneier; Ram Shankar Siva Kumar
  In addition to their security properties, adversarial machine-learning
attacks and defenses have political dimensions. They enable or foreclose
certain options for both the subjects of the machine learning systems and for
those who deploy them, creating risks for civil liberties and human rights. In
this paper, we draw on insights from science and technology studies,
anthropology, and human rights literature, to inform how defenses against
adversarial attacks can be used to suppress dissent and limit attempts to
investigate machine learning systems. To make this concrete, we use real-world
examples of how attacks such as perturbation, model inversion, or membership
inference can be used for socially desirable ends. Although the predictions of
this analysis may seem dire, there is hope. Efforts to address human rights
concerns in the commercial spyware industry provide guidance for similar
measures to ensure ML systems serve democratic, not authoritarian ends


http://arxiv.org/abs/2002.00760
FastWordBug: A Fast Method To Generate Adversarial Text Against NLP Applications.
Dou Goodman; Lv Zhonghou; Wang minghua
  In this paper, we present a novel algorithm, FastWordBug, to efficiently
generate small text perturbations in a black-box setting that forces a
sentiment analysis or text classification mode to make an incorrect prediction.
By combining the part of speech attributes of words, we propose a scoring
method that can quickly identify important words that affect text
classification. We evaluate FastWordBug on three real-world text datasets and
two state-of-the-art machine learning models under black-box setting. The
results show that our method can significantly reduce the accuracy of the
model, and at the same time, we can call the model as little as possible, with
the highest attack efficiency. We also attack two popular real-world cloud
services of NLP, and the results show that our method works as well.


http://arxiv.org/abs/2001.11569
Tiny Noise Can Make an EEG-Based Brain-Computer Interface Speller Output Anything.
Xiao Zhang; Dongrui Wu; Lieyun Ding; Hanbin Luo; Chin-Teng Lin; Tzyy-Ping Jung; Ricardo Chavarriaga
  An electroencephalogram (EEG) based brain-computer interface (BCI) speller
allows a user to input text to a computer by thought. It is particularly useful
to severely disabled individuals, e.g., amyotrophic lateral sclerosis patients,
who have no other effective means of communication with another person or a
computer. Most studies so far focused on making EEG-based BCI spellers faster
and more reliable; however, few have considered their security. Here we show
that P300 and steady-state visual evoked potential BCI spellers are very
vulnerable, i.e., they can be severely attacked by adversarial perturbations,
which are too tiny to be noticed when added to EEG signals, but can mislead the
spellers to spell anything the attacker wants. The consequence could range from
merely user frustration to severe misdiagnosis in clinical applications. We
hope our research can attract more attention to the security of EEG-based BCI
spellers, and more broadly, EEG-based BCIs, which has received little attention
before.


http://arxiv.org/abs/2001.10999
A4 : Evading Learning-based Adblockers.
Shitong Zhu; Zhongjie Wang; Xun Chen; Shasha Li; Umar Iqbal; Zhiyun Qian; Kevin S. Chan; Srikanth V. Krishnamurthy; Zubair Shafiq
  Efforts by online ad publishers to circumvent traditional ad blockers towards
regaining fiduciary benefits, have been demonstrably successful. As a result,
there have recently emerged a set of adblockers that apply machine learning
instead of manually curated rules and have been shown to be more robust in
blocking ads on websites including social media sites such as Facebook. Among
these, AdGraph is arguably the state-of-the-art learning-based adblocker. In
this paper, we develop A4, a tool that intelligently crafts adversarial samples
of ads to evade AdGraph. Unlike the popular research on adversarial samples
against images or videos that are considered less- to un-restricted, the
samples that A4 generates preserve application semantics of the web page, or
are actionable. Through several experiments we show that A4 can bypass AdGraph
about 60% of the time, which surpasses the state-of-the-art attack by a
significant margin of 84.3%; in addition, changes to the visual layout of the
web page due to these perturbations are imperceptible. We envision the
algorithmic framework proposed in A4 is also promising in improving adversarial
attacks against other learning-based web applications with similar
requirements.


http://arxiv.org/abs/2001.11108
D2M: Dynamic Defense and Modeling of Adversarial Movement in Networks.
Scott Freitas; Andrew Wicker; Duen Horng Chau; Joshua Neil
  Given a large enterprise network of devices and their authentication history
(e.g., device logons), how can we quantify network vulnerability to lateral
attack and identify at-risk devices? We systematically address these problems
through D2M, the first framework that models lateral attacks on enterprise
networks using multiple attack strategies developed with researchers,
engineers, and threat hunters in the Microsoft Defender Advanced Threat
Protection group. These strategies integrate real-world adversarial actions
(e.g., privilege escalation) to generate attack paths: a series of compromised
machines. Leveraging these attack paths and a novel Monte-Carlo method, we
formulate network vulnerability as a probabilistic function of the network
topology, distribution of access credentials and initial penetration point. To
identify machines at risk to lateral attack, we propose a suite of five fast
graph mining techniques, including a novel technique called AnomalyShield
inspired by node immunization research. Using three real-world authentication
graphs from Microsoft and Los Alamos National Laboratory (up to 223,399
authentications), we report the first experimental results on network
vulnerability to lateral attack, demonstrating D2M's unique potential to
empower IT admins to develop robust user access credential policies.


http://arxiv.org/abs/2001.11064
Just Noticeable Difference for Machines to Generate Adversarial Images.
Adil Kaan Akan; Mehmet Ali Genc; Fatos T. Yarman Vural
  One way of designing a robust machine learning algorithm is to generate
authentic adversarial images which can trick the algorithms as much as
possible. In this study, we propose a new method to generate adversarial images
which are very similar to true images, yet, these images are discriminated from
the original ones and are assigned into another category by the model. The
proposed method is based on a popular concept of experimental psychology,
called, Just Noticeable Difference. We define Just Noticeable Difference for a
machine learning model and generate a least perceptible difference for
adversarial images which can trick a model. The suggested model iteratively
distorts a true image by gradient descent method until the machine learning
algorithm outputs a false label. Deep Neural Networks are trained for object
detection and classification tasks. The cost function includes regularization
terms to generate just noticeably different adversarial images which can be
detected by the model. The adversarial images generated in this study looks
more natural compared to the output of state of the art adversarial image
generators.


http://arxiv.org/abs/2001.11055
Semantic Adversarial Perturbations using Learnt Representations.
Isaac Dunn; Tom Melham; Daniel Kroening
  Adversarial examples for image classifiers are typically created by searching
for a suitable norm-constrained perturbation to the pixels of an image.
However, such perturbations represent only a small and rather contrived subset
of possible adversarial inputs; robustness to norm-constrained pixel
perturbations alone is insufficient. We introduce a novel method for the
construction of a rich new class of semantic adversarial examples. Leveraging
the hierarchical feature representations learnt by generative models, our
procedure makes adversarial but realistic changes at different levels of
semantic granularity. Unlike prior work, this is not an ad-hoc algorithm
targeting a fixed category of semantic property. For instance, our approach
perturbs the pose, location, size, shape, colour and texture of the objects in
an image without manual encoding of these concepts. We demonstrate this new
attack by creating semantic adversarial examples that fool state-of-the-art
classifiers on the MNIST and ImageNet datasets.


http://arxiv.org/abs/2001.11137
Adversarial Attacks on Convolutional Neural Networks in Facial Recognition Domain.
Yigit Alparslan; Jeremy Keim-Shenk; Shweta Khade; Rachel Greenstadt
  Numerous recent studies have demonstrated how Deep Neural Network (DNN)
classifiers can be fooled by adversarial examples, in which an attacker adds
perturbations to an original sample, causing the classifier to misclassify the
sample. Adversarial attacks that render DNNs vulnerable in real life represent
a serious threat, given the consequences of improperly functioning autonomous
vehicles, malware filters, or biometric authentication systems. In this paper,
we apply Fast Gradient Sign Method to introduce perturbations to a facial image
dataset and then test the output on a different classifier that we trained
ourselves, to analyze transferability of this method. Next, we craft a variety
of different attack algorithms on a facial image dataset, with the intention of
developing untargeted black-box approaches assuming minimal adversarial
knowledge, to further assess the robustness of DNNs in the facial recognition
realm. We explore modifying single optimal pixels by a large amount, or
modifying all pixels by a smaller amount, or combining these two attack
approaches. While our single-pixel attacks achieved about a 15% average
decrease in classifier confidence level for the actual class, the all-pixel
attacks were more successful and achieved up to an 84% average decrease in
confidence, along with an 81.6% misclassification rate, in the case of the
attack that we tested with the highest levels of perturbation. Even with these
high levels of perturbation, the face images remained fairly clearly
identifiable to a human. We hope our research may help to advance the study of
adversarial attacks on DNNs and defensive mechanisms to counteract them,
particularly in the facial recognition domain.


http://arxiv.org/abs/2001.10648
Modelling and Quantifying Membership Information Leakage in Machine Learning.
Farhad Farokhi; Mohamed Ali Kaafar
  Machine learning models have been shown to be vulnerable to membership
inference attacks, i.e., inferring whether individuals' data have been used for
training models. The lack of understanding about factors contributing success
of these attacks motivates the need for modelling membership information
leakage using information theory and for investigating properties of machine
learning models and training algorithms that can reduce membership information
leakage. We use conditional mutual information leakage to measure the amount of
information leakage from the trained machine learning model about the presence
of an individual in the training dataset. We devise an upper bound for this
measure of information leakage using Kullback--Leibler divergence that is more
amenable to numerical computation. We prove a direct relationship between the
Kullback--Leibler membership information leakage and the probability of success
for a hypothesis-testing adversary examining whether a particular data record
belongs to the training dataset of a machine learning model. We show that the
mutual information leakage is a decreasing function of the training dataset
size and the regularization weight. We also prove that, if the sensitivity of
the machine learning model (defined in terms of the derivatives of the fitness
with respect to model parameters) is high, more membership information is
potentially leaked. This illustrates that complex models, such as deep neural
networks, are more susceptible to membership inference attacks in comparison to
simpler models with fewer degrees of freedom. We show that the amount of the
membership information leakage is reduced by
$\mathcal{O}(\log^{1/2}(\delta^{-1})\epsilon^{-1})$ when using Gaussian
$(\epsilon,\delta)$-differentially-private additive noises.


http://arxiv.org/abs/2001.10916
Interpreting Machine Learning Malware Detectors Which Leverage N-gram Analysis.
William Briguglio; Sherif Saad
  In cyberattack detection and prevention systems, cybersecurity analysts
always prefer solutions that are as interpretable and understandable as
rule-based or signature-based detection. This is because of the need to tune
and optimize these solutions to mitigate and control the effect of false
positives and false negatives. Interpreting machine learning models is a new
and open challenge. However, it is expected that an interpretable machine
learning solution will be domain-specific. For instance, interpretable
solutions for machine learning models in healthcare are different than
solutions in malware detection. This is because the models are complex, and
most of them work as a black-box. Recently, the increased ability for malware
authors to bypass antimalware systems has forced security specialists to look
to machine learning for creating robust detection systems. If these systems are
to be relied on in the industry, then, among other challenges, they must also
explain their predictions. The objective of this paper is to evaluate the
current state-of-the-art ML models interpretability techniques when applied to
ML-based malware detectors. We demonstrate interpretability techniques in
practice and evaluate the effectiveness of existing interpretability techniques
in the malware analysis domain.


http://arxiv.org/abs/2001.09993
Generating Natural Adversarial Hyperspectral examples with a modified Wasserstein GAN.
Jean-Christophe OBELIX Burnel; Kilian OBELIX Fatras; Nicolas OBELIX Courty
  Adversarial examples are a hot topic due to their abilities to fool a
classifier's prediction. There are two strategies to create such examples, one
uses the attacked classifier's gradients, while the other only requires access
to the clas-sifier's prediction. This is particularly appealing when the
classifier is not full known (black box model). In this paper, we present a new
method which is able to generate natural adversarial examples from the true
data following the second paradigm. Based on Generative Adversarial Networks
(GANs) [5], it reweights the true data empirical distribution to encourage the
classifier to generate ad-versarial examples. We provide a proof of concept of
our method by generating adversarial hyperspectral signatures on a remote
sensing dataset.


http://arxiv.org/abs/2001.09598
FakeLocator: Robust Localization of GAN-Based Face Manipulations via Semantic Segmentation Networks with Bells and Whistles.
Yihao Huang; Felix Juefei-Xu; Run Wang; Xiaofei Xie; Lei Ma; Jianwen Li; Weikai Miao; Yang Liu; Geguang Pu
  Nowadays, full face synthesis and partial face manipulation by virtue of the
generative adversarial networks (GANs) have raised wide public concern. In the
digital media forensics area, detecting and ultimately locating the image
forgery have become imperative. Although many methods focus on fake detection,
only a few put emphasis on the localization of the fake regions. Through
analyzing the imperfection in the upsampling procedures of the GAN-based
methods and recasting the fake localization problem as a modified semantic
segmentation one, our proposed FakeLocator can obtain high localization
accuracy, at full resolution, on manipulated facial images. To the best of our
knowledge, this is the very first attempt to solve the GAN-based fake
localization problem with a semantic segmentation map. As an improvement, the
real-numbered segmentation map proposed by us preserves more information of
fake regions. For this new type segmentation map, we also find suitable loss
functions for it. Experimental results on the CelebA and FFHQ databases with
seven different SOTA GAN-based face generation methods show the effectiveness
of our method. Compared with the baseline, our method performs several times
better on various metrics. Moreover, the proposed method is robust against
various real-world facial image degradations such as JPEG compression,
low-resolution, noise, and blur.


http://arxiv.org/abs/2001.09684
Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning.
Inaam Ilahi; Muhammad Usama; Junaid Qadir; Muhammad Umar Janjua; Ala Al-Fuqaha; Dinh Thai Hoang; Dusit Niyato
  Deep Reinforcement Learning (DRL) has numerous applications in the real world
thanks to its outstanding ability in quickly adapting to the surrounding
environments. Despite its great advantages, DRL is susceptible to adversarial
attacks, which precludes its use in real-life critical systems and applications
(e.g., smart grids, traffic controls, and autonomous vehicles) unless its
vulnerabilities are addressed and mitigated. Thus, this paper provides a
comprehensive survey that discusses emerging attacks in DRL-based systems and
the potential countermeasures to defend against these attacks. We first cover
some fundamental backgrounds about DRL and present emerging adversarial attacks
on machine learning techniques. We then investigate more details of the
vulnerabilities that the adversary can exploit to attack DRL along with the
state-of-the-art countermeasures to prevent such attacks. Finally, we highlight
open issues and research challenges for developing solutions to deal with
attacks for DRL-based intelligent systems.


http://arxiv.org/abs/2001.09610
Practical Fast Gradient Sign Attack against Mammographic Image Classifier.
Ibrahim Yilmaz
  Artificial intelligence (AI) has been a topic of major research for many
years. Especially, with the emergence of deep neural network (DNN), these
studies have been tremendously successful. Today machines are capable of making
faster, more accurate decision than human. Thanks to the great development of
machine learning (ML) techniques, ML have been used many different fields such
as education, medicine, malware detection, autonomous car etc. In spite of
having this degree of interest and much successful research, ML models are
still vulnerable to adversarial attacks. Attackers can manipulate clean data in
order to fool the ML classifiers to achieve their desire target. For instance;
a benign sample can be modified as a malicious sample or a malicious one can be
altered as benign while this modification can not be recognized by human
observer. This can lead to many financial losses, or serious injuries, even
deaths. The motivation behind this paper is that we emphasize this issue and
want to raise awareness. Therefore, the security gap of mammographic image
classifier against adversarial attack is demonstrated. We use mamographic
images to train our model then evaluate our model performance in terms of
accuracy. Later on, we poison original dataset and generate adversarial samples
that missclassified by the model. We then using structural similarity index
(SSIM) analyze similarity between clean images and adversarial images. Finally,
we show how successful we are to misuse by using different poisoning factors.


http://arxiv.org/abs/2001.09486
Ensemble Noise Simulation to Handle Uncertainty about Gradient-based Adversarial Attacks.
Rehana Mahfuz; Rajeev Sahay; Aly El Gamal
  Gradient-based adversarial attacks on neural networks can be crafted in a
variety of ways by varying either how the attack algorithm relies on the
gradient, the network architecture used for crafting the attack, or both. Most
recent work has focused on defending classifiers in a case where there is no
uncertainty about the attacker's behavior (i.e., the attacker is expected to
generate a specific attack using a specific network architecture). However, if
the attacker is not guaranteed to behave in a certain way, the literature lacks
methods in devising a strategic defense. We fill this gap by simulating the
attacker's noisy perturbation using a variety of attack algorithms based on
gradients of various classifiers. We perform our analysis using a
pre-processing Denoising Autoencoder (DAE) defense that is trained with the
simulated noise. We demonstrate significant improvements in post-attack
accuracy, using our proposed ensemble-trained defense, compared to a situation
where no effort is made to handle uncertainty.


http://arxiv.org/abs/2002.03751
Weighted Average Precision: Adversarial Example Detection in the Visual Perception of Autonomous Vehicles.
Yilan Li; Senem Velipasalar
  Recent works have shown that neural networks are vulnerable to carefully
crafted adversarial examples (AE). By adding small perturbations to input
images, AEs are able to make the victim model predicts incorrect outputs.
Several research work in adversarial machine learning started to focus on the
detection of AEs in autonomous driving. However, the existing studies either
use preliminary assumption on outputs of detections or ignore the tracking
system in the perception pipeline. In this paper, we firstly propose a novel
distance metric for practical autonomous driving object detection outputs.
Then, we bridge the gap between the current AE detection research and the
real-world autonomous systems by providing a temporal detection algorithm,
which takes the impact of tracking system into consideration. We perform
evaluation on Berkeley Deep Drive (BDD) and CityScapes datasets to show how our
approach outperforms existing single-frame-mAP based AE detections by
increasing 17.76% accuracy of performance.


http://arxiv.org/abs/2001.09388
AI-Powered GUI Attack and Its Defensive Methods.
Ning Yu; Zachary Tuttle; Carl Jake Thurnau; Emmanuel Mireku
  Since the first Graphical User Interface (GUI) prototype was invented in the
1970s, GUI systems have been deployed into various personal computer systems
and server platforms. Recently, with the development of artificial intelligence
(AI) technology, malicious malware powered by AI is emerging as a potential
threat to GUI systems. This type of AI-based cybersecurity attack, targeting at
GUI systems, is explored in this paper. It is twofold: (1) A malware is
designed to attack the existing GUI system by using AI-based object recognition
techniques. (2) Its defensive methods are discovered by generating adversarial
examples and other methods to alleviate the threats from the intelligent GUI
attack. The results have shown that a generic GUI attack can be implemented and
performed in a simple way based on current AI techniques and its
countermeasures are temporary but effective to mitigate the threats of GUI
attack so far.


http://arxiv.org/abs/2001.09395
Analyzing the Noise Robustness of Deep Neural Networks.
Kelei Cao; Mengchen Liu; Hang Su; Jing Wu; Jun Zhu; Shixia Liu
  Adversarial examples, generated by adding small but intentionally
imperceptible perturbations to normal examples, can mislead deep neural
networks (DNNs) to make incorrect predictions. Although much work has been done
on both adversarial attack and defense, a fine-grained understanding of
adversarial examples is still lacking. To address this issue, we present a
visual analysis method to explain why adversarial examples are misclassified.
The key is to compare and analyze the datapaths of both the adversarial and
normal examples. A datapath is a group of critical neurons along with their
connections. We formulate the datapath extraction as a subset selection problem
and solve it by constructing and training a neural network. A multi-level
visualization consisting of a network-level visualization of data flows, a
layer-level visualization of feature maps, and a neuron-level visualization of
learned features, has been designed to help investigate how datapaths of
adversarial and normal examples diverge and merge in the prediction process. A
quantitative evaluation and a case study were conducted to demonstrate the
promise of our method to explain the misclassification of adversarial examples.


http://arxiv.org/abs/2001.08883
When Wireless Security Meets Machine Learning: Motivation, Challenges, and Research Directions.
Yalin E. Sagduyu; Yi Shi; Tugba Erpek; William Headley; Bryse Flowers; George Stantchev; Zhuo Lu
  Wireless systems are vulnerable to various attacks such as jamming and
eavesdropping due to the shared and broadcast nature of wireless medium. To
support both attack and defense strategies, machine learning (ML) provides
automated means to learn from and adapt to wireless communication
characteristics that are hard to capture by hand-crafted features and models.
This article discusses motivation, background, and scope of research efforts
that bridge ML and wireless security. Motivated by research directions surveyed
in the context of ML for wireless security, ML-based attack and defense
solutions and emerging adversarial ML techniques in the wireless domain are
identified along with a roadmap to foster research efforts in bridging ML and
wireless security.


http://arxiv.org/abs/2001.08855
Privacy for All: Demystify Vulnerability Disparity of Differential Privacy against Membership Inference Attack.
Bo Zhang; Ruotong Yu; Haipei Sun; Yanying Li; Jun Xu; Hui Wang
  Machine learning algorithms, when applied to sensitive data, pose a potential
threat to privacy. A growing body of prior work has demonstrated that
membership inference attack (MIA) can disclose specific private information in
the training data to an attacker. Meanwhile, the algorithmic fairness of
machine learning has increasingly caught attention from both academia and
industry. Algorithmic fairness ensures that the machine learning models do not
discriminate a particular demographic group of individuals (e.g., black and
female people). Given that MIA is indeed a learning model, it raises a serious
concern if MIA ``fairly'' treats all groups of individuals equally. In other
words, whether a particular group is more vulnerable against MIA than the other
groups. This paper examines the algorithmic fairness issue in the context of
MIA and its defenses. First, for fairness evaluation, it formalizes the
notation of vulnerability disparity (VD) to quantify the difference of MIA
treatment on different demographic groups. Second, it evaluates VD on four
real-world datasets, and shows that VD indeed exists in these datasets. Third,
it examines the impacts of differential privacy, as a defense mechanism of MIA,
on VD. The results show that although DP brings significant change on VD, it
cannot eliminate VD completely. Therefore, fourth, it designs a new mitigation
algorithm named FAIRPICK to reduce VD. An extensive set of experimental results
demonstrate that FAIRPICK can effectively reduce VD for both with and without
the DP deployment.


http://arxiv.org/abs/2001.08389
Towards Robust DNNs: An Taylor Expansion-Based Method for Generating Powerful Adversarial Examples.
Ya-guan Qian; Xi-Ming Zhang; Bin Wang; Wei Li; Jian-Hai Chen; Wu-Jie Zhou; Jing-Sheng Lei
  Although deep neural networks (DNNs) have achieved successful applications in
many fields, they are vulnerable to adversarial examples. Adversarial training
is one of the most effective methods to improve the robustness of DNNs, and it
is generally considered as a minimax point problem that minimizes the loss
function and maximizes the perturbation. Therefore, powerful adversarial
examples can effectively simulate perturbation maximization to solve the
minimax problem. In paper, a novel method was proposed to generate more
powerful adversarial examples for robust adversarial training. The main idea is
to approximates the output of DNNs in the input neighborhood by using the
Taylor expansion, and then optimizes it by using the Lagrange multiplier method
to generate adversarial examples. The experiment results show it can
effectively improve the robust of DNNs trained with these powerful adversarial
examples.


http://arxiv.org/abs/2001.08444
On the human evaluation of audio adversarial examples.
Jon Vadillo; Roberto Santana
  Human-machine interaction is increasingly dependent on speech communication.
Machine Learning models are usually applied to interpret human speech commands.
However, these models can be fooled by adversarial examples, which are inputs
intentionally perturbed to produce a wrong prediction without being noticed.
While much research has been focused on developing new techniques to generate
adversarial perturbations, less attention has been given to aspects that
determine whether and how the perturbations are noticed by humans. This
question is relevant since high fooling rates of proposed adversarial
perturbation strategies are only valuable if the perturbations are not
detectable. In this paper we investigate to which extent the distortion metrics
proposed in the literature for audio adversarial examples, and which are
commonly applied to evaluate the effectiveness of methods for generating these
attacks, are a reliable measure of the human perception of the perturbations.
Using an analytical framework, and an experiment in which 18 subjects evaluate
audio adversarial examples, we demonstrate that the metrics employed by
convention are not a reliable measure of the perceptual similarity of
adversarial examples in the audio domain.


http://arxiv.org/abs/2001.07933
Adversarial Attack on Community Detection by Hiding Individuals.
Jia Li; Honglei Zhang; Zhichao Han; Yu Rong; Hong Cheng; Junzhou Huang
  It has been demonstrated that adversarial graphs, i.e., graphs with
imperceptible perturbations added, can cause deep graph models to fail on
node/graph classification tasks. In this paper, we extend adversarial graphs to
the problem of community detection which is much more difficult. We focus on
black-box attack and aim to hide targeted individuals from the detection of
deep graph community detection models, which has many applications in
real-world scenarios, for example, protecting personal privacy in social
networks and understanding camouflage patterns in transaction networks. We
propose an iterative learning framework that takes turns to update two modules:
one working as the constrained graph generator and the other as the surrogate
community detection model. We also find that the adversarial graphs generated
by our method can be transferred to other learning based community detection
models.


http://arxiv.org/abs/2001.07645
SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation.
Jesse Sun; Fatemeh Darbeha; Mark Zaidi; Bo Wang
  Medical image segmentation is a difficult but important task for many
clinical operations such as cardiac bi-ventricular volume estimation. More
recently, there has been a shift to utilizing deep learning and fully
convolutional neural networks (CNNs) to perform image segmentation that has
yielded state-of-the-art results in many public benchmark datasets. Despite the
progress of deep learning in medical image segmentation, standard CNNs are
still not fully adopted in clinical settings as they lack robustness and
interpretability. Shapes are generally more meaningful features than solely
textures of images, which are features regular CNNs learn, causing a lack of
robustness. Likewise, previous works surrounding model interpretability have
been focused on post hoc gradient-based saliency methods. However,
gradient-based saliency methods typically require additional computations post
hoc and have been shown to be unreliable for interpretability. Thus, we present
a new architecture called Shape Attentive U-Net (SAUNet) which focuses on model
interpretability and robustness. The proposed architecture attempts to address
these limitations by the use of a secondary shape stream that captures rich
shape-dependent information in parallel with the regular texture stream.
Furthermore, we suggest multi-resolution saliency maps can be learned using our
dual-attention decoder module which allows for multi-level interpretability and
mitigates the need for additional computations post hoc. Our method also
achieves state-of-the-art results on the two large public cardiac MRI image
segmentation datasets of SUN09 and AC17.


http://arxiv.org/abs/2001.08103
Secure and Robust Machine Learning for Healthcare: A Survey.
Adnan Qayyum; Junaid Qadir; Muhammad Bilal; Ala Al-Fuqaha
  Recent years have witnessed widespread adoption of machine learning (ML)/deep
learning (DL) techniques due to their superior performance for a variety of
healthcare applications ranging from the prediction of cardiac arrest from
one-dimensional heart signals to computer-aided diagnosis (CADx) using
multi-dimensional medical images. Notwithstanding the impressive performance of
ML/DL, there are still lingering doubts regarding the robustness of ML/DL in
healthcare settings (which is traditionally considered quite challenging due to
the myriad security and privacy issues involved), especially in light of recent
results that have shown that ML/DL are vulnerable to adversarial attacks. In
this paper, we present an overview of various application areas in healthcare
that leverage such techniques from security and privacy point of view and
present associated challenges. In addition, we present potential methods to
ensure secure and privacy-preserving ML for healthcare applications. Finally,
we provide insight into the current research challenges and promising
directions for future research.


http://arxiv.org/abs/2001.07685
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.
Kihyuk Sohn; David Berthelot; Chun-Liang Li; Zizhao Zhang; Nicholas Carlini; Ekin D. Cubuk; Alex Kurakin; Han Zhang; Colin Raffel
  Semi-supervised learning (SSL) provides an effective means of leveraging
unlabeled data to improve a model's performance. In this paper, we demonstrate
the power of a simple combination of two common SSL methods: consistency
regularization and pseudo-labeling. Our algorithm, FixMatch, first generates
pseudo-labels using the model's predictions on weakly-augmented unlabeled
images. For a given image, the pseudo-label is only retained if the model
produces a high-confidence prediction. The model is then trained to predict the
pseudo-label when fed a strongly-augmented version of the same image. Despite
its simplicity, we show that FixMatch achieves state-of-the-art performance
across a variety of standard semi-supervised learning benchmarks, including
94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just
4 labels per class. Since FixMatch bears many similarities to existing SSL
methods that achieve worse performance, we carry out an extensive ablation
study to tease apart the experimental factors that are most important to
FixMatch's success. We make our code available at
https://github.com/google-research/fixmatch.


http://arxiv.org/abs/2001.07792
GhostImage: Perception Domain Attacks against Vision-based Object Classification Systems.
Yanmao Man; Ming Li; Ryan Gerdes
  In vision-based object classification systems, imaging sensors perceive the
environment and then objects are detected and classified for decision-making
purposes. Vulnerabilities in the perception domain enable an attacker to inject
false data into the sensor which could lead to unsafe consequences. In this
work, we focus on camera-based systems and propose GhostImage attacks, with the
goal of either creating a fake perceived object or obfuscating the object's
image that leads to wrong classification results. This is achieved by remotely
projecting adversarial patterns into camera-perceived images, exploiting two
common effects in optical imaging systems, namely lens flare/ghost effects, and
auto-exposure control. To improve the robustness of the attack to channel
perturbations, we generate optimal input patterns by integrating adversarial
machine learning techniques with a trained end-to-end channel model. We realize
GhostImage attacks with a projector, and conducted comprehensive experiments,
using three different image datasets, in indoor and outdoor environments, and
three different cameras. We demonstrate that GhostImage attacks are applicable
to both autonomous driving and security surveillance scenarios. Experiment
results show that, depending on the projector-camera distance, attack success
rates can reach as high as 100%.


http://arxiv.org/abs/2001.07631
Generate High-Resolution Adversarial Samples by Identifying Effective Features.
Sizhe Chen; Peidong Zhang; Chengjin Sun; Jia Cai; Xiaolin Huang
  As the prevalence of deep learning in computer vision, adversarial samples
that weaken the neural networks emerge in large numbers, revealing their
deep-rooted defects. Most adversarial attacks calculate an imperceptible
perturbation in image space to fool the DNNs. In this strategy, the
perturbation looks like noise and thus could be mitigated. Attacks in feature
space produce semantic perturbation, but they could only deal with low
resolution samples. The reason lies in the great number of coupled features to
express a high-resolution image. In this paper, we propose Attack by
Identifying Effective Features (AIEF), which learns different weights for
features to attack. Effective features, those with great weights, influence the
victim model much but distort the image little, and thus are more effective for
attack. By attacking mostly on them, AIEF produces high resolution adversarial
samples with acceptable distortions. We demonstrate the effectiveness of AIEF
by attacking on different tasks with different generative models.


http://arxiv.org/abs/2001.07769
Massif: Interactive Interpretation of Adversarial Attacks on Deep Learning.
Nilaksh Polo Das; Haekyu Polo Park; Zijie J. Polo Wang; Fred Polo Hohman; Robert Polo Firstman; Emily Polo Rogers; Duen Polo Horng; Chau
  Deep neural networks (DNNs) are increasingly powering high-stakes
applications such as autonomous cars and healthcare; however, DNNs are often
treated as "black boxes" in such applications. Recent research has also
revealed that DNNs are highly vulnerable to adversarial attacks, raising
serious concerns over deploying DNNs in the real world. To overcome these
deficiencies, we are developing Massif, an interactive tool for deciphering
adversarial attacks. Massif identifies and interactively visualizes neurons and
their connections inside a DNN that are strongly activated or suppressed by an
adversarial attack. Massif provides both a high-level, interpretable overview
of the effect of an attack on a DNN, and a low-level, detailed description of
the affected neurons. These tightly coupled views in Massif help people better
understand which input features are most vulnerable or important for correct
predictions.


http://arxiv.org/abs/2001.07820
Elephant in the Room: An Evaluation Framework for Assessing Adversarial Examples in NLP.
Ying Xu; Xu Zhong; Antonio Jose Jimeno Yepes; Jey Han Lau
  An adversarial example is an input transformed by small perturbations that
machine learning models consistently misclassify. While there are a number of
methods proposed to generate adversarial examples for text data, it is not
trivial to assess the quality of these adversarial examples, as minor
perturbations (such as changing a word in a sentence) can lead to a significant
shift in their meaning, readability and classification label. In this paper, we
propose an evaluation framework to assess the quality of adversarial examples
based on the aforementioned properties. We experiment with five benchmark
attacking methods and an alternative approach based on an auto-encoder, and
found that these methods generate adversarial examples with poor readability
and content preservation. We also learned that there are multiple factors that
can influence the attacking performance, such as the the length of text
examples and the input domain.


http://arxiv.org/abs/2001.06309
Cyber Attack Detection thanks to Machine Learning Algorithms.
Antoine Delplace; Sheryl Hermoso; Kristofer Anandita
  Cybersecurity attacks are growing both in frequency and sophistication over
the years. This increasing sophistication and complexity call for more
advancement and continuous innovation in defensive strategies. Traditional
methods of intrusion detection and deep packet inspection, while still largely
used and recommended, are no longer sufficient to meet the demands of growing
security threats. As computing power increases and cost drops, Machine Learning
is seen as an alternative method or an additional mechanism to defend against
malwares, botnets, and other attacks. This paper explores Machine Learning as a
viable solution by examining its capabilities to classify malicious traffic in
a network.
  First, a strong data analysis is performed resulting in 22 extracted features
from the initial Netflow datasets. All these features are then compared with
one another through a feature selection process. Then, our approach analyzes
five different machine learning algorithms against NetFlow dataset containing
common botnets. The Random Forest Classifier succeeds in detecting more than
95% of the botnets in 8 out of 13 scenarios and more than 55% in the most
difficult datasets. Finally, insight is given to improve and generalize the
results, especially through a bootstrapping technique.


http://arxiv.org/abs/2001.06099
Code-Bridged Classifier (CBC): A Low or Negative Overhead Defense for Making a CNN Classifier Robust Against Adversarial Attacks.
Farnaz Behnia; Ali Mirzaeian; Mohammad Sabokrou; Sai Manoj; Tinoosh Mohsenin; Khaled N. Khasawneh; Liang Zhao; Houman Homayoun; Avesta Sasan
  In this paper, we propose Code-Bridged Classifier (CBC), a framework for
making a Convolutional Neural Network (CNNs) robust against adversarial attacks
without increasing or even by decreasing the overall models' computational
complexity. More specifically, we propose a stacked encoder-convolutional
model, in which the input image is first encoded by the encoder module of a
denoising auto-encoder, and then the resulting latent representation (without
being decoded) is fed to a reduced complexity CNN for image classification. We
illustrate that this network not only is more robust to adversarial examples
but also has a significantly lower computational complexity when compared to
the prior art defenses.


http://arxiv.org/abs/2001.06325
Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNet.
Sizhe Chen; Zhengbao He; Chengjin Sun; Xiaolin Huang
  Adversarial attacks on deep neural networks (DNNs) have been found for
several years. However, the existing adversarial attacks have high success
rates only when the information of the attacked DNN is well-known or could be
estimated by structure similarity or massive queries. In this paper, we propose
an \emph{Attack on Attention} (AoA), a semantic feature commonly shared by
DNNs. The transferability of AoA is quite high. With no more than 10 queries of
the decision only, AoA can achieve almost 100\% success rate when attacking on
many popular DNNs. Even without query, AoA could keep a surprisingly high
attack performance. We apply AoA to generate 96020 adversarial samples from
ImageNet to defeat many neural networks, and thus name the dataset as
\emph{DAmageNet}. 20 well-trained DNNs are tested on DAmageNet. Without
adversarial training, most of the tested DNNs have an error rate over 90\%.
DAmageNet is the first universal adversarial dataset and it could serve as a
benchmark for robustness testing and adversarial training.


http://arxiv.org/abs/2001.05873
A Little Fog for a Large Turn.
Harshitha Machiraju; Vineeth N Balasubramanian
  Small, carefully crafted perturbations called adversarial perturbations can
easily fool neural networks. However, these perturbations are largely additive
and not naturally found. We turn our attention to the field of Autonomous
navigation wherein adverse weather conditions such as fog have a drastic effect
on the predictions of these systems. These weather conditions are capable of
acting like natural adversaries that can help in testing models. To this end,
we introduce a general notion of adversarial perturbations, which can be
created using generative models and provide a methodology inspired by
Cycle-Consistent Generative Adversarial Networks to generate adversarial
weather conditions for a given image. Our formulation and results show that
these images provide a suitable testbed for steering models used in Autonomous
navigation models. Our work also presents a more natural and general definition
of Adversarial perturbations based on Perceptual Similarity.


http://arxiv.org/abs/2001.07523
The gap between theory and practice in function approximation with deep neural networks.
Ben Adcock; Nick Dexter
  Deep learning (DL) is transforming whole industries as complicated
decision-making processes are being automated by Deep Neural Networks (DNNs)
trained on real-world data. Driven in part by a rapidly-expanding literature on
DNN approximation theory showing that DNNs can approximate a rich variety of
functions, these tools are increasingly being considered for problems in
scientific computing. Yet, unlike more traditional algorithms in this field,
relatively little is known about DNNs from the principles of numerical
analysis, namely, stability, accuracy, computational efficiency and sample
complexity. In this paper we introduce a computational framework for examining
DNNs in practice, and use it to study their empirical performance with regard
to these issues. We examine the performance of DNNs of different widths and
depths on a variety of test functions in various dimensions, including smooth
and piecewise smooth functions. We also compare DL against best-in-class
methods for smooth function approximation based on compressed sensing. Our main
conclusion is that there is a crucial gap between the approximation theory of
DNNs and their practical performance, with trained DNNs performing relatively
poorly on functions for which there are strong approximation results (e.g.
smooth functions), yet performing well in comparison to best-in-class methods
for other functions. Finally, we present a novel practical existence theorem,
which asserts the existence of a DNN architecture and training procedure which
offers the same performance as current best-in-class schemes. This result
indicates the potential for practical DNN approximation, and the need for
future research into practical architecture design and training strategies.


http://arxiv.org/abs/2001.06057
Increasing the robustness of DNNs against image corruptions by playing the Game of Noise.
Evgenia Rusak; Lukas Schott; Roland S. Zimmermann; Julian Bitterwolf; Oliver Bringmann; Matthias Bethge; Wieland Brendel
  The human visual system is remarkably robust against a wide range of
naturally occurring variations and corruptions like rain or snow. In contrast,
the performance of modern image recognition models strongly degrades when
evaluated on previously unseen corruptions. Here, we demonstrate that a simple
but properly tuned training with additive Gaussian and Speckle noise
generalizes surprisingly well to unseen corruptions, easily reaching the
previous state of the art on the corruption benchmark ImageNet-C (with
ResNet50) and on MNIST-C. We build on top of these strong baseline results and
show that an adversarial training of the recognition model against uncorrelated
worst-case noise distributions leads to an additional increase in performance.
This regularization can be combined with previously proposed defense methods
for further improvement.


http://arxiv.org/abs/2001.04974
Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation.
Chuteng Zhou; Prad Kadambi; Matthew Mattina; Paul N. Whatmough
  The success of deep learning has brought forth a wave of interest in computer
hardware design to better meet the high demands of neural network inference. In
particular, analog computing hardware has been heavily motivated specifically
for accelerating neural networks, based on either electronic, optical or
photonic devices, which may well achieve lower power consumption than
conventional digital electronics. However, these proposed analog accelerators
suffer from the intrinsic noise generated by their physical components, which
makes it challenging to achieve high accuracy on deep neural networks. Hence,
for successful deployment on analog accelerators, it is essential to be able to
train deep neural networks to be robust to random continuous noise in the
network weights, which is a somewhat new challenge in machine learning. In this
paper, we advance the understanding of noisy neural networks. We outline how a
noisy neural network has reduced learning capacity as a result of loss of
mutual information between its input and output. To combat this, we propose
using knowledge distillation combined with noise injection during training to
achieve more noise robust networks, which is demonstrated experimentally across
different networks and datasets, including ImageNet. Our method achieves models
with as much as two times greater noise tolerance compared with the previous
best attempts, which is a significant step towards making analog hardware
practical for deep learning.


http://arxiv.org/abs/2001.05574
Advbox: a toolbox to generate adversarial examples that fool neural networks.
Dou Goodman; Hao Xin; Wang Yang; Wu Yuesheng; Xiong Junfeng; Zhang Huan
  In recent years, neural networks have been extensively deployed for computer
vision tasks, particularly visual classification problems, where new algorithms
reported to achieve or even surpass the human performance. Recent studies have
shown that they are all vulnerable to the attack of adversarial examples. Small
and often imperceptible perturbations to the input images are sufficient to
fool the most powerful neural networks. \emph{Advbox} is a toolbox to generate
adversarial examples that fool neural networks in PaddlePaddle, PyTorch,
Caffe2, MxNet, Keras, TensorFlow and it can benchmark the robustness of machine
learning models. Compared to previous work, our platform supports black box
attacks on Machine-Learning-as-a-service, as well as more attack scenarios,
such as Face Recognition Attack, Stealth T-shirt, and Deepfake Face Detect. The
code is licensed under the Apache 2.0 license and is openly available at
https://github.com/advboxes/AdvBox.


http://arxiv.org/abs/2001.04011
Membership Inference Attacks Against Object Detection Models.
Yeachan Park; Myungjoo Kang
  Machine learning models can leak information about the dataset they trained.
In this paper, we present the first membership inference attack against
black-boxed object detection models that determines whether given data records
are used in training. To attack the object detection model, we devise a noble
method named as Canvas Method, drawing predicted bounding boxes on an empty
image for attack model input. In experiments, we successfully reveal the
membership status of privately sensitive data trained by one-stage and
two-stage detection models. We then propose defense strategies and also conduct
a transfer attack between models and datasets. Our results show that object
detection models are vulnerable to inference attacks as other models.


http://arxiv.org/abs/2001.04051
An Adversarial Approach for the Robust Classification of Pneumonia from Chest Radiographs.
Joseph D. Janizek; Gabriel Erion; Alex J. DeGrave; Su-In Lee
  While deep learning has shown promise in the domain of disease classification
from medical images, models based on state-of-the-art convolutional neural
network architectures often exhibit performance loss due to dataset shift.
Models trained using data from one hospital system achieve high predictive
performance when tested on data from the same hospital, but perform
significantly worse when they are tested in different hospital systems.
Furthermore, even within a given hospital system, deep learning models have
been shown to depend on hospital- and patient-level confounders rather than
meaningful pathology to make classifications. In order for these models to be
safely deployed, we would like to ensure that they do not use confounding
variables to make their classification, and that they will work well even when
tested on images from hospitals that were not included in the training data. We
attempt to address this problem in the context of pneumonia classification from
chest radiographs. We propose an approach based on adversarial optimization,
which allows us to learn more robust models that do not depend on confounders.
Specifically, we demonstrate improved out-of-hospital generalization
performance of a pneumonia classifier by training a model that is invariant to
the view position of chest radiographs (anterior-posterior vs.
posterior-anterior). Our approach leads to better predictive performance on
external hospital data than both a standard baseline and previously proposed
methods to handle confounding, and also suggests a method for identifying
models that may rely on confounders. Code available at
https://github.com/suinleelab/cxr_adv.


http://arxiv.org/abs/2001.03994
Fast is better than free: Revisiting adversarial training.
Eric Wong; Leslie Rice; J. Zico Kolter
  Adversarial training, a method for learning robust deep networks, is
typically assumed to be more expensive than traditional training due to the
necessity of constructing adversarial examples via a first-order method like
projected gradient decent (PGD). In this paper, we make the surprising
discovery that it is possible to train empirically robust models using a much
weaker and cheaper adversary, an approach that was previously believed to be
ineffective, rendering the method no more costly than standard training in
practice. Specifically, we show that adversarial training with the fast
gradient sign method (FGSM), when combined with random initialization, is as
effective as PGD-based training but has significantly lower cost. Furthermore
we show that FGSM adversarial training can be further accelerated by using
standard techniques for efficient training of deep networks, allowing us to
learn a robust CIFAR10 classifier with 45% robust accuracy to PGD attacks with
$\epsilon=8/255$ in 6 minutes, and a robust ImageNet classifier with 43% robust
accuracy at $\epsilon=2/255$ in 12 hours, in comparison to past work based on
"free" adversarial training which took 10 and 50 hours to reach the same
respective thresholds. Finally, we identify a failure mode referred to as
"catastrophic overfitting" which may have caused previous attempts to use FGSM
adversarial training to fail. All code for reproducing the experiments in this
paper as well as pretrained model weights are at
https://github.com/locuslab/fast_adversarial.


http://arxiv.org/abs/2001.05286
Exploring and Improving Robustness of Multi Task Deep Neural Networks via Domain Agnostic Defenses.
Kashyap Coimbatore Murali
  In this paper, we explore the robustness of the Multi-Task Deep Neural
Networks (MT-DNN) against non-targeted adversarial attacks across Natural
Language Understanding (NLU) tasks as well as some possible ways to defend
against them. Liu et al., have shown that the Multi-Task Deep Neural Network,
due to the regularization effect produced when training as a result of its
cross task data, is more robust than a vanilla BERT model trained only on one
task (1.1%-1.5% absolute difference). We further show that although the MT-DNN
has generalized better, making it easily transferable across domains and tasks,
it can still be compromised as after only 2 attacks (1-character and
2-character) the accuracy drops by 42.05% and 32.24% for the SNLI and SciTail
tasks. Finally, we propose a domain agnostic defense which restores the model's
accuracy (36.75% and 25.94% respectively) as opposed to a general-purpose
defense or an off-the-shelf spell checker.


http://arxiv.org/abs/2001.03754
Sparse Black-box Video Attack with Reinforcement Learning.
Huanqian Yan; Xingxing Wei; Bo Li
  Adversarial attacks on video recognition models have been explored recently.
However, most existing works treat each video frame equally and ignore their
temporal interactions. To overcome this drawback, a few methods try to select
some key frames, and then perform attacks based on them. Unfortunately, their
selecting strategy is independent with the attacking step, therefore the
resulting performance is limited. In this paper, we aim to attack video
recognition task in the black-box setting. The difference is, we think the
frame selection phase is closely relevant with the attacking phase. The
reasonable key frames should be adjusted according to the feedback of attacking
threat models. Based on this idea, we formulate the black-box video attacks
into the Reinforcement Learning (RL) framework. Specifically, the environment
in RL is set as the threat models, and the agent in RL plays the role of frame
selecting and video attacking simultaneously. By continuously querying the
threat models and receiving the feedback of predicted probabilities (reward),
the agent adjusts its frame selection strategy and performs attacks (action).
Step by step, the optimal key frames are selected and the smallest adversarial
perturbations are achieved. We conduct a series of experiments with two
mainstream video recognition models: C3D and LRCN on the public UCF-101 and
HMDB-51 datasets. The results demonstrate that the proposed method can
significantly reduce the perturbation of adversarial examples and attacking on
the sparse video frames can have better attack effectiveness than attacking on
each frame.


http://arxiv.org/abs/2001.03662
ReluDiff: Differential Verification of Deep Neural Networks.
Brandon Paulsen; Jingbo Wang; Chao Wang
  As deep neural networks are increasingly being deployed in practice, their
efficiency has become an important issue. While there are compression
techniques for reducing the network's size, energy consumption and
computational requirement, they only demonstrate empirically that there is no
loss of accuracy, but lack formal guarantees of the compressed network, e.g.,
in the presence of adversarial examples. Existing verification techniques such
as Reluplex, ReluVal, and DeepPoly provide formal guarantees, but they are
designed for analyzing a single network instead of the relationship between two
networks. To fill the gap, we develop a new method for differential
verification of two closely related networks. Our method consists of a fast but
approximate forward interval analysis pass followed by a backward pass that
iteratively refines the approximation until the desired property is verified.
We have two main innovations. During the forward pass, we exploit structural
and behavioral similarities of the two networks to more accurately bound the
difference between the output neurons of the two networks. Then in the backward
pass, we leverage the gradient differences to more accurately compute the most
beneficial refinement. Our experiments show that, compared to state-of-the-art
verification tools, our method can achieve orders-of-magnitude speedup and
prove many more properties than existing tools.


http://arxiv.org/abs/2001.03311
Guess First to Enable Better Compression and Adversarial Robustness.
Sicheng Zhu; Bang An; Shiyu Niu
  Machine learning models are generally vulnerable to adversarial examples,
which is in contrast to the robustness of humans. In this paper, we try to
leverage one of the mechanisms in human recognition and propose a bio-inspired
classification framework in which model inference is conditioned on label
hypothesis. We provide a class of training objectives for this framework and an
information bottleneck regularizer which utilizes the advantage that label
information can be discarded during inference. This framework enables better
compression of the mutual information between inputs and latent representations
without loss of learning capacity, at the cost of tractable inference
complexity. Better compression and elimination of label information further
bring better adversarial robustness without loss of natural accuracy, which is
demonstrated in the experiment.


http://arxiv.org/abs/2001.02438
To Transfer or Not to Transfer: Misclassification Attacks Against Transfer Learned Text Classifiers.
Bijeeta Pal; Shruti Tople
  Transfer learning --- transferring learned knowledge --- has brought a
paradigm shift in the way models are trained. The lucrative benefits of
improved accuracy and reduced training time have shown promise in training
models with constrained computational resources and fewer training samples.
Specifically, publicly available text-based models such as GloVe and BERT that
are trained on large corpus of datasets have seen ubiquitous adoption in
practice. In this paper, we ask, "can transfer learning in text prediction
models be exploited to perform misclassification attacks?" As our main
contribution, we present novel attack techniques that utilize unintended
features learnt in the teacher (public) model to generate adversarial examples
for student (downstream) models. To the best of our knowledge, ours is the
first work to show that transfer learning from state-of-the-art word-based and
sentence-based teacher models increase the susceptibility of student models to
misclassification attacks. First, we propose a novel word-score based attack
algorithm for generating adversarial examples against student models trained
using context-free word-level embedding model. On binary classification tasks
trained using the GloVe teacher model, we achieve an average attack accuracy of
97% for the IMDB Movie Reviews and 80% for the Fake News Detection. For
multi-class tasks, we divide the Newsgroup dataset into 6 and 20 classes and
achieve an average attack accuracy of 75% and 41% respectively. Next, we
present length-based and sentence-based misclassification attacks for the Fake
News Detection task trained using a context-aware BERT model and achieve 78%
and 39% attack accuracy respectively. Thus, our results motivate the need for
designing training techniques that are robust to unintended feature learning,
specifically for transfer learned models.


http://arxiv.org/abs/2001.02378
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius.
Runtian Zhai; Chen Dan; Di He; Huan Zhang; Boqing Gong; Pradeep Ravikumar; Cho-Jui Hsieh; Liwei Wang
  Adversarial training is one of the most popular ways to learn robust models
but is usually attack-dependent and time costly. In this paper, we propose the
MACER algorithm, which learns robust models without using adversarial training
but performs better than all existing provable l2-defenses. Recent work shows
that randomized smoothing can be used to provide a certified l2 radius to
smoothed classifiers, and our algorithm trains provably robust smoothed
classifiers via MAximizing the CErtified Radius (MACER). The attack-free
characteristic makes MACER faster to train and easier to optimize. In our
experiments, we show that our method can be applied to modern deep neural
networks on a wide range of datasets, including Cifar-10, ImageNet, MNIST, and
SVHN. For all tasks, MACER spends less training time than state-of-the-art
adversarial training algorithms, and the learned models achieve larger average
certified radius.


http://arxiv.org/abs/2001.03460
Transferability of Adversarial Examples to Attack Cloud-based Image Classifier Service.
Dou Goodman
  In recent years, Deep Learning(DL) techniques have been extensively deployed
for computer vision tasks, particularly visual classification problems, where
new algorithms reported to achieve or even surpass the human performance. While
many recent works demonstrated that DL models are vulnerable to adversarial
examples. Fortunately, generating adversarial examples usually requires
white-box access to the victim model, and real-world cloud-based image
classification services are more complex than white-box classifier,the
architecture and parameters of DL models on cloud platforms cannot be obtained
by the attacker. The attacker can only access the APIs opened by cloud
platforms. Thus, keeping models in the cloud can usually give a (false) sense
of security. In this paper, we mainly focus on studying the security of
real-world cloud-based image classification services. Specifically, (1) We
propose a novel attack method, Fast Featuremap Loss PGD (FFL-PGD) attack based
on Substitution model, which achieves a high bypass rate with a very limited
number of queries. Instead of millions of queries in previous studies, our
method finds the adversarial examples using only two queries per image; and (2)
we make the first attempt to conduct an extensive empirical study of black-box
attacks against real-world cloud-based classification services. Through
evaluations on four popular cloud platforms including Amazon, Google,
Microsoft, Clarifai, we demonstrate that FFL-PGD attack has a success rate over
90\% among different classification services. (3) We discuss the possible
defenses to address these security challenges in cloud-based classification
services. Our defense technology is mainly divided into model training stage
and image preprocessing stage.


http://arxiv.org/abs/2001.01987
Softmax-based Classification is k-means Clustering: Formal Proof, Consequences for Adversarial Attacks, and Improvement through Centroid Based Tailoring.
Sibylle Hess; Wouter Duivesteijn; Decebal Mocanu
  We formally prove the connection between k-means clustering and the
predictions of neural networks based on the softmax activation layer. In
existing work, this connection has been analyzed empirically, but it has never
before been mathematically derived. The softmax function partitions the
transformed input space into cones, each of which encompasses a class. This is
equivalent to putting a number of centroids in this transformed space at equal
distance from the origin, and k-means clustering the data points by proximity
to these centroids. Softmax only cares in which cone a data point falls, and
not how far from the centroid it is within that cone. We formally prove that
networks with a small Lipschitz modulus (which corresponds to a low
susceptibility to adversarial attacks) map data points closer to the cluster
centroids, which results in a mapping to a k-means-friendly space. To leverage
this knowledge, we propose Centroid Based Tailoring as an alternative to the
softmax function in the last layer of a neural network. The resulting Gauss
network has similar predictive accuracy as traditional networks, but is less
susceptible to one-pixel attacks; while the main contribution of this paper is
theoretical in nature, the Gauss network contributes empirical auxiliary
benefits.


http://arxiv.org/abs/2001.02297
Generating Semantic Adversarial Examples via Feature Manipulation.
Shuo Wang; Shangyu Chen; Tianle Chen; Surya Nepal; Carsten Rudolph; Marthie Grobler
  The vulnerability of deep neural networks to adversarial attacks has been
widely demonstrated (e.g., adversarial example attacks). Traditional attacks
perform unstructured pixel-wise perturbation to fool the classifier. An
alternative approach is to have perturbations in the latent space. However,
such perturbations are hard to control due to the lack of interpretability and
disentanglement. In this paper, we propose a more practical adversarial attack
by designing structured perturbation with semantic meanings. Our proposed
technique manipulates the semantic attributes of images via the disentangled
latent codes. The intuition behind our technique is that images in similar
domains have some commonly shared but theme-independent semantic attributes,
e.g. thickness of lines in handwritten digits, that can be bidirectionally
mapped to disentangled latent codes. We generate adversarial perturbation by
manipulating a single or a combination of these latent codes and propose two
unsupervised semantic manipulation approaches: vector-based disentangled
representation and feature map-based disentangled representation, in terms of
the complexity of the latent codes and smoothness of the reconstructed images.
We conduct extensive experimental evaluations on real-world image data to
demonstrate the power of our attacks for black-box classifiers. We further
demonstrate the existence of a universal, image-agnostic semantic adversarial
example.


http://arxiv.org/abs/2001.01506
Deceiving Image-to-Image Translation Networks for Autonomous Driving with Adversarial Perturbations.
Lin Wang; Wonjune Cho; Kuk-Jin Yoon
  Deep neural networks (DNNs) have achieved impressive performance on handling
computer vision problems, however, it has been found that DNNs are vulnerable
to adversarial examples. For such reason, adversarial perturbations have been
recently studied in several respects. However, most previous works have focused
on image classification tasks, and it has never been studied regarding
adversarial perturbations on Image-to-image (Im2Im) translation tasks, showing
great success in handling paired and/or unpaired mapping problems in the field
of autonomous driving and robotics. This paper examines different types of
adversarial perturbations that can fool Im2Im frameworks for autonomous driving
purpose. We propose both quasi-physical and digital adversarial perturbations
that can make Im2Im models yield unexpected results. We then empirically
analyze these perturbations and show that they generalize well under both
paired for image synthesis and unpaired settings for style transfer. We also
validate that there exist some perturbation thresholds over which the Im2Im
mapping is disrupted or impossible. The existence of these perturbations
reveals that there exist crucial weaknesses in Im2Im models. Lastly, we show
that our methods illustrate how perturbations affect the quality of outputs,
pioneering the improvement of the robustness of current SOTA networks for
autonomous driving.


http://arxiv.org/abs/2001.01172
The Human Visual System and Adversarial AI.
Yaoshiang Ho; Samuel Wookey
  This paper applies theories about the Human Visual System to make Adversarial
AI more effective. To date, Adversarial AI has modeled perceptual distances
between clean and adversarial examples of images using Lp norms. These norms
have the benefit of simple mathematical description and reasonable
effectiveness in approximating perceptual distance. However, in prior decades,
other areas of image processing have moved beyond simpler models like Mean
Squared Error (MSE) towards more complex models that better approximate the
Human Visual System (HVS). We demonstrate a proof of concept of incorporating
HVS models into Adversarial AI.


http://arxiv.org/abs/2001.00483
Reject Illegal Inputs with Generative Classifier Derived from Any Discriminative Classifier.
Xin Wang
  Generative classifiers have been shown promising to detect illegal inputs
including adversarial examples and out-of-distribution samples. Supervised Deep
Infomax~(SDIM) is a scalable end-to-end framework to learn generative
classifiers. In this paper, we propose a modification of SDIM termed
SDIM-\emph{logit}. Instead of training generative classifier from scratch,
SDIM-\emph{logit} first takes as input the logits produced any given
discriminative classifier, and generate logit representations; then a
generative classifier is derived by imposing statistical constraints on logit
representations. SDIM-\emph{logit} could inherit the performance of the
discriminative classifier without loss. SDIM-\emph{logit} incurs a negligible
number of additional parameters, and can be efficiently trained with base
classifiers fixed. We perform \emph{classification with rejection}, where test
samples whose class conditionals are smaller than pre-chosen thresholds will be
rejected without predictions. Experiments on illegal inputs, including
adversarial examples, samples with common corruptions, and
out-of-distribution~(OOD) samples show that allowed to reject a portion of test
samples, SDIM-\emph{logit} significantly improves the performance on the left
test sets.


http://arxiv.org/abs/2001.01587
Exploring Adversarial Attack in Spiking Neural Networks with Spike-Compatible Gradient.
Ling Liang; Xing Hu; Lei Deng; Yujie Wu; Guoqi Li; Yufei Ding; Peng Li; Yuan Xie
  Recently, backpropagation through time inspired learning algorithms are
widely introduced into SNNs to improve the performance, which brings the
possibility to attack the models accurately given Spatio-temporal gradient
maps. We propose two approaches to address the challenges of gradient input
incompatibility and gradient vanishing. Specifically, we design a gradient to
spike converter to convert continuous gradients to ternary ones compatible with
spike inputs. Then, we design a gradient trigger to construct ternary gradients
that can randomly flip the spike inputs with a controllable turnover rate, when
meeting all zero gradients. Putting these methods together, we build an
adversarial attack methodology for SNNs trained by supervised algorithms.
Moreover, we analyze the influence of the training loss function and the firing
threshold of the penultimate layer, which indicates a "trap" region under the
cross-entropy loss that can be escaped by threshold tuning. Extensive
experiments are conducted to validate the effectiveness of our solution.
Besides the quantitative analysis of the influence factors, we evidence that
SNNs are more robust against adversarial attack than ANNs. This work can help
reveal what happens in SNN attack and might stimulate more research on the
security of SNN models and neuromorphic devices.


http://arxiv.org/abs/2001.00308
Ensembles of Many Diverse Weak Defenses can be Strong: Defending Deep Neural Networks Against Adversarial Attacks.
Ying Meng; Jianhai Su; Jason O'Kane; Pooyan Jamshidi
  Despite achieving state-of-the-art performance across many domains, machine
learning systems are highly vulnerable to subtle adversarial perturbations.
Although defense approaches have been proposed in recent years, many have been
bypassed by even weak adversarial attacks. An early
study~\cite{he2017adversarial} shows that ensembles created by combining
multiple weak defenses (i.e., input data transformations) are still weak. We
show that it is indeed possible to construct effective ensembles using weak
defenses to block adversarial attacks. However, to do so requires a diverse set
of such weak defenses. In this work, we propose Athena, an extensible framework
for building effective defenses to adversarial attacks against machine learning
systems. Here we conducted a comprehensive empirical study to evaluate several
realizations of Athena. More specifically, we evaluated the effectiveness of 5
ensemble strategies with a diverse set of many weak defenses that comprise
transforming the inputs (e.g., rotation, shifting, noising, denoising, and many
more) before feeding them to target deep neural network (DNN) classifiers. We
evaluate the effectiveness of the ensembles with adversarial examples generated
by 9 various adversaries (i.e., FGSM, CW, etc.) in 4 threat models (i.e.,
zero-knowledge, black-box, gray-box, white-box) on MNIST. We also explain, via
a comprehensive empirical study, why building defenses based on the idea of
many diverse weak defenses works, when it is most effective, and what its
inherent limitations and overhead are.


http://arxiv.org/abs/1912.13258
Automated Testing for Deep Learning Systems with Differential Behavior Criteria.
Yuan Gao; Yiqiang Han
  In this work, we conducted a study on building an automated testing system
for deep learning systems based on differential behavior criteria. The
automated testing goals were achieved by jointly optimizing two objective
functions: maximizing differential behaviors from models under testing and
maximizing neuron coverage. By observing differential behaviors from three
pre-trained models during each testing iteration, the input image that
triggered erroneous feedback was registered as a corner-case. The generated
corner-cases can be used to examine the robustness of DNNs and consequently
improve model accuracy. A project called DeepXplore was also used as a baseline
model. After we fully implemented and optimized the baseline system, we
explored its application as an augmenting training dataset with newly generated
corner cases. With the GTRSB dataset, by retraining the model based on
automated generated corner cases, the accuracy of three generic models
increased by 259.2%, 53.6%, and 58.3%, respectively. Further, to extend the
capability of automated testing, we explored other approaches based on
differential behavior criteria to generate photo-realistic images for deep
learning systems. One approach was to apply various transformations to the seed
images for the deep learning framework. The other approach was to utilize the
Generative Adversarial Networks (GAN) technique, which was implemented on MNIST
and Driving datasets. The style transferring capability has been observed very
effective in adding additional visual effects, replacing image elements, and
style-shifting (virtual image to real images). The GAN-based testing sample
generation system was shown to be the next frontier for automated testing for
deep learning systems.


http://arxiv.org/abs/2001.00071
Protecting GANs against privacy attacks by preventing overfitting.
Sumit Mukherjee; Yixi Xu; Anusua Trivedi; Juan Lavista Ferres
  Generative Adversarial Networks (GANs) have made releasing of synthetic
images a viable approach to share data without releasing the original dataset.
It has been shown that such synthetic data can be used for a variety of
downstream tasks such as training classifiers that would otherwise require the
original dataset to be shared. However, recent work has shown that the GAN
models and their synthetically generated data can be used to infer the training
set membership by an adversary who has access to the entire dataset and some
auxiliary information. Here we develop a new GAN architecture (privGAN) which
provides protection against this mode of attack while leading to negligible
loss in downstream performances. Our architecture explicitly prevents
overfitting to the training set thereby providing implicit protection against
white-box attacks. The main contributions of this paper are: i) we propose a
novel GAN architecture that can generate synthetic data in a privacy preserving
manner and demonstrate the effectiveness of our model against white--box
attacks on several benchmark datasets, ii) we provide a theoretical
understanding of the optimal solution of the GAN loss function, iii) we
demonstrate on two common benchmark datasets that synthetic images generated by
privGAN lead to negligible loss in downstream performance when compared against
non--private GANs. While we have focosued on benchmarking privGAN exclusively
of image datasets, the architecture of privGAN is not exclusive to image
datasets and can be easily extended to other types of datasets.


http://arxiv.org/abs/2001.00116
Erase and Restore: Simple, Accurate and Resilient Detection of $L_2$ Adversarial Examples.
Fei Zuo; Qiang Zeng
  By adding carefully crafted perturbations to input images, adversarial
examples (AEs) can be generated to mislead neural-network-based image
classifiers. $L_2$ adversarial perturbations by Carlini and Wagner (CW) are
regarded as among the most effective attacks. While many countermeasures
against AEs have been proposed, detection of adaptive CW $L_2$ AEs has been
very inaccurate. Our observation is that those deliberately altered pixels in
an $L_2$ AE, altogether, exert their malicious influence. By randomly erasing
some pixels from an $L_2$ AE and then restoring it with an inpainting
technique, such an AE, before and after the steps, tends to have different
classification results, while a benign sample does not show this symptom. Based
on this, we propose a novel AE detection technique, Erase and Restore (E\&R),
that exploits the limitation of $L_2$ attacks. On two popular image datasets,
CIFAR-10 and ImageNet, our experiments show that the proposed technique is able
to detect over 98% of the AEs generated by CW and other $L_2$ algorithms and
has a very low false positive rate on benign images. Moreover, our approach
demonstrate strong resilience to adaptive attacks. While adding noises and
inpainting each have been well studied, by combining them together, we deliver
a simple, accurate and resilient detection technique against adaptive $L_2$
AEs.


http://arxiv.org/abs/2001.00030
Quantum Adversarial Machine Learning.
Sirui Lu; Lu-Ming Duan; Dong-Ling Deng
  Adversarial machine learning is an emerging field that focuses on studying
vulnerabilities of machine learning approaches in adversarial settings and
developing techniques accordingly to make learning robust to adversarial
manipulations. It plays a vital role in various machine learning applications
and has attracted tremendous attention across different communities recently.
In this paper, we explore different adversarial scenarios in the context of
quantum machine learning. We find that, similar to traditional classifiers
based on classical neural networks, quantum learning systems are likewise
vulnerable to crafted adversarial examples, independent of whether the input
data is classical or quantum. In particular, we find that a quantum classifier
that achieves nearly the state-of-the-art accuracy can be conclusively deceived
by adversarial examples obtained via adding imperceptible perturbations to the
original legitimate samples. This is explicitly demonstrated with quantum
adversarial learning in different scenarios, including classifying real-life
images (e.g., handwritten digit images in the dataset MNIST), learning phases
of matter (such as, ferromagnetic/paramagnetic orders and symmetry protected
topological phases), and classifying quantum data. Furthermore, we show that
based on the information of the adversarial examples at hand, practical defense
strategies can be designed to fight against a number of different attacks. Our
results uncover the notable vulnerability of quantum machine learning systems
to adversarial perturbations, which not only reveals a novel perspective in
bridging machine learning and quantum physics in theory but also provides
valuable guidance for practical applications of quantum classifiers based on
both near-term and future quantum technologies.


http://arxiv.org/abs/2001.05844
Adversarial Example Generation using Evolutionary Multi-objective Optimization.
Takahiro Suzuki; Shingo Takeshita; Satoshi Ono
  This paper proposes Evolutionary Multi-objective Optimization (EMO)-based
Adversarial Example (AE) design method that performs under black-box setting.
Previous gradient-based methods produce AEs by changing all pixels of a target
image, while previous EC-based method changes small number of pixels to produce
AEs. Thanks to EMO's property of population based-search, the proposed method
produces various types of AEs involving ones locating between AEs generated by
the previous two approaches, which helps to know the characteristics of a
target model or to know unknown attack patterns. Experimental results showed
the potential of the proposed method, e.g., it can generate robust AEs and,
with the aid of DCT-based perturbation pattern generation, AEs for high
resolution images.


http://arxiv.org/abs/1912.12859
Defending from adversarial examples with a two-stream architecture.
Hao Ge; Xiaoguang Tu; Mei Xie; Zheng Ma
  In recent years, deep learning has shown impressive performance on many
tasks. However, recent researches showed that deep learning systems are
vulnerable to small, specially crafted perturbations that are imperceptible to
humans. Images with such perturbations are the so called adversarial examples,
which have proven to be an indisputable threat to the DNN based applications.
The lack of better understanding of the DNNs has prevented the development of
efficient defenses against adversarial examples. In this paper, we propose a
two-stream architecture to protect CNN from attacking by adversarial examples.
Our model draws on the idea of "two-stream" which commonly used in the security
field, and successfully defends different kinds of attack methods by the
differences of "high-resolution" and "low-resolution" networks in feature
extraction. We provide a reasonable interpretation on why our two-stream
architecture is difficult to defeat, and show experimentally that our method is
hard to defeat with state-of-the-art attacks. We demonstrate that our
two-stream architecture is robust to adversarial examples built by currently
known attacking algorithms.


http://arxiv.org/abs/1912.12510
Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices.
Chandramouli Shama Sastry; Sageev Oore
  When presented with Out-of-Distribution (OOD) examples, deep neural networks
yield confident, incorrect predictions. Detecting OOD examples is challenging,
and the potential risks are high. In this paper, we propose to detect OOD
examples by identifying inconsistencies between activity patterns and class
predicted. We find that characterizing activity patterns by Gram matrices and
identifying anomalies in gram matrix values can yield high OOD detection rates.
We identify anomalies in the gram matrices by simply comparing each value with
its respective range observed over the training data. Unlike many approaches,
this can be used with any pre-trained softmax classifier and does not require
access to OOD data for fine-tuning hyperparameters, nor does it require OOD
access for inferring parameters. The method is applicable across a variety of
architectures and vision datasets and, for the important and surprisingly hard
task of detecting far-from-distribution out-of-distribution examples, it
generally performs better than or equal to state-of-the-art OOD detection
methods (including those that do assume access to OOD examples).


http://arxiv.org/abs/1912.12463
Search Based Repair of Deep Neural Networks.
Jeongju Sohn; Sungmin Kang; Shin Yoo
  Deep Neural Networks (DNNs) are being adopted in various domains, including
safety critical ones. The wide-spread adoption also calls for ways to guide the
testing of their accuracy and robustness, for which various test adequacy
criteria and input generation methods have been recently introduced. In this
paper, we explore the natural subsequent step: given an input that reveals
unexpected behaviour in a trained DNN, we propose to repair the DNN using
input-output pairs as a specification. This paper introduces Arachne, a novel
program repair technique for DNNs. Arachne first performs sensitivity based
fault localisation to limit the number of neural weights it has to modify.
Subsequently, Arachne uses Particle Swarm Optimisation (PSO) to directly
optimise the localised neural weights until the behaviour is corrected. An
empirical study using three different benchmark datasets shows that Arachne can
reduce the instances of the most frequent misclassification type committed by a
pre-trained CIFAR-10 classifier by 27.5%, without any need for additional
training data. Patches generated by Arachne tend to be more focused on the
targeted misbehaviour than DNN retraining, which is more disruptive to
non-targeted behaviour. The overall results suggest the feasibility of patching
DNNs using Arachne until they can be retrained properly.


http://arxiv.org/abs/1912.11852
Benchmarking Adversarial Robustness.
Yinpeng Dong; Qi-An Fu; Xiao Yang; Tianyu Pang; Hang Su; Zihao Xiao; Jun Zhu
  Deep neural networks are vulnerable to adversarial examples, which becomes
one of the most important research problems in the development of deep
learning. While a lot of efforts have been made in recent years, it is of great
significance to perform correct and complete evaluations of the adversarial
attack and defense algorithms. In this paper, we establish a comprehensive,
rigorous, and coherent benchmark to evaluate adversarial robustness on image
classification tasks. After briefly reviewing plenty of representative attack
and defense methods, we perform large-scale experiments with two robustness
curves as the fair-minded evaluation criteria to fully understand the
performance of these methods. Based on the evaluation results, we draw several
important findings and provide insights for future research.


http://arxiv.org/abs/1912.11969
Efficient Adversarial Training with Transferable Adversarial Examples.
Haizhong Zheng; Ziqi Zhang; Juncheng Gu; Honglak Lee; Atul Prakash
  Adversarial training is an effective defense method to protect classification
models against adversarial attacks. However, one limitation of this approach is
that it can require orders of magnitude additional training time due to high
cost of generating strong adversarial examples during training. In this paper,
we first show that there is high transferability between models from
neighboring epochs in the same training process, i.e., adversarial examples
from one epoch continue to be adversarial in subsequent epochs. Leveraging this
property, we propose a novel method, Adversarial Training with Transferable
Adversarial Examples (ATTA), that can enhance the robustness of trained models
and greatly improve the training efficiency by accumulating adversarial
perturbations through epochs. Compared to state-of-the-art adversarial training
methods, ATTA enhances adversarial accuracy by up to 7.2% on CIFAR10 and
requires 12~14x less training time on MNIST and CIFAR10 datasets with
comparable model robustness.


http://arxiv.org/abs/1912.11464
Attack-Resistant Federated Learning with Residual-based Reweighting.
Shuhao Fu; Chulin Xie; Bo Li; Qifeng Chen
  Federated learning has a variety of applications in multiple domains by
utilizing private training data stored on different devices. However, the
aggregation process in federated learning is highly vulnerable to adversarial
attacks so that the global model may behave abnormally under attacks. To tackle
this challenge, we present a novel aggregation algorithm with residual-based
reweighting to defend federated learning. Our aggregation algorithm combines
repeated median regression with the reweighting scheme in iteratively
reweighted least squares. Our experiments show that our aggregation algorithm
outperforms other alternative algorithms in the presence of label-flipping,
backdoor, and Gaussian noise attacks. We also provide theoretical guarantees
for our aggregation algorithm.


http://arxiv.org/abs/1912.11372
Analysis of Moving Target Defense Against False Data Injection Attacks on Power Grid.
Zhenyong Zhang; Ruilong Deng; Member; IEEE; David K. Y. Yau; Senior Member; IEEE; Peng Cheng; Member; IEEE; Jiming Chen; Fellow; IEEE
  Recent studies have considered thwarting false data injection (FDI) attacks
against state estimation in power grids by proactively perturbing branch
susceptances. This approach is known as moving target defense (MTD). However,
despite of the deployment of MTD, it is still possible for the attacker to
launch stealthy FDI attacks generated with former branch susceptances. In this
paper, we prove that, an MTD has the capability to thwart all FDI attacks
constructed with former branch susceptances only if (i) the number of branches
$l$ in the power system is not less than twice that of the system states $n$
(i.e., $l \geq 2n$, where $n + 1$ is the number of buses); (ii) the
susceptances of more than $n$ branches, which cover all buses, are perturbed.
Moreover, we prove that the state variable of a bus that is only connected by a
single branch (no matter it is perturbed or not) can always be modified by the
attacker. Nevertheless, in order to reduce the attack opportunities of
potential attackers, we first exploit the impact of the susceptance
perturbation magnitude on the dimension of the \emph{stealthy attack space}, in
which the attack vector is constructed with former branch susceptances. Then,
we propose that, by perturbing an appropriate set of branches, we can minimize
the dimension of the \emph{stealthy attack space} and maximize the number of
covered buses. Besides, we consider the increasing operation cost caused by the
activation of MTD. Finally, we conduct extensive simulations to illustrate our
findings with IEEE standard test power systems.


http://arxiv.org/abs/1912.11279
Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer.
Hongyan Chang; Virat Shejwalkar; Reza Shokri; Amir Houmansadr
  Collaborative (federated) learning enables multiple parties to train a model
without sharing their private data, but through repeated sharing of the
parameters of their local models. Despite its advantages, this approach has
many known privacy and security weaknesses and performance overhead, in
addition to being limited only to models with homogeneous architectures. Shared
parameters leak a significant amount of information about the local (and
supposedly private) datasets. Besides, federated learning is severely
vulnerable to poisoning attacks, where some participants can adversarially
influence the aggregate parameters. Large models, with high dimensional
parameter vectors, are in particular highly susceptible to privacy and security
attacks: curse of dimensionality in federated learning. We argue that sharing
parameters is the most naive way of information exchange in collaborative
learning, as they open all the internal state of the model to inference
attacks, and maximize the model's malleability by stealthy poisoning attacks.
We propose Cronus, a robust collaborative machine learning framework. The
simple yet effective idea behind designing Cronus is to control, unify, and
significantly reduce the dimensions of the exchanged information between
parties, through robust knowledge transfer between their black-box local
models. We evaluate all existing federated learning algorithms against
poisoning attacks, and we show that Cronus is the only secure method, due to
its tight robustness guarantee. Treating local models as black-box, reduces the
information leakage through models, and enables us using existing
privacy-preserving algorithms that mitigate the risk of information leakage
through the model's output (predictions). Cronus also has a significantly lower
sample complexity, compared to federated learning, which does not bind its
security to the number of participants.


http://arxiv.org/abs/1912.11460
Characterizing the Decision Boundary of Deep Neural Networks.
Hamid Karimi; Tyler Derr; Jiliang Tang
  Deep neural networks and in particular, deep neural classifiers have become
an integral part of many modern applications. Despite their practical success,
we still have limited knowledge of how they work and the demand for such an
understanding is evergrowing. In this regard, one crucial aspect of deep neural
network classifiers that can help us deepen our knowledge about their
decision-making behavior is to investigate their decision boundaries.
Nevertheless, this is contingent upon having access to samples populating the
areas near the decision boundary. To achieve this, we propose a novel approach
we call Deep Decision boundary Instance Generation (DeepDIG). DeepDIG utilizes
a method based on adversarial example generation as an effective way of
generating samples near the decision boundary of any deep neural network model.
Then, we introduce a set of important principled characteristics that take
advantage of the generated instances near the decision boundary to provide
multifaceted understandings of deep neural networks. We have performed
extensive experiments on multiple representative datasets across various deep
neural network models and characterized their decision boundaries.


http://arxiv.org/abs/1912.12106
White Noise Analysis of Neural Networks.
Ali Borji; Sikun Lin
  A white noise analysis of modern deep neural networks is presented to unveil
their biases at the whole network level or the single neuron level. Our
analysis is based on two popular and related methods in psychophysics and
neurophysiology namely classification images and spike triggered analysis.
These methods have been widely used to understand the underlying mechanisms of
sensory systems in humans and monkeys. We leverage them to investigate the
inherent biases of deep neural networks and to obtain a first-order
approximation of their functionality. We emphasize on CNNs since they are
currently the state of the art methods in computer vision and are a decent
model of human visual processing. In addition, we study multi-layer
perceptrons, logistic regression, and recurrent neural networks. Experiments
over four classic datasets, MNIST, Fashion-MNIST, CIFAR-10, and ImageNet, show
that the computed bias maps resemble the target classes and when used for
classification lead to an over twofold performance than the chance level.
Further, we show that classification images can be used to attack a black-box
classifier and to detect adversarial patch attacks. Finally, we utilize spike
triggered averaging to derive the filters of CNNs and explore how the behavior
of a network changes when neurons in different layers are modulated. Our effort
illustrates a successful example of borrowing from neurosciences to study ANNs
and highlights the importance of cross-fertilization and synergy across machine
learning, deep learning, and computational neuroscience.


http://arxiv.org/abs/1912.11188
Adversarial AutoAugment.
Xinyu Zhang; Qiang Wang; Jian Zhang; Zhao Zhong
  Data augmentation (DA) has been widely utilized to improve generalization in
training deep neural networks. Recently, human-designed data augmentation has
been gradually replaced by automatically learned augmentation policy. Through
finding the best policy in well-designed search space of data augmentation,
AutoAugment can significantly improve validation accuracy on image
classification tasks. However, this approach is not computationally practical
for large-scale problems. In this paper, we develop an adversarial method to
arrive at a computationally-affordable solution called Adversarial AutoAugment,
which can simultaneously optimize target related object and augmentation policy
search loss. The augmentation policy network attempts to increase the training
loss of a target network through generating adversarial augmentation policies,
while the target network can learn more robust features from harder examples to
improve the generalization. In contrast to prior work, we reuse the computation
in target network training for policy evaluation, and dispense with the
retraining of the target network. Compared to AutoAugment, this leads to about
12x reduction in computing cost and 11x shortening in time overhead on
ImageNet. We show experimental results of our approach on CIFAR-10/CIFAR-100,
ImageNet, and demonstrate significant performance improvements over
state-of-the-art. On CIFAR-10, we achieve a top-1 test error of 1.36%, which is
the currently best performing single model. On ImageNet, we achieve a leading
performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D
without extra data.


http://arxiv.org/abs/1912.11171
Geometry-aware Generation of Adversarial and Cooperative Point Clouds.
Yuxin Wen; Jiehong Lin; Ke Chen; Kui Jia
  Recent studies show that machine learning models are vulnerable to
adversarial examples. In 2D image domain, these examples are obtained by adding
imperceptible noises to natural images. This paper studies adversarial
generation of point clouds by learning to deform those approximating object
surfaces of certain categories. As 2D manifolds embedded in the 3D Euclidean
space, object surfaces enjoy the general properties of smoothness and fairness.
We thus argue that in order to achieve imperceptible surface shape
deformations, adversarial point clouds should have the same properties with
similar degrees of smoothness/fairness to the benign ones, while being close to
the benign ones as well when measured under certain distance metrics of point
clouds. To this end, we propose a novel loss function to account for
imperceptible, geometry-aware deformations of point clouds, and use the
proposed loss in an adversarial objective to attack representative models of
point set classifiers. Experiments show that our proposed method achieves
stronger attacks than existing methods, without introduction of noticeable
outliers and surface irregularities. In this work, we also investigate an
opposite direction that learns to deform point clouds of object surfaces in the
same geometry-aware, but cooperative manner. Cooperatively generated point
clouds are more favored by machine learning models in terms of improved
classification confidence or accuracy. We present experiments verifying that
our proposed objective succeeds in learning cooperative shape deformations.


http://arxiv.org/abs/1912.10375
AdvCodec: Towards A Unified Framework for Adversarial Text Generation.
Boxin Wang; Hengzhi Pei; Han Liu; Bo Li
  While there has been great interest in generating imperceptible adversarial
examples in continuous data domain (e.g. image and audio) to explore the model
vulnerabilities, generating \emph{adversarial text} in the discrete domain is
still challenging. The main contribution of this paper is to propose a general
targeted attack framework AdvCodec for adversarial text generation which
addresses the challenge of discrete input space and is easily adapted to
general natural language processing (NLP) tasks. In particular, we propose a
tree-based autoencoder to encode discrete text data into continuous vector
space, upon which we optimize the adversarial perturbation. A tree-based
decoder is then applied to ensure the grammar correctness of the generated
text. It also enables the flexibility of making manipulations on different
levels of text, such as sentence (AdvCodec(sent)) and word (AdvCodec(word))
levels. We consider multiple attacking scenarios, including appending an
adversarial sentence or adding unnoticeable words to a given paragraph, to
achieve the arbitrary targeted attack. To demonstrate the effectiveness of the
proposed method, we consider two most representative NLP tasks: sentiment
analysis and question answering (QA). Extensive experimental results and human
studies show that AdvCodec generated adversarial text can successfully attack
the neural models without misleading the human. In particular, our attack
causes a BERT-based sentiment classifier accuracy to drop from 0.703$ to 0.006,
and a BERT-based QA model's F1 score to drop from 88.62 to 33.21 (with best
targeted attack F1 score as 46.54). Furthermore, we show that the white-box
generated adversarial texts can transfer across other black-box models,
shedding light on an effective way to examine the robustness of existing NLP
models.


http://arxiv.org/abs/1912.10154
Measuring Dataset Granularity.
Yin Cui; Zeqi Gu; Dhruv Mahajan; der Maaten Laurens van; Serge Belongie; Ser-Nam Lim
  Despite the increasing visibility of fine-grained recognition in our field,
"fine-grained'' has thus far lacked a precise definition. In this work,
building upon clustering theory, we pursue a framework for measuring dataset
granularity. We argue that dataset granularity should depend not only on the
data samples and their labels, but also on the distance function we choose. We
propose an axiomatic framework to capture desired properties for a dataset
granularity measure and provide examples of measures that satisfy these
properties. We assess each measure via experiments on datasets with
hierarchical labels of varying granularity. When measuring granularity in
commonly used datasets with our measure, we find that certain datasets that are
widely considered fine-grained in fact contain subsets of considerable size
that are substantially more coarse-grained than datasets generally regarded as
coarse-grained. We also investigate the interplay between dataset granularity
with a variety of factors and find that fine-grained datasets are more
difficult to learn from, more difficult to transfer to, more difficult to
perform few-shot learning with, and more vulnerable to adversarial attacks.


http://arxiv.org/abs/1912.09899
Certified Robustness for Top-k Predictions against Adversarial Perturbations via Randomized Smoothing.
Jinyuan Jia; Xiaoyu Cao; Binghui Wang; Neil Zhenqiang Gong
  It is well-known that classifiers are vulnerable to adversarial
perturbations. To defend against adversarial perturbations, various certified
robustness results have been derived. However, existing certified robustnesses
are limited to top-1 predictions. In many real-world applications, top-$k$
predictions are more relevant. In this work, we aim to derive certified
robustness for top-$k$ predictions. In particular, our certified robustness is
based on randomized smoothing, which turns any classifier to a new classifier
via adding noise to an input example. We adopt randomized smoothing because it
is scalable to large-scale neural networks and applicable to any classifier. We
derive a tight robustness in $\ell_2$ norm for top-$k$ predictions when using
randomized smoothing with Gaussian noise. We find that generalizing the
certified robustness from top-1 to top-$k$ predictions faces significant
technical challenges. We also empirically evaluate our method on CIFAR10 and
ImageNet. For example, our method can obtain an ImageNet classifier with a
certified top-5 accuracy of 62.8\% when the $\ell_2$-norms of the adversarial
perturbations are less than 0.5 (=127/255). Our code is publicly available at:
\url{https://github.com/jjy1994/Certify_Topk}.


http://arxiv.org/abs/1912.10013
secml: A Python Library for Secure and Explainable Machine Learning.
Marco Melis; Ambra Demontis; Maura Pintor; Angelo Sotgiu; Battista Biggio
  We present secml, an open-source Python library for secure and explainable
machine learning. It implements the most popular attacks against machine
learning, including not only test-time evasion attacks to generate adversarial
examples against deep neural networks, but also training-time poisoning attacks
against support vector machines and many other algorithms. These attacks enable
evaluating the security of learning algorithms and of the corresponding
defenses under both white-box and black-box threat models. To this end, secml
provides built-in functions to compute security evaluation curves, showing how
quickly classification performance decreases against increasing adversarial
perturbations of the input data. secml also includes explainability methods to
help understand why adversarial attacks succeed against a given model, by
visualizing the most influential features and training prototypes contributing
to each decision. It is distributed under the Apache License 2.0, and hosted at
https://gitlab.com/secml/secml.


http://arxiv.org/abs/1912.10185
Jacobian Adversarially Regularized Networks for Robustness.
Alvin Chan; Yi Tay; Yew Soon Ong; Jie Fu
  Adversarial examples are crafted with imperceptible perturbations with the
intent to fool neural networks. Against such attacks, adversarial training and
its variants stand as the strongest defense to date. Previous studies have
pointed out that robust models that have undergone adversarial training tend to
produce more salient and interpretable Jacobian matrices than their non-robust
counterparts. A natural question is whether a model trained with an objective
to produce salient Jacobian can result in better robustness. This paper answers
this question with affirmative empirical results. We propose Jacobian
Adversarially Regularized Networks (JARN) as a method to optimize the saliency
of a classifier's Jacobian by adversarially regularizing the model's Jacobian
to resemble natural training images. Image classifiers trained with JARN show
improved robust accuracy compared to standard models on the MNIST, SVHN and
CIFAR-10 datasets, uncovering a new angle to boost robustness without using
adversarial training examples.


http://arxiv.org/abs/1912.09855
Explainability and Adversarial Robustness for RNNs.
Alexander Hartl; Maximilian Bachl; Joachim Fabini; Tanja Zseby
  Recurrent Neural Networks (RNNs) yield attractive properties for constructing
Intrusion Detection Systems (IDSs) for network data. With the rise of
ubiquitous Machine Learning (ML) systems, malicious actors have been catching
up quickly to find new ways to exploit ML vulnerabilities for profit. Recently
developed adversarial ML techniques focus on computer vision and their
applicability to network traffic is not straightforward: Network packets expose
fewer features than an image, are sequential and impose several constraints on
their features.
  We show that despite these completely different characteristics, adversarial
samples can be generated reliably for RNNs. To understand a classifier's
potential for misclassification, we extend existing explainability techniques
and propose new ones, suitable particularly for sequential data. Applying them
shows that already the first packets of a communication flow are of crucial
importance and are likely to be targeted by attackers. Feature importance
methods show that even relatively unimportant features can be effectively
abused to generate adversarial samples. Since traditional evaluation metrics
such as accuracy are not sufficient for quantifying the adversarial threat, we
propose the Adversarial Robustness Score (ARS) for comparing IDSs, capturing a
common notion of adversarial robustness, and show that an adversarial training
procedure can significantly and successfully reduce the attack surface.


http://arxiv.org/abs/1912.09670
Adversarial symmetric GANs: bridging adversarial samples and adversarial networks.
Faqiang Liu; Mingkun Xu; Guoqi Li; Jing Pei; Luping Shi; Rong Zhao
  Generative adversarial networks have achieved remarkable performance on
various tasks but suffer from training instability. Despite many training
strategies proposed to improve training stability, this issue remains as a
challenge. In this paper, we investigate the training instability from the
perspective of adversarial samples and reveal that adversarial training on fake
samples is implemented in vanilla GANs, but adversarial training on real
samples has long been overlooked. Consequently, the discriminator is extremely
vulnerable to adversarial perturbation and the gradient given by the
discriminator contains non-informative adversarial noises, which hinders the
generator from catching the pattern of real samples. Here, we develop
adversarial symmetric GANs (AS-GANs) that incorporate adversarial training of
the discriminator on real samples into vanilla GANs, making adversarial
training symmetrical. The discriminator is therefore more robust and provides
more informative gradient with less adversarial noise, thereby stabilizing
training and accelerating convergence. The effectiveness of the AS-GANs is
verified on image generation on CIFAR-10 , CelebA, and LSUN with varied network
architectures. Not only the training is more stabilized, but the FID scores of
generated samples are consistently improved by a large margin compared to the
baseline. The bridging of adversarial samples and adversarial networks provides
a new approach to further develop adversarial networks.


http://arxiv.org/abs/1912.10834
Does Symbolic Knowledge Prevent Adversarial Fooling?
Stefano Teso
  Arguments in favor of injecting symbolic knowledge into neural architectures
abound. When done right, constraining a sub-symbolic model can substantially
improve its performance and sample complexity and prevent it from predicting
invalid configurations. Focusing on deep probabilistic (logical) graphical
models -- i.e., constrained joint distributions whose parameters are determined
(in part) by neural nets based on low-level inputs -- we draw attention to an
elementary but unintended consequence of symbolic knowledge: that the resulting
constraints can propagate the negative effects of adversarial examples.


http://arxiv.org/abs/1912.10833
A New Ensemble Method for Concessively Targeted Multi-model Attack.
Ziwen He; Wei Wang; Xinsheng Xuan; Jing Dong; Tieniu Tan
  It is well known that deep learning models are vulnerable to adversarial
examples crafted by maliciously adding perturbations to original inputs. There
are two types of attacks: targeted attack and non-targeted attack, and most
researchers often pay more attention to the targeted adversarial examples.
However, targeted attack has a low success rate, especially when aiming at a
robust model or under a black-box attack protocol. In this case, non-targeted
attack is the last chance to disable AI systems. Thus, in this paper, we
propose a new attack mechanism which performs the non-targeted attack when the
targeted attack fails. Besides, we aim to generate a single adversarial sample
for different deployed models of the same task, e.g. image classification
models. Hence, for this practical application, we focus on attacking ensemble
models by dividing them into two groups: easy-to-attack and robust models. We
alternately attack these two groups of models in the non-targeted or targeted
manner. We name it a bagging and stacking ensemble (BAST) attack. The BAST
attack can generate an adversarial sample that fails multiple models
simultaneously. Some of the models classify the adversarial sample as a target
label, and other models which are not attacked successfully may give wrong
labels at least. The experimental results show that the proposed BAST attack
outperforms the state-of-the-art attack methods on the new defined criterion
that considers both targeted and non-targeted attack performance.


http://arxiv.org/abs/1912.09533
Towards Verifying Robustness of Neural Networks Against Semantic Perturbations.
Jeet Lily Mohapatra; Lily Tsui-Wei; Weng; Pin-Yu Chen; Sijia Liu; Luca Daniel
  Verifying robustness of neural networks given a specified threat model is a
fundamental yet challenging task. While current verification methods mainly
focus on the L_p-norm-ball threat model of the input instances, robustness
verification against semantic adversarial attacks inducing large L_p-norm
perturbations such as color shifting and lighting adjustment are beyond their
capacity. To bridge this gap, we propose Semantify-NN, a model-agnostic and
generic robustness verification approach against semantic perturbations for
neural networks. By simply inserting our proposed semantic perturbation layers
(SP-layers) to the input layer of any given model, Semantify-NN is
model-agnostic, and any $L_p$-norm-ball based verification tools can be used to
verify the model robustness against semantic perturbations. We illustrate the
principles of designing the SP-layers and provide examples including semantic
perturbations to image classification in the space of hue, saturation,
lightness, brightness, contrast and rotation, respectively. Experimental
results on various network architectures and different datasets demonstrate the
superior verification performance of Semantify-NN over L_p-norm-based
verification frameworks that naively convert semantic perturbation to L_p-norm.
To the best of our knowledge, Semantify-NN is the first framework to support
robustness verification against a wide range of semantic perturbations.


http://arxiv.org/abs/1912.12170
Mitigating large adversarial perturbations on X-MAS (X minus Moving Averaged Samples).
Woohyung Chun; Sung-Min Hong; Junho Huh; Inyup Kang
  We propose the scheme that mitigates an adversarial perturbation $\epsilon$
on the adversarial example $X_{adv}$ ($=$ $X$ $\pm$ $\epsilon$) by subtracting
the estimated perturbation $\hat{\epsilon}$ from $X$ $+$ $\epsilon$ and adding
$\hat{\epsilon}$ to $X$ $-$ $\epsilon$. The estimated perturbation
$\hat{\epsilon}$ comes from the difference between $X_{adv}$ and its
moving-averaged outcome $W_{avg}*X_{adv}$ where $W_{avg}$ is $N \times N$
moving average kernel that all the coefficients are one. Usually, the adjacent
samples of an image are close to each other such that we can let $X$ $\approx$
$W_{avg}*X$ (naming this relation after X-MAS[X minus Moving Averaged Sample]).
Since the X-MAS relation is approximately zero, the estimated perturbation can
be less than the adversarial perturbation. The scheme is also extended to do
the multi-level mitigation by configuring the mitigated adversarial example
$X_{adv}$ $\pm$ $\hat{\epsilon}$ as a new adversarial example to be mitigated.
The multi-level mitigation gets $X_{adv}$ closer to $X$ with a smaller (i.e.
mitigated) perturbation than original unmitigated perturbation by setting
$W_{avg} * X_{adv}$ ($<$ $X$ $+$ $W_{avg}*\epsilon$ if $X$ $\approx$
$W_{avg}*X$) as the boundary condition that the multi-level mitigation cannot
cross over (i.e. decreasing $\epsilon$ cannot go below and increasing
$\epsilon$ cannot go beyond). With the multi-level mitigation, we can get high
prediction accuracies even in the adversarial example having a large
perturbation (i.e. $\epsilon$ $\geq$ $16$). The proposed scheme is evaluated
with adversarial examples crafted by the Iterative FGSM (Fast Gradient Sign
Method) on ResNet-50 trained with ImageNet dataset.


http://arxiv.org/abs/1912.09064
Optimization-Guided Binary Diversification to Mislead Neural Networks for Malware Detection.
Mahmood Sharif; Keane Lucas; Lujo Bauer; Michael K. Reiter; Saurabh Shintre
  Motivated by the transformative impact of deep neural networks (DNNs) on
different areas (e.g., image and speech recognition), researchers and
anti-virus vendors are proposing end-to-end DNNs for malware detection from raw
bytes that do not require manual feature engineering. Given the security
sensitivity of the task that these DNNs aim to solve, it is important to assess
their susceptibility to evasion.
  In this work, we propose an attack that guides binary-diversification tools
via optimization to mislead DNNs for malware detection while preserving the
functionality of binaries. Unlike previous attacks on such DNNs, ours
manipulates instructions that are a functional part of the binary, which makes
it particularly challenging to defend against. We evaluated our attack against
three DNNs in white-box and black-box settings, and found that it can often
achieve success rates near 100%. Moreover, we found that our attack can fool
some commercial anti-viruses, in certain cases with a success rate of 85%. We
explored several defenses, both new and old, and identified some that can
successfully prevent over 80% of our evasion attempts. However, these defenses
may still be susceptible to evasion by adaptive attackers, and so we advocate
for augmenting malware-detection systems with methods that do not rely on
machine learning.


http://arxiv.org/abs/1912.09059
$n$-ML: Mitigating Adversarial Examples via Ensembles of Topologically Manipulated Classifiers.
Mahmood Sharif; Lujo Bauer; Michael K. Reiter
  This paper proposes a new defense called $n$-ML against adversarial examples,
i.e., inputs crafted by perturbing benign inputs by small amounts to induce
misclassifications by classifiers. Inspired by $n$-version programming, $n$-ML
trains an ensemble of $n$ classifiers, and inputs are classified by a vote of
the classifiers in the ensemble. Unlike prior such approaches, however, the
classifiers in the ensemble are trained specifically to classify adversarial
examples differently, rendering it very difficult for an adversarial example to
obtain enough votes to be misclassified. We show that $n$-ML roughly retains
the benign classification accuracies of state-of-the-art models on the MNIST,
CIFAR10, and GTSRB datasets, while simultaneously defending against adversarial
examples with better resilience than the best defenses known to date and, in
most cases, with lower classification-time overhead.


http://arxiv.org/abs/1912.09405
Perturbations on the Perceptual Ball.
Andrew Elliott; Stephen Law; Chris Russell
  We present a simple regularisation of Adversarial Perturbations based upon
the perceptual loss. While the resulting perturbations remain imperceptible to
the human eye, they differ from existing adversarial perturbations in two
important regards: (i) our resulting perturbations are semi-sparse, and
typically make alterations to objects and regions of interest leaving the
background static; (ii) our perturbations do not alter the distribution of data
in the image and are undetectable by state-of-the-art methods. As such this
work reinforces the connection between explainable AI and adversarial
perturbations. We demonstrate the merits of our approach by evaluating on
standard explainability benchmark and by defeating a recent test for detecting
adversarial perturbations.


http://arxiv.org/abs/1912.08981
Identifying Adversarial Sentences by Analyzing Text Complexity.
Hoang-Quoc Nguyen-Son; Tran Phuong Thao; Seira Hidano; Shinsaku Kiyomoto
  Attackers create adversarial text to deceive both human perception and the
current AI systems to perform malicious purposes such as spam product reviews
and fake political posts. We investigate the difference between the adversarial
and the original text to prevent the risk. We prove that the text written by a
human is more coherent and fluent. Moreover, the human can express the idea
through the flexible text with modern words while a machine focuses on
optimizing the generated text by the simple and common words. We also suggest a
method to identify the adversarial text by extracting the features related to
our findings. The proposed method achieves high performance with 82.0% of
accuracy and 18.4% of equal error rate, which is better than the existing
methods whose the best accuracy is 77.0% corresponding to the error rate 22.8%.


http://arxiv.org/abs/1912.08954
An Adversarial Perturbation Oriented Domain Adaptation Approach for Semantic Segmentation.
Jihan Yang; Ruijia Xu; Ruiyu Li; Xiaojuan Qi; Xiaoyong Shen; Guanbin Li; Liang Lin
  We focus on Unsupervised Domain Adaptation (UDA) for the task of semantic
segmentation. Recently, adversarial alignment has been widely adopted to match
the marginal distribution of feature representations across two domains
globally. However, this strategy fails in adapting the representations of the
tail classes or small objects for semantic segmentation since the alignment
objective is dominated by head categories or large objects. In contrast to
adversarial alignment, we propose to explicitly train a domain-invariant
classifier by generating and defensing against pointwise feature space
adversarial perturbations. Specifically, we firstly perturb the intermediate
feature maps with several attack objectives (i.e., discriminator and
classifier) on each individual position for both domains, and then the
classifier is trained to be invariant to the perturbations. By perturbing each
position individually, our model treats each location evenly regardless of the
category or object size and thus circumvents the aforementioned issue.
Moreover, the domain gap in feature space is reduced by extrapolating source
and target perturbed features towards each other with attack on the domain
discriminator. Our approach achieves the state-of-the-art performance on two
challenging domain adaptation tasks for semantic segmentation: GTA5 ->
Cityscapes and SYNTHIA -> Cityscapes.


http://arxiv.org/abs/1912.08865
Adversarial VC-dimension and Sample Complexity of Neural Networks.
Zetong Qi; T. J. Wilder
  Adversarial attacks during the testing phase of neural networks pose a
challenge for the deployment of neural networks in security critical settings.
These attacks can be performed by adding noise that is imperceptible to humans
on top of the original data. By doing so, an attacker can create an adversarial
sample, which will cause neural networks to misclassify. In this paper, we seek
to understand the theoretical limits of what can be learned by neural networks
in the presence of an adversary. We first defined the hypothesis space of a
neural network, and showed the relationship between the growth number of the
entire neural network and the growth number of each neuron. Combine that with
the adversarial Vapnik-Chervonenkis(VC)-dimension of halfspace classifiers, we
concluded the adversarial VC-dimension of the neural networks with sign
activation functions.


http://arxiv.org/abs/1912.09303
SIGMA : Strengthening IDS with GAN and Metaheuristics Attacks.
Simon Msika; Alejandro Quintero; Foutse Khomh
  An Intrusion Detection System (IDS) is a key cybersecurity tool for network
administrators as it identifies malicious traffic and cyberattacks. With the
recent successes of machine learning techniques such as deep learning, more and
more IDS are now using machine learning algorithms to detect attacks faster.
However, these systems lack robustness when facing previously unseen types of
attacks. With the increasing number of new attacks, especially against Internet
of Things devices, having a robust IDS able to spot unusual and new attacks
becomes necessary.
  This work explores the possibility of leveraging generative adversarial
models to improve the robustness of machine learning based IDS. More
specifically, we propose a new method named SIGMA, that leverages adversarial
examples to strengthen IDS against new types of attacks. Using Generative
Adversarial Networks (GAN) and metaheuristics, SIGMA %Our method consists in
generates adversarial examples, iteratively, and uses it to retrain a machine
learning-based IDS, until a convergence of the detection rate (i.e. until the
detection system is not improving anymore). A round of improvement consists of
a generative phase, in which we use GANs and metaheuristics to generate
instances ; an evaluation phase in which we calculate the detection rate of
those newly generated attacks ; and a training phase, in which we train the IDS
with those attacks. We have evaluated the SIGMA method for four standard
machine learning classification algorithms acting as IDS, with a combination of
GAN and a hybrid local-search and genetic algorithm, to generate new datasets
of attacks. Our results show that SIGMA can successfully generate adversarial
attacks against different machine learning based IDS. Also, using SIGMA, we can
improve the performance of an IDS to up to 100\% after as little as two rounds
of improvement.


http://arxiv.org/abs/1912.08639
Detecting Adversarial Attacks On Audio-Visual Speech Recognition.
Pingchuan Ma; Stavros Petridis; Maja Pantic
  Adversarial attacks pose a threat to deep learning models. However, research
on adversarial detection methods, especially in the multi-modal domain, is very
limited. In this work, we propose an efficient and straightforward detection
method based on the temporal correlation between audio and video streams. The
main idea is that the correlation between audio and video in adversarial
examples will be lower than benign examples due to added adversarial noise. We
use the synchronisation confidence score as a proxy for audio-visual
correlation and based on it we can detect adversarial attacks. To the best of
our knowledge, this is the first work on detection of adversarial attacks on
audio-visual speech recognition models. We apply recent adversarial attacks on
two audio-visual speech recognition models trained on the GRID and LRW
datasets. The experimental results demonstrated that the proposed approach is
an effective way for detecting such attacks.


http://arxiv.org/abs/1912.08166
APRICOT: A Dataset of Physical Adversarial Attacks on Object Detection.
Anneliese Braunegg; Amartya Chakraborty; Michael Krumdick; Nicole Lape; Sara Leary; Keith Manville; Elizabeth Merkhofer; Laura Strickhart; Matthew Walmer
  Physical adversarial attacks threaten to fool object detection systems, but
reproducible research on the real-world effectiveness of physical patches and
how to defend against them requires a publicly available benchmark dataset. We
present APRICOT, a collection of over 1,000 annotated photographs of printed
adversarial patches in public locations. The patches target several object
categories for three COCO-trained detection models, and the photos represent
natural variation in position, distance, lighting conditions, and viewing
angle. Our analysis suggests that maintaining adversarial robustness in
uncontrolled settings is highly challenging, but it is still possible to
produce targeted detections under white-box and sometimes black-box settings.
We establish baselines for defending against adversarial patches through
several methods, including a detector supervised with synthetic data and
unsupervised methods such as kernel density estimation, Bayesian uncertainty,
and reconstruction error. Our results suggest that adversarial patches can be
effectively flagged, both in a high-knowledge, attack-specific scenario, and in
an unsupervised setting where patches are detected as anomalies in natural
images. This dataset and the described experiments provide a benchmark for
future research on the effectiveness of and defenses against physical
adversarial objects in the wild.


http://arxiv.org/abs/1912.07742
CAG: A Real-time Low-cost Enhanced-robustness High-transferability Content-aware Adversarial Attack Generator.
Huy Phan; Yi Xie; Siyu Liao; Jie Chen; Bo Yuan
  Deep neural networks (DNNs) are vulnerable to adversarial attack despite
their tremendous success in many AI fields. Adversarial attack is a method that
causes the intended misclassfication by adding imperceptible perturbations to
legitimate inputs. Researchers have developed numerous types of adversarial
attack methods. However, from the perspective of practical deployment, these
methods suffer from several drawbacks such as long attack generating time, high
memory cost, insufficient robustness and low transferability. We propose a
Content-aware Adversarial Attack Generator (CAG) to achieve real-time,
low-cost, enhanced-robustness and high-transferability adversarial attack.
First, as a type of generative model-based attack, CAG shows significant
speedup (at least 500 times) in generating adversarial examples compared to the
state-of-the-art attacks such as PGD and C\&W. CAG only needs a single
generative model to perform targeted attack to any targeted class. Because CAG
encodes the label information into a trainable embedding layer, it differs from
prior generative model-based adversarial attacks that use $n$ different copies
of generative models for $n$ different targeted classes. As a result, CAG
significantly reduces the required memory cost for generating adversarial
examples. CAG can generate adversarial perturbations that focus on the critical
areas of input by integrating the class activation maps information in the
training process, and hence improve the robustness of CAG attack against the
state-of-art adversarial defenses. In addition, CAG exhibits high
transferability across different DNN classifier models in black-box attack
scenario by introducing random dropout in the process of generating
perturbations. Extensive experiments on different datasets and DNN models have
verified the real-time, low-cost, enhanced-robustness, and high-transferability
benefits of CAG.


http://arxiv.org/abs/1912.07748
MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking.
Rushil Anirudh; Jayaraman J. Thiagarajan; Bhavya Kailkhura; Timo Bremer
  In the past few years, Generative Adversarial Networks (GANs) have
dramatically advanced our ability to represent and parameterize
high-dimensional, non-linear image manifolds. As a result, they have been
widely adopted across a variety of applications, ranging from challenging
inverse problems like image completion, to problems such as anomaly detection
and adversarial defense. A recurring theme in many of these applications is the
notion of projecting an image observation onto the manifold that is inferred by
the generator. In this context, Projected Gradient Descent (PGD) has been the
most popular approach, which essentially optimizes for a latent vector that
minimizes the discrepancy between a generated image and the given observation.
However, PGD is a brittle optimization technique that fails to identify the
right projection (or latent vector) when the observation is corrupted, or
perturbed even by a small amount. Such corruptions are common in the real
world, for example images in the wild come with unknown crops, rotations,
missing pixels, or other kinds of non-linear distributional shifts which break
current encoding methods, rendering downstream applications unusable. To
address this, we propose corruption mimicking -- a new robust projection
technique, that utilizes a surrogate network to approximate the unknown
corruption directly at test time, without the need for additional supervision
or data augmentation. The proposed method is significantly more robust than PGD
and other competing methods under a wide variety of corruptions, thereby
enabling a more effective use of GANs in real-world applications. More
importantly, we show that our approach produces state-of-the-art performance in
several GAN-based applications -- anomaly detection, domain adaptation, and
adversarial defense, that benefit from an accurate projection.


http://arxiv.org/abs/1912.07458
On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration.
Kanil Patel; William Beluch; Dan Zhang; Michael Pfeiffer; Bin Yang
  Uncertainty estimates help to identify ambiguous, novel, or anomalous inputs,
but the reliable quantification of uncertainty has proven to be challenging for
modern deep networks. To improve uncertainty estimation, we propose On-Manifold
Adversarial Data Augmentation or OMADA, which specifically attempts to generate
the most challenging examples by following an on-manifold adversarial attack
path in the latent space of an autoencoder-based generative model that closely
approximates decision boundaries between two or more classes. On a variety of
datasets and for multiple network architectures, OMADA consistently yields more
accurate and better calibrated classifiers than baseline models, and
outperforms competing approaches such as Mixup and CutMix, as well as achieving
similar performance to (at times better than) post-processing calibration
methods such as temperature scaling. Variants of OMADA can employ different
sampling schemes for ambiguous on-manifold examples based on the entropy of
their estimated soft labels, which exhibit specific strengths for
generalization, calibration of predicted uncertainty, or detection of
out-of-distribution inputs.


http://arxiv.org/abs/1912.07561
Constructing a provably adversarially-robust classifier from a high accuracy one.
Grzegorz Głuch; Rüdiger Urbanke
  Modern machine learning models with very high accuracy have been shown to be
vulnerable to small, adversarially chosen perturbations of the input. Given
black-box access to a high-accuracy classifier $f$, we show how to construct a
new classifier $g$ that has high accuracy and is also robust to adversarial
$\ell_2$-bounded perturbations. Our algorithm builds upon the framework of
\textit{randomized smoothing} that has been recently shown to outperform all
previous defenses against $\ell_2$-bounded adversaries. Using techniques like
random partitions and doubling dimension, we are able to bound the adversarial
error of $g$ in terms of the optimum error. In this paper we focus on our
conceptual contribution, but we do present two examples to illustrate our
framework. We will argue that, under some assumptions, our bounds are optimal
for these cases.


http://arxiv.org/abs/1912.07160
DAmageNet: A Universal Adversarial Dataset.
Sizhe Chen; Xiaolin Huang; Zhengbao He; Chengjin Sun
  It is now well known that deep neural networks (DNNs) are vulnerable to
adversarial attack. Adversarial samples are similar to the clean ones, but are
able to cheat the attacked DNN to produce incorrect predictions in high
confidence. But most of the existing adversarial attacks have high success rate
only when the information of the attacked DNN is well-known or could be
estimated by massive queries. A promising way is to generate adversarial
samples with high transferability. By this way, we generate 96020 transferable
adversarial samples from original ones in ImageNet. The average difference,
measured by root means squared deviation, is only around 3.8 on average.
However, the adversarial samples are misclassified by various models with an
error rate up to 90\%. Since the images are generated independently with the
attacked DNNs, this is essentially zero-query adversarial attack. We call the
dataset \emph{DAmageNet}, which is the first universal adversarial dataset that
beats many models trained in ImageNet. By finding the drawbacks, DAmageNet
could serve as a benchmark to study and improve robustness of DNNs. DAmageNet
could be downloaded in http://www.pami.sjtu.edu.cn/Show/56/122.


http://arxiv.org/abs/1912.06960
What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance.
Mahmoud Afifi; Michael S Brown
  There is active research targeting local image manipulations that can fool
deep neural networks (DNNs) into producing incorrect results. This paper
examines a type of global image manipulation that can produce similar adverse
effects. Specifically, we explore how strong color casts caused by incorrectly
applied computational color constancy - referred to as white balance (WB) in
photography - negatively impact the performance of DNNs targeting image
segmentation and classification. In addition, we discuss how existing image
augmentation methods used to improve the robustness of DNNs are not well suited
for modeling WB errors. To address this problem, a novel augmentation method is
proposed that can emulate accurate color constancy degradation. We also explore
pre-processing training and testing images with a recent WB correction
algorithm to reduce the effects of incorrectly white-balanced images. We
examine both augmentation and pre-processing strategies on different datasets
and demonstrate notable improvements on the CIFAR-10, CIFAR-100, and ADE20K
datasets.


http://arxiv.org/abs/1912.06872
Towards Robust Toxic Content Classification.
Keita Kurita; Anna Belova; Antonios Anastasopoulos
  Toxic content detection aims to identify content that can offend or harm its
recipients. Automated classifiers of toxic content need to be robust against
adversaries who deliberately try to bypass filters. We propose a method of
generating realistic model-agnostic attacks using a lexicon of toxic tokens,
which attempts to mislead toxicity classifiers by diluting the toxicity signal
either by obfuscating toxic tokens through character-level perturbations, or by
injecting non-toxic distractor tokens. We show that these realistic attacks
reduce the detection recall of state-of-the-art neural toxicity detectors,
including those using ELMo and BERT, by more than 50% in some cases. We explore
two approaches for defending against such attacks. First, we examine the effect
of training on synthetically noised data. Second, we propose the Contextual
Denoising Autoencoder (CDAE): a method for learning robust representations that
uses character-level and contextual information to denoise perturbed tokens. We
show that the two approaches are complementary, improving robustness to both
character-level perturbations and distractors, recovering a considerable
portion of the lost accuracy. Finally, we analyze the robustness
characteristics of the most competitive methods and outline practical
considerations for improving toxicity detectors.


http://arxiv.org/abs/1912.06409
Potential adversarial samples for white-box attacks.
Amir Nazemi; Paul Fieguth
  Deep convolutional neural networks can be highly vulnerable to small
perturbations of their inputs, potentially a major issue or limitation on
system robustness when using deep networks as classifiers. In this paper we
propose a low-cost method to explore marginal sample data near trained
classifier decision boundaries, thus identifying potential adversarial samples.
By finding such adversarial samples it is possible to reduce the search space
of adversarial attack algorithms while keeping a reasonable successful
perturbation rate. In our developed strategy, the potential adversarial samples
represent only 61% of the test data, but in fact cover more than 82% of the
adversarial samples produced by iFGSM and 92% of the adversarial samples
successfully perturbed by DeepFool on CIFAR10.


http://arxiv.org/abs/1912.05683
Learning to Model Aspects of Hearing Perception Using Neural Loss Functions.
Prateek Verma; Jonathan Berger
  We present a framework to model the perceived quality of audio signals by
combining convolutional architectures, with ideas from classical signal
processing, and describe an approach to enhancing perceived acoustical quality.
We demonstrate the approach by transforming the sound of an inexpensive musical
with degraded sound quality to that of a high-quality musical instrument
without the need for parallel data which is often hard to collect. We adapt the
classical approach of a simple adaptive EQ filtering to the objective criterion
learned by a neural architecture and optimize it to get the signal of our
interest. Since we learn adaptive masks depending on the signal of interest as
opposed to a fixed transformation for all the inputs, we show that shallow
neural architectures can achieve the desired result. A simple constraint on the
objective and the initialization helps us in avoiding adversarial examples,
which otherwise would have produced noisy, unintelligible audio. We believe
that the current framework proposed has enormous applications, in a variety of
problems where one can learn a loss function depending on the problem, using a
neural architecture and optimize it after it has been learned.


http://arxiv.org/abs/1912.05661
Gabor Layers Enhance Network Robustness.
Juan C. Pérez; Motasem Alfarra; Guillaume Jeanneret; Adel Bibi; Ali Thabet; Bernard Ghanem; Pablo Arbeláez
  We revisit the benefits of merging classical vision concepts with deep
learning models. In particular, we explore the effect on robustness against
adversarial attacks of replacing the first layers of various deep architectures
with Gabor layers, i.e. convolutional layers with filters that are based on
learnable Gabor parameters. We observe that architectures enhanced with Gabor
layers gain a consistent boost in robustness over regular models and preserve
high generalizing test performance, even though these layers come at a
negligible increase in the number of parameters. We then exploit the closed
form expression of Gabor filters to derive an expression for a Lipschitz
constant of such filters, and harness this theoretical result to develop a
regularizer we use during training to further enhance network robustness. We
conduct extensive experiments with various architectures (LeNet, AlexNet, VGG16
and WideResNet) on several datasets (MNIST, SVHN, CIFAR10 and CIFAR100) and
demonstrate large empirical robustness gains. Furthermore, we experimentally
show how our regularizer provides consistent robustness improvements.


http://arxiv.org/abs/1912.05699
What it Thinks is Important is Important: Robustness Transfers through Input Gradients.
Alvin Chan; Yi Tay; Yew-Soon Ong
  Adversarial perturbations are imperceptible changes to input pixels that can
change the prediction of deep learning models. Learned weights of models robust
to such perturbations are previously found to be transferable across different
tasks but this applies only if the model architecture for the source and target
tasks is the same. Input gradients characterize how small changes at each input
pixel affect the model output. Using only natural images, we show here that
training a student model's input gradients to match those of a robust teacher
model can gain robustness close to a strong baseline that is robustly trained
from scratch. Through experiments in MNIST, CIFAR-10, CIFAR-100 and
Tiny-ImageNet, we show that our proposed method, input gradient adversarial
matching, can transfer robustness across different tasks and even across
different model architectures. This demonstrates that directly targeting the
semantics of input gradients is a feasible way towards adversarial robustness.


http://arxiv.org/abs/1912.05333
An Efficient Approach for Using Expectation Maximization Algorithm in Capsule Networks.
Moein Hasani; Amin Nasim Saravi; Hassan Khotanlou
  Capsule Networks (CapsNets) are brand-new architectures that have shown
ground-breaking results in certain areas of Computer Vision (CV). In 2017,
Hinton and his team introduced CapsNets with routing-by-agreement in "Sabour et
al" and in a more recent paper "Matrix Capsules with EM Routing" they proposed
a more complete architecture with Expectation-Maximization (EM) algorithm.
Unlike the traditional convolutional neural networks (CNNs), this architecture
is able to preserve the pose of the objects in the picture. Due to this
characteristic, it has been able to beat the previous state-of-theart results
on the smallNORB dataset, which includes samples with various view points.
Also, this architecture is more robust to white box adversarial attacks.
However, CapsNets have two major drawbacks. They can't perform as well as CNNs
on complex datasets and, they need a huge amount of time for training. We try
to mitigate these shortcomings by finding optimum settings of EM routing
iterations for training CapsNets. Unlike the past studies, we use un-equal
numbers of EM routing iterations for different stages of the CapsNet. For our
research, we use three datasets: Yale face dataset, Belgium Traffic Sign
dataset, and Fashion-MNIST dataset.


http://arxiv.org/abs/1912.05391
Detecting and Correcting Adversarial Images Using Image Processing Operations and Convolutional Neural Networks.
Huy H. Nguyen; Minoru Kuribayashi; Junichi Yamagishi; Isao Echizen
  Deep neural networks (DNNs) have achieved excellent performance on several
tasks and have been widely applied in both academia and industry. However, DNNs
are vulnerable to adversarial machine learning attacks, in which noise is added
to the input to change the network output. We have devised two methods for
detecting adversarial images; one based on statistical image processing and one
based on convolutional neural network in which the final softmax layer is
removed during training. In addition to detection, the image-processing-based
method can be used to reduce adversarial noise in images and thereby restore
the image labels, which is crucial to restoring the normal functionalities of
DNN-based systems. Testing using an adversarial machine learning database we
created for generating several types of attack using images from the ImageNet
Large Scale Visual Recognition Challenge database demonstrated the efficiency
of our proposed methods for both detection and correction even when training
was done from scratch on a small database.


http://arxiv.org/abs/1912.05945
Towards a Robust Classifier: An MDL-Based Method for Generating Adversarial Examples.
Behzad Asadi; Vijay Varadharajan
  We address the problem of adversarial examples in machine learning where an
adversary tries to misguide a classifier by making functionality-preserving
modifications to original samples. We assume a black-box scenario where the
adversary has access to only the feature set, and the final hard-decision
output of the classifier. We propose a method to generate adversarial examples
using the minimum description length (MDL) principle. Our final aim is to
improve the robustness of the classifier by considering generated examples in
rebuilding the classifier. We evaluate our method for the application of static
malware detection in portable executable (PE) files. We consider API calls of
PE files as their distinguishing features where the feature vector is a binary
vector representing the presence-absence of API calls. In our method, we first
create a dataset of benign samples by querying the target classifier. We next
construct a code table of frequent patterns for the compression of this dataset
using the MDL principle. We finally generate an adversarial example
corresponding to a malware sample by selecting and adding a pattern from the
benign code table to the malware sample. The selected pattern is the one that
minimizes the length of the compressed adversarial example given the code
table. This modification preserves the functionalities of the original malware
sample as all original API calls are kept, and only some new API calls are
added. Considering a neural network, we show that the evasion rate is 78.24
percent for adversarial examples compared to 8.16 percent for original malware
samples. This shows the effectiveness of our method in generating examples that
need to be considered in rebuilding the classifier.


http://arxiv.org/abs/1912.04792
Training Provably Robust Models by Polyhedral Envelope Regularization.
Chen Liu; Mathieu Salzmann; Sabine Süsstrunk
  Training certifiable neural networks enables one to obtain models with
robustness guarantees against adversarial attacks. In this work, we introduce a
framework to bound the adversary-free region in the neighborhood of the input
data by a polyhedral envelope, which yields finer-grained certified robustness.
We further introduce polyhedral envelope regularization (PER) to encourage
larger polyhedral envelopes and thus improve the provable robustness of the
models. We demonstrate the flexibility and effectiveness of our framework on
standard benchmarks; it applies to networks of different architectures and
general activation functions. Compared with the state-of-the-art methods, PER
has very little computational overhead and better robustness guarantees without
over-regularizing the model.


http://arxiv.org/abs/1912.04538
Appending Adversarial Frames for Universal Video Attack.
Zhikai Chen; Lingxi Xie; Shanmin Pang; Yong He; Qi Tian
  There have been many efforts in attacking image classification models with
adversarial perturbations, but the same topic on video classification has not
yet been thoroughly studied. This paper presents a novel idea of video-based
attack, which appends a few dummy frames (e.g., containing the texts of `thanks
for watching') to a video clip and then adds adversarial perturbations only on
these new frames. Our approach enjoys three major benefits, namely, a high
success rate, a low perceptibility, and a strong ability in transferring across
different networks. These benefits mostly come from the common dummy frame
which pushes all samples towards the boundary of classification. On the other
hand, such attacks are easily to be concealed since most people would not
notice the abnormality behind the perturbed video clips. We perform experiments
on two popular datasets with six state-of-the-art video classification models,
and demonstrate the effectiveness of our approach in the scenario of universal
video attacks.


http://arxiv.org/abs/1912.04497
Feature Losses for Adversarial Robustness.
Kirthi Shankar Sivamani
  Deep learning has made tremendous advances in computer vision tasks such as
image classification. However, recent studies have shown that deep learning
models are vulnerable to specifically crafted adversarial inputs that are
quasi-imperceptible to humans. In this work, we propose a novel approach to
defending adversarial attacks. We employ an input processing technique based on
denoising autoencoders as a defense. It has been shown that the input
perturbations grow and accumulate as noise in feature maps while propagating
through a convolutional neural network (CNN). We exploit the noisy feature maps
by using an additional subnetwork to extract image feature maps and train an
auto-encoder on perceptual losses of these feature maps. This technique
achieves close to state-of-the-art results on defending MNIST and CIFAR10
datasets, but more importantly, shows a new way of employing a defense that
cannot be trivially trained end-to-end by the attacker. Empirical results
demonstrate the effectiveness of this approach on the MNIST and CIFAR10
datasets on simple as well as iterative LP attacks. Our method can be applied
as a preprocessing technique to any off the shelf CNN.


http://arxiv.org/abs/1912.03790
Hardening Random Forest Cyber Detectors Against Adversarial Attacks.
Giovanni Apruzzese; Mauro Andreolini; Michele Colajanni; Mirco Marchetti
  Machine learning algorithms are effective in several applications, but they
are not as much successful when applied to intrusion detection in cyber
security. Due to the high sensitivity to their training data, cyber detectors
based on machine learning are vulnerable to targeted adversarial attacks that
involve the perturbation of initial samples. Existing defenses assume
unrealistic scenarios; their results are underwhelming in non-adversarial
settings; or they can be applied only to machine learning algorithms that
perform poorly for cyber security. We present an original methodology for
countering adversarial perturbations targeting intrusion detection systems
based on random forests. As a practical application, we integrate the proposed
defense method in a cyber detector analyzing network traffic. The experimental
results on millions of labelled network flows show that the new detector has a
twofold value: it outperforms state-of-the-art detectors that are subject to
adversarial attacks; it exhibits robust results both in adversarial and
non-adversarial scenarios.


http://arxiv.org/abs/1912.03829
Amora: Black-box Adversarial Morphing Attack.
Run Wang; Felix Juefei-Xu; Xiaofei Xie; Lei Ma; Yihao Huang; Yang Liu
  Nowadays, digital facial content manipulation has become ubiquitous and
realistic with the unprecedented success of generative adversarial networks
(GANs) in image synthesis. Unfortunately, face recognition (FR) systems suffer
from severe security concerns due to facial image manipulations. In this paper,
we investigate and introduce a new type of adversarial attack to evade FR
systems by manipulating facial content, called adversarial morphing attack
(a.k.a. Amora). In contrast to adversarial noise attack that perturbs pixel
intensity values by adding human-imperceptible noise, our proposed adversarial
morphing attack is a semantic attack that perturbs pixels spatially in a
coherent manner. To tackle the black-box attack problem, we have devised a
simple yet effective joint dictionary learning pipeline to obtain a proprietary
optical flow field for each attack. We have quantitatively and qualitatively
demonstrated the effectiveness of our adversarial morphing attack at various
levels of morphing intensity on two popular FR systems with smiling facial
expression manipulations. Both open-set and closed-set experimental results
indicate that a novel black-box adversarial attack based on local deformation
is possible, which is vastly different from additive noise based attacks. The
findings of this work may pave a new research direction towards a more thorough
understanding and investigation of image-based adversarial attacks and
defenses.


http://arxiv.org/abs/1912.03609
Exploring the Back Alleys: Analysing The Robustness of Alternative Neural Network Architectures against Adversarial Attacks.
Yi Xiang Marcus Tan; Yuval Elovici; Alexander Binder
  Recent discoveries in the field of adversarial machine learning have shown
that Artificial Neural Networks (ANNs) are susceptible to adversarial attacks.
These attacks cause misclassification of specially crafted adversarial samples.
In light of this phenomenon, it is worth investigating whether other types of
neural networks are less susceptible to adversarial attacks. In this work, we
applied standard attack methods originally aimed at conventional ANNs, towards
stochastic ANNs and also towards Spiking Neural Networks (SNNs), across three
different datasets namely MNIST, CIFAR-10 and Patch Camelyon. We analysed their
adversarial robustness against attacks performed in the raw image space of the
different model variants. We employ a variety of attacks namely Basic Iterative
Method (BIM), Carlini & Wagner L2 attack (CWL2) and Boundary attack. Our
results suggests that SNNs and stochastic ANNs exhibit some degree of
adversarial robustness as compared to their ANN counterparts under certain
attack methods. Namely, we found that the Boundary and the state-of-the-art
CWL2 attacks are largely ineffective against stochastic ANNs. Following this
observation, we proposed a modified version of the CWL2 attack and analysed the
impact of this attack on the models' adversarial robustness. Our results
suggest that with this modified CWL2 attack, many models are more easily fooled
as compared to the vanilla CWL2 attack, albeit observing an increase in L2
norms of adversarial perturbations. Lastly, we also investigate the resilience
of alternative neural networks against adversarial samples transferred from
ResNet18. We show that the modified CWL2 attack provides an improved
cross-architecture transferability compared to other attacks.


http://arxiv.org/abs/1912.03192
Achieving Robustness in the Wild via Adversarial Mixing with Disentangled Representations.
Sven Gowal; Chongli Qin; Po-Sen Huang; Taylan Cemgil; Krishnamurthy Dvijotham; Timothy Mann; Pushmeet Kohli
  Recent research has made the surprising finding that state-of-the-art deep
learning models sometimes fail to generalize to small variations of the input.
Adversarial training has been shown to be an effective approach to overcome
this problem. However, its application has been limited to enforcing invariance
to analytically defined transformations like $\ell_p$-norm bounded
perturbations. Such perturbations do not necessarily cover plausible real-world
variations that preserve the semantics of the input (such as a change in
lighting conditions). In this paper, we propose a novel approach to express and
formalize robustness to these kinds of real-world transformations of the input.
The two key ideas underlying our formulation are (1) leveraging disentangled
representations of the input to define different factors of variations, and (2)
generating new input images by adversarially composing the representations of
different images. We use a StyleGAN model to demonstrate the efficacy of this
framework. Specifically, we leverage the disentangled latent representations
computed by a StyleGAN model to generate perturbations of an image that are
similar to real-world variations (like adding make-up, or changing the
skin-tone of a person) and train models to be invariant to these perturbations.
Extensive experiments show that our method improves generalization and reduces
the effect of spurious correlations (reducing the error rate of a "smile"
detector by 21% for example).


http://arxiv.org/abs/1912.03406
Principal Component Properties of Adversarial Samples.
Malhar Jere; Sandro Herbig; Christine Lind; Farinaz Koushanfar
  Deep Neural Networks for image classification have been found to be
vulnerable to adversarial samples, which consist of sub-perceptual noise added
to a benign image that can easily fool trained neural networks, posing a
significant risk to their commercial deployment. In this work, we analyze
adversarial samples through the lens of their contributions to the principal
components of each image, which is different than prior works in which authors
performed PCA on the entire dataset. We investigate a number of
state-of-the-art deep neural networks trained on ImageNet as well as several
attacks for each of the networks. Our results demonstrate empirically that
adversarial samples across several attacks have similar properties in their
contributions to the principal components of neural network inputs. We propose
a new metric for neural networks to measure their robustness to adversarial
samples, termed the (k,p) point. We utilize this metric to achieve 93.36%
accuracy in detecting adversarial samples independent of architecture and
attack type for models trained on ImageNet.


http://arxiv.org/abs/1912.03430
Does Interpretability of Neural Networks Imply Adversarial Robustness?
Adam Noack; Isaac Ahern; Dejing Dou; Boyang Li
  The success of deep neural networks is clouded by two issues: (1) a
vulnerability to adversarial examples and (2) a tendency to be uninterpretable.
Interestingly, recent empirical evidence in the literature as well as
theoretical analysis on simple models suggest these two seemingly disparate
issues are actually connected. In particular, robust models tend to be more
interpretable than non-robust models.
  In this paper, we provide evidence for the claim that this relationship is
bidirectional. Viz., models that are optimized to have interpretable gradients
are more robust to adversarial examples than models trained in a standard
manner. With further analysis and experiments on standard image classification
datasets, we identify two factors behind this phenomenon---namely the
suppression of the gradient's magnitude and the selective use of features
guided by high-quality interpretations---which explain model behaviors under
various regularization and target interpretation settings.


http://arxiv.org/abs/1912.02918
Detection of Face Recognition Adversarial Attacks.
Fabio Valerio Massoli; Fabio Carrara; Giuseppe Amato; Fabrizio Falchi
  Deep Learning methods have become state-of-the-art for solving tasks such as
Face Recognition (FR). Unfortunately, despite their success, it has been
pointed out that these learning models are exposed to adversarial inputs -
images to which an imperceptible amount of noise for humans is added to
maliciously fool a neural network - thus limiting their adoption in real-world
applications. While it is true that an enormous effort has been spent in order
to train robust models against this type of threat, adversarial detection
techniques have recently started to draw attention within the scientific
community. A detection approach has the advantage that it does not require to
re-train any model, thus it can be added on top of any system. In this context,
we present our work on adversarial samples detection in forensics mainly
focused on detecting attacks against FR systems in which the learning model is
typically used only as a features extractor. Thus, in these cases, train a more
robust classifier might not be enough to defence a FR system. In this frame,
the contribution of our work is four-fold: i) we tested our recently proposed
adversarial detection approach against classifier attacks, i.e. adversarial
samples crafted to fool a FR neural network acting as a classifier; ii) using a
k-Nearest Neighbor (kNN) algorithm as a guidance, we generated deep features
attacks against a FR system based on a DL model acting as features extractor,
followed by a kNN which gives back the query identity based on features
similarity; iii) we used the deep features attacks to fool a FR system on the
1:1 Face Verification task and we showed their superior effectiveness with
respect to classifier attacks in fooling such type of system; iv) we used the
detectors trained on classifier attacks to detect deep features attacks, thus
showing that such approach is generalizable to different types of offensives.


http://arxiv.org/abs/1912.02386
The Search for Sparse, Robust Neural Networks.
Justin Cosentino; Federico Zaiter; Dan Pei; Jun Zhu
  Recent work on deep neural network pruning has shown there exist sparse
subnetworks that achieve equal or improved accuracy, training time, and loss
using fewer network parameters when compared to their dense counterparts.
Orthogonal to pruning literature, deep neural networks are known to be
susceptible to adversarial examples, which may pose risks in security- or
safety-critical applications. Intuition suggests that there is an inherent
trade-off between sparsity and robustness such that these characteristics could
not co-exist. We perform an extensive empirical evaluation and analysis testing
the Lottery Ticket Hypothesis with adversarial training and show this approach
enables us to find sparse, robust neural networks. Code for reproducing
experiments is available here:
https://github.com/justincosentino/robust-sparse-networks.


http://arxiv.org/abs/1912.02598
Region-Wise Attack: On Efficient Generation of Robust Physical Adversarial Examples.
Bo Luo; Qiang Xu
  Deep neural networks (DNNs) are shown to be susceptible to adversarial
example attacks. Most existing works achieve this malicious objective by
crafting subtle pixel-wise perturbations, and they are difficult to launch in
the physical world due to inevitable transformations (e.g., different
photographic distances and angles). Recently, there are a few research works on
generating physical adversarial examples, but they generally require the
details of the model a priori, which is often impractical. In this work, we
propose a novel physical adversarial attack for arbitrary black-box DNN models,
namely Region-Wise Attack. To be specific, we present how to efficiently search
for region-wise perturbations to the inputs and determine their shapes,
locations and colors via both top-down and bottom-up techniques. In addition,
we introduce two fine-tuning techniques to further improve the robustness of
our attack. Experimental results demonstrate the efficacy and robustness of the
proposed Region-Wise Attack in real world.


http://arxiv.org/abs/1912.01810
Learning with Multiplicative Perturbations.
Xiulong Yang; Shihao Ji
  Adversarial Training (AT) and Virtual Adversarial Training (VAT) are the
regularization techniques that train Deep Neural Networks (DNNs) with
adversarial examples generated by adding small but worst-case perturbations to
input examples. In this paper, we propose xAT and xVAT, new adversarial
training algorithms, that generate \textbf{multiplicative} perturbations to
input examples for robust training of DNNs. Such perturbations are much more
perceptible and interpretable than their \textbf{additive} counterparts
exploited by AT and VAT. Furthermore, the multiplicative perturbations can be
generated transductively or inductively while the standard AT and VAT only
support a transductive implementation. We conduct a series of experiments that
analyze the behavior of the multiplicative perturbations and demonstrate that
xAT and xVAT match or outperform state-of-the-art classification accuracies
across multiple established benchmarks while being about 30\% faster than their
additive counterparts. Furthermore, the resulting DNNs also demonstrate
distinct weight distributions.


http://arxiv.org/abs/1912.02258
A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks.
Prithviraj Dasgupta; Joseph B. Collins
  Machine learning techniques are currently used extensively for automating
various cybersecurity tasks. Most of these techniques utilize supervised
learning algorithms that rely on training the algorithm to classify incoming
data into different categories, using data encountered in the relevant domain.
A critical vulnerability of these algorithms is that they are susceptible to
adversarial attacks where a malicious entity called an adversary deliberately
alters the training data to misguide the learning algorithm into making
classification errors. Adversarial attacks could render the learning algorithm
unsuitable to use and leave critical systems vulnerable to cybersecurity
attacks. Our paper provides a detailed survey of the state-of-the-art
techniques that are used to make a machine learning algorithm robust against
adversarial attacks using the computational framework of game theory. We also
discuss open problems and challenges and possible directions for further
research that would make deep machine learning-based systems more robust and
reliable for cybersecurity tasks.


http://arxiv.org/abs/1912.02153
Walking on the Edge: Fast, Low-Distortion Adversarial Examples.
Hanwei Zhang; Yannis Avrithis; Teddy Furon; Laurent Amsaleg
  Adversarial examples of deep neural networks are receiving ever increasing
attention because they help in understanding and reducing the sensitivity to
their input. This is natural given the increasing applications of deep neural
networks in our everyday lives. When white-box attacks are almost always
successful, it is typically only the distortion of the perturbations that
matters in their evaluation.
  In this work, we argue that speed is important as well, especially when
considering that fast attacks are required by adversarial training. Given more
time, iterative methods can always find better solutions. We investigate this
speed-distortion trade-off in some depth and introduce a new attack called
boundary projection (BP) that improves upon existing methods by a large margin.
Our key idea is that the classification boundary is a manifold in the image
space: we therefore quickly reach the boundary and then optimize distortion on
this manifold.


http://arxiv.org/abs/1912.02184
Towards Robust Image Classification Using Sequential Attention Models.
Daniel Zoran; Mike Chrzanowski; Po-Sen Huang; Sven Gowal; Alex Mott; Pushmeet Kohl
  In this paper we propose to augment a modern neural-network architecture with
an attention model inspired by human perception. Specifically, we adversarially
train and analyze a neural model incorporating a human inspired, visual
attention component that is guided by a recurrent top-down sequential process.
Our experimental evaluation uncovers several notable findings about the
robustness and behavior of this new model. First, introducing attention to the
model significantly improves adversarial robustness resulting in
state-of-the-art ImageNet accuracies under a wide range of random targeted
attack strengths. Second, we show that by varying the number of attention steps
(glances/fixations) for which the model is unrolled, we are able to make its
defense capabilities stronger, even in light of stronger attacks --- resulting
in a "computational race" between the attacker and the defender. Finally, we
show that some of the adversarial examples generated by attacking our model are
quite different from conventional adversarial examples --- they contain global,
salient and spatially coherent structures coming from the target class that
would be recognizable even to a human, and work by distracting the attention of
the model away from the main object in the original image.


http://arxiv.org/abs/1912.02316
Scratch that! An Evolution-based Adversarial Attack against Neural Networks.
Malhar Jere; Briland Hitaj; Gabriela Ciocarlie; Farinaz Koushanfar
  Recent research has shown that Deep Neural Networks (DNNs) for image
classification are vulnerable to adversarial attacks. However, most works on
adversarial samples utilize sub-perceptual noise that, while invisible or
slightly visible to humans, often covers the entire image. Additionally, most
of these attacks often require knowledge of the neural network architecture and
its parameters, and the ability to calculate the gradients of the parameters
with respect to the inputs. In this work, we show that it is possible to attack
neural networks in a highly restricted threat setting, where attackers have no
knowledge of the neural network (i.e., in a black-box setting) and can only
modify highly localized adversarial noise in the form of randomly chosen
straight lines or scratches. Our Adversarial Scratches attack method covers
only 1-2% of the image pixels and are generated using the Covariance Matrix
Adaptation Evolutionary Strategy, a purely black-box method that does not
require knowledge of the neural network architecture and its gradients. Against
ImageNet models, Adversarial Scratches requires 3 times fewer queries than
GenAttack (without any optimizations) and 73 times fewer queries than ZOO, both
prior state-of-the-art black-box attacks. We successfully deceive
state-of-the-art Inception-v3, ResNet-50 and VGG-19 models trained on ImageNet
with deceiving rates of 75.8%, 62.7%, and 45% respectively, with fewer queries
than several state-of-the-art black-box attacks, while modifying less than 2%
of the image pixels. Additionally, we provide a new threat scenario for neural
networks, demonstrate a new attack surface that can be used to perform
adversarial attacks, and discuss its potential implications.


http://arxiv.org/abs/1912.01667
A Survey of Black-Box Adversarial Attacks on Computer Vision Models.
Siddhant Bhambri; Sumanyu Muku; Avinash Tulasi; Arun Balaji Buduru
  Machine learning has seen tremendous advances in the past few years, which
has lead to deep learning models being deployed in varied applications of
day-to-day life. Attacks on such models using perturbations, particularly in
real-life scenarios, pose a severe challenge to their applicability, pushing
research into the direction which aims to enhance the robustness of these
models. After the introduction of these perturbations by Szegedy et al. [1],
significant amount of research has focused on the reliability of such models,
primarily in two aspects - white-box, where the adversary has access to the
targeted model and related parameters; and the black-box, which resembles a
real-life scenario with the adversary having almost no knowledge of the model
to be attacked. To provide a comprehensive security cover, it is essential to
identify, study, and build defenses against such attacks. Hence, in this paper,
we propose to present a comprehensive comparative study of various black-box
adversarial attacks and defense techniques.


http://arxiv.org/abs/1912.01978
FANNet: Formal Analysis of Noise Tolerance, Training Bias and Input Sensitivity in Neural Networks.
Mahum Naseer; Mishal Fatima Minhas; Faiq Khalid; Muhammad Abdullah Hanif; Osman Hasan; Muhammad Shafique
  With a constant improvement in the network architectures and training
methodologies, Neural Networks (NNs) are increasingly being deployed in
real-world Machine Learning systems. However, despite their impressive
performance on "known inputs", these NNs can fail absurdly on the "unseen
inputs", especially if these real-time inputs deviate from the training dataset
distributions, or contain certain types of input noise. This indicates the low
noise tolerance of NNs, which is a major reason for the recent increase of
adversarial attacks. This is a serious concern, particularly for
safety-critical applications, where inaccurate results lead to dire
consequences. We propose a novel methodology that leverages model checking for
the Formal Analysis of Neural Network (FANNet) under different input noise
ranges. Our methodology allows us to rigorously analyze the noise tolerance of
NNs, their input node sensitivity, and the effects of training bias on their
performance, e.g., in terms of classification accuracy. For evaluation, we use
a feed-forward fully-connected NN architecture trained for the Leukemia
classification. Our experimental results show $\pm 11\%$ noise tolerance for
the given trained network, identify the most sensitive input nodes, and confirm
the biasness of the available training dataset.


http://arxiv.org/abs/1912.01171
Universal Adversarial Perturbations for CNN Classifiers in EEG-Based BCIs.
Zihan Liu; Xiao Zhang; Dongrui Wu
  Multiple convolutional neural network (CNN) classifiers have been proposed
for electroencephalogram (EEG) based brain-computer interfaces (BCIs). However,
CNN models have been found vulnerable to universal adversarial perturbations
(UAPs), which are small and example-independent, yet powerful enough to degrade
the performance of a CNN model, when added to a benign example. This paper
proposes a novel total loss minimization (TLM) approach to generate UAPs for
EEG-based BCIs. Experimental results demonstrate the effectiveness of TLM on
three popular CNN classifiers for both target and non-target attacks. We also
verify the transferability of UAPs in EEG-based BCI systems. To our knowledge,
this is the first study on UAPs of CNN classifiers in EEG-based BCIs, and also
the first study on UAPs for target attacks. UAPs are easy to construct, and can
attack BCIs in real-time, exposing a critical security concern of BCIs.


http://arxiv.org/abs/1912.01149
Cost-Aware Robust Tree Ensembles for Security Applications.
Yizheng Chen; Shiqi Wang; Weifan Jiang; Asaf Cidon; Suman Jana
  Features of security classifiers have various costs to be manipulated. The
costs are asymmetric across features and to the directions of changes, which
cannot be precisely captured by existing cost models based on $L_p$-norm
robustness. In this paper, we utilize such domain knowledge to increase the
evasion cost against security classifiers, specifically, tree ensemble models
that are widely used by security tasks. We propose a new cost modeling method
to capture the domain knowledge of features as constraint, and then we
integrate the cost-driven constraint into the node construction process to
train robust tree ensembles. During the training process, we use the constraint
to find data points that are likely to be perturbed given the costs of the
features, and we optimize the quality of the trees using a new robust training
algorithm. Our cost-aware training method can be applied to different types of
tree ensembles, including random forest model that cannot be robustly trained
by previous methods. Using twitter spam detection as the security application,
our evaluation results show that training cost-aware robust model can rank high
cost features as the most important ones, and increase the adaptive attack cost
by 6.4X compared to the baseline.


http://arxiv.org/abs/1912.00888
Deep Neural Network Fingerprinting by Conferrable Adversarial Examples.
Nils Lukas; Yuxuan Zhang; Florian Kerschbaum
  In Machine Learning as a Service, a provider trains a deep neural network and
provides many users access to it. However, the hosted (source) model is
susceptible to model stealing attacks where an adversary derives a
\emph{surrogate model} from API access to the source model. For post hoc
detection of such attacks, the provider needs a robust method to determine
whether a suspect model is a surrogate of their model or not. We propose a
fingerprinting method for deep neural networks that extracts a set of inputs
from the source model so that only surrogates agree with the source model on
the classification of such inputs. These inputs are a specifically crafted
subclass of targeted transferable adversarial examples which we call
conferrable adversarial examples that transfer exclusively from a source model
to its surrogates. We propose new methods to generate these conferrable
adversarial examples and use them as our fingerprint. Our fingerprint is the
first to be successfully tested as robust against distillation attacks, and our
experiments show that this robustness extends to robustness against weaker
removal attacks such as fine-tuning, ensemble attacks, and adversarial
retraining. We even protect against a powerful adversary with white-box access
to the source model, whereas the defender only needs black-box access to the
surrogate model. We conduct our experiments on the CINIC dataset and a subset
of ImageNet32 with 100 classes.


http://arxiv.org/abs/1912.00330
Adversary A3C for Robust Reinforcement Learning.
Zhaoyuan Gu; Zhenzhong Jia; Howie Choset
  Asynchronous Advantage Actor Critic (A3C) is an effective Reinforcement
Learning (RL) algorithm for a wide range of tasks, such as Atari games and
robot control. The agent learns policies and value function through
trial-and-error interactions with the environment until converging to an
optimal policy. Robustness and stability are critical in RL; however, neural
network can be vulnerable to noise from unexpected sources and is not likely to
withstand very slight disturbances. We note that agents generated from mild
environment using A3C are not able to handle challenging environments. Learning
from adversarial examples, we proposed an algorithm called Adversary Robust A3C
(AR-A3C) to improve the agent's performance under noisy environments. In this
algorithm, an adversarial agent is introduced to the learning process to make
it more robust against adversarial disturbances, thereby making it more
adaptive to noisy environments. Both simulations and real-world experiments are
carried out to illustrate the stability of the proposed algorithm. The AR-A3C
algorithm outperforms A3C in both clean and noisy environments.


http://arxiv.org/abs/1912.00466
A Method for Computing Class-wise Universal Adversarial Perturbations.
Tejus Gupta; Abhishek Sinha; Nupur Kumari; Mayank Singh; Balaji Krishnamurthy
  We present an algorithm for computing class-specific universal adversarial
perturbations for deep neural networks. Such perturbations can induce
misclassification in a large fraction of images of a specific class. Unlike
previous methods that use iterative optimization for computing a universal
perturbation, the proposed method employs a perturbation that is a linear
function of weights of the neural network and hence can be computed much
faster. The method does not require any training data and has no
hyper-parameters. The attack obtains 34% to 51% fooling rate on
state-of-the-art deep neural networks on ImageNet and transfers across models.
We also study the characteristics of the decision boundaries learned by
standard and adversarially trained models to understand the universal
adversarial perturbations.


http://arxiv.org/abs/1912.00461
AdvPC: Transferable Adversarial Perturbations on 3D Point Clouds.
Abdullah Hamdi; Sara Rojas; Ali Thabet; Bernard Ghanem
  Deep neural networks are vulnerable to adversarial attacks, in which
imperceptible perturbations to their input lead to erroneous network
predictions. This phenomenon has been extensively studied in the image domain,
and only recently extended to 3D point clouds. In this work, we present novel
data-driven adversarial attacks against 3D point cloud networks. We aim to
address the following problems in current 3D point cloud adversarial attacks:
they do not transfer well between different networks, and they are easy to
defend against simple statistical methods. To this extent, we develop new point
cloud attacks (we dub AdvPC) that exploit input data distributions. These
attacks lead to perturbations that are resilient against current defenses while
remaining highly transferable compared to state-of-the-art attacks. We test our
attacks using four popular point cloud networks: PointNet, PointNet++ (MSG and
SSG), and DGCNN. Our proposed attack enables an increase in the transferability
of up to 20 points for some networks. It also increases the ability to break
defenses of up to 23 points on ModelNet40 data.


http://arxiv.org/abs/1912.05021
Design and Interpretation of Universal Adversarial Patches in Face Detection.
Xiao Yang; Fangyun Wei; Hongyang Zhang; Xiang Ming; Jun Zhu
  We consider universal adversarial patches for faces - small visual elements
whose addition to a face image reliably destroys the performance of face
detectors. Unlike previous work that mostly focused on the algorithmic design
of adversarial examples in terms of improving the success rate as an attacker,
in this work we show an interpretation of such patches that can prevent the
state-of-the-art face detectors from detecting the real faces. We investigate a
phenomenon: patches designed to suppress real face detection appear face-like.
This phenomenon holds generally across different initialization, locations,
scales of patches, backbones, and state-of-the-art face detection frameworks.
We propose new optimization-based approaches to automatic design of universal
adversarial patches for varying goals of the attack, including scenarios in
which true positives are suppressed without introducing false positives. Our
proposed algorithms perform well on real-world datasets, deceiving
state-of-the-art face detectors in terms of multiple precision/recall metrics
and transferring between different detection frameworks.


http://arxiv.org/abs/1912.00181
Error-Correcting Neural Network.
Yang Song; Qiyu Kang; Wee Peng Tay
  Error-correcting output codes (ECOC) is an ensemble method combining a set of
binary classifiers for multi-class learning problems. However, in traditional
ECOC framework, the binary classifiers are trained independently. To explore
the interaction between the binary classifiers, we construct an error
correction network (ECN) that jointly trains all binary classifiers while
maximizing the ensemble diversity to improve its robustness against adversarial
attacks. An ECN is built based on a code matrix which is generated by
maximizing the error tolerance, i.e., the minimum Hamming distance between any
two rows, as well as the ensemble diversity, i.e., the variation of information
between any two columns. Though ECN inherently promotes the diversity between
the binary classifiers as each ensemble member solves a different
classification problem (specified by the corresponding column of the code
matrix), we empirically show that the ensemble diversity can be further
improved by forcing the weight matrices learned by ensemble members to be
orthogonal. The ECN is trained in end-to-end fashion and can be complementary
to other defense approaches including adversarial training. We show empirically
that ECN is effective against the state-of-the-art while-box attacks while
maintaining good accuracy on normal examples.


http://arxiv.org/abs/1912.00049
Square Attack: a query-efficient black-box adversarial attack via random search.
Maksym Andriushchenko; Francesco Croce; Nicolas Flammarion; Matthias Hein
  We propose the Square Attack, a new score-based black-box $l_2$ and
$l_\infty$ adversarial attack that does not rely on local gradient information
and thus is not affected by gradient masking. The Square Attack is based on a
randomized search scheme where we select localized square-shaped updates at
random positions so that the $l_\infty$- or $l_2$-norm of the perturbation is
approximately equal to the maximal budget at each step. Our method is
algorithmically transparent, robust to the choice of hyperparameters, and is
significantly more query efficient compared to the more complex
state-of-the-art methods. In particular, on ImageNet we improve the average
query efficiency for various deep networks by a factor of at least $2$ and up
to $7$ compared to the recent state-of-the-art $l_\infty$-attack of Meunier et
al. while having a higher success rate. The Square Attack can even be
competitive to gradient-based white-box attacks in terms of success rate.
Moreover, we show its utility by breaking a recently proposed defense based on
randomization. The code of our attack is available at
https://github.com/max-andr/square-attack


http://arxiv.org/abs/1911.12562
Towards Privacy and Security of Deep Learning Systems: A Survey.
Yingzhe He; Guozhu Meng; Kai Chen; Xingbo Hu; Jinwen He
  Deep learning has gained tremendous success and great popularity in the past
few years. However, recent research found that it is suffering several inherent
weaknesses, which can threaten the security and privacy of the stackholders.
Deep learning's wide use further magnifies the caused consequences. To this
end, lots of research has been conducted with the purpose of exhaustively
identifying intrinsic weaknesses and subsequently proposing feasible
mitigation. Yet few is clear about how these weaknesses are incurred and how
effective are these attack approaches in assaulting deep learning. In order to
unveil the security weaknesses and aid in the development of a robust deep
learning system, we are devoted to undertaking a comprehensive investigation on
attacks towards deep learning, and extensively evaluating these attacks in
multiple views. In particular, we focus on four types of attacks associated
with security and privacy of deep learning: model extraction attack, model
inversion attack, poisoning attack and adversarial attack. For each type of
attack, we construct its essential workflow as well as adversary capabilities
and attack goals. Many pivot metrics are devised for evaluating the attack
approaches, by which we perform a quantitative and qualitative analysis. From
the analysis, we have identified significant and indispensable factors in an
attack vector, \eg, how to reduce queries to target models, what distance used
for measuring perturbation. We spot light on 17 findings covering these
approaches' merits and demerits, success probability, deployment complexity and
prospects. Moreover, we discuss other potential security weaknesses and
possible mitigation which can inspire relevant researchers in this area.


http://arxiv.org/abs/1911.11932
Survey of Attacks and Defenses on Edge-Deployed Neural Networks.
Mihailo Isakov; Vijay Gadepally; Karen M. Gettings; Michel A. Kinsy
  Deep Neural Network (DNN) workloads are quickly moving from datacenters onto
edge devices, for latency, privacy, or energy reasons. While datacenter
networks can be protected using conventional cybersecurity measures, edge
neural networks bring a host of new security challenges. Unlike classic IoT
applications, edge neural networks are typically very compute and memory
intensive, their execution is data-independent, and they are robust to noise
and faults. Neural network models may be very expensive to develop, and can
potentially reveal information about the private data they were trained on,
requiring special care in distribution. The hidden states and outputs of the
network can also be used in reconstructing user inputs, potentially violating
users' privacy. Furthermore, neural networks are vulnerable to adversarial
attacks, which may cause misclassifications and violate the integrity of the
output. These properties add challenges when securing edge-deployed DNNs,
requiring new considerations, threat models, priorities, and approaches in
securely and privately deploying DNNs to the edge. In this work, we cover the
landscape of attacks on, and defenses, of neural networks deployed in edge
devices and provide a taxonomy of attacks and defenses targeting edge DNNs.


http://arxiv.org/abs/1911.11881
An Adaptive View of Adversarial Robustness from Test-time Smoothing Defense.
Chao Tang; Yifei Fan; Anthony Yezzi
  The safety and robustness of learning-based decision-making systems are under
threats from adversarial examples, as imperceptible perturbations can mislead
neural networks to completely different outputs. In this paper, we present an
adaptive view of the issue via evaluating various test-time smoothing defense
against white-box untargeted adversarial examples. Through controlled
experiments with pretrained ResNet-152 on ImageNet, we first illustrate the
non-monotonic relation between adversarial attacks and smoothing defenses. Then
at the dataset level, we observe large variance among samples and show that it
is easy to inflate accuracy (even to 100%) or build large-scale (i.e., with
size ~10^4) subsets on which a designated method outperforms others by a large
margin. Finally at the sample level, as different adversarial examples require
different degrees of defense, the potential advantages of iterative methods are
also discussed. We hope this paper reveal useful behaviors of test-time
defenses, which could help improve the evaluation process for adversarial
robustness in the future.


http://arxiv.org/abs/1911.11946
Can Attention Masks Improve Adversarial Robustness?
Pratik Vaishnavi; Tianji Cong; Kevin Eykholt; Atul Prakash; Amir Rahmati
  Deep Neural Networks (DNNs) are known to be susceptible to adversarial
examples. Adversarial examples are maliciously crafted inputs that are designed
to fool a model, but appear normal to human beings. Recent work has shown that
pixel discretization can be used to make classifiers for MNIST highly robust to
adversarial examples. However, pixel discretization fails to provide
significant protection on more complex datasets. In this paper, we take the
first step towards reconciling these contrary findings. Focusing on the
observation that discrete pixelization in MNIST makes the background completely
black and foreground completely white, we hypothesize that the important
property for increasing robustness is the elimination of image background using
attention masks before classifying an object. To examine this hypothesis, we
create foreground attention masks for two different datasets, GTSRB and
MS-COCO. Our initial results suggest that using attention mask leads to
improved robustness. On the adversarially trained classifiers, we see an
adversarial robustness increase of over 20% on MS-COCO.


http://arxiv.org/abs/1911.11746
Defending Against Adversarial Machine Learning.
Alison Jenkins
  An Adversarial System to attack and an Authorship Attribution System (AAS) to
defend itself against the attacks are analyzed. Defending a system against
attacks from an adversarial machine learner can be done by randomly switching
between models for the system, by detecting and reacting to changes in the
distribution of normal inputs, or by using other methods. Adversarial machine
learning is used to identify a system that is being used to map system inputs
to outputs. Three types of machine learners are using for the model that is
being attacked. The machine learners that are used to model the system being
attacked are a Radial Basis Function Support Vector Machine, a Linear Support
Vector Machine, and a Feedforward Neural Network. The feature masks are evolved
using accuracy as the fitness measure. The system defends itself against
adversarial machine learning attacks by identifying inputs that do not match
the probability distribution of normal inputs. The system also defends itself
against adversarial attacks by randomly switching between the feature masks
being used to map system inputs to outputs.


http://arxiv.org/abs/1911.11484
Using Depth for Pixel-Wise Detection of Adversarial Attacks in Crowd Counting.
Weizhe Liu; Mathieu Salzmann; Pascal Fua
  State-of-the-art methods for counting people in crowded scenes rely on deep
networks to estimate crowd density. While effective, deep learning approaches
are vulnerable to adversarial attacks, which, in a crowd-counting context, can
lead to serious security issues. However, attack and defense mechanisms have
been virtually unexplored in regression tasks, let alone for crowd density
estimation.
  In this paper, we investigate the effectiveness of existing attack strategies
on crowd-counting networks, and introduce a simple yet effective pixel-wise
detection mechanism. It builds on the intuition that, when attacking a
multitask network, in our case estimating crowd density and scene depth, both
outputs will be perturbed, and thus the second one can be used for detection
purposes. We will demonstrate that this significantly outperforms heuristic and
uncertainty-based strategies.


http://arxiv.org/abs/1911.11253
Playing it Safe: Adversarial Robustness with an Abstain Option.
Cassidy Laidlaw; Soheil Feizi
  We explore adversarial robustness in the setting in which it is acceptable
for a classifier to abstain---that is, output no class---on adversarial
examples. Adversarial examples are small perturbations of normal inputs to a
classifier that cause the classifier to give incorrect output; they present
security and safety challenges for machine learning systems. In many
safety-critical applications, it is less costly for a classifier to abstain on
adversarial examples than to give incorrect output for them. We first introduce
a novel objective function for adversarial robustness with an abstain option
which characterizes an explicit tradeoff between robustness and accuracy. We
then present a simple baseline in which an adversarially-trained classifier
abstains on all inputs within a certain distance of the decision boundary,
which we theoretically and experimentally evaluate. Finally, we propose
Combined Abstention Robustness Learning (CARL), a method for jointly learning a
classifier and the region of the input space on which it should abstain. We
explore different variations of the PGD and DeepFool adversarial attacks on
CARL in the abstain setting. Evaluating against these attacks, we demonstrate
that training with CARL results in a more accurate, robust, and efficient
classifier than the baseline.


http://arxiv.org/abs/1911.10891
ColorFool: Semantic Adversarial Colorization.
Ali Shahin Shamsabadi; Ricardo Sanchez-Matilla; Andrea Cavallaro
  Adversarial attacks that generate small L_p-norm perturbations to mislead
classifiers have limited success in black-box settings and with unseen
classifiers. These attacks are also fragile with defenses that use denoising
filters and to adversarial training procedures. Instead, adversarial attacks
that generate unrestricted perturbations are more robust to defenses, are
generally more successful in black-box settings and are more transferable to
unseen classifiers. However, unrestricted perturbations may be noticeable to
humans. In this paper, we propose a content-based black-box adversarial attack
that generates unrestricted perturbations by exploiting image semantics to
selectively modify colors within chosen ranges that are perceived as natural by
humans. We show that the proposed approach, ColorFool, outperforms in terms of
success rate, robustness to defense frameworks and transferability five
state-of-the-art adversarial attacks on two different tasks, scene and object
classification, when attacking three state-of-the-art deep neural networks
using three standard datasets. We will make the code of the proposed approach
and the whole evaluation framework publicly available.


http://arxiv.org/abs/1911.10875
Adversarial Attack with Pattern Replacement.
Ziang Dong; Liang Mao; Shiliang Sun
  We propose a generative model for adversarial attack. The model generates
subtle but predictive patterns from the input. To perform an attack, it
replaces the patterns of the input with those generated based on examples from
some other class. We demonstrate our model by attacking CNN on MNIST.


http://arxiv.org/abs/1911.11219
One Man's Trash is Another Man's Treasure: Resisting Adversarial Examples by Adversarial Examples.
Chang Xiao; Changxi Zheng
  Modern image classification systems are often built on deep neural networks,
which suffer from adversarial examples--images with deliberately crafted,
imperceptible noise to mislead the network's classification. To defend against
adversarial examples, a plausible idea is to obfuscate the network's gradient
with respect to the input image. This general idea has inspired a long line of
defense methods. Yet, almost all of them have proven vulnerable. We revisit
this seemingly flawed idea from a radically different perspective. We embrace
the omnipresence of adversarial examples and the numerical procedure of
crafting them, and turn this harmful attacking process into a useful defense
mechanism. Our defense method is conceptually simple: before feeding an input
image for classification, transform it by finding an adversarial example on a
pre-trained external model. We evaluate our method against a wide range of
possible attacks. On both CIFAR-10 and Tiny ImageNet datasets, our method is
significantly more robust than state-of-the-art methods. Particularly, in
comparison to adversarial training, our method offers lower training cost as
well as stronger robustness.


http://arxiv.org/abs/1911.10695
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks.
Minghao Guo; Yuzhe Yang; Rui Xu; Ziwei Liu; Dahua Lin
  Recent advances in adversarial attacks uncover the intrinsic vulnerability of
modern deep neural networks. Since then, extensive efforts have been devoted to
enhancing the robustness of deep networks via specialized learning algorithms
and loss functions. In this work, we take an architectural perspective and
investigate the patterns of network architectures that are resilient to
adversarial attacks. To obtain the large number of networks needed for this
study, we adopt one-shot neural architecture search, training a large network
for once and then finetuning the sub-networks sampled therefrom. The sampled
architectures together with the accuracies they achieve provide a rich basis
for our study. Our "robust architecture Odyssey" reveals several valuable
observations: 1) densely connected patterns result in improved robustness; 2)
under computational budget, adding convolution operations to direct connection
edge is effective; 3) flow of solution procedure (FSP) matrix is a good
indicator of network robustness. Based on these observations, we discover a
family of robust architectures (RobNets). On various datasets, including CIFAR,
SVHN, Tiny-ImageNet, and ImageNet, RobNets exhibit superior robustness
performance to other widely used architectures. Notably, RobNets substantially
improve the robust accuracy (~5% absolute gains) under both white-box and
black-box attacks, even with fewer parameter numbers. Code is available at
https://github.com/gmh14/RobNets.


http://arxiv.org/abs/1911.10561
Time-aware Gradient Attack on Dynamic Network Link Prediction.
Jinyin Chen; Jian Zhang; Zhi Chen; Min Du; Feifei Li; Qi Xuan
  In network link prediction, it is possible to hide a target link from being
predicted with a small perturbation on network structure. This observation may
be exploited in many real world scenarios, for example, to preserve privacy, or
to exploit financial security. There have been many recent studies to generate
adversarial examples to mislead deep learning models on graph data. However,
none of the previous work has considered the dynamic nature of real-world
systems. In this work, we present the first study of adversarial attack on
dynamic network link prediction (DNLP). The proposed attack method, namely
time-aware gradient attack (TGA), utilizes the gradient information generated
by deep dynamic network embedding (DDNE) across different snapshots to rewire a
few links, so as to make DDNE fail to predict target links. We implement TGA in
two ways: one is based on traversal search, namely TGA-Tra; and the other is
simplified with greedy search for efficiency, namely TGA-Gre. We conduct
comprehensive experiments which show the outstanding performance of TGA in
attacking DNLP algorithms.


http://arxiv.org/abs/1911.10364
Universal Adversarial Perturbations to Understand Robustness of Texture vs. Shape-biased Training.
Kenneth T. Co; Luis Muñoz-González; Leslie Kanthan; Ben Glocker; Emil C. Lupu
  Convolutional Neural Networks (CNNs) used on image classification tasks such
as ImageNet have been shown to be biased towards recognizing textures rather
than shapes. Recent work has attempted to alleviate this by augmenting the
training dataset with shape-based examples to create Stylized-ImageNet.
However, in this paper we show that models trained on this modified dataset
remain as vulnerable to Universal Adversarial Perturbations (UAPs) as those
trained in ImageNet. We use UAPs to evaluate, compare, and understand the
robustness of CNN models with varying degrees of shape-based training. We also
find that a posteriori fine-tuning on ImageNet negates features learned from
training on Stylized-ImageNet. This study reveals an important current
limitation and highlights the need for further research into robustness of CNNs
for visual recognition.


http://arxiv.org/abs/1911.10435
Robust Assessment of Real-World Adversarial Examples.
Brett Jefferson; Carlos Ortiz Marrero
  We explore rigorous, systematic, and controlled experimental evaluation of
adversarial examples in the real world and propose a testing regimen for
evaluation of real world adversarial objects. We show that for small scene/
environmental perturbations, large adversarial performance differences exist.
Current state of adversarial reporting exists largely as a frequency count over
a dynamic collections of scenes. Our work underscores the need for either a
more complete report or a score that incorporates scene changes and baseline
performance for models and environments tested by adversarial developers. We
put forth a score that attempts to address the above issues in a
straight-forward exemplar application for multiple generated adversary
examples. We contribute the following: 1. a testbed for adversarial assessment,
2. a score for adversarial examples, and 3. a collection of additional
evaluations on testbed data.


http://arxiv.org/abs/1911.10258
Bounding Singular Values of Convolution Layers.
Sahil Singla; Soheil Feizi
  In deep neural networks, the spectral norm of the Jacobian of a layer bounds
the factor by which the norm of a signal changes during forward or backward
propagation. Spectral norm regularization has also been shown to improve the
generalization and robustness of deep networks. However, existing methods to
compute the spectral norm of the jacobian of convolution layers either rely on
heuristics (but are efficient in computation) or are exact (but computationally
expensive to be used during training). In this work, we resolve these issues by
deriving an upper bound on the spectral norm of a standard 2D multi-channel
convolution layer. Our method provides a provable bound that is differentiable
and can be computed efficiently during training with negligible overhead. We
show that our spectral bound is an effective regularizer and can be used to
bound the lipschitz constant and the curvature (eigenvalues of the Hessian) of
neural network. Through experiments on MNIST and CIFAR-10, we demonstrate the
effectiveness of our spectral bound in improving the generalization and
provable robustness of deep networks against adversarial examples. Our code is
available at \url{https://github.com/singlasahil14/CONV-SV}.


http://arxiv.org/abs/1911.11616
Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reduction.
Yantao Lu; Yunhan Jia; Jianyu Wang; Bai Li; Weiheng Chai; Lawrence Carin; Senem Velipasalar
  Neural networks are known to be vulnerable to carefully crafted adversarial
examples, and these malicious samples often transfer, i.e., they remain
adversarial even against other models. Although great efforts have been delved
into the transferability across models, surprisingly, less attention has been
paid to the cross-task transferability, which represents the real-world
cybercriminal's situation, where an ensemble of different defense/detection
mechanisms need to be evaded all at once. In this paper, we investigate the
transferability of adversarial examples across a wide range of real-world
computer vision tasks, including image classification, object detection,
semantic segmentation, explicit content detection, and text detection. Our
proposed attack minimizes the ``dispersion'' of the internal feature map, which
overcomes existing attacks' limitation of requiring task-specific loss
functions and/or probing a target model. We conduct evaluation on open source
detection and segmentation models as well as four different computer vision
tasks provided by Google Cloud Vision (GCV) APIs, to show how our approach
outperforms existing attacks by degrading performance of multiple CV tasks by a
large margin with only modest perturbations linf=16.


http://arxiv.org/abs/1911.10008
Attack Agnostic Statistical Method for Adversarial Detection.
Sambuddha Saha; Aashish Kumar; Pratyush Sahay; George Jose; Srinivas Kruthiventi; Harikrishna Muralidhara
  Deep Learning based AI systems have shown great promise in various domains
such as vision, audio, autonomous systems (vehicles, drones), etc. Recent
research on neural networks has shown the susceptibility of deep networks to
adversarial attacks - a technique of adding small perturbations to the inputs
which can fool a deep network into misclassifying them. Developing defenses
against such adversarial attacks is an active research area, with some
approaches proposing robust models that are immune to such adversaries, while
other techniques attempt to detect such adversarial inputs. In this paper, we
present a novel statistical approach for adversarial detection in image
classification. Our approach is based on constructing a per-class feature
distribution and detecting adversaries based on comparison of features of a
test image with the feature distribution of its class. For this purpose, we
make use of various statistical distances such as ED (Energy Distance), MMD
(Maximum Mean Discrepancy) for adversarial detection, and analyze the
performance of each metric. We experimentally show that our approach achieves
good adversarial detection performance on MNIST and CIFAR-10 datasets
irrespective of the attack method, sample size and the degree of adversarial
perturbation.


http://arxiv.org/abs/1911.10182
Universal adversarial examples in speech command classification.
Jon Vadillo; Roberto Santana
  Adversarial examples are inputs intentionally perturbed with the aim of
forcing a machine learning model to produce a wrong prediction, while the
changes are not easily detectable by a human. Although this topic has been
intensively studied in the image domain, classification tasks in the audio
domain have received less attention. In this paper we address the existence of
universal perturbations for speech command classification. We provide evidence
that universal attacks can be generated for speech command classification
tasks, which are able to generalize across different models to a significant
extent. Additionally, a novel analytical framework is proposed for the
evaluation of universal perturbations under different levels of universality,
demonstrating that the feasibility of generating effective perturbations
decreases as the universality level increases. Finally, we propose a more
detailed and rigorous framework to measure the amount of distortion introduced
by the perturbations, demonstrating that the methods employed by convention are
not realistic in audio-based problems.


http://arxiv.org/abs/1911.10291
Invert and Defend: Model-based Approximate Inversion of Generative Adversarial Networks for Secure Inference.
Wei-An Lin; Yogesh Balaji; Pouya Samangouei; Rama Chellappa
  Inferring the latent variable generating a given test sample is a challenging
problem in Generative Adversarial Networks (GANs). In this paper, we propose
InvGAN - a novel framework for solving the inference problem in GANs, which
involves training an encoder network capable of inverting a pre-trained
generator network without access to any training data. Under mild assumptions,
we theoretically show that using InvGAN, we can approximately invert the
generations of any latent code of a trained GAN model. Furthermore, we
empirically demonstrate the superiority of our inference scheme by quantitative
and qualitative comparisons with other methods that perform a similar task. We
also show the effectiveness of our framework in the problem of adversarial
defenses where InvGAN can successfully be used as a projection-based defense
mechanism. Additionally, we show how InvGAN can be used to implement
reparameterization white-box attacks on projection-based defense mechanisms.
Experimental validation on several benchmark datasets demonstrate the efficacy
of our method in achieving improved performance on several white-box and
black-box attacks. Our code is available at
https://github.com/yogeshbalaji/InvGAN.


http://arxiv.org/abs/1911.09449
Heuristic Black-box Adversarial Attacks on Video Recognition Models.
Zhipeng Wei; Jingjing Chen; Xingxing Wei; Linxi Jiang; Tat-Seng Chua; Fengfeng Zhou; Yu-Gang Jiang
  We study the problem of attacking video recognition models in the black-box
setting, where the model information is unknown and the adversary can only make
queries to detect the predicted top-1 class and its probability. Compared with
the black-box attack on images, attacking videos is more challenging as the
computation cost for searching the adversarial perturbations on a video is much
higher due to its high dimensionality. To overcome this challenge, we propose a
heuristic black-box attack model that generates adversarial perturbations only
on the selected frames and regions. More specifically, a heuristic-based
algorithm is proposed to measure the importance of each frame in the video
towards generating the adversarial examples. Based on the frames' importance,
the proposed algorithm heuristically searches a subset of frames where the
generated adversarial example has strong adversarial attack ability while keeps
the perturbations lower than the given bound. Besides, to further boost the
attack efficiency, we propose to generate the perturbations only on the salient
regions of the selected frames. In this way, the generated perturbations are
sparse in both temporal and spatial domains. Experimental results of attacking
two mainstream video recognition methods on the UCF-101 dataset and the HMDB-51
dataset demonstrate that the proposed heuristic black-box adversarial attack
method can significantly reduce the computation cost and lead to more than 28\%
reduction in query numbers for the untargeted attack on both datasets.


http://arxiv.org/abs/1911.09665
Adversarial Examples Improve Image Recognition.
Cihang Xie; Mingxing Tan; Boqing Gong; Jiang Wang; Alan Yuille; Quoc V. Le
  Adversarial examples are commonly viewed as a threat to ConvNets. Here we
present an opposite perspective: adversarial examples can be used to improve
image recognition models if harnessed in the right manner. We propose AdvProp,
an enhanced adversarial training scheme which treats adversarial examples as
additional examples, to prevent overfitting. Key to our method is the usage of
a separate auxiliary batch norm for adversarial examples, as they have
different underlying distributions to normal examples.
  We show that AdvProp improves a wide range of models on various image
recognition tasks and performs better when the models are bigger. For instance,
by applying AdvProp to the latest EfficientNet-B7 [28] on ImageNet, we achieve
significant improvements on ImageNet (+0.7%), ImageNet-C (+6.5%), ImageNet-A
(+7.0%), Stylized-ImageNet (+4.8%). With an enhanced EfficientNet-B8, our
method achieves the state-of-the-art 85.5% ImageNet top-1 accuracy without
extra data. This result even surpasses the best model in [20] which is trained
with 3.5B Instagram images (~3000X more than ImageNet) and ~9.4X more
parameters. Models are available at
https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.


http://arxiv.org/abs/1911.09272
Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablation.
Alexander Levine; Soheil Feizi
  Recently, techniques have been developed to provably guarantee the robustness
of a classifier to adversarial perturbations of bounded L_1 and L_2 magnitudes
by using randomized smoothing: the robust classification is a consensus of base
classifications on randomly noised samples where the noise is additive. In this
paper, we extend this technique to the L_0 threat model. We propose an
efficient and certifiably robust defense against sparse adversarial attacks by
randomly ablating input features, rather than using additive noise.
Experimentally, on MNIST, we can certify the classifications of over 50% of
images to be robust to any distortion of at most 8 pixels. This is comparable
to the observed empirical robustness of unprotected classifiers on MNIST to
modern L_0 attacks, demonstrating the tightness of the proposed robustness
certificate. We also evaluate our certificate on ImageNet and CIFAR-10. Our
certificates represent an improvement on those provided in a concurrent work
(Lee et al. 2019) which uses random noise rather than ablation (median
certificates of 8 pixels versus 4 pixels on MNIST; 16 pixels versus 1 pixel on
ImageNet.) Additionally, we empirically demonstrate that our classifier is
highly robust to modern sparse adversarial attacks on MNIST. Our
classifications are robust, in median, to adversarial perturbations of up to 31
pixels, compared to 22 pixels reported as the state-of-the-art defense, at the
cost of a slight decrease (around 2.3%) in the classification accuracy. Code is
available at https://github.com/alevine0/randomizedAblation/.


http://arxiv.org/abs/1911.08790
Analysis of Deep Networks for Monocular Depth Estimation Through Adversarial Attacks with Proposal of a Defense Method.
Junjie Hu; Takayuki Okatani
  In this paper, we consider adversarial attacks against a system of monocular
depth estimation (MDE) based on convolutional neural networks (CNNs). The
motivation is two-fold. One is to study the security of MDE systems, which has
not been actively considered in the community. The other is to improve our
understanding of the computational mechanism of CNNs performing MDE. Toward
this end, we apply the method recently proposed for visualization of MDE to
defending attacks. It trains another CNN to predict a saliency map from an
input image, such that the CNN for MDE continues to accurately estimate the
depth map from the image with its non-salient part masked out. We report the
following findings. First, unsurprisingly, attacks by IFGSM (or equivalently
PGD) succeed in making the CNNs yield inaccurate depth estimates. Second, the
attacks can be defended by masking out non-salient pixels, indicating that the
attacks function by perturbing mostly non-salient pixels. However, the
prediction of saliency maps is itself vulnerable to the attacks, even though it
is not the direct target of the attacks. We show that the attacks can be
defended by using a saliency map predicted by a CNN trained to be robust to the
attacks. These results provide an effective defense method as well as a clue to
understanding the computational mechanism of CNNs for MDE.


http://arxiv.org/abs/1911.09058
Fine-grained Synthesis of Unrestricted Adversarial Examples.
Omid Poursaeed; Tianxing Jiang; Harry Yang; Serge Belongie; Ser-Nam Lim
  We propose a novel approach for generating unrestricted adversarial examples
by manipulating fine-grained aspects of image generation. Unlike existing
unrestricted attacks that typically hand-craft geometric transformations, we
learn stylistic and stochastic modifications leveraging state-of-the-art
generative models. This allows us to manipulate an image in a controlled,
fine-grained manner without being bounded by a norm threshold. Our model can be
used for both targeted and non-targeted unrestricted attacks. We demonstrate
that our attacks can bypass certified defenses, yet our adversarial images look
indistinguishable from natural images as verified by human evaluation.
Adversarial training can be used as an effective defense without degrading
performance of the model on clean images. We perform experiments on LSUN and
CelebA-HQ as high resolution datasets to validate efficacy of our proposed
approach.


http://arxiv.org/abs/1911.08723
Deep Minimax Probability Machine.
Lirong He; Ziyi Guo; Kaizhu Huang; Zenglin Xu
  Deep neural networks enjoy a powerful representation and have proven
effective in a number of applications. However, recent advances show that deep
neural networks are vulnerable to adversarial attacks incurred by the so-called
adversarial examples. Although the adversarial example is only slightly
different from the input sample, the neural network classifies it as the wrong
class. In order to alleviate this problem, we propose the Deep Minimax
Probability Machine (DeepMPM), which applies MPM to deep neural networks in an
end-to-end fashion. In a worst-case scenario, MPM tries to minimize an upper
bound of misclassification probabilities, considering the global information
(i.e., mean and covariance information of each class). DeepMPM can be more
robust since it learns the worst-case bound on the probability of
misclassification of future data. Experiments on two real-world datasets can
achieve comparable classification performance with CNN, while can be more
robust on adversarial attacks.


http://arxiv.org/abs/1911.08635
Logic-inspired Deep Neural Networks.
Minh Le
  Deep neural networks have achieved impressive performance and become de-facto
standard in many tasks. However, phenomena such as adversarial examples and
fooling examples hint that the generalization they make is flawed. We argue
that the problem roots in their distributed and connected nature and propose
remedies inspired by propositional logic. Our experiments show that the
proposed models are more local and better at resisting fooling and adversarial
examples. By means of an ablation analysis, we reveal insights into adversarial
examples and suggest a new hypothesis on their origins.


http://arxiv.org/abs/1911.08696
Where is the Bottleneck of Adversarial Learning with Unlabeled Data?
Jingfeng Zhang; Bo Han; Gang Niu; Tongliang Liu; Masashi Sugiyama
  Deep neural networks (DNNs) are incredibly brittle due to adversarial
examples. To robustify DNNs, adversarial training was proposed, which requires
large-scale but well-labeled data. However, it is quite expensive to annotate
large-scale data well. To compensate for this shortage, several seminal works
are utilizing large-scale unlabeled data. In this paper, we observe that
seminal works do not perform well, since the quality of pseudo labels on
unlabeled data is quite poor, especially when the amount of unlabeled data is
significantly larger than that of labeled data. We believe that the quality of
pseudo labels is the bottleneck of adversarial learning with unlabeled data. To
tackle this bottleneck, we leverage deep co-training, which trains two deep
networks and encourages two networks diverged by exploiting peer's adversarial
examples. Based on deep co-training, we propose robust co-training (RCT) for
adversarial learning with unlabeled data. We conduct comprehensive experiments
on CIFAR-10 and SVHN datasets. Empirical results demonstrate that our RCT can
significantly outperform baselines (e.g., robust self-training (RST)) in both
standard test accuracy and robust test accuracy w.r.t. different datasets,
different network structures, and different types of adversarial training.


http://arxiv.org/abs/1911.08654
Adversarial Robustness of Flow-Based Generative Models.
Phillip Pope; Yogesh Balaji; Soheil Feizi
  Flow-based generative models leverage invertible generator functions to fit a
distribution to the training data using maximum likelihood. Despite their use
in several application domains, robustness of these models to adversarial
attacks has hardly been explored. In this paper, we study adversarial
robustness of flow-based generative models both theoretically (for some simple
models) and empirically (for more complex ones). First, we consider a linear
flow-based generative model and compute optimal sample-specific and universal
adversarial perturbations that maximally decrease the likelihood scores. Using
this result, we study the robustness of the well-known adversarial training
procedure, where we characterize the fundamental trade-off between model
robustness and accuracy. Next, we empirically study the robustness of two
prominent deep, non-linear, flow-based generative models, namely GLOW and
RealNVP. We design two types of adversarial attacks; one that minimizes the
likelihood scores of in-distribution samples, while the other that maximizes
the likelihood scores of out-of-distribution ones. We find that GLOW and
RealNVP are extremely sensitive to both types of attacks. Finally, using a
hybrid adversarial training procedure, we significantly boost the robustness of
these generative models.


http://arxiv.org/abs/1911.08432
Defective Convolutional Layers Learn Robust CNNs.
Tiange Luo; Tianle Cai; Mengxiao Zhang; Siyu Chen; Di He; Liwei Wang
  Robustness of convolutional neural networks has recently been highlighted by
the adversarial examples, i.e., inputs added with well-designed perturbations
which are imperceptible to humans but can cause the network to give incorrect
outputs. Recent research suggests that the noises in adversarial examples break
the textural structure, which eventually leads to wrong predictions by
convolutional neural networks. To help a convolutional neural network make
predictions relying less on textural information, we propose defective
convolutional layers which contain defective neurons whose activations are set
to be a constant function. As the defective neurons contain no information and
are far different from the standard neurons in its spatial neighborhood, the
textural features cannot be accurately extracted and the model has to seek for
other features for classification, such as the shape. We first show that
predictions made by the defective CNN are less dependent on textural
information, but more on shape information, and further find that adversarial
examples generated by the defective CNN appear to have semantic shapes.
Experimental results demonstrate the defective CNN has higher defense ability
than the standard CNN against various types of attack. In particular, it
achieves state-of-the-art performance against transfer-based attacks without
applying any adversarial training.


http://arxiv.org/abs/1911.08644
Generate (non-software) Bugs to Fool Classifiers.
Hiromu Yakura; Youhei Akimoto; Jun Sakuma
  In adversarial attacks intended to confound deep learning models, most
studies have focused on limiting the magnitude of the modification so that
humans do not notice the attack. On the other hand, during an attack against
autonomous cars, for example, most drivers would not find it strange if a small
insect image were placed on a stop sign, or they may overlook it. In this
paper, we present a systematic approach to generate natural adversarial
examples against classification models by employing such natural-appearing
perturbations that imitate a certain object or signal. We first show the
feasibility of this approach in an attack against an image classifier by
employing generative adversarial networks that produce image patches that have
the appearance of a natural object to fool the target model. We also introduce
an algorithm to optimize placement of the perturbation in accordance with the
input image, which makes the generation of adversarial examples fast and likely
to succeed. Moreover, we experimentally show that the proposed approach can be
extended to the audio domain, for example, to generate perturbations that sound
like the chirping of birds to fool a speech classifier.


http://arxiv.org/abs/1911.07682
A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories.
Zhaohui Che; Ali Borji; Guangtao Zhai; Suiyi Ling; Jing Li; Patrick Le Callet
  Deep neural networks are vulnerable to adversarial attacks.


http://arxiv.org/abs/1911.08053
A novel method for identifying the deep neural network model with the Serial Number.
XiangRui Xu; YaQin Li; Cao Yuan
  Deep neural network (DNN) with the state of art performance has emerged as a
viable and lucrative business service. However, those impressive performances
require a large number of computational resources, which comes at a high cost
for the model creators. The necessity for protecting DNN models from illegal
reproducing and distribution appears salient now. Recently, trigger-set
watermarking, breaking the white-box restriction, relying on adversarial
training pre-defined (incorrect) labels for crafted inputs, and subsequently
using them to verify the model authenticity, has been the main topic of DNN
ownership verification. While these methods have successfully demonstrated
robustness against removal attacks, few are effective against the tampering
attacks from competitors forging the fake watermarks and dogging in the
manager. In this paper, we put forth a new framework of the trigger-set
watermark by embedding a unique Serial Number (relatedness less original
labels) to the deep neural network for model ownership identification, which is
both robust to model pruning and resist to tampering attacks. Experiment
results demonstrate that the DNN Serial Number only incurs slight accuracy
degradation of the original performance and is valid for ownership
verification.


http://arxiv.org/abs/1911.08011
Adversarial Attacks on Grid Events Classification: An Adversarial Machine Learning Approach.
Iman Niazazari; Hanif Livani
  With the ever-increasing reliance on data for data-driven applications in
power grids, such as event cause analysis, the authenticity of data streams has
become crucially important. The data can be prone to adversarial stealthy
attacks aiming to manipulate the data such that residual-based bad data
detectors cannot detect them, and the perception of system operators or event
classifiers changes about the actual event. This paper investigates the impact
of adversarial attacks on convolutional neural network-based event cause
analysis frameworks. We have successfully verified the ability of adversaries
to maliciously misclassify events through stealthy data manipulations. The
vulnerability assessment is studied with respect to the number of compromised
measurements. Furthermore, a defense mechanism to robustify the performance of
the event cause analysis is proposed. The effectiveness of adversarial attacks
on changing the output of the framework is studied using the data generated by
real-time digital simulator (RTDS) under different scenarios such as type of
attacks and level of access to data.


http://arxiv.org/abs/1911.07989
WITCHcraft: Efficient PGD attacks with random step size.
Ping-Yeh Chiang; Jonas Geiping; Micah Goldblum; Tom Goldstein; Renkun Ni; Steven Reich; Ali Shafahi
  State-of-the-art adversarial attacks on neural networks use expensive
iterative methods and numerous random restarts from different initial points.
Iterative FGSM-based methods without restarts trade off performance for
computational efficiency because they do not adequately explore the image space
and are highly sensitive to the choice of step size. We propose a variant of
Projected Gradient Descent (PGD) that uses a random step size to improve
performance without resorting to expensive random restarts. Our method, Wide
Iterative Stochastic crafting (WITCHcraft), achieves results superior to the
classical PGD attack on the CIFAR-10 and MNIST data sets but without additional
computational cost. This simple modification of PGD makes crafting attacks more
economical, which is important in situations like adversarial training where
attacks need to be crafted in real time.


http://arxiv.org/abs/1911.08090
Deep Detector Health Management under Adversarial Campaigns.
Javier Echauz; Keith Kenemer; Sarfaraz Hussein; Jay Dhaliwal; Saurabh Shintre; Slawomir Grzonkowski; Andrew Gardner
  Machine learning models are vulnerable to adversarial inputs that induce
seemingly unjustifiable errors. As automated classifiers are increasingly used
in industrial control systems and machinery, these adversarial errors could
grow to be a serious problem. Despite numerous studies over the past few years,
the field of adversarial ML is still considered alchemy, with no practical
unbroken defenses demonstrated to date, leaving PHM practitioners with few
meaningful ways of addressing the problem. We introduce turbidity detection as
a practical superset of the adversarial input detection problem, coping with
adversarial campaigns rather than statistically invisible one-offs. This
perspective is coupled with ROC-theoretic design guidance that prescribes an
inexpensive domain adaptation layer at the output of a deep learning model
during an attack campaign. The result aims to approximate the Bayes optimal
mitigation that ameliorates the detection model's degraded health. A
proactively reactive type of prognostics is achieved via Monte Carlo simulation
of various adversarial campaign scenarios, by sampling from the model's own
turbidity distribution to quickly deploy the correct mitigation during a
real-world campaign.


http://arxiv.org/abs/1911.07201
Countering Inconsistent Labelling by Google's Vision API for Rotated Images.
Aman Apte; Aritra Bandyopadhyay; K Akhilesh Shenoy; Jason Peter Andrews; Aditya Rathod; Manish Agnihotri; Aditya Jajodia
  Google's Vision API analyses images and provides a variety of output
predictions, one such type is context-based labelling. In this paper, it is
shown that adversarial examples that cause incorrect label prediction and
spoofing can be generated by rotating the images. Due to the black-boxed nature
of the API, a modular context-based pre-processing pipeline is proposed
consisting of a Res-Net50 model, that predicts the angle by which the image
must be rotated to correct its orientation. The pipeline successfully performs
the correction whilst maintaining the image's resolution and feeds it to the
API which generates labels similar to the original correctly oriented image and
using a Percentage Error metric, the performance of the corrected images as
compared to its rotated counter-parts is found to be significantly higher.
These observations imply that the API can benefit from such a pre-processing
pipeline to increase robustness to rotational perturbances.


http://arxiv.org/abs/1911.07421
Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models.
Tong Che; Xiaofeng Liu; Site Li; Yubin Ge; Ruixiang Zhang; Caiming Xiong; Yoshua Bengio
  AI Safety is a major concern in many deep learning applications such as
autonomous driving. Given a trained deep learning model, an important natural
problem is how to reliably verify the model's prediction. In this paper, we
propose a novel framework --- deep verifier networks (DVN) to verify the inputs
and outputs of deep discriminative models with deep generative models. Our
proposed model is based on conditional variational auto-encoders with
disentanglement constraints. We give both intuitive and theoretical
justifications of the model. Our verifier network is trained independently with
the prediction model, which eliminates the need of retraining the verifier
network for a new model. We test the verifier network on out-of-distribution
detection and adversarial example detection problems, as well as anomaly
detection problems in structured prediction tasks such as image caption
generation. We achieve state-of-the-art results in all of these problems.


http://arxiv.org/abs/1911.07198
Smoothed Inference for Adversarially-Trained Models.
Yaniv Nemcovsky; Evgenii Zheltonozhskii; Chaim Baskin; Brian Chmiel; Maxim Fishman; Alex M. Bronstein; Avi Mendelson
  Deep neural networks are known to be vulnerable to adversarial attacks.
Current methods of defense from such attacks are based on either implicit or
explicit regularization, e.g., adversarial training. Randomized smoothing, the
averaging of the classifier outputs over a random distribution centered in the
sample, has been shown to guarantee the performance of a classifier subject to
bounded perturbations of the input. In this work, we study the application of
randomized smoothing as a way to improve performance on unperturbed data as
well as to increase robustness to adversarial attacks. The proposed technique
can be applied on top of any existing adversarial defense, but works
particularly well with the randomized approaches. We examine its performance on
common white-box (PGD) and black-box (transfer and NAttack) attacks on CIFAR-10
and CIFAR-100, substantially outperforming previous art for most scenarios and
comparable on others. For example, we achieve 60.4% accuracy under a PGD attack
on CIFAR-10 using ResNet-20, outperforming previous art by 11.7%. Since our
method is based on sampling, it lends itself well for trading-off between the
model inference complexity and its performance. A reference implementation of
the proposed techniques is provided at https://github.com/yanemcovsky/SIAM


http://arxiv.org/abs/1911.07107
SMART: Skeletal Motion Action Recognition aTtack.
He Wang; Feixiang He; Zexi Peng; Yongliang Yang; Tianjia Shao; Kun Zhou; David Hogg
  Adversarial attack has inspired great interest in computer vision, by showing
that classification-based solutions are prone to imperceptible attack in many
tasks. In this paper, we propose a method, SMART, to attack action recognizers
which rely on 3D skeletal motions. Our method involves an innovative perceptual
loss which ensures the imperceptibility of the attack. Empirical studies
demonstrate that SMART is effective in both white-box and black-box scenarios.
Its generalizability is evidenced on a variety of action recognizers and
datasets. Its versatility is shown in different attacking strategies. Its
deceitfulness is proven in extensive perceptual studies. Finally, SMART shows
that adversarial attack on 3D skeletal motion, one type of time-series data, is
significantly different from traditional adversarial attack problems.


http://arxiv.org/abs/1911.07015
Suspicion-Free Adversarial Attacks on Clustering Algorithms.
Anshuman Chhabra; Abhishek Roy; Prasant Mohapatra
  Clustering algorithms are used in a large number of applications and play an
important role in modern machine learning-- yet, adversarial attacks on
clustering algorithms seem to be broadly overlooked unlike supervised learning.
In this paper, we seek to bridge this gap by proposing a black-box adversarial
attack for clustering models for linearly separable clusters. Our attack works
by perturbing a single sample close to the decision boundary, which leads to
the misclustering of multiple unperturbed samples, named spill-over adversarial
samples. We theoretically show the existence of such adversarial samples for
the K-Means clustering. Our attack is especially strong as (1) we ensure the
perturbed sample is not an outlier, hence not detectable, and (2) the exact
metric used for clustering is not known to the attacker. We theoretically
justify that the attack can indeed be successful without the knowledge of the
true metric. We conclude by providing empirical results on a number of
datasets, and clustering algorithms. To the best of our knowledge, this is the
first work that generates spill-over adversarial samples without the knowledge
of the true metric ensuring that the perturbed sample is not an outlier, and
theoretically proves the above.


http://arxiv.org/abs/1911.07140
Black-Box Adversarial Attack with Transferable Model-based Embedding.
Zhichao Huang; Tong Zhang
  We present a new method for black-box adversarial attack. Unlike previous
methods that combined transfer-based and scored-based methods by using the
gradient or initialization of a surrogate white-box model, this new method
tries to learn a low-dimensional embedding using a pretrained model, and then
performs efficient search within the embedding space to attack an unknown
target network. The method produces adversarial perturbations with high level
semantic patterns that are easily transferable. We show that this approach can
greatly improve the query efficiency of black-box adversarial attack across
different target network architectures. We evaluate our approach on MNIST,
ImageNet and Google Cloud Vision API, resulting in a significant reduction on
the number of queries. We also attack adversarially defended networks on
CIFAR10 and ImageNet, where our method not only reduces the number of queries,
but also improves the attack success rate.


http://arxiv.org/abs/1911.06968
Defensive Few-shot Adversarial Learning.
Wenbin Li; Lei Wang; Xingxing Zhang; Jing Huo; Yang Gao; Jiebo Luo
  The robustness of deep learning models against adversarial attacks has
received increasing attention in recent years. However, both deep learning and
adversarial training rely on the availability of a large amount of labeled data
and usually do not generalize well to new, unseen classes when only a few
training samples are accessible. To address this problem, we explicitly
introduce a new challenging problem -- how to learn a robust deep model with
limited training samples per class, called defensive few-shot learning in this
paper. Simply employing the existing adversarial training techniques in the
literature cannot solve this problem. This is because few-shot learning needs
to learn transferable knowledge from disjoint auxiliary data, and thus it is
invalid to assume the sample-level distribution consistency between the
training and test sets as commonly assumed in existing adversarial training
techniques. In this paper, instead of assuming such a distribution consistency,
we propose to make this assumption at a task-level in the episodic training
paradigm in order to better transfer the defense knowledge. Furthermore, inside
each task, we design a task-conditioned distribution constraint to narrow the
distribution gap between clean and adversarial examples at a sample-level.
These give rise to a novel mechanism called multi-level distribution based
adversarial training (MDAT) for learning transferable adversarial defense. In
addition, a unified $\mathcal{F}_{\beta}$ score is introduced to evaluate
different defense methods under the same principle. Extensive experiments
demonstrate that MDAT achieves higher effectiveness and robustness over
existing alternatives in the few-shot case.


http://arxiv.org/abs/1911.06479
On Model Robustness Against Adversarial Examples.
Shufei Zhang; Kaizhu Huang; Zenglin Xu
  We study the model robustness against adversarial examples, referred to as
small perturbed input data that may however fool many state-of-the-art deep
learning models. Unlike previous research, we establish a novel theory
addressing the robustness issue from the perspective of stability of the loss
function in the small neighborhood of natural examples. We propose to exploit
an energy function to describe the stability and prove that reducing such
energy guarantees the robustness against adversarial examples. We also show
that the traditional training methods including adversarial training with the
$l_2$ norm constraint (AT) and Virtual Adversarial Training (VAT) tend to
minimize the lower bound of our proposed energy function. We make an analysis
showing that minimization of such lower bound can however lead to insufficient
robustness within the neighborhood around the input sample. Furthermore, we
design a more rational method with the energy regularization which proves to
achieve better robustness than previous methods. Through a series of
experiments, we demonstrate the superiority of our model on both supervised
tasks and semi-supervised tasks. In particular, our proposed adversarial
framework achieves the best performance compared with previous adversarial
training methods on benchmark datasets MNIST, CIFAR-10, and SVHN. Importantly,
they demonstrate much better robustness against adversarial examples than all
the other comparison methods.


http://arxiv.org/abs/1911.06587
Learning To Characterize Adversarial Subspaces.
Xiaofeng Mao; Yuefeng Chen; Yuhong Li; Yuan He; Hui Xue
  Deep Neural Networks (DNNs) are known to be vulnerable to the maliciously
generated adversarial examples. To detect these adversarial examples, previous
methods use artificially designed metrics to characterize the properties of
\textit{adversarial subspaces} where adversarial examples lie. However, we find
these methods are not working in practical attack detection scenarios. Because
the artificially defined features are lack of robustness and show limitation in
discriminative power to detect strong attacks. To solve this problem, we
propose a novel adversarial detection method which identifies adversaries by
adaptively learning reasonable metrics to characterize adversarial subspaces.
As auxiliary context information, \textit{k} nearest neighbors are used to
represent the surrounded subspace of the detected sample. We propose an
innovative model called Neighbor Context Encoder (NCE) to learn from \textit{k}
neighbors context and infer if the detected sample is normal or adversarial. We
conduct thorough experiment on CIFAR-10, CIFAR-100 and ImageNet dataset. The
results demonstrate that our approach surpasses all existing methods under
three settings: \textit{attack-aware black-box detection},
\textit{attack-unaware black-box detection} and \textit{white-box detection}.


http://arxiv.org/abs/1911.06502
Simple iterative method for generating targeted universal adversarial perturbations.
Hokuto Hirano; Kazuhiro Takemoto
  Deep neural networks (DNNs) are vulnerable to adversarial attacks. In
particular, a single perturbation known as the universal adversarial
perturbation (UAP) can foil most classification tasks conducted by DNNs. Thus,
different methods for generating UAPs are required to fully evaluate the
vulnerability of DNNs. A realistic evaluation would be with cases that consider
targeted attacks; wherein the generated UAP causes DNN to classify an input
into a specific class. However, the development of UAPs for targeted attacks
has largely fallen behind that of UAPs for non-targeted attacks. Therefore, we
propose a simple iterative method to generate UAPs for targeted attacks. Our
method combines the simple iterative method for generating non-targeted UAPs
and the fast gradient sign method for generating a targeted adversarial
perturbation for an input. We applied the proposed method to state-of-the-art
DNN models for image classification and proved the existence of almost
imperceptible UAPs for targeted attacks; further, we demonstrated that such
UAPs are easily generatable.


http://arxiv.org/abs/1911.06591
AdvKnn: Adversarial Attacks On K-Nearest Neighbor Classifiers With Approximate Gradients.
Xiaodan Li; Yuefeng Chen; Yuan He; Hui Xue
  Deep neural networks have been shown to be vulnerable to adversarial
examples---maliciously crafted examples that can trigger the target model to
misbehave by adding imperceptible perturbations. Existing attack methods for
k-nearest neighbor~(kNN) based algorithms either require large perturbations or
are not applicable for large k. To handle this problem, this paper proposes a
new method called AdvKNN for evaluating the adversarial robustness of kNN-based
models. Firstly, we propose a deep kNN block to approximate the output of kNN
methods, which is differentiable thus can provide gradients for attacks to
cross the decision boundary with small distortions. Second, a new consistency
learning for distribution instead of classification is proposed for the
effectiveness in distribution based methods. Extensive experimental results
indicate that the proposed method significantly outperforms state of the art in
terms of attack success rate and the added perturbations.


http://arxiv.org/abs/1912.01487
Adversarial Embedding: A robust and elusive Steganography and Watermarking technique.
Salah Ghamizi; Maxime Cordy; Mike Papadakis; Yves Le Traon
  We propose adversarial embedding, a new steganography and watermarking
technique that embeds secret information within images. The key idea of our
method is to use deep neural networks for image classification and adversarial
attacks to embed secret information within images. Thus, we use the attacks to
embed an encoding of the message within images and the related deep neural
network outputs to extract it. The key properties of adversarial attacks
(invisible perturbations, nontransferability, resilience to tampering) offer
guarantees regarding the confidentiality and the integrity of the hidden
messages. We empirically evaluate adversarial embedding using more than 100
models and 1,000 messages. Our results confirm that our embedding passes
unnoticed by both humans and steganalysis methods, while at the same time
impedes illicit retrieval of the message (less than 13% recovery rate when the
interceptor has some knowledge about our model), and is resilient to soft and
(to some extent) aggressive image tampering (up to 100% recovery rate under
jpeg compression). We further develop our method by proposing a new type of
adversarial attack which improves the embedding density (amount of hidden
information) of our method to up to 10 bits per pixel.


http://arxiv.org/abs/1911.06470
Self-supervised Adversarial Training.
Kejiang Chen; Hang Zhou; Yuefeng Chen; Xiaofeng Mao; Yuhong Li; Yuan He; Hui Xue; Weiming Zhang; Nenghai Yu
  Recent work has demonstrated that neural networks are vulnerable to
adversarial examples. To escape from the predicament, many works try to harden
the model in various ways, in which adversarial training is an effective way
which learns robust feature representation so as to resist adversarial attacks.
Meanwhile, the self-supervised learning aims to learn robust and semantic
embedding from data itself. With these views, we introduce self-supervised
learning to against adversarial examples in this paper. Specifically, the
self-supervised representation coupled with k-Nearest Neighbour is proposed for
classification. To further strengthen the defense ability, self-supervised
adversarial training is proposed, which maximizes the mutual information
between the representations of original examples and the corresponding
adversarial examples. Experimental results show that the self-supervised
representation outperforms its supervised version in respect of robustness and
self-supervised adversarial training can further improve the defense ability
efficiently.


http://arxiv.org/abs/1911.06285
DomainGAN: Generating Adversarial Examples to Attack Domain Generation Algorithm Classifiers.
Isaac Corley; Jonathan Lwowski; Justin Hoffman
  Domain Generation Algorithms (DGAs) are frequently used to generate numerous
domains for use by botnets. These domains are often utilized as rendezvous
points for servers that malware has command and control over. There are many
algorithms that are used to generate domains, however many of these algorithms
are simplistic and easily detected by traditional machine learning techniques.
In this paper, three variants of Generative Adversarial Networks (GANs) are
optimized to generate domains which have similar characteristics of benign
domains, resulting in domains which greatly evade several state-of-the-art deep
learning based DGA classifiers. We additionally provide a detailed analysis
into offensive usability for each variant with respect to repeated and existing
domain collisions. Finally, we fine-tune the state-of-the-art DGA classifiers
by adding GAN generated samples to their original training datasets and analyze
the changes in performance. Our results conclude that GAN based DGAs are
superior in evading DGA classifiers in comparison to traditional DGAs, and of
the variants, the Wasserstein GAN with Gradient Penalty (WGANGP) is the highest
performing DGA for uses both offensively and defensively.


http://arxiv.org/abs/1911.07931
CAGFuzz: Coverage-Guided Adversarial Generative Fuzzing Testing of Deep Learning Systems.
Pengcheng Zhang; Qiyin Dai; Patrizio Pelliccione
  Deep Learning systems (DL) based on Deep Neural Networks (DNNs) are more and
more used in various aspects of our life, including unmanned vehicles, speech
processing, and robotics. However, due to the limited dataset and the
dependence on manual labeling data, DNNs often fail to detect their erroneous
behaviors, which may lead to serious problems. Several approaches have been
proposed to enhance the input examples for testing DL systems. However, they
have the following limitations. First, they design and generate adversarial
examples from the perspective of model, which may cause low generalization
ability when they are applied to other models. Second, they only use surface
feature constraints to judge the difference between the adversarial example
generated and the original example. The deep feature constraints, which contain
high-level semantic information, such as image object category and scene
semantics are completely neglected. To address these two problems, in this
paper, we propose CAGFuzz, a Coverage-guided Adversarial Generative Fuzzing
testing approach, which generates adversarial examples for a targeted DNN to
discover its potential defects. First, we train an adversarial case generator
(AEG) from the perspective of general data set. Second, we extract the depth
features of the original and adversarial examples, and constrain the
adversarial examples by cosine similarity to ensure that the semantic
information of adversarial examples remains unchanged. Finally, we retrain
effective adversarial examples to improve neuron testing coverage rate. Based
on several popular data sets, we design a set of dedicated experiments to
evaluate CAGFuzz. The experimental results show that CAGFuzz can improve the
neuron coverage rate, detect hidden errors, and also improve the accuracy of
the target DNN.


http://arxiv.org/abs/1911.05904
There is Limited Correlation between Coverage and Robustness for Deep Neural Networks.
Yizhen Dong; Peixin Zhang; Jingyi Wang; Shuang Liu; Jun Sun; Jianye Hao; Xinyu Wang; Li Wang; Jin Song Dong; Dai Ting
  Deep neural networks (DNN) are increasingly applied in safety-critical
systems, e.g., for face recognition, autonomous car control and malware
detection. It is also shown that DNNs are subject to attacks such as
adversarial perturbation and thus must be properly tested. Many coverage
criteria for DNN since have been proposed, inspired by the success of code
coverage criteria for software programs. The expectation is that if a DNN is a
well tested (and retrained) according to such coverage criteria, it is more
likely to be robust. In this work, we conduct an empirical study to evaluate
the relationship between coverage, robustness and attack/defense metrics for
DNN. Our study is the largest to date and systematically done based on 100 DNN
models and 25 metrics. One of our findings is that there is limited correlation
between coverage and robustness, i.e., improving coverage does not help improve
the robustness. Our dataset and implementation have been made available to
serve as a benchmark for future studies on testing DNN.


http://arxiv.org/abs/1911.05916
Adversarial Margin Maximization Networks.
Ziang Yan; Yiwen Guo; Changshui Zhang
  The tremendous recent success of deep neural networks (DNNs) has sparked a
surge of interest in understanding their predictive ability. Unlike the human
visual system which is able to generalize robustly and learn with little
supervision, DNNs normally require a massive amount of data to learn new
concepts. In addition, research works also show that DNNs are vulnerable to
adversarial examples-maliciously generated images which seem perceptually
similar to the natural ones but are actually formed to fool learning models,
which means the models have problem generalizing to unseen data with certain
type of distortions. In this paper, we analyze the generalization ability of
DNNs comprehensively and attempt to improve it from a geometric point of view.
We propose adversarial margin maximization (AMM), a learning-based
regularization which exploits an adversarial perturbation as a proxy. It
encourages a large margin in the input space, just like the support vector
machines. With a differentiable formulation of the perturbation, we train the
regularized DNNs simply through back-propagation in an end-to-end manner.
Experimental results on various datasets (including MNIST, CIFAR-10/100, SVHN
and ImageNet) and different DNN architectures demonstrate the superiority of
our method over previous state-of-the-arts. Code and models for reproducing our
results will be made publicly available.


http://arxiv.org/abs/1911.05153
Improving Robustness of Task Oriented Dialog Systems.
Arash Einolghozati; Sonal Gupta; Mrinal Mohit; Rushin Shah
  Task oriented language understanding in dialog systems is often modeled using
intents (task of a query) and slots (parameters for that task). Intent
detection and slot tagging are, in turn, modeled using sentence classification
and word tagging techniques respectively. Similar to adversarial attack
problems with computer vision models discussed in existing literature, these
intent-slot tagging models are often over-sensitive to small variations in
input -- predicting different and often incorrect labels when small changes are
made to a query, thus reducing their accuracy and reliability. However,
evaluating a model's robustness to these changes is harder for language since
words are discrete and an automated change (e.g. adding `noise') to a query
sometimes changes the meaning and thus labels of a query. In this paper, we
first describe how to create an adversarial test set to measure the robustness
of these models. Furthermore, we introduce and adapt adversarial training
methods as well as data augmentation using back-translation to mitigate these
issues. Our experiments show that both techniques improve the robustness of the
system substantially and can be combined to yield the best results.


http://arxiv.org/abs/1911.04681
On Robustness to Adversarial Examples and Polynomial Optimization.
Pranjal Awasthi; Abhratanu Dutta; Aravindan Vijayaraghavan
  We study the design of computationally efficient algorithms with provable
guarantees, that are robust to adversarial (test time) perturbations. While
there has been an proliferation of recent work on this topic due to its
connections to test time robustness of deep networks, there is limited
theoretical understanding of several basic questions like (i) when and how can
one design provably robust learning algorithms? (ii) what is the price of
achieving robustness to adversarial examples in a computationally efficient
manner?
  The main contribution of this work is to exhibit a strong connection between
achieving robustness to adversarial examples, and a rich class of polynomial
optimization problems, thereby making progress on the above questions. In
particular, we leverage this connection to (a) design computationally efficient
robust algorithms with provable guarantees for a large class of hypothesis,
namely linear classifiers and degree-2 polynomial threshold functions (PTFs),
(b) give a precise characterization of the price of achieving robustness in a
computationally efficient manner for these classes, (c) design efficient
algorithms to certify robustness and generate adversarial attacks in a
principled manner for 2-layer neural networks. We empirically demonstrate the
effectiveness of these attacks on real data.


http://arxiv.org/abs/1911.05268
Adversarial Examples in Modern Machine Learning: A Review.
Rey Reza Wiyatno; Anqi Xu; Ousmane Dia; Berker Archy de
  Recent research has found that many families of machine learning models are
vulnerable to adversarial examples: inputs that are specifically designed to
cause the target model to produce erroneous outputs. In this survey, we focus
on machine learning models in the visual domain, where methods for generating
and detecting such examples have been most extensively studied. We explore a
variety of adversarial attack methods that apply to image-space content, real
world adversarial attacks, adversarial defenses, and the transferability
property of adversarial examples. We also discuss strengths and weaknesses of
various methods of adversarial attack and defense. Our aim is to provide an
extensive coverage of the field, furnishing the reader with an intuitive
understanding of the mechanics of adversarial attack and defense mechanisms and
enlarging the community of researchers studying this fundamental set of
problems.


http://arxiv.org/abs/1911.06155
RNN-Test: Adversarial Testing Framework for Recurrent Neural Network Systems.
Jianmin Guo; Yue Zhao; Xueying Han; Yu Jiang; Jiaguang Sun
  While huge efforts have been investigated in the adversarial testing of
convolutional neural networks (CNN), the testing for recurrent neural networks
(RNN) is still limited to the classification context and leave threats for vast
sequential application domains. In this work, we propose a generic adversarial
testing framework RNN-Test. First, based on the distinctive structure of RNNs,
we define three novel coverage metrics to measure the testing completeness and
guide the generation of adversarial inputs. Second, we propose the state
inconsistency orientation to generate the perturbations by maximizing the
inconsistency of the hidden states of RNN cells. Finally, we combine
orientations with coverage guidance to produce minute perturbations. Given the
RNN model and the sequential inputs, RNN-Test will modify one character or one
word out of the whole inputs based on the perturbations obtained, so as to lead
the RNN to produce wrong outputs. For evaluation, we apply RNN-Test on two
models of common RNN structure - the PTB language model and the spell checker
model. RNN-Test efficiently reduces the performance of the PTB language model
by increasing its test perplexity by 58.11%, and finds numbers of incorrect
behaviors of the spell checker model with the success rate of 73.44% on
average. With our customization, RNN-Test using the redefined neuron coverage
as guidance could achieve 35.71% higher perplexity than original strategy of
DeepXplore.


http://arxiv.org/abs/1911.06269
Few-Features Attack to Fool Machine Learning Models through Mask-Based GAN.
Feng Chen; Yunkai Shang; Bo Xu; Jincheng Hu
  GAN is a deep-learning based generative approach to generate contents such as
images, languages and speeches. Recently, studies have shown that GAN can also
be applied to generative adversarial attack examples to fool the
machine-learning models. In comparison with the previous non-learning
adversarial example attack approaches, the GAN-based adversarial attack example
approach can generate the adversarial samples quickly using the GAN
architecture every time facing a new sample after training, but meanwhile needs
to perturb the attack samples in great quantities, which results in the
unpractical application in reality. To address this issue, we propose a new
approach, named Few-Feature-Attack-GAN (FFA-GAN). FFA-GAN has a significant
time-consuming advantage than the non-learning adversarial samples approaches
and a better non-zero-features performance than the GANbased adversarial sample
approaches. FFA-GAN can automatically generate the attack samples in the
black-box attack through the GAN architecture instead of the evolutional
algorithms or the other non-learning approaches. Besides, we introduce the mask
mechanism into the generator network of the GAN architecture to optimize the
constraint issue, which can also be regarded as the sparsity problem of the
important features. During the training, the different weights of losses of the
generator are set in the different training phases to ensure the divergence of
the two above mentioned parallel networks of the generator. Experiments are
made respectively on the structured data sets KDD-Cup 1999 and CIC-IDS 2017, in
which the dimensions of the data are relatively low, and also on the
unstructured data sets MNIST and CIFAR-10 with the data of the relatively high
dimensions. The results of the experiments demonstrate the effectiveness and
the robustness of our proposed approach.


http://arxiv.org/abs/1911.04657
CALPA-NET: Channel-pruning-assisted Deep Residual Network for Steganalysis of Digital Images.
Shunquan Tan; Weilong Wu; Zilong Shao; Qiushi Li; Bin Li; Jiwu Huang
  Over the past few years, detection performance improvements of deep-learning
based steganalyzers have been usually achieved through structure expansion.
However, excessive expanded structure results in huge computational cost,
storage overheads, and consequently difficulty in training and deployment. In
this paper we propose CALPA-NET, a ChAnneL-Pruning-Assisted deep residual
network architecture search approach to shrink the network structure of
existing vast, over-parameterized deep-learning based steganalyzers. By
providing theoretical rationale, we demonstrate that the broad inverted-pyramid
structure of existing deep-learning based steganalyzers contradicts the
well-established model diversity oriented philosophy and is not suitable for
steganalysis. Then a hybrid criterion combined with two network pruning schemes
is introduced to adaptively shrink every involved convolutional layer in a
data-driven manner. The resulting network architecture presents a slender
bottleneck-like structure. We have conducted extensive experiments on
BOSSBase+BOWS2 dataset and even a large-scale subset extracted from ImageNet
CLS-LOC dataset. The experimental results show that the model structure
generated by our proposed CALPA-NET can achieve comparative detection
performance with less than two percent of parameters and about one third FLOPs
compared to the original steganalytic model. The new model possesses even
better adaptivity, transferability, and scalability.


http://arxiv.org/abs/1911.05072
Learning From Brains How to Regularize Machines.
Zhe Li; Wieland Brendel; Edgar Y. Walker; Erick Cobos; Taliah Muhammad; Jacob Reimer; Matthias Bethge; Fabian H. Sinz; Xaq Pitkow; Andreas S. Tolias
  Despite impressive performance on numerous visual tasks, Convolutional Neural
Networks (CNNs) --- unlike brains --- are often highly sensitive to small
perturbations of their input, e.g. adversarial noise leading to erroneous
decisions. We propose to regularize CNNs using large-scale neuroscience data to
learn more robust neural features in terms of representational similarity. We
presented natural images to mice and measured the responses of thousands of
neurons from cortical visual areas. Next, we denoised the notoriously variable
neural activity using strong predictive models trained on this large corpus of
responses from the mouse visual system, and calculated the representational
similarity for millions of pairs of images from the model's predictions. We
then used the neural representation similarity to regularize CNNs trained on
image classification by penalizing intermediate representations that deviated
from neural ones. This preserved performance of baseline models when
classifying images under standard benchmarks, while maintaining substantially
higher performance compared to baseline or control models when classifying
noisy images. Moreover, the models regularized with cortical representations
also improved model robustness in terms of adversarial attacks. This
demonstrates that regularizing with neural data can be an effective tool to
create an inductive bias towards more robust inference.


http://arxiv.org/abs/1911.04636
Robust Design of Deep Neural Networks against Adversarial Attacks based on Lyapunov Theory.
Arash Rahnama; Andre T. Nguyen; Edward Raff
  Deep neural networks (DNNs) are vulnerable to subtle adversarial
perturbations applied to the input. These adversarial perturbations, though
imperceptible, can easily mislead the DNN. In this work, we take a control
theoretic approach to the problem of robustness in DNNs. We treat each
individual layer of the DNN as a nonlinear dynamical system and use Lyapunov
theory to prove stability and robustness locally. We then proceed to prove
stability and robustness globally for the entire DNN. We develop empirically
tight bounds on the response of the output layer, or any hidden layer, to
adversarial perturbations added to the input, or the input of hidden layers.
Recent works have proposed spectral norm regularization as a solution for
improving robustness against l2 adversarial attacks. Our results give new
insights into how spectral norm regularization can mitigate the adversarial
effects. Finally, we evaluate the power of our approach on a variety of data
sets and network architectures and against some of the well-known adversarial
attacks.


http://arxiv.org/abs/1911.04429
GraphDefense: Towards Robust Graph Convolutional Networks.
Xiaoyun Wang; Xuanqing Liu; Cho-Jui Hsieh
  In this paper, we study the robustness of graph convolutional networks
(GCNs). Despite the good performance of GCNs on graph semi-supervised learning
tasks, previous works have shown that the original GCNs are very unstable to
adversarial perturbations. In particular, we can observe a severe performance
degradation by slightly changing the graph adjacency matrix or the features of
a few nodes, making it unsuitable for security-critical applications. Inspired
by the previous works on adversarial defense for deep neural networks, and
especially adversarial training algorithm, we propose a method called
GraphDefense to defend against the adversarial perturbations. In addition, for
our defense method, we could still maintain semi-supervised learning settings,
without a large label rate. We also show that adversarial training in features
is equivalent to adversarial training for edges with a small perturbation. Our
experiments show that the proposed defense methods successfully increase the
robustness of Graph Convolutional Networks. Furthermore, we show that with
careful design, our proposed algorithm can scale to large graphs, such as
Reddit dataset.


http://arxiv.org/abs/1911.03677
A Reinforced Generation of Adversarial Samples for Neural Machine Translation.
Wei Zou; Shujian Huang; Jun Xie; Xinyu Dai; Jiajun Chen
  Neural machine translation systems tend to fail on less de-cent inputs
despite its great efficacy, which may greatly harm the credibility of these
systems. Fathoming how and when neural-based systems fail in such cases is
critical for industrial maintenance. Instead of collecting and analyzing bad
cases using limited handcrafted error features, here we investigate this issue
by generating adversarial samples via a new paradigm based on reinforcement
learning. Our paradigm could expose pitfalls for a given performance metric,
e.g.BLEU, and could target any given neural machine translation architecture.
We conduct experiments of adversarial attacks on two mainstream neural machine
translation architectures, RNN-search and Transformer. The results show that
our method efficiently produces stable attacks with meaning-preserving
adversarial samples. We also present a qualitative and quantitative analysis
for the preference pattern of the attack, showing its capability of pitfall
exposure.


http://arxiv.org/abs/1911.03614
Improving Machine Reading Comprehension via Adversarial Training.
Ziqing Yang; Yiming Cui; Wanxiang Che; Ting Liu; Shijin Wang; Guoping Hu
  Adversarial training (AT) as a regularization method has proved its
effectiveness in various tasks, such as image classification and text
classification. Though there are successful applications of AT in many tasks of
natural language processing (NLP), the mechanism behind it is still unclear. In
this paper, we aim to apply AT on machine reading comprehension (MRC) and study
its effects from multiple perspectives. We experiment with three different
kinds of RC tasks: span-based RC, span-based RC with unanswerable questions and
multi-choice RC. The experimental results show that the proposed method can
improve the performance significantly and universally on SQuAD1.1, SQuAD2.0 and
RACE. With virtual adversarial training (VAT), we explore the possibility of
improving the RC models with semi-supervised learning and prove that examples
from a different task are also beneficial. We also find that AT helps little in
defending against artificial adversarial examples, but AT helps the model to
learn better on examples that contain more low-frequency words.


http://arxiv.org/abs/1911.03784
Adaptive versus Standard Descent Methods and Robustness Against Adversarial Examples.
Marc Khoury
  Adversarial examples are a pervasive phenomenon of machine learning models
where seemingly imperceptible perturbations to the input lead to
misclassifications for otherwise statistically accurate models. In this paper
we study how the choice of optimization algorithm influences the robustness of
the resulting classifier to adversarial examples. Specifically we show an
example of a learning problem for which the solution found by adaptive
optimization algorithms exhibits qualitatively worse robustness properties
against both $L_{2}$- and $L_{\infty}$-adversaries than the solution found by
non-adaptive algorithms. Then we fully characterize the geometry of the loss
landscape of $L_{2}$-adversarial training in least-squares linear regression.
The geometry of the loss landscape is subtle and has important consequences for
optimization algorithms. Finally we provide experimental evidence which
suggests that non-adaptive methods consistently produce more robust models than
adaptive methods.


http://arxiv.org/abs/1911.03849
Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policy.
Xinghua Qu; Zhu Sun; Yew-Soon Ong; Abhishek Gupta; Pengfei Wei
  Recent studies have revealed that neural network-based policies can be easily
fooled by adversarial examples. However, while most prior works analyze the
effects of perturbing every pixel of every frame assuming white-box policy
access, in this paper we take a more restrictive view towards adversary
generation - with the goal of unveiling the limits of a model's vulnerability.
In particular, we explore minimalistic attacks by defining three key settings:
(1) black-box policy access: where the attacker only has access to the input
(state) and output (action probability) of an RL policy; (2) fractional-state
adversary: where only several pixels are perturbed, with the extreme case being
a single-pixel adversary; and (3) tactically-chanced attack: where only
significant frames are tactically chosen to be attacked. We formulate the
adversarial attack by accommodating the three key settings and explore their
potency on six Atari games by examining four fully trained state-of-the-art
policies. In Breakout, for example, we surprisingly find that: (i) all policies
showcase significant performance degradation by merely modifying 0.01% of the
input state, and (ii) the policy trained by DQN is totally deceived by
perturbation to only 1% frames.


http://arxiv.org/abs/1911.04278
Intrusion Detection for Industrial Control Systems: Evaluation Analysis and Adversarial Attacks.
Giulio Zizzo; Chris Hankin; Sergio Maffeis; Kevin Jones
  Neural networks are increasingly used in security applications for intrusion
detection on industrial control systems. In this work we examine two areas that
must be considered for their effective use. Firstly, is their vulnerability to
adversarial attacks when used in a time series setting. Secondly, is potential
over-estimation of performance arising from data leakage artefacts.
  To investigate these areas we implement a long short-term memory (LSTM) based
intrusion detection system (IDS) which effectively detects cyber-physical
attacks on a water treatment testbed representing a strong baseline IDS.
  For investigating adversarial attacks we model two different white box
attackers. The first attacker is able to manipulate sensor readings on a subset
of the Secure Water Treatment (SWaT) system. By creating a stream of
adversarial data the attacker is able to hide the cyber-physical attacks from
the IDS. For the cyber-physical attacks which are detected by the IDS, the
attacker required on average 2.48 out of 12 total sensors to be compromised for
the cyber-physical attacks to be hidden from the IDS. The second attacker model
we explore is an $L_{\infty}$ bounded attacker who can send fake readings to
the IDS, but to remain imperceptible, limits their perturbations to the
smallest $L_{\infty}$ value needed.
  Additionally, we examine data leakage problems arising from tuning for $F_1$
score on the whole SWaT attack set and propose a method to tune detection
parameters that does not utilise any attack data. If attack after-effects are
accounted for then our new parameter tuning method achieved an $F_1$ score of
0.811$\pm$0.0103.


http://arxiv.org/abs/1911.07922
Patch augmentation: Towards efficient decision boundaries for neural networks.
Marcus D. Bloice; Andreas Holzinger
  In this paper we propose a new augmentation technique, called patch
augmentation, that, in our experiments, improves model accuracy and makes
networks more robust to adversarial attacks. In brief, this data-independent
approach creates new image data based on image/label pairs, where a patch from
one of the two images in the pair is superimposed on to the other image,
creating a new augmented sample. The new image's label is a linear combination
of the image pair's corresponding labels. Initial experiments show a several
percentage point increase in accuracy on CIFAR-10, from a baseline of 80.6% to
86.8%. CIFAR-100 sees better improvements still. Networks trained using patch
augmentation are also more robust to adversarial attacks, which we demonstrate
using the Fast Gradient Sign Method. An adversarial misclassification, or
adversarial attack, occurs when an image that should seemingly be easily
classified correctly by the network is suddenly classified as belonging to a
completely different class-and with high confidence. Such occurrences are
difficult to diagnose and are a cause of much concern in artificial
intelligence research, as any model trained with empirical risk minimisation
seems to be vulnerable to such attacks. The ease at which neural networks are
susceptible to adversarial perturbations are partially the result of images
lying close to the decision boundaries that are typically learned by neural
networks during their training. Patch augmentation is an attempt to train more
efficient decision boundaries.


http://arxiv.org/abs/1911.03109
Domain Robustness in Neural Machine Translation.
Mathias Müller; Annette Rios; Rico Sennrich
  Translating text that diverges from the training domain is a key challenge
for neural machine translation (NMT). Domain robustness - the generalization of
models to unseen test domains - is low compared to statistical machine
translation. In this paper, we investigate the performance of NMT on
out-of-domain test sets, and ways to improve it.
  We observe that hallucination (translations that are fluent but unrelated to
the source) is common in out-of-domain settings, and we empirically compare
methods that improve adequacy (reconstruction), out-of-domain translation
(subword regularization), or robustness against adversarial examples (defensive
distillation), as well as noisy channel models.
  In experiments on German to English OPUS data, and German to Romansh, a
low-resource scenario, we find that several methods improve domain robustness,
reconstruction standing out as a method that not only improves automatic
scores, but also shows improvements in a manual assessments of adequacy, albeit
at some loss in fluency. However, out-of-domain performance is still relatively
low and domain robustness remains an open problem.


http://arxiv.org/abs/1911.03078
Adversarial Attacks on GMM i-vector based Speaker Verification Systems.
Xu Li; Jinghua Zhong; Xixin Wu; Jianwei Yu; Xunying Liu; Helen Meng
  This work investigates the vulnerability of Gaussian Mix-ture Model (GMM)
i-vector based speaker verification (SV) systems to adversarial attacks, and
the transferability of adversarial samples crafted from GMM i-vector based
systemsto x-vector based systems. In detail, we formulate the GMM i-vector
based system as a scoring function, and leverage the fast gradient sign method
(FGSM) to generate adversarial samples through this function. These adversarial
samples are used to attack both GMM i-vector and x-vector based systems. We
measure the vulnerability of the systems by the degradation of equal error rate
and false acceptance rate. Experimental results show that GMM i-vector based
systems are seriously vulnerable to adversarial attacks, and the generated
adversarial samples are proved to be transferable and pose threats to neural
network speaker embedding based systems (e.g. x-vector systems).


http://arxiv.org/abs/1911.03274
Imperceptible Adversarial Attacks on Tabular Data.
Vincent Ballet; Xavier Renard; Jonathan Aigrain; Thibault Laugel; Pascal Frossard; Marcin Detyniecki
  Security of machine learning models is a concern as they may face adversarial
attacks for unwarranted advantageous decisions. While research on the topic has
mainly been focusing on the image domain, numerous industrial applications, in
particular in finance, rely on standard tabular data. In this paper, we discuss
the notion of adversarial examples in the tabular domain. We propose a
formalization based on the imperceptibility of attacks in the tabular domain
leading to an approach to generate imperceptible adversarial examples.
Experiments show that we can generate imperceptible adversarial examples with a
high fooling rate.


http://arxiv.org/abs/1911.04606
White-Box Target Attack for EEG-Based BCI Regression Problems.
Lubin Meng; Chin-Teng Lin; Tzyy-Ring Jung; Dongrui Wu
  Machine learning has achieved great success in many applications, including
electroencephalogram (EEG) based brain-computer interfaces (BCIs).
Unfortunately, many machine learning models are vulnerable to adversarial
examples, which are crafted by adding deliberately designed perturbations to
the original inputs. Many adversarial attack approaches for classification
problems have been proposed, but few have considered target adversarial attacks
for regression problems. This paper proposes two such approaches. More
specifically, we consider white-box target attacks for regression problems,
where we know all information about the regression model to be attacked, and
want to design small perturbations to change the regression output by a
pre-determined amount. Experiments on two BCI regression problems verified that
both approaches are effective. Moreover, adversarial examples generated from
both approaches are also transferable, which means that we can use adversarial
examples generated from one known regression model to attack an unknown
regression model, i.e., to perform black-box attacks. To our knowledge, this is
the first study on adversarial attacks for EEG-based BCI regression problems,
which calls for more attention on the security of BCI systems.


http://arxiv.org/abs/1911.04338
Active Learning for Black-Box Adversarial Attacks in EEG-Based Brain-Computer Interfaces.
Xue Jiang; Xiao Zhang; Dongrui Wu
  Deep learning has made significant breakthroughs in many fields, including
electroencephalogram (EEG) based brain-computer interfaces (BCIs). However,
deep learning models are vulnerable to adversarial attacks, in which
deliberately designed small perturbations are added to the benign input samples
to fool the deep learning model and degrade its performance. This paper
considers transferability-based black-box attacks, where the attacker trains a
substitute model to approximate the target model, and then generates
adversarial examples from the substitute model to attack the target model.
Learning a good substitute model is critical to the success of these attacks,
but it requires a large number of queries to the target model. We propose a
novel framework which uses query synthesis based active learning to improve the
query efficiency in training the substitute model. Experiments on three
convolutional neural network (CNN) classifiers and three EEG datasets
demonstrated that our method can improve the attack success rate with the same
number of queries, or, in other words, our method requires fewer queries to
achieve a desired attack performance. To our knowledge, this is the first work
that integrates active learning and adversarial attacks for EEG-based BCIs.


http://arxiv.org/abs/1911.02466
Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distance.
Zhengyu Zhao; Zhuoran Liu; Martha Larson
  The success of image perturbations that are designed to fool image
classification is assessed in terms of both adversarial effect and visual
imperceptibility. In this work, we investigate the contribution of human color
perception to perturbations that are not noticeable. Our basic insight is that
perceptual color distance makes it possible to drop the conventional assumption
that imperceptible perturbations should strive for small $L_p$ norms in RGB
space. Our first approach, Perceptual Color distance C&W (PerC-C&W), extends
the widely-used C&W approach and produces larger RGB perturbations. PerC-C&W is
able to maintain adversarial strength, while contributing to imperceptibility.
Our second approach, Perceptual Color distance Alternating Loss (PerC-AL),
achieves the same outcome, but does so more efficiently by alternating between
the classification loss and perceptual color difference when updating
perturbations. Experimental evaluation shows PerC approaches improve robustness
and transferability of perturbations over conventional approaches and also
demonstrates that the PerC distance can provide added value on top of existing
structure-based approaches to creating image perturbations.


http://arxiv.org/abs/1911.02508
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods.
Dylan Slack; Sophie Hilgard; Emily Jia; Sameer Singh; Himabindu Lakkaraju
  As machine learning black boxes are increasingly being deployed in domains
such as healthcare and criminal justice, there is growing emphasis on building
tools and techniques for explaining these black boxes in an interpretable
manner. Such explanations are being leveraged by domain experts to diagnose
systematic errors and underlying biases of black boxes. In this paper, we
demonstrate that post hoc explanations techniques that rely on input
perturbations, such as LIME and SHAP, are not reliable. Specifically, we
propose a novel scaffolding technique that effectively hides the biases of any
given classifier by allowing an adversarial entity to craft an arbitrary
desired explanation. Our approach can be used to scaffold any biased classifier
in such a way that its predictions on the input data distribution still remain
biased, but the post hoc explanations of the scaffolded classifier look
innocuous. Using extensive evaluation with multiple real-world datasets
(including COMPAS), we demonstrate how extremely biased (racist) classifiers
crafted by our framework can easily fool popular explanation techniques such as
LIME and SHAP into generating innocuous explanations which do not reflect the
underlying biases.


http://arxiv.org/abs/1911.02621
The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey.
Olakunle Ibitoye; Rana Abou-Khamis; Ashraf Matrawy; M. Omair Shafiq
  Machine learning models have made many decision support systems to be faster,
more accurate and more efficient. However, applications of machine learning in
network security face more disproportionate threat of active adversarial
attacks compared to other domains. This is because machine learning
applications in network security such as malware detection, intrusion
detection, and spam filtering are by themselves adversarial in nature. In what
could be considered an arms race between attackers and defenders, adversaries
constantly probe machine learning systems with inputs which are explicitly
designed to bypass the system and induce a wrong prediction. In this survey, we
first provide a taxonomy of machine learning techniques, styles, and
algorithms. We then introduce a classification of machine learning in network
security applications. Next, we examine various adversarial attacks against
machine learning in network security and introduce two classification
approaches for adversarial attacks in network security. First, we classify
adversarial attacks in network security based on a taxonomy of network security
applications. Secondly, we categorize adversarial attacks in network security
into a problem space vs. feature space dimensional classification model. We
then analyze the various defenses against adversarial attacks on machine
learning-based network security applications. We conclude by introducing an
adversarial risk model and evaluate several existing adversarial attacks
against machine learning in network security using the risk model. We also
identify where each attack classification resides within the adversarial risk
model


http://arxiv.org/abs/1911.02360
Reversible Adversarial Example based on Reversible Image Transformation.
Zhaoxia Yin; Hua Wang; Weiming Zhang
  At present there are many companies that take the most advanced Deep Neural
Networks (DNNs) to classify and analyze photos we upload to social networks or
the cloud. In order to prevent users privacy from leakage, the attack
characteristics of the adversarial example can be exploited to make these
models misjudged. In this paper, we take advantage of reversible image
transformation to construct reversible adversarial example, which is still an
adversarial example to DNNs. It not only allows DNNs to extract the wrong
information, but also can be recovered to its original image without any
distortion. Experimental results show that reversible adversarial examples
obtained by our method have higher attack success rates while ensuring that the
reversible image quality is still high. Moreover, the proposed method is easy
to operate, suitable for practical applications.


http://arxiv.org/abs/1911.01670
Adversarial Enhancement for Community Detection in Complex Networks.
Jiajun Zhou; Zhi Chen; Min Du; Lihong Chen; Shanqing Yu; Feifei Li; Guanrong Chen; Qi Xuan
  Community detection plays a significant role in network analysis. However, it
also faces numerous challenges like adversarial attacks. How to further improve
the performance and robustness of community detection for real-world networks
has raised great concerns. In this paper, we propose a concept of adversarial
enhancement for community detection, and present two adversarial enhancement
algorithms: one is named adversarial enhancement via genetic algorithm (AE-GA),
in which the modularity and the number of clusters are used to design a fitness
function to solve the resolution limit problem; and the other is called
adversarial enhancement via vertex similarity (AE-VS), integrating multiple
information of community structures captured by diverse vertex similarities,
which scales well on large-scale networks. The two algorithms are tested along
with six existing community detection algorithms on four real-world networks.
Comprehensive experimental results show that, by comparing with two traditional
enhancement strategies, our methods help six community detection algorithms
achieve more significant performance improvement. Moreover, experiments on the
corresponding adversarial networks indicate that our methods can rebuild the
network structure destroyed by adversarial attacks to certain extent, achieving
stronger defense against community detection deception.


http://arxiv.org/abs/1911.01952
Test Metrics for Recurrent Neural Networks.
Wei Huang; Youcheng Sun; James Sharp; Xiaowei Huang
  Recurrent neural networks (RNNs) have been applied to a broad range of
application areas such as natural language processing, drug discovery, and
video recognition. This paper develops a coverage-guided test framework,
including three test metrics and a mutation-based test case generation method,
for the validation of a major class of RNNs, i.e., long short-term memory
networks (LSTMs). The test metrics are designed with respect to the internal
structures of the LSTM layers to quantify the information of the forget gate,
the one-step information change of an aggregate hidden state, and the
multi-step information evolution of positive and negative aggregate hidden
state, respectively. We apply the test framework to several typical LSTM
applications, including a network trained on IMDB movie reviews for sentiment
analysis, a network trained on MNIST dataset for image classification, and a
network trained on a lipophilicity dataset for scientific machine learning. Our
experimental results show that the coverage-guided testing can be used to not
only extensively exploit the behaviour of the LSTM layer in order to discover
the safety loopholes (such as adversarial examples) but also help understand
the internal mechanism of how the LSTM layer processes data.


http://arxiv.org/abs/1911.01921
DLA: Dense-Layer-Analysis for Adversarial Example Detection.
Philip Sperl; Ching-Yu Kao; Peng Chen; Konstantin Böttinger
  In recent years Deep Neural Networks (DNNs) have achieved remarkable results
and even showed super-human capabilities in a broad range of domains. This led
people to trust in DNNs' classifications and resulting actions even in
security-sensitive environments like autonomous driving.
  Despite their impressive achievements, DNNs are known to be vulnerable to
adversarial examples. Such inputs contain small perturbations to intentionally
fool the attacked model.
  In this paper, we present a novel end-to-end framework to detect such attacks
during classification without influencing the target model's performance.
Inspired by recent research in neuron-coverage guided testing we show that
dense layers of DNNs carry security-sensitive information. With a secondary DNN
we analyze the activation patterns of the dense layers during classification
runtime, which enables effective and real-time detection of adversarial
examples.
  Our prototype implementation successfully detects adversarial examples in
image, natural language, and audio processing. Thereby, we cover a variety of
target DNNs, including Long Short Term Memory (LSTM) architectures. In
addition, to effectively defend against state-of-the-art attacks, our approach
generalizes between different sets of adversarial examples. Thus, our method
most likely enables us to detect even future, yet unknown attacks. Finally,
during white-box adaptive attacks, we show our method cannot be easily
bypassed.


http://arxiv.org/abs/1911.02142
Intriguing Properties of Adversarial ML Attacks in the Problem Space.
Fabio Pierazzi; Feargus Pendlebury; Jacopo Cortellazzi; Lorenzo Cavallaro
  Recent research efforts on adversarial ML have investigated problem-space
attacks, focusing on the generation of real evasive objects in domains where,
unlike images, there is no clear inverse mapping to the feature space (e.g.,
software). However, the design, comparison, and real-world implications of
problem-space attacks remain underexplored. This paper makes two major
contributions. First, we propose a novel formalization for adversarial ML
evasion attacks in the problem-space, which includes the definition of a
comprehensive set of constraints on available transformations, preserved
semantics, robustness to preprocessing, and plausibility. We shed light on the
relationship between feature space and problem space, and we introduce the
concept of side-effect features as the byproduct of the inverse feature-mapping
problem. This enables us to define and prove necessary and sufficient
conditions for the existence of problem-space attacks. We further demonstrate
the expressive power of our formalization by using it to describe several
attacks from related literature across different domains. Second, building on
our formalization, we propose a novel problem-space attack on Android malware
that overcomes past limitations. Experiments on a dataset with 170K Android
apps from 2017 and 2018 show the practical feasibility of evading a
state-of-the-art malware classifier along with its hardened version. Our
results demonstrate that "adversarial-malware as a service" is a realistic
threat, as we automatically generate thousands of realistic and inconspicuous
adversarial applications at scale, where on average it takes only a few minutes
to generate an adversarial app. Our formalization of problem-space attacks
paves the way to more principled research in this domain.


http://arxiv.org/abs/1911.01043
Persistency of Excitation for Robustness of Neural Networks.
Kamil Nar; S. Shankar Sastry
  When an online learning algorithm is used to estimate the unknown parameters
of a model, the signals interacting with the parameter estimates should not
decay too quickly for the optimal values to be discovered correctly. This
requirement is referred to as persistency of excitation, and it arises in
various contexts, such as optimization with stochastic gradient methods,
exploration for multi-armed bandits, and adaptive control of dynamical systems.
While training a neural network, the iterative optimization algorithm involved
also creates an online learning problem, and consequently, correct estimation
of the optimal parameters requires persistent excitation of the network
weights. In this work, we analyze the dynamics of the gradient descent
algorithm while training a two-layer neural network with two different loss
functions, the squared-error loss and the cross-entropy loss; and we obtain
conditions to guarantee persistent excitation of the network weights. We then
show that these conditions are difficult to satisfy when a multi-layer network
is trained for a classification task, for the signals in the intermediate
layers of the network become low-dimensional during training and fail to remain
persistently exciting. To provide a remedy, we delve into the classical
regularization terms used for linear models, reinterpret them as a means to
ensure persistent excitation of the model parameters, and propose an algorithm
for neural networks by building an analogy. The results in this work shed some
light on why adversarial examples have become a challenging problem for neural
networks, why merely augmenting training data sets will not be an effective
approach to address them, and why there may not exist a data-independent
regularization term for neural networks, which involve only the model
parameters but not the training data.


http://arxiv.org/abs/1911.01559
A Tale of Evil Twins: Adversarial Inputs versus Poisoned Models.
Ren Pang; Hua Shen; Xinyang Zhang; Shouling Ji; Yevgeniy Vorobeychik; Xiapu Luo; Alex Liu; Ting Wang
  Despite their tremendous success in a range of domains, deep learning systems
are inherently susceptible to two types of manipulations: adversarial inputs --
maliciously crafted samples that deceive target deep neural network (DNN)
models, and poisoned models -- adversely forged DNNs that misbehave on
pre-defined inputs. While prior work has intensively studied the two attack
vectors in parallel, there is still a lack of understanding about their
fundamental connections: what are the dynamic interactions between the two
attack vectors? what are the implications of such interactions for optimizing
existing attacks? what are the potential countermeasures against the enhanced
attacks? Answering these key questions is crucial for assessing and mitigating
the holistic vulnerabilities of DNNs deployed in realistic settings.
  Here we take a solid step towards this goal by conducting the first
systematic study of the two attack vectors within a unified framework.
Specifically, (i) we develop a new attack model that jointly optimizes
adversarial inputs and poisoned models; (ii) with both analytical and empirical
evidence, we reveal that there exist intriguing "mutual reinforcement" effects
between the two attack vectors -- leveraging one vector significantly amplifies
the effectiveness of the other; (iii) we demonstrate that such effects enable a
large design spectrum for the adversary to enhance the existing attacks that
exploit both vectors (e.g., backdoor attacks), such as maximizing the attack
evasiveness with respect to various detection methods; (iv) finally, we discuss
potential countermeasures against such optimized attacks and their technical
challenges, pointing to several promising research directions.


http://arxiv.org/abs/1911.01172
Fast-UAP: An Algorithm for Speeding up Universal Adversarial Perturbation Generation with Orientation of Perturbation Vectors.
Jiazhu Dai; Le Shu
  Convolutional neural networks (CNN) have become one of the most popular
machine learning tools and are being applied in various tasks, however, CNN
models are vulnerable to universal perturbations, which are usually
human-imperceptible but can cause natural images to be misclassified with high
probability. One of the state-of-the-art algorithms to generate universal
perturbations is known as UAP. UAP only aggregates the minimal perturbations in
every iteration, which will lead to generated universal perturbation whose
magnitude cannot rise up efficiently and cause a slow generation. In this
paper, we proposed an optimized algorithm to improve the performance of
crafting universal perturbations based on orientation of perturbation vectors.
At each iteration, instead of choosing minimal perturbation vector with respect
to each image, we aggregate the current instance of universal perturbation with
the perturbation which has similar orientation to the former so that the
magnitude of the aggregation will rise up as large as possible at every
iteration. The experiment results show that we get universal perturbations in a
shorter time and with a smaller number of training images. Furthermore, we
observe in experiments that universal perturbations generated by our proposed
algorithm have an average increment of fooling rate by 8% ~ 9% in white-box
attacks and black-box attacks comparing with universal perturbations generated
by UAP.


http://arxiv.org/abs/1911.01840
Who is Real Bob? Adversarial Attacks on Speaker Recognition Systems.
Guangke Chen; Sen Chen; Lingling Fan; Xiaoning Du; Zhe Zhao; Fu Song; Yang Liu
  Speaker recognition (SR) is widely used in our daily life as a biometric
authentication mechanism. The popularity of SR brings in serious security
concerns, as demonstrated by recent adversarial attacks. However, the impacts
of such threats in the practical black-box setting are still open, since
current attacks consider the white-box setting only.
  In this paper, we conduct the first comprehensive and systematic study of the
adversarial attacks on SR systems (SRSs) to understand their security weakness
in the practical black-box setting. For this purpose, we propose an adversarial
attack, named FakeBob, to craft adversarial samples. Specifically, we formulate
the adversarial sample generation as an optimization problem, incorporated with
the confidence of adversarial samples and maximal distortion to balance between
the strength and imperceptibility of adversarial voices. One key contribution
is to propose a novel algorithm to estimate the score threshold, a feature in
SRSs, and use it in the optimization problem to solve the optimization problem.
We demonstrate that FakeBob achieves close to 100% targeted attack success rate
on both open-source and commercial systems. We further demonstrate that FakeBob
is also effective (at least 65% untargeted success rate) on both open-source
and commercial systems when playing over the air in the physical world.
Moreover, we have conducted a human study which reveals that it is hard for
human to differentiate the speakers of the original and adversarial voices.
Last but not least, we show that three promising defense methods for
adversarial attack from the speech recognition domain become ineffective on
SRSs against FakeBob, which calls for more effective defense methods. We
highlight that our study peeks into the security implications of adversarial
attacks on SRSs, and realistically fosters to improve the security robustness
of SRSs.


http://arxiv.org/abs/1911.00870
Improved Detection of Adversarial Attacks via Penetration Distortion Maximization.
Shai Rozenberg; Gal Elidan; Ran El-Yaniv
  This paper is concerned with the defense of deep models against adversarial
attacks. We developan adversarial detection method, which is inspired by the
certificate defense approach, and capturesthe idea of separating class clusters
in the embedding space to increase the margin. The resultingdefense is
intuitive, effective, scalable, and can be integrated into any given neural
classificationmodel. Our method demonstrates state-of-the-art (detection)
performance under all threat models.


http://arxiv.org/abs/1911.00650
Automatic Detection of Generated Text is Easiest when Humans are Fooled.
Daphne Ippolito; Daniel Duckworth; Chris Callison-Burch; Douglas Eck
  Recent advancements in neural language modelling make it possible to rapidly
generate vast amounts of human-sounding text. The capabilities of humans and
automatic discriminators to detect machine-generated text have been a large
source of research interest, but humans and machines rely on different cues to
make their decisions. Here, we perform careful benchmarking and analysis of
three popular sampling-based decoding strategies---top-$k$, nucleus sampling,
and untruncated random sampling---and show that improvements in decoding
methods have primarily optimized for fooling humans. This comes at the expense
of introducing statistical abnormalities that make detection easy for automatic
systems. We also show that though both human and automatic detector performance
improve with longer excerpt length, even multi-sentence excerpts can fool
expert human raters over 30% of the time. Our findings reveal the importance of
using both human and automatic detectors to assess the humanness of text
generation systems.


http://arxiv.org/abs/1911.00660
Security of Facial Forensics Models Against Adversarial Attacks.
Rong Huang; Fuming Fang; Huy H. Nguyen; Junichi Yamagishi; Isao Echizen
  Deep neural networks (DNNs) have been used in forensics to identify fake
facial images. We investigated several DNN-based forgery forensics models
(FFMs) to determine whether they are secure against adversarial attacks. We
experimentally demonstrated the existence of individual adversarial
perturbations (IAPs) and universal adversarial perturbations (UAPs) that can
lead a well-performed FFM to misbehave. Based on iterative procedure, gradient
information is used to generate two kinds of IAPs that can be used to fabricate
classification and segmentation outputs. In contrast, UAPs are generated on the
basis of over-firing. We designed a new objective function that encourages
neurons to over-fire, which makes UAP generation feasible even without using
training data. Experiments demonstrated the transferability of UAPs across
unseen datasets and unseen FFMs. Moreover, we are the first to conduct
subjective assessment for imperceptibility of the adversarial perturbations,
revealing that the crafted UAPs are visually negligible. There findings provide
a baseline for evaluating the adversarial security of FFMs.


http://arxiv.org/abs/1910.14655
Enhancing Certifiable Robustness via a Deep Model Ensemble.
Huan Zhang; Minhao Cheng; Cho-Jui Hsieh
  We propose an algorithm to enhance certified robustness of a deep model
ensemble by optimally weighting each base model. Unlike previous works on using
ensembles to empirically improve robustness, our algorithm is based on
optimizing a guaranteed robustness certificate of neural networks. Our proposed
ensemble framework with certified robustness, RobBoost, formulates the optimal
model selection and weighting task as an optimization problem on a lower bound
of classification margin, which can be efficiently solved using coordinate
descent. Experiments show that our algorithm can form a more robust ensemble
than naively averaging all available models using robustly trained MNIST or
CIFAR base models. Additionally, our ensemble typically has better accuracy on
clean (unperturbed) data. RobBoost allows us to further improve certified
robustness and clean accuracy by creating an ensemble of already certified
models.


http://arxiv.org/abs/1910.14356
Certifiable Robustness to Graph Perturbations.
Aleksandar Bojchevski; Stephan Günnemann
  Despite the exploding interest in graph neural networks there has been little
effort to verify and improve their robustness. This is even more alarming given
recent findings showing that they are extremely vulnerable to adversarial
attacks on both the graph structure and the node attributes. We propose the
first method for verifying certifiable (non-)robustness to graph perturbations
for a general class of models that includes graph neural networks and
label/feature propagation. By exploiting connections to PageRank and Markov
decision processes our certificates can be efficiently (and under many threat
models exactly) computed. Furthermore, we investigate robust training
procedures that increase the number of certifiably robust nodes while
maintaining or improving the clean predictive accuracy.


http://arxiv.org/abs/1911.00126
Adversarial Music: Real World Audio Adversary Against Wake-word Detection System.
Juncheng B. Li; Shuhui Qu; Xinjian Li; Joseph Szurley; J. Zico Kolter; Florian Metze
  Voice Assistants (VAs) such as Amazon Alexa or Google Assistant rely on
wake-word detection to respond to people's commands, which could potentially be
vulnerable to audio adversarial examples. In this work, we target our attack on
the wake-word detection system, jamming the model with some inconspicuous
background music to deactivate the VAs while our audio adversary is present. We
implemented an emulated wake-word detection system of Amazon Alexa based on
recent publications. We validated our models against the real Alexa in terms of
wake-word detection accuracy. Then we computed our audio adversaries with
consideration of expectation over transform and we implemented our audio
adversary with a differentiable synthesizer. Next, we verified our audio
adversaries digitally on hundreds of samples of utterances collected from the
real world. Our experiments show that we can effectively reduce the recognition
F1 score of our emulated model from 93.4% to 11.0%. Finally, we tested our
audio adversary over the air, and verified it works effectively against Alexa,
reducing its F1 score from 92.5% to 11.0%.; We also verified that
non-adversarial music does not disable Alexa as effectively as our music at the
same sound level. To the best of our knowledge, this is the first real-world
adversarial attack against a commercial-grade VA wake-word detection system.
Our code and demo videos can be accessed at
\url{https://www.junchengbillyli.com/AdversarialMusic}


http://arxiv.org/abs/1910.14184
Universal Adversarial Perturbations Against Person Re-Identification.
Wenjie Ding; Xing Wei; Xiaopeng Hong; Rongrong Ji; Yihong Gong
  Person re-identification (re-ID) has made great progress and achieved high
performance in recent years with the development of deep learning. However, as
an application related to security issues, there are few researches considering
the safety of person re-ID systems. In this paper, we attempt to explore the
robustness of current person re-ID models against adversarial samples.
Specifically, we attack the re-ID models using universal adversarial
perturbations (UAPs), which are especially dangerous to the surveillance
systems because it could fool most pedestrian images with a little overhead.
Existing methods for UAPs mainly consider classification models, while the
tasks in open set scenarios like re-ID are rarely explored. Re-ID attack is
different from classification ones in the sense that the former discards
decision boundary during test and cares more about the ranking list. Therefore,
we propose an effective method to train UAPs against person re-ID models from
the global list-wise perspective. Furthermore, to increase the impact of attack
to different models and datasets, we propose a novel UAPs learning method based
on total variation minimization. Extensive experiments validate the
effectiveness of our proposed method.


http://arxiv.org/abs/1910.14107
Investigating Resistance of Deep Learning-based IDS against Adversaries using min-max Optimization.
Rana Abou Khamis; Omair Shafiq; Ashraf Matrawy
  With the growth of adversarial attacks against machine learning models,
several concerns have emerged about potential vulnerabilities in designing deep
neural network-based intrusion detection systems (IDS). In this paper, we study
the resilience of deep learning-based intrusion detection systems against
adversarial attacks. We apply the min-max (or saddle-point) approach to train
intrusion detection systems against adversarial attack samples in NSW-NB 15
dataset. We have the max approach for generating adversarial samples that
achieves maximum loss and attack deep neural networks. On the other side, we
utilize the existing min approach [2] [9] as a defense strategy to optimize
intrusion detection systems that minimize the loss of the incorporated
adversarial samples during the adversarial training. We study and measure the
effectiveness of the adversarial attack methods as well as the resistance of
the adversarially trained models against such attacks. We find that the
adversarial attack methods that were designed in binary domains can be used in
continuous domains and exhibit different misclassification levels. We finally
show that principal component analysis (PCA) based feature reduction can boost
the robustness in intrusion detection system (IDS) using a deep neural network
(DNN).


http://arxiv.org/abs/1910.13222
Adversarial Example in Remote Sensing Image Recognition.
Li Chen; Guowei Zhu; Qi Li; Haifeng Li
  With the wide application of remote sensing technology in various fields, the
accuracy and security requirements for remote sensing images (RSIs) recognition
are also increasing. In recent years, due to the rapid development of deep
learning in the field of image recognition, RSI recognition models based on
deep convolution neural networks (CNNs) outperform traditional hand-craft
feature techniques. However, CNNs also pose security issues when they show
their capability of accurate classification. By adding a very small variation
of the adversarial perturbation to the input image, the CNN model can be caused
to produce erroneous results with extremely high confidence, and the
modification of the image is not perceived by the human eye. This added
adversarial perturbation image is called an adversarial example, which poses a
serious security problem for systems based on CNN model recognition results.
This paper, for the first time, analyzes adversarial example problem of RSI
recognition under CNN models. In the experiments, we used different attack
algorithms to fool multiple high-accuracy RSI recognition models trained on
multiple RSI datasets. The results show that RSI recognition models are also
vulnerable to adversarial examples, and the models with different structures
trained on the same RSI dataset also have different vulnerabilities. For each
RSI dataset, the number of features also affects the vulnerability of the
model. Many features are good for defensive adversarial examples. Further, we
find that the attacked class of RSI has an attack selectivity property. The
misclassification of adversarial examples of the RSIs are related to the
similarity of the original classes in the CNN feature space. In addition,
adversarial examples in RSI recognition are of great significance for the
security of remote sensing applications, showing a huge potential for future
research.


http://arxiv.org/abs/1910.13025
Active Subspace of Neural Networks: Structural Analysis and Universal Attacks.
Chunfeng Cui; Kaiqi Zhang; Talgat Daulbaev; Julia Gusak; Ivan Oseledets; Zheng Zhang
  Active subspace is a model reduction method widely used in the uncertainty
quantification community. In this paper, we propose analyzing the internal
structure and vulnerability and deep neural networks using active subspace.
Firstly, we employ the active subspace to measure the number of "active
neurons" at each intermediate layer and reduce the number of neurons from
several thousands to several dozens. This motivates us to change the network
structure and to develop a new and more compact network, referred to as
{ASNet}, that has significantly fewer model parameters. Secondly, we propose
analyzing the vulnerability of a neural network using active subspace and
finding an additive universal adversarial attack vector that can misclassify a
dataset with a high probability. Our experiments on CIFAR-10 show that ASNet
can achieve 23.98$\times$ parameter and 7.30$\times$ flops reduction. The
universal active subspace attack vector can achieve around 20% higher attack
ratio compared with the existing approach in all of our numerical experiments.
The PyTorch codes for this paper are available online.


http://arxiv.org/abs/1910.12908
Certified Adversarial Robustness for Deep Reinforcement Learning.
Björn Lütjens; Michael Everett; Jonathan P. How
  Deep Neural Network-based systems are now the state-of-the-art in many
robotics tasks, but their application in safety-critical domains remains
dangerous without formal guarantees on network robustness. Small perturbations
to sensor inputs (from noise or adversarial examples) are often enough to
change network-based decisions, which was already shown to cause an autonomous
vehicle to swerve into oncoming traffic. In light of these dangers, numerous
algorithms have been developed as defensive mechanisms from these adversarial
inputs, some of which provide formal robustness guarantees or certificates.
This work leverages research on certified adversarial robustness to develop an
online certified defense for deep reinforcement learning algorithms. The
proposed defense computes guaranteed lower bounds on state-action values during
execution to identify and choose the optimal action under a worst-case
deviation in input space due to possible adversaries or noise. The approach is
demonstrated on a Deep Q-Network policy and is shown to increase robustness to
noise and adversaries in pedestrian collision avoidance scenarios and a classic
control task.


http://arxiv.org/abs/1910.12196
Open the Boxes of Words: Incorporating Sememes into Textual Adversarial Attack.
Yuan Zang; Chenghao Yang; Fanchao Qi; Zhiyuan Liu; Meng Zhang; Qun Liu; Maosong Sun
  Adversarial attack is carried out to reveal the vulnerability of deep neural
networks. Word substitution is a class of effective adversarial textual attack
method, which has been extensively explored. However, all existing studies
utilize word embeddings or thesauruses to find substitutes. In this paper, we
incorporate sememes, the minimum semantic units, into adversarial attack. We
propose an efficient sememe-based word substitution strategy and integrate it
into a genetic attack algorithm. In experiments, we employ our attack method to
attack LSTM and BERT on both Chinese and English sentiment analysis as well as
natural language inference benchmark datasets. Experimental results demonstrate
our model achieves better attack success rates and less modification than the
baseline methods based on word embedding or synonym. Furthermore, we find our
attack model can bring more robustness enhancement to the target model with
adversarial training.


http://arxiv.org/abs/1910.12227
EdgeFool: An Adversarial Image Enhancement Filter.
Ali Shahin Shamsabadi; Changjae Oh; Andrea Cavallaro
  Adversarial examples are intentionally perturbed images that mislead
classifiers. These images can, however, be easily detected using denoising
algorithms, when high-frequency spatial perturbations are used, or can be
noticed by humans, when perturbations are large. In this paper, we propose
EdgeFool, an adversarial image enhancement filter that learns structure-aware
adversarial perturbations. EdgeFool generates adversarial images with
perturbations that enhance image details via training a fully convolutional
neural network end-to-end with a multi-task loss function. This loss function
accounts for both image detail enhancement and class misleading objectives. We
evaluate EdgeFool on three classifiers (ResNet-50, ResNet-18 and AlexNet) using
two datasets (ImageNet and Private-Places365) and compare it with six
adversarial methods (DeepFool, SparseFool, Carlini-Wagner, SemanticAdv,
Non-targeted and Private Fast Gradient Sign Methods). Code is available at
https://github.com/smartcameras/EdgeFool.git.


http://arxiv.org/abs/1911.00927
Spot Evasion Attacks: Adversarial Examples for License Plate Recognition Systems with Convolutional Neural Networks.
Ya-guan Qian; Dan-feng Ma; Bin Wang; Jun Pan; Jia-min Wang; Jian-hai Chen; Wu-jie Zhou; Jing-sheng Lei
  Recent studies have shown convolution neural networks (CNNs) for image
recognition are vulnerable to evasion attacks with carefully manipulated
adversarial examples. Previous work primarily focused on how to generate
adversarial examples closed to source images, by introducing pixel-level
perturbations into the whole or specific part of images. In this paper, we
propose an evasion attack on CNN classifiers in the context of License Plate
Recognition (LPR), which adds predetermined perturbations to specific regions
of license plate images, simulating some sort of naturally formed spots (such
as sludge, etc.). Therefore, the problem is modeled as an optimization process
searching for optimal perturbation positions, which is different from previous
work that consider pixel values as decision variables. Notice that this is a
complex nonlinear optimization problem, and we use a genetic-algorithm based
approach to obtain optimal perturbation positions. In experiments, we use the
proposed algorithm to generate various adversarial examples in the form of
rectangle, circle, ellipse and spots cluster. Experimental results show that
these adversarial examples are almost ignored by human eyes, but can fool
HyperLPR with high attack success rate over 93%. Therefore, we believe that
this kind of spot evasion attacks would pose a great threat to current LPR
systems, and needs to be investigated further by the security community.


http://arxiv.org/abs/1910.12084
Detection of Adversarial Attacks and Characterization of Adversarial Subspace.
Mohammad Esmaeilpour; Patrick Cardinal; Alessandro Lameiras Koerich
  Adversarial attacks have always been a serious threat for any data-driven
model. In this paper, we explore subspaces of adversarial examples in unitary
vector domain, and we propose a novel detector for defending our models trained
for environmental sound classification. We measure chordal distance between
legitimate and malicious representation of sounds in unitary space of
generalized Schur decomposition and show that their manifolds lie far from each
other. Our front-end detector is a regularized logistic regression which
discriminates eigenvalues of legitimate and adversarial spectrograms. The
experimental results on three benchmarking datasets of environmental sounds
represented by spectrograms reveal high detection rate of the proposed detector
for eight types of adversarial attacks and outperforms other detection
approaches.


http://arxiv.org/abs/1910.12163
Understanding and Quantifying Adversarial Examples Existence in Linear Classification.
Xupeng Shi; A. Adam Ding
  State-of-art deep neural networks (DNN) are vulnerable to attacks by
adversarial examples: a carefully designed small perturbation to the input,
that is imperceptible to human, can mislead DNN. To understand the root cause
of adversarial examples, we quantify the probability of adversarial example
existence for linear classifiers. Previous mathematical definition of
adversarial examples only involves the overall perturbation amount, and we
propose a more practical relevant definition of strong adversarial examples
that separately limits the perturbation along the signal direction also. We
show that linear classifiers can be made robust to strong adversarial examples
attack in cases where no adversarial robust linear classifiers exist under the
previous definition. The quantitative formulas are confirmed by numerical
experiments using a linear support vector machine (SVM) classifier. The results
suggest that designing general strong-adversarial-robust learning systems is
feasible but only through incorporating human knowledge of the underlying
classification problem.


http://arxiv.org/abs/1910.12165
Adversarial Defense Via Local Flatness Regularization.
Jia Xu; Yiming Li; Yong Jiang; Shu-Tao Xia
  Adversarial defense is a popular and important research area. Due to its
intrinsic mechanism, one of the most straightforward and effective ways of
defending attacks is to analyze the property of loss surface in the input
space. In this paper, we define the local flatness of the loss surface as the
maximum value of the chosen norm of the gradient regarding to the input within
a neighborhood centered on the benign sample, and discuss the relationship
between the local flatness and adversarial vulnerability. Based on the
analysis, we propose a novel defense approach via regularizing the local
flatness, dubbed local flatness regularization (LFR). We also demonstrate the
effectiveness of the proposed method from other perspectives, such as human
visual mechanism, and analyze the relationship between LFR and other related
methods theoretically. Experiments are conducted to verify our theory and
demonstrate the superiority of the proposed method.


http://arxiv.org/abs/1910.12392
Effectiveness of random deep feature selection for securing image manipulation detectors against adversarial examples.
Mauro Barni; Ehsan Nowroozi; Benedetta Tondi; Bowen Zhang
  We investigate if the random feature selection approach proposed in [1] to
improve the robustness of forensic detectors to targeted attacks, can be
extended to detectors based on deep learning features. In particular, we study
the transferability of adversarial examples targeting an original CNN image
manipulation detector to other detectors (a fully connected neural network and
a linear SVM) that rely on a random subset of the features extracted from the
flatten layer of the original network. The results we got by considering three
image manipulation detection tasks (resizing, median filtering and adaptive
histogram equalization), two original network architectures and three classes
of attacks, show that feature randomization helps to hinder attack
transferability, even if, in some cases, simply changing the architecture of
the detector, or even retraining the detector is enough to prevent the
transferability of the attacks.


http://arxiv.org/abs/1910.11603
MediaEval 2019: Concealed FGSM Perturbations for Privacy Preservation.
Panagiotis Linardos; Suzanne Little; Kevin McGuinness
  This work tackles the Pixel Privacy task put forth by MediaEval 2019. Our
goal is to manipulate images in a way that conceals them from automatic scene
classifiers while preserving the original image quality. We use the fast
gradient sign method, which normally has a corrupting influence on image
appeal, and devise two methods to minimize the damage. The first approach uses
a map of pixel locations that are either salient or flat, and directs
perturbations away from them. The second approach subtracts the gradient of an
aesthetics evaluation model from the gradient of the attack model to guide the
perturbations towards a direction that preserves appeal. We make our code
available at: https://git.io/JesXr.


http://arxiv.org/abs/1910.11585
Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?
Ali Shafahi; Amin Ghiasi; Furong Huang; Tom Goldstein
  Adversarial training is one of the strongest defenses against adversarial
attacks, but it requires adversarial examples to be generated for every
mini-batch during optimization. The expense of producing these examples during
training often precludes adversarial training from use on complex image
datasets. In this study, we explore the mechanisms by which adversarial
training improves classifier robustness, and show that these mechanisms can be
effectively mimicked using simple regularization methods, including label
smoothing and logit squeezing. Remarkably, using these simple regularization
methods in combination with Gaussian noise injection, we are able to achieve
strong adversarial robustness -- often exceeding that of adversarial training
-- using no adversarial examples.


http://arxiv.org/abs/1910.10994
ATZSL: Defensive Zero-Shot Recognition in the Presence of Adversaries.
Xingxing Zhang; Shupeng Gui; Zhenfeng Zhu; Yao Zhao; Ji Liu
  Zero-shot learning (ZSL) has received extensive attention recently especially
in areas of fine-grained object recognition, retrieval, and image captioning.
Due to the complete lack of training samples and high requirement of defense
transferability, the ZSL model learned is particularly vulnerable against
adversarial attacks. Recent work also showed adversarially robust
generalization requires more data. This may significantly affect the robustness
of ZSL. However, very few efforts have been devoted towards this direction. In
this paper, we take an initial attempt, and propose a generic formulation to
provide a systematical solution (named ATZSL) for learning a robust ZSL model.
It is capable of achieving better generalization on various adversarial objects
recognition while only losing a negligible performance on clean images for
unseen classes, by casting ZSL into a min-max optimization problem. To address
it, we design a defensive relation prediction network, which can bridge the
seen and unseen class domains via attributes to generalize prediction and
defense strategy. Additionally, our framework can be extended to deal with the
poisoned scenario of unseen class attributes. An extensive group of experiments
are then presented, demonstrating that ATZSL obtains remarkably more favorable
trade-off between model transferability and robustness, over currently
available alternatives under various settings.


http://arxiv.org/abs/1910.10679
A Useful Taxonomy for Adversarial Robustness of Neural Networks.
Leslie N. Smith
  Adversarial attacks and defenses are currently active areas of research for
the deep learning community. A recent review paper divided the defense
approaches into three categories; gradient masking, robust optimization, and
adversarial example detection. We divide gradient masking and robust
optimization differently: (1) increasing intra-class compactness and
inter-class separation of the feature vectors improves adversarial robustness,
and (2) marginalization or removal of non-robust image features also improves
adversarial robustness. By reframing these topics differently, we provide a
fresh perspective that provides insight into the underlying factors that enable
training more robust networks and can help inspire novel solutions. In
addition, there are several papers in the literature of adversarial defenses
that claim there is a cost for adversarial robustness, or a trade-off between
robustness and accuracy but, under this proposed taxonomy, we hypothesis that
this is not universal. We follow up on our taxonomy with several challenges to
the deep learning research community that builds on the connections and
insights in this paper.


http://arxiv.org/abs/1910.10783
Wasserstein Smoothing: Certified Robustness against Wasserstein Adversarial Attacks.
Alexander Levine; Soheil Feizi
  In the last couple of years, several adversarial attack methods based on
different threat models have been proposed for the image classification
problem. Most existing defenses consider additive threat models in which sample
perturbations have bounded L_p norms. These defenses, however, can be
vulnerable against adversarial attacks under non-additive threat models. An
example of an attack method based on a non-additive threat model is the
Wasserstein adversarial attack proposed by Wong et al. (2019), where the
distance between an image and its adversarial example is determined by the
Wasserstein metric ("earth-mover distance") between their normalized pixel
intensities. Until now, there has been no certifiable defense against this type
of attack. In this work, we propose the first defense with certified robustness
against Wasserstein Adversarial attacks using randomized smoothing. We develop
this certificate by considering the space of possible flows between images, and
representing this space such that Wasserstein distance between images is
upper-bounded by L_1 distance in this flow-space. We can then apply existing
randomized smoothing certificates for the L_1 metric. In MNIST and CIFAR-10
datasets, we find that our proposed defense is also practically effective,
demonstrating significantly improved accuracy under Wasserstein adversarial
attack compared to unprotected models.


http://arxiv.org/abs/1910.10053
Attacking Optical Flow.
Anurag Ranjan; Joel Janai; Andreas Geiger; Michael J. Black
  Deep neural nets achieve state-of-the-art performance on the problem of
optical flow estimation. Since optical flow is used in several safety-critical
applications like self-driving cars, it is important to gain insights into the
robustness of those techniques. Recently, it has been shown that adversarial
attacks easily fool deep neural networks to misclassify objects. The robustness
of optical flow networks to adversarial attacks, however, has not been studied
so far. In this paper, we extend adversarial patch attacks to optical flow
networks and show that such attacks can compromise their performance. We show
that corrupting a small patch of less than 1% of the image size can
significantly affect optical flow estimates. Our attacks lead to noisy flow
estimates that extend significantly beyond the region of the attack, in many
cases even completely erasing the motion of objects in the scene. While
networks using an encoder-decoder architecture are very sensitive to these
attacks, we found that networks using a spatial pyramid architecture are less
affected. We analyse the success and failure of attacking both architectures by
visualizing their feature maps and comparing them to classical optical flow
techniques which are robust to these attacks. We also demonstrate that such
attacks are practical by placing a printed pattern into real scenes.


http://arxiv.org/abs/1910.10013
Adversarial Example Detection by Classification for Deep Speech Recognition.
Saeid Samizade; Zheng-Hua Tan; Chao Shen; Xiaohong Guan
  Machine Learning systems are vulnerable to adversarial attacks and will
highly likely produce incorrect outputs under these attacks. There are
white-box and black-box attacks regarding to adversary's access level to the
victim learning algorithm. To defend the learning systems from these attacks,
existing methods in the speech domain focus on modifying input signals and
testing the behaviours of speech recognizers. We, however, formulate the
defense as a classification problem and present a strategy for systematically
generating adversarial example datasets: one for white-box attacks and one for
black-box attacks, containing both adversarial and normal examples. The
white-box attack is a gradient-based method on Baidu DeepSpeech with the
Mozilla Common Voice database while the black-box attack is a gradient-free
method on a deep model-based keyword spotting system with the Google Speech
Command dataset. The generated datasets are used to train a proposed
Convolutional Neural Network (CNN), together with cepstral features, to detect
adversarial examples. Experimental results show that, it is possible to
accurately distinct between adversarial and normal examples for known attacks,
in both single-condition and multi-condition training settings, while the
performance degrades dramatically for unknown attacks. The adversarial datasets
and the source code are made publicly available.


http://arxiv.org/abs/1910.10106
Cross-Representation Transferability of Adversarial Attacks: From Spectrograms to Audio Waveforms.
Karl M. Koerich; Mohammad Esmailpour; Sajjad Abdoli; Alceu S. Jr. Britto; Alessandro L. Koerich
  This paper shows the susceptibility of spectrogram-based audio classifiers to
adversarial attacks and the transferability of such attacks to audio waveforms.
Some commonly used adversarial attacks to images have been applied to
Mel-frequency and short-time Fourier transform spectrograms, and such perturbed
spectrograms are able to fool a 2D convolutional neural network (CNN). Such
attacks produce perturbed spectrograms that are visually imperceptible by
humans. Furthermore, the audio waveforms reconstructed from the perturbed
spectrograms are also able to fool a 1D CNN trained on the original audio.
Experimental results on a dataset of western music have shown that the 2D CNN
achieves up to 81.87% of mean accuracy on legitimate examples and such
performance drops to 12.09% on adversarial examples. Likewise, the 1D CNN
achieves up to 78.29% of mean accuracy on original audio samples and such
performance drops to 27.91% on adversarial audio waveforms reconstructed from
the perturbed spectrograms.


http://arxiv.org/abs/1910.09821
Structure Matters: Towards Generating Transferable Adversarial Images.
Dan Peng; Zizhan Zheng; Linhao Luo; Xiaofeng Zhang
  Recent works on adversarial examples for image classification focus on
directly modifying pixels with minor perturbations. The small perturbation
requirement is imposed to ensure the generated adversarial examples being
natural and realistic to humans, which, however, puts a curb on the attack
space thus limiting the attack ability and transferability especially for
systems protected by a defense mechanism. In this paper, we propose the novel
concepts of structure patterns and structure-aware perturbations that relax the
small perturbation constraint while still keeping images natural. The key idea
of our approach is to allow perceptible deviation in adversarial examples while
keeping structure patterns that are central to a human classifier. Built upon
these concepts, we propose a \emph{structure-preserving attack (SPA)} for
generating natural adversarial examples with extremely high transferability.
Empirical results on the MNIST and the CIFAR10 datasets show that SPA exhibits
strong attack ability in both the white-box and black-box setting even defenses
are applied. Moreover, with the integration of PGD or CW attack, its attack
ability escalates sharply under the white-box setting, without losing the
outstanding transferability inherited from SPA.


http://arxiv.org/abs/1910.09239
Recovering Localized Adversarial Attacks.
Jan Philip Göpfert; Heiko Wersing; Barbara Hammer
  Deep convolutional neural networks have achieved great successes over recent
years, particularly in the domain of computer vision. They are fast,
convenient, and -- thanks to mature frameworks -- relatively easy to implement
and deploy. However, their reasoning is hidden inside a black box, in spite of
a number of proposed approaches that try to provide human-understandable
explanations for the predictions of neural networks. It is still a matter of
debate which of these explainers are best suited for which situations, and how
to quantitatively evaluate and compare them. In this contribution, we focus on
the capabilities of explainers for convolutional deep neural networks in an
extreme situation: a setting in which humans and networks fundamentally
disagree. Deep neural networks are susceptible to adversarial attacks that
deliberately modify input samples to mislead a neural network's classification,
without affecting how a human observer interprets the input. Our goal with this
contribution is to evaluate explainers by investigating whether they can
identify adversarially attacked regions of an image. In particular, we
quantitatively and qualitatively investigate the capability of three popular
explainers of classifications -- classic salience, guided backpropagation, and
LIME -- with respect to their ability to identify regions of attack as the
explanatory regions for the (incorrect) prediction in representative examples
from image classification. We find that LIME outperforms the other explainers.


http://arxiv.org/abs/1910.09464
Learning to Learn by Zeroth-Order Oracle.
Yangjun Ruan; Yuanhao Xiong; Sashank Reddi; Sanjiv Kumar; Cho-Jui Hsieh
  In the learning to learn (L2L) framework, we cast the design of optimization
algorithms as a machine learning problem and use deep neural networks to learn
the update rules. In this paper, we extend the L2L framework to zeroth-order
(ZO) optimization setting, where no explicit gradient information is available.
Our learned optimizer, modeled as recurrent neural network (RNN), first
approximates gradient by ZO gradient estimator and then produces parameter
update utilizing the knowledge of previous iterations. To reduce high variance
effect due to ZO gradient estimator, we further introduce another RNN to learn
the Gaussian sampling rule and dynamically guide the query direction sampling.
Our learned optimizer outperforms hand-designed algorithms in terms of
convergence rate and final solution on both synthetic and practical ZO
optimization tasks (in particular, the black-box adversarial attack task, which
is one of the most widely used tasks of ZO optimization). We finally conduct
extensive analytical experiments to demonstrate the effectiveness of our
proposed optimizer.


http://arxiv.org/abs/1910.09338
An Alternative Surrogate Loss for PGD-based Adversarial Testing.
Sven Gowal; Jonathan Uesato; Chongli Qin; Po-Sen Huang; Timothy Mann; Pushmeet Kohli
  Adversarial testing methods based on Projected Gradient Descent (PGD) are
widely used for searching norm-bounded perturbations that cause the inputs of
neural networks to be misclassified. This paper takes a deeper look at these
methods and explains the effect of different hyperparameters (i.e., optimizer,
step size and surrogate loss). We introduce the concept of MultiTargeted
testing, which makes clever use of alternative surrogate losses, and explain
when and how MultiTargeted is guaranteed to find optimal perturbations.
Finally, we demonstrate that MultiTargeted outperforms more sophisticated
methods and often requires less iterative steps than other variants of PGD
found in the literature. Notably, MultiTargeted ranks first on MadryLab's
white-box MNIST and CIFAR-10 leaderboards, reducing the accuracy of their MNIST
model to 88.36% (with $\ell_\infty$ perturbations of $\epsilon = 0.3$) and the
accuracy of their CIFAR-10 model to 44.03% (at $\epsilon = 8/255$).
MultiTargeted also ranks first on the TRADES leaderboard reducing the accuracy
of their CIFAR-10 model to 53.07% (with $\ell_\infty$ perturbations of
$\epsilon = 0.031$).


http://arxiv.org/abs/1910.08910
Enhancing Recurrent Neural Networks with Sememes.
Yujia Qin; Fanchao Qi; Sicong Ouyang; Zhiyuan Liu; Cheng Yang; Yasheng Wang; Qun Liu; Maosong Sun
  Sememes, the minimum semantic units of human languages, have been
successfully utilized in various natural language processing applications.
However, most existing studies exploit sememes in specific tasks and few
efforts are made to utilize sememes more fundamentally. In this paper, we
propose to incorporate sememes into recurrent neural networks (RNNs) to improve
their sequence modeling ability, which is beneficial to all kinds of downstream
tasks. We design three different sememe incorporation methods and employ them
in typical RNNs including LSTM, GRU and their bidirectional variants. For
evaluation, we use several benchmark datasets involving PTB and WikiText-2 for
language modeling, SNLI for natural language inference. Experimental results
show evident and consistent improvement of our sememe-incorporated models
compared with vanilla RNNs, which proves the effectiveness of our sememe
incorporation methods. Moreover, we find the sememe-incorporated models have
great robustness and outperform adversarial training in defending adversarial
attack. All the code and data of this work will be made available to the
public.


http://arxiv.org/abs/1910.08716
Adversarial Attacks on Spoofing Countermeasures of automatic speaker verification.
Songxiang Liu; Haibin Wu; Hung-yi Lee; Helen Meng
  High-performance spoofing countermeasure systems for automatic speaker
verification (ASV) have been proposed in the ASVspoof 2019 challenge. However,
the robustness of such systems under adversarial attacks has not been studied
yet. In this paper, we investigate the vulnerability of spoofing
countermeasures for ASV under both white-box and black-box adversarial attacks
with the fast gradient sign method (FGSM) and the projected gradient descent
(PGD) method. We implement high-performing countermeasure models in the
ASVspoof 2019 challenge and conduct adversarial attacks on them. We compare
performance of black-box attacks across spoofing countermeasure models with
different network architectures and different amount of model parameters. The
experimental results show that all implemented countermeasure models are
vulnerable to FGSM and PGD attacks under the scenario of white-box attack. The
more dangerous black-box attacks also prove to be effective by the experimental
results.


http://arxiv.org/abs/1910.08640
Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?
Simran Kaur; Jeremy Cohen; Zachary C. Lipton
  For a standard convolutional neural network, optimizing over the input pixels
to maximize the score of some target class will generally produce a
grainy-looking version of the original image. However, researchers have
demonstrated that for adversarially-trained neural networks, this optimization
produces images that uncannily resemble the target class. In this paper, we
show that these "perceptually-aligned gradients" also occur under randomized
smoothing, an alternative means of constructing adversarially-robust
classifiers. Our finding suggests that perceptually-aligned gradients may be a
general property of robust classifiers, rather than a specific property of
adversarially-trained neural networks. We hope that our results will inspire
research aimed at explaining this link between perceptually-aligned gradients
and adversarial robustness.


http://arxiv.org/abs/1910.08681
Spatial-aware Online Adversarial Perturbations Against Visual Object Tracking.
Qing Guo; Xiaofei Xie; Lei Ma; Zhongguo Li; Wei Feng; Yang Liu
  Adversarial attacks of deep neural networks have been intensively studied on
image, audio, natural language, patch, and pixel classification tasks.
Nevertheless, as a typical, while important real-world application, the
adversarial attacks of online video object tracking that traces an object's
moving trajectory instead of its category are rarely explored. In this paper,
we identify a new task for the adversarial attack to visual object tracking:
online generating imperceptible perturbations that mislead trackers along an
incorrect (Untargeted Attack, UA) or specified trajectory (Targeted Attack,
TA). To this end, we first propose a spatial-aware basic attack by adapting
existing attack methods, i.e., FGSM, BIM, and C\&W, and comprehensively analyze
the attacking performance. We identify that online object tracking poses two
new challenges: 1) it is difficult to generate imperceptible perturbations that
can transfer across time/frames, and 2) real-time trackers require the attack
to satisfy a certain level of efficiency. To address these challenges, we
further propose the online incremental attack (OIA) that performs
spatial-temporal sparse incremental perturbations online and makes the
adversarial attack less perceptible. In addition, as an optimization-based
method, OIA quickly converges to very small losses within several iterations by
considering historical incremental perturbations, making it much more efficient
than the basic attacks. The in-depth evaluation on the state-of-the-art
trackers (i.e., SiamRPN with Alex, MobileNetv2, and ResNet-50) for OTB100 and
VOT2018 demonstrates the effectiveness and transferability of OIA in misleading
existing trackers under both UA and TA with minor perturbations.


http://arxiv.org/abs/1910.08623
A Fast Saddle-Point Dynamical System Approach to Robust Deep Learning.
Yasaman Esfandiari; Aditya Balu; Keivan Ebrahimi; Umesh Vaidya; Nicola Elia; Soumik Sarkar
  Recent focus on robustness to adversarial attacks for deep neural networks
produced a large variety of algorithms for training robust models. Most of the
effective algorithms involve solving the min-max optimization problem for
training robust models (min step) under worst-case attacks (max step). However,
they often suffer from high computational cost from running several inner
maximization iterations (to find an optimal attack) inside every outer
minimization iteration. Therefore, it becomes difficult to readily apply such
algorithms for moderate to large size real world data sets. To alleviate this,
we explore the effectiveness of iterative descent-ascent algorithms where the
maximization and minimization steps are executed in an alternate fashion to
simultaneously obtain the worst-case attack and the corresponding robust model.
Specifically, we propose a novel discrete-time dynamical system-based algorithm
that aims to find the saddle point of a min-max optimization problem in the
presence of uncertainties. Under the assumptions that the cost function is
convex and uncertainties enter concavely in the robust learning problem, we
analytically show that our algorithm converges asymptotically to the robust
optimal solution under a general adversarial budget constraints as induced by
$\ell_p$ norm, for $1\leq p\leq \infty$. Based on our proposed analysis, we
devise a fast robust training algorithm for deep neural networks. Although such
training involves highly non-convex robust optimization problems, empirical
results show that the algorithm can achieve significant robustness compared to
other state-of-the-art robust models on benchmark data sets.


http://arxiv.org/abs/1910.11099
Evading Real-Time Person Detectors by Adversarial T-shirt.
Kaidi Xu; Gaoyuan Zhang; Sijia Liu; Quanfu Fan; Mengshu Sun; Hongge Chen; Pin-Yu Chen; Yanzhi Wang; Xue Lin
  It is known that deep neural networks (DNNs) could be vulnerable to
adversarial attacks. The so-called physical adversarial examples deceive
DNN-based decision makers by attaching adversarial patches to real objects.
However, most of the existing works on physical adversarial attacks focus on
static objects such as glass frame, stop sign and image attached to a
cardboard. In this work, we proposed adversarial T-shirt, a robust physical
adversarial example for evading person detectors even if it suffers from
deformation due toa moving person's pose change. To the best of our knowledge,
the effect of deformation is first modeled for designing physical adversarial
examples with respect to non-rigid objects such as T-shirts. We show that the
proposed method achieves 79% and 63% attack success rates in digital and
physical worlds respectively against YOLOv2. In contrast, the state-of-the-art
physical attack method to fool a person detector only achieves 27% attack
success rate. Furthermore, by leveraging min-max optimization, we extend our
method to the ensemble attack setting against object detectors YOLOv2 and
Faster R-CNN simultaneously.


http://arxiv.org/abs/1910.08051
Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets.
Yogesh Balaji; Tom Goldstein; Judy Hoffman
  Adversarial training is by far the most successful strategy for improving
robustness of neural networks to adversarial attacks. Despite its success as a
defense mechanism, adversarial training fails to generalize well to unperturbed
test set. We hypothesize that this poor generalization is a consequence of
adversarial training with uniform perturbation radius around every training
sample. Samples close to decision boundary can be morphed into a different
class under a small perturbation budget, and enforcing large margins around
these samples produce poor decision boundaries that generalize poorly.
Motivated by this hypothesis, we propose instance adaptive adversarial training
-- a technique that enforces sample-specific perturbation margins around every
training sample. We show that using our approach, test accuracy on unperturbed
samples improve with a marginal drop in robustness. Extensive experiments on
CIFAR-10, CIFAR-100 and Imagenet datasets demonstrate the effectiveness of our
proposed approach.


http://arxiv.org/abs/1910.08108
Enforcing Linearity in DNN succours Robustness and Adversarial Image Generation.
Anindya Sarkar; Nikhil Kumar Gupta; Raghu Iyengar
  Recent studies on the adversarial vulnerability of neural networks have shown
that models trained with the objective of minimizing an upper bound on the
worst-case loss over all possible adversarial perturbations improve robustness
against adversarial attacks. Beside exploiting adversarial training framework,
we show that by enforcing a Deep Neural Network (DNN) to be linear in
transformed input and feature space improves robustness significantly. We also
demonstrate that by augmenting the objective function with Local Lipschitz
regularizer boost robustness of the model further. Our method outperforms most
sophisticated adversarial training methods and achieves state of the art
adversarial accuracy on MNIST, CIFAR10 and SVHN dataset. In this paper, we also
propose a novel adversarial image generation method by leveraging Inverse
Representation Learning and Linearity aspect of an adversarially trained deep
neural network classifier.


http://arxiv.org/abs/1910.08536
LanCe: A Comprehensive and Lightweight CNN Defense Methodology against Physical Adversarial Attacks on Embedded Multimedia Applications.
Zirui Xu; Fuxun Yu; Xiang Chen
  Recently, adversarial attacks can be applied to the physical world, causing
practical issues to various Convolutional Neural Networks (CNNs) powered
applications. Most existing physical adversarial attack defense works only
focus on eliminating explicit perturbation patterns from inputs, ignoring
interpretation to CNN's intrinsic vulnerability. Therefore, they lack the
expected versatility to different attacks and thereby depend on considerable
data processing costs. In this paper, we propose LanCe -- a comprehensive and
lightweight CNN defense methodology against different physical adversarial
attacks. By interpreting CNN's vulnerability, we find that non-semantic
adversarial perturbations can activate CNN with significantly abnormal
activations and even overwhelm other semantic input patterns' activations. We
improve the CNN recognition process by adding a self-verification stage to
detect the potential adversarial input with only one CNN inference cost. Based
on the detection result, we further propose a data recovery methodology to
defend the physical adversarial attacks. We apply such defense methodology into
both image and audio CNN recognition scenarios and analyze the computational
complexity for each scenario, respectively. Experiments show that our
methodology can achieve an average 91% successful rate for attack detection and
89% accuracy recovery. Moreover, it is at most 3x faster compared with the
state-of-the-art defense methods, making it feasible to resource-constrained
embedded systems, such as mobile devices.


http://arxiv.org/abs/1910.07629
A New Defense Against Adversarial Images: Turning a Weakness into a Strength.
Tao Yu; Shengyuan Hu; Chuan Guo; Wei-Lun Chao; Kilian Q. Weinberger
  Natural images are virtually surrounded by low-density misclassified regions
that can be efficiently discovered by gradient-guided search --- enabling the
generation of adversarial images. While many techniques for detecting these
attacks have been proposed, they are easily bypassed when the adversary has
full knowledge of the detection mechanism and adapts the attack strategy
accordingly. In this paper, we adopt a novel perspective and regard the
omnipresence of adversarial perturbations as a strength rather than a weakness.
We postulate that if an image has been tampered with, these adversarial
directions either become harder to find with gradient methods or have
substantially higher density than for natural images. We develop a practical
test for this signature characteristic to successfully detect adversarial
attacks, achieving unprecedented accuracy under the white-box setting where the
adversary is given full knowledge of our detection mechanism.


http://arxiv.org/abs/1910.06813
Improving Robustness of time series classifier with Neural ODE guided gradient based data augmentation.
Anindya Sarkar; Anirudh Sunder Raj; Raghu Sesha Iyengar
  Exploring adversarial attack vectors and studying their effects on machine
learning algorithms has been of interest to researchers. Deep neural networks
working with time series data have received lesser interest compared to their
image counterparts in this context. In a recent finding, it has been revealed
that current state-of-the-art deep learning time series classifiers are
vulnerable to adversarial attacks. In this paper, we introduce two local
gradient based and one spectral density based time series data augmentation
techniques. We show that a model trained with data obtained using our
techniques obtains state-of-the-art classification accuracy on various time
series benchmarks. In addition, it improves the robustness of the model against
some of the most common corruption techniques,such as Fast Gradient Sign Method
(FGSM) and Basic Iterative Method (BIM).


http://arxiv.org/abs/1910.07416
Understanding Misclassifications by Attributes.
Sadaf Gulshad; Zeynep Akata; Jan Hendrik Metzen; Arnold Smeulders
  In this paper, we aim to understand and explain the decisions of deep neural
networks by studying the behavior of predicted attributes when adversarial
examples are introduced. We study the changes in attributes for clean as well
as adversarial images in both standard and adversarially robust networks. We
propose a metric to quantify the robustness of an adversarially robust network
against adversarial attacks. In a standard network, attributes predicted for
adversarial images are consistent with the wrong class, while attributes
predicted for the clean images are consistent with the true class. In an
adversarially robust network, the attributes predicted for adversarial images
classified correctly are consistent with the true class. Finally, we show that
the ability to robustify a network varies for different datasets. For the fine
grained dataset, it is higher as compared to the coarse-grained dataset.
Additionally, the ability to robustify a network increases with the increase in
adversarial noise.


http://arxiv.org/abs/1910.07517
Adversarial Examples for Models of Code.
Noam Yefet; Uri Alon; Eran Yahav
  We introduce a novel approach for attacking trained models of code with
adversarial examples. The main idea is to force a given trained model to make a
prediction of the adversary's choice by introducing small perturbations that do
not change program semantics. We find these perturbations by deriving the
desired prediction with respect to the model's inputs while holding the model
weights constant and following the gradients to slightly modify the input.
  To defend a model against such attacks, we propose placing a defensive model
in front of the downstream model. The defensive model detects unlikely
mutations and masks them before feeding the input to the downstream model.
  We show that our attack succeeds in changing a prediction to the adversary's
desire ("targeted attack") up to 89% of the times, and succeeds in changing a
given prediction to any incorrect prediction ("non-targeted attack") 94% of the
times. By using our proposed defense, the success rate of the attack drops
drastically for both targeted and non-targeted attacks, with a minor penalty of
2% relative degradation in accuracy while not performing under attack.


http://arxiv.org/abs/1910.07067
On adversarial patches: real-world attack on ArcFace-100 face recognition system.
Mikhail Pautov; Grigorii Melnikov; Edgar Kaziakhmedov; Klim Kireev; Aleksandr Petiushko
  Recent works showed the vulnerability of image classifiers to adversarial
attacks in the digital domain. However, the majority of attacks involve adding
small perturbation to an image to fool the classifier. Unfortunately, such
procedures can not be used to conduct a real-world attack, where adding an
adversarial attribute to the photo is a more practical approach. In this paper,
we study the problem of real-world attacks on face recognition systems. We
examine security of one of the best public face recognition systems,
LResNet100E-IR with ArcFace loss, and propose a simple method to attack it in
the physical world. The method suggests creating an adversarial patch that can
be printed, added as a face attribute and photographed; the photo of a person
with such attribute is then passed to the classifier such that the classifier's
recognized class changes from correct to the desired one. Proposed generating
procedure allows projecting adversarial patches not only on different areas of
the face, such as nose or forehead but also on some wearable accessory, such as
eyeglasses.


http://arxiv.org/abs/1910.06296
DeepSearch: Simple and Effective Blackbox Fuzzing of Deep Neural Networks.
Fuyuan Zhang; Sankalan Pal Chowdhury; Maria Christakis
  Although deep neural networks have been successful in image classification,
they are prone to adversarial attacks. To generate misclassified inputs, there
has emerged a wide variety of techniques, such as black- and whitebox testing
of neural networks. In this paper, we present DeepSearch, a novel
blackbox-fuzzing technique for image classifiers. Despite its simplicity,
DeepSearch is shown to be more effective in finding adversarial examples than
closely related black- and whitebox approaches. DeepSearch is additionally able
to generate the most subtle adversarial examples in comparison to these
approaches.


http://arxiv.org/abs/1910.06259
Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks.
David Stutz; Matthias Hein; Bernt Schiele
  Adversarial training yields robust models against a specific threat model,
e.g., $L_\infty$ adversarial examples. Typically robustness does not generalize
to previously unseen threat models, e.g., other $L_p$ norms, or larger
perturbations. Our confidence-calibrated adversarial training (CCAT) tackles
this problem by biasing the model towards low confidence predictions on
adversarial examples. By allowing to reject examples with low confidence,
robustness generalizes beyond the threat model employed during training. CCAT,
trained only on $L_\infty$ adversarial examples, increases robustness against
larger $L_\infty$, $L_2$, $L_1$ and $L_0$ attacks, adversarial frames, distal
adversarial examples and corrupted examples and yields better clean accuracy
compared to adversarial training. For thorough evaluation we developed novel
white- and black-box attacks directly attacking CCAT by maximizing confidence.
For each threat model, we use $7$ attacks with up to $50$ restarts and $5000$
iterations and report worst-case robust test error, extended to our
confidence-thresholded setting, across all attacks.


http://arxiv.org/abs/1910.06513
ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization.
Xiangyi Chen; Sijia Liu; Kaidi Xu; Xingguo Li; Xue Lin; Mingyi Hong; David Cox
  The adaptive momentum method (AdaMM), which uses past gradients to update
descent directions and learning rates simultaneously, has become one of the
most popular first-order optimization methods for solving machine learning
problems. However, AdaMM is not suited for solving black-box optimization
problems, where explicit gradient forms are difficult or infeasible to obtain.
In this paper, we propose a zeroth-order AdaMM (ZO-AdaMM) algorithm, that
generalizes AdaMM to the gradient-free regime. We show that the convergence
rate of ZO-AdaMM for both convex and nonconvex optimization is roughly a factor
of $O(\sqrt{d})$ worse than that of the first-order AdaMM algorithm, where $d$
is problem size. In particular, we provide a deep understanding on why
Mahalanobis distance matters in convergence of ZO-AdaMM and other AdaMM-type
methods. As a byproduct, our analysis makes the first step toward understanding
adaptive learning rate methods for nonconvex constrained optimization.
Furthermore, we demonstrate two applications, designing per-image and universal
adversarial attacks from black-box neural networks, respectively. We perform
extensive experiments on ImageNet and empirically show that ZO-AdaMM converges
much faster to a solution of high accuracy compared with $6$ state-of-the-art
ZO optimization methods.


http://arxiv.org/abs/1910.06838
Man-in-the-Middle Attacks against Machine Learning Classifiers via Malicious Generative Models.
Derek Derui; Wang; Chaoran Li; Sheng Wen; Surya Nepal; Yang Xiang
  Deep Neural Networks (DNNs) are vulnerable to deliberately crafted
adversarial examples. In the past few years, many efforts have been spent on
exploring query-optimisation attacks to find adversarial examples of either
black-box or white-box DNN models, as well as the defending countermeasures
against those attacks. In this work, we explore vulnerabilities of DNN models
under the umbrella of Man-in-the-Middle (MitM) attacks, which has not been
investigated before. From the perspective of an MitM adversary, the
aforementioned adversarial example attacks are not viable anymore. First, such
attacks must acquire the outputs from the models by multiple times before
actually launching attacks, which is difficult for the MitM adversary in
practice. Second, such attacks are one-off and cannot be directly generalised
onto new data examples, which decreases the rate of return for the attacker. In
contrast, using generative models to craft adversarial examples on the fly can
mitigate the drawbacks. However, the adversarial capability of the generative
models, such as Variational Auto-Encoder (VAE), has not been extensively
studied. Therefore, given a classifier, we investigate using a VAE decoder to
either transform benign inputs to their adversarial counterparts or decode
outputs from benign VAE encoders to be adversarial examples. The proposed
method can endue more capability to MitM attackers. Based on our evaluation,
the proposed attack can achieve above 95% success rate on both MNIST and
CIFAR10 datasets, which is better or comparable with state-of-the-art
query-optimisation attacks. At the meantime, the attack is 104 times faster
than the query-optimisation attacks.


http://arxiv.org/abs/1910.06261
Real-world adversarial attack on MTCNN face detection system.
Edgar Kaziakhmedov; Klim Kireev; Grigorii Melnikov; Mikhail Pautov; Aleksandr Petiushko
  Recent studies proved that deep learning approaches achieve remarkable
results on face detection task. On the other hand, the advances gave rise to a
new problem associated with the security of the deep convolutional neural
network models unveiling potential risks of DCNNs based applications. Even
minor input changes in the digital domain can result in the network being
fooled. It was shown then that some deep learning-based face detectors are
prone to adversarial attacks not only in a digital domain but also in the real
world. In the paper, we investigate the security of the well-known cascade CNN
face detection system - MTCNN and introduce an easily reproducible and a robust
way to attack it. We propose different face attributes printed on an ordinary
white and black printer and attached either to the medical face mask or to the
face directly. Our approach is capable of breaking the MTCNN detector in a
real-world scenario.


http://arxiv.org/abs/1910.05513
On Robustness of Neural Ordinary Differential Equations.
Hanshu Yan; Jiawei Du; Vincent Y. F. Tan; Jiashi Feng
  Neural ordinary differential equations (ODEs) have been attracting increasing
attention in various research domains recently. There have been some works
studying optimization issues and approximation capabilities of neural ODEs, but
their robustness is still yet unclear. In this work, we fill this important gap
by exploring robustness properties of neural ODEs both empirically and
theoretically. We first present an empirical study on the robustness of the
neural ODE-based networks (ODENets) by exposing them to inputs with various
types of perturbations and subsequently investigating the changes of the
corresponding outputs. In contrast to conventional convolutional neural
networks (CNNs), we find that the ODENets are more robust against both random
Gaussian perturbations and adversarial attack examples. We then provide an
insightful understanding of this phenomenon by exploiting a certain desirable
property of the flow of a continuous-time ODE, namely that integral curves are
non-intersecting. Our work suggests that, due to their intrinsic robustness, it
is promising to use neural ODEs as a basic block for building robust deep
network models. To further enhance the robustness of vanilla neural ODEs, we
propose the time-invariant steady neural ODE (TisODE), which regularizes the
flow on perturbed data via the time-invariant property and the imposition of a
steady-state constraint. We show that the TisODE method outperforms vanilla
neural ODEs and also can work in conjunction with other state-of-the-art
architectural methods to build more robust deep networks.


http://arxiv.org/abs/1910.05262
Hear "No Evil", See "Kenansville": Efficient and Transferable Black-Box Attacks on Speech Recognition and Voice Identification Systems.
Hadi Abdullah; Muhammad Sajidur Rahman; Washington Garcia; Logan Blue; Kevin Warren; Anurag Swarnim Yadav; Tom Shrimpton; Patrick Traynor
  Automatic speech recognition and voice identification systems are being
deployed in a wide array of applications, from providing control mechanisms to
devices lacking traditional interfaces, to the automatic transcription of
conversations and authentication of users. Many of these applications have
significant security and privacy considerations. We develop attacks that force
mistranscription and misidentification in state of the art systems, with
minimal impact on human comprehension. Processing pipelines for modern systems
are comprised of signal preprocessing and feature extraction steps, whose
output is fed to a machine-learned model. Prior work has focused on the models,
using white-box knowledge to tailor model-specific attacks. We focus on the
pipeline stages before the models, which (unlike the models) are quite similar
across systems. As such, our attacks are black-box and transferable, and
demonstrably achieve mistranscription and misidentification rates as high as
100% by modifying only a few frames of audio. We perform a study via Amazon
Mechanical Turk demonstrating that there is no statistically significant
difference between human perception of regular and perturbed audio. Our
findings suggest that models may learn aspects of speech that are generally not
perceived by human subjects, but that are crucial for model accuracy. We also
find that certain English language phonemes (in particular, vowels) are
significantly more susceptible to our attack. We show that the attacks are
effective when mounted over cellular networks, where signals are subject to
degradation due to transcoding, jitter, and packet loss.


http://arxiv.org/abs/1910.05018
Verification of Neural Networks: Specifying Global Robustness using Generative Models.
Nathanaël Fijalkow; Mohit Kumar Gupta
  The success of neural networks across most machine learning tasks and the
persistence of adversarial examples have made the verification of such models
an important quest. Several techniques have been successfully developed to
verify robustness, and are now able to evaluate neural networks with thousands
of nodes. The main weakness of this approach is in the specification:
robustness is asserted on a validation set consisting of a finite set of
examples, i.e. locally.
  We propose a notion of global robustness based on generative models, which
asserts the robustness on a very large and representative set of examples. We
show how this can be used for verifying neural networks. In this paper we
experimentally explore the merits of this approach, and show how it can be used
to construct realistic adversarial examples.


http://arxiv.org/abs/1910.04819
Information Robust Dirichlet Networks for Predictive Uncertainty Estimation.
Theodoros Tsiligkaridis
  Precise estimation of uncertainty in predictions for AI systems is a critical
factor in ensuring trust and safety. Conventional neural networks tend to be
overconfident as they do not account for uncertainty during training. In
contrast to Bayesian neural networks that learn approximate distributions on
weights to infer prediction confidence, we propose a novel method, Information
Robust Dirichlet networks, that learns the Dirichlet distribution on prediction
probabilities by minimizing the expected $L_p$ norm of the prediction error and
an information divergence loss that penalizes information flow towards
incorrect classes, while simultaneously maximizing differential entropy of
small adversarial perturbations to provide accurate uncertainty estimates.
Properties of the new cost function are derived to indicate how improved
uncertainty estimation is achieved. Experiments using real datasets show that
our technique outperforms state-of-the-art neural networks, by a large margin,
for estimating in-distribution and out-of-distribution uncertainty, and
detecting adversarial examples.


http://arxiv.org/abs/1910.04618
Universal Adversarial Perturbation for Text Classification.
Hang Gao; Tim Oates
  Given a state-of-the-art deep neural network text classifier, we show the
existence of a universal and very small perturbation vector (in the embedding
space) that causes natural text to be misclassified with high probability.
Unlike images on which a single fixed-size adversarial perturbation can be
found, text is of variable length, so we define the "universality" as
"token-agnostic", where a single perturbation is applied to each token,
resulting in different perturbations of flexible sizes at the sequence level.
We propose an algorithm to compute universal adversarial perturbations, and
show that the state-of-the-art deep neural networks are highly vulnerable to
them, even though they keep the neighborhood of tokens mostly preserved. We
also show how to use these adversarial perturbations to generate adversarial
text samples. The surprising existence of universal "token-agnostic"
adversarial perturbations may reveal important properties of a text classifier.


http://arxiv.org/abs/1910.03850
Learning deep forest with multi-scale Local Binary Pattern features for face anti-spoofing.
Rizhao Cai; Changsheng Chen
  Face Anti-Spoofing (FAS) is significant for the security of face recognition
systems. Convolutional Neural Networks (CNNs) have been introduced to the field
of the FAS and have achieved competitive performance. However, CNN-based
methods are vulnerable to the adversarial attack. Attackers could generate
adversarial-spoofing examples to circumvent a CNN-based face liveness detector.
Studies about the transferability of the adversarial attack reveal that
utilizing handcrafted feature-based methods could improve security in a
system-level. Therefore, handcrafted feature-based methods are worth our
exploration. In this paper, we introduce the deep forest, which is proposed as
an alternative towards CNNs by Zhou et al., in the problem of the FAS. To the
best of our knowledge, this is the first attempt at exploiting the deep forest
in the problem of FAS. Moreover, we propose to re-devise the representation
constructing by using LBP descriptors rather than the Grained-Scanning
Mechanism in the original scheme. Our method achieves competitive results. On
the benchmark database IDIAP REPLAY-ATTACK, 0\% Equal Error Rate (EER) is
achieved. This work provides a competitive option in a fusing scheme for
improving system-level security and offers important ideas to those who want to
explore methods besides CNNs.


http://arxiv.org/abs/1910.03810
Adversarial Learning of Deepfakes in Accounting.
Marco Schreyer; Timur Sattarov; Bernd Reimer; Damian Borth
  Nowadays, organizations collect vast quantities of accounting relevant
transactions, referred to as 'journal entries', in 'Enterprise Resource
Planning' (ERP) systems. The aggregation of those entries ultimately defines an
organization's financial statement. To detect potential misstatements and
fraud, international audit standards demand auditors to directly assess journal
entries using 'Computer Assisted AuditTechniques' (CAATs). At the same time,
discoveries in deep learning research revealed that machine learning models are
vulnerable to 'adversarial attacks'. It also became evident that such attack
techniques can be misused to generate 'Deepfakes' designed to directly attack
the perception of humans by creating convincingly altered media content. The
research of such developments and their potential impact on the finance and
accounting domain is still in its early stage. We believe that it is of vital
relevance to investigate how such techniques could be maliciously misused in
this sphere. In this work, we show an adversarial attack against CAATs using
deep neural networks. We first introduce a real-world 'thread model' designed
to camouflage accounting anomalies such as fraudulent journal entries. Second,
we show that adversarial autoencoder neural networks are capable of learning a
human interpretable model of journal entries that disentangles the entries
latent generative factors. Finally, we demonstrate how such a model can be
maliciously misused by a perpetrator to generate robust 'adversarial' journal
entries that mislead CAATs.


http://arxiv.org/abs/1910.03916
Deep Latent Defence.
Giulio Zizzo; Chris Hankin; Sergio Maffeis; Kevin Jones
  Deep learning methods have shown state of the art performance in a range of
tasks from computer vision to natural language processing. However, it is well
known that such systems are vulnerable to attackers who craft inputs in order
to cause misclassification. The level of perturbation an attacker needs to
introduce in order to cause such a misclassification can be extremely small,
and often imperceptible. This is of significant security concern, particularly
where misclassification can cause harm to humans.
  We thus propose Deep Latent Defence, an architecture which seeks to combine
adversarial training with a detection system. At its core Deep Latent Defence
has a adversarially trained neural network. A series of encoders take the
intermediate layer representation of data as it passes though the network and
project it to a latent space which we use for detecting adversarial samples via
a $k$-nn classifier. We present results using both grey and white box
attackers, as well as an adaptive $L_{\infty}$ bounded attack which was
constructed specifically to try and evade our defence. We find that even under
the strongest attacker model that we have investigated our defence is able to
offer significant defensive benefits.


http://arxiv.org/abs/1910.04279
Adversarial Training: embedding adversarial perturbations into the parameter space of a neural network to build a robust system.
Shixian Wen; Laurent Itti
  Adversarial training, in which a network is trained on both adversarial and
clean examples, is one of the most trusted defense methods against adversarial
attacks. However, there are three major practical difficulties in implementing
and deploying this method - expensive in terms of extra memory and computation
costs; accuracy trade-off between clean and adversarial examples; and lack of
diversity of adversarial perturbations. Classical adversarial training uses
fixed, precomputed perturbations in adversarial examples (input space). In
contrast, we introduce dynamic adversarial perturbations into the parameter
space of the network, by adding perturbation biases to the fully connected
layers of deep convolutional neural network. During training, using only clean
images, the perturbation biases are updated in the Fast Gradient Sign Direction
to automatically create and store adversarial perturbations by recycling the
gradient information computed. The network learns and adjusts itself
automatically to these learned adversarial perturbations. Thus, we can achieve
adversarial training with negligible cost compared to requiring a training set
of adversarial example images. In addition, if combined with classical
adversarial training, our perturbation biases can alleviate accuracy trade-off
difficulties, and diversify adversarial perturbations.


http://arxiv.org/abs/1910.03468
Directional Adversarial Training for Cost Sensitive Deep Learning Classification Applications.
Matteo Terzi; Gian Antonio Susto; Pratik Chaudhari
  In many real-world applications of Machine Learning it is of paramount
importance not only to provide accurate predictions, but also to ensure certain
levels of robustness. Adversarial Training is a training procedure aiming at
providing models that are robust to worst-case perturbations around predefined
points. Unfortunately, one of the main issues in adversarial training is that
robustness w.r.t. gradient-based attackers is always achieved at the cost of
prediction accuracy. In this paper, a new algorithm, called Wasserstein
Projected Gradient Descent (WPGD), for adversarial training is proposed. WPGD
provides a simple way to obtain cost-sensitive robustness, resulting in a finer
control of the robustness-accuracy trade-off. Moreover, WPGD solves an optimal
transport problem on the output space of the network and it can efficiently
discover directions where robustness is required, allowing to control the
directional trade-off between accuracy and robustness. The proposed WPGD is
validated in this work on image recognition tasks with different benchmark
datasets and architectures. Moreover, real world-like datasets are often
unbalanced: this paper shows that when dealing with such type of datasets, the
performance of adversarial training are mainly affected in term of standard
accuracy.


http://arxiv.org/abs/1910.03624
SmoothFool: An Efficient Framework for Computing Smooth Adversarial Perturbations.
Ali Dabouei; Sobhan Soleymani; Fariborz Taherkhani; Jeremy Dawson; Nasser M. Nasrabadi
  Deep neural networks are susceptible to adversarial manipulations in the
input domain. The extent of vulnerability has been explored intensively in
cases of $\ell_p$-bounded and $\ell_p$-minimal adversarial perturbations.
However, the vulnerability of DNNs to adversarial perturbations with specific
statistical properties or frequency-domain characteristics has not been
sufficiently explored. In this paper, we study the smoothness of perturbations
and propose SmoothFool, a general and computationally efficient framework for
computing smooth adversarial perturbations. Through extensive experiments, we
validate the efficacy of the proposed method for both the white-box and
black-box attack scenarios. In particular, we demonstrate that: (i) there exist
extremely smooth adversarial perturbations for well-established and widely used
network architectures, (ii) smoothness significantly enhances the robustness of
perturbations against state-of-the-art defense mechanisms, (iii) smoothness
improves the transferability of adversarial perturbations across both data
points and network architectures, and (iv) class categories exhibit a variable
range of susceptibility to smooth perturbations. Our results suggest that
smooth APs can play a significant role in exploring the vulnerability extent of
DNNs to adversarial examples.


http://arxiv.org/abs/1910.02673
Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork.
Yulong Wang; Xiaolin Hu; Hang Su
  We propose a novel perspective to understand deep neural networks in an
interpretable disentanglement form. For each semantic class, we extract a
class-specific functional subnetwork from the original full model, with
compressed structure while maintaining comparable prediction performance. The
structure representations of extracted subnetworks display a resemblance to
their corresponding class semantic similarities. We also apply extracted
subnetworks in visual explanation and adversarial example detection tasks by
merely replacing the original full model with class-specific subnetworks.
Experiments demonstrate that this intuitive operation can effectively improve
explanation saliency accuracy for gradient-based explanation methods, and
increase the detection rate for confidence score-based adversarial example
detection methods.


http://arxiv.org/abs/1910.02354
Unrestricted Adversarial Attacks for Semantic Segmentation.
Guangyu Shen; Chengzhi Mao; Junfeng Yang; Baishakhi Ray
  Semantic segmentation is one of the most impactful applications of machine
learning; however, their robustness under adversarial attack is not well
studied. In this paper, we focus on generating unrestricted adversarial
examples for semantic segmentation models. We demonstrate a simple yet
effective method to generate unrestricted adversarial examples using
conditional generative adversarial networks (CGAN) without any hand-crafted
metric. The na\"ive implementation of CGAN, however, yields inferior image
quality and low attack success rate. Instead, we leverage the SPADE
(Spatially-adaptive denormalization) structure with an additional loss item,
which is able to generate effective adversarial attacks in a single step. We
validate our approach on the well studied Cityscapes and ADE20K datasets, and
demonstrate that our synthetic adversarial examples are not only realistic, but
also improve the attack success rate by up to 41.0\% compared with the state of
the art adversarial attack methods including PGD attack.


http://arxiv.org/abs/1910.02244
Yet another but more efficient black-box adversarial attack: tiling and evolution strategies.
Laurent Meunier; Jamal Atif; Olivier Teytaud
  We introduce a new black-box attack achieving state of the art performances.
Our approach is based on a new objective function, borrowing ideas from
$\ell_\infty$-white box attacks, and particularly designed to fit
derivative-free optimization requirements. It only requires to have access to
the logits of the classifier without any other information which is a more
realistic scenario. Not only we introduce a new objective function, we extend
previous works on black box adversarial attacks to a larger spectrum of
evolution strategies and other derivative-free optimization methods. We also
highlight a new intriguing property that deep neural networks are not robust to
single shot tiled attacks. Our models achieve, with a budget limited to
$10,000$ queries, results up to $99.2\%$ of success rate against InceptionV3
classifier with $630$ queries to the network on average in the untargeted
attacks setting, which is an improvement by $90$ queries of the current state
of the art. In the targeted setting, we are able to reach, with a limited
budget of $100,000$, $100\%$ of success rate with a budget of $6,662$ queries
on average, i.e. we need $800$ queries less than the current state of the art.


http://arxiv.org/abs/1910.02125
Requirements for Developing Robust Neural Networks.
John S. Hyatt; Michael S. Lee
  Validation accuracy is a necessary, but not sufficient, measure of a neural
network classifier's quality. High validation accuracy during development does
not guarantee that a model is free of serious flaws, such as vulnerability to
adversarial attacks or a tendency to misclassify (with high confidence) data it
was not trained on. The model may also be incomprehensible to a human or base
its decisions on unreasonable criteria. These problems, which are not unique to
classifiers, have been the focus of a substantial amount of recent research.
However, they are not prioritized during model development, which almost always
optimizes on validation accuracy to the exclusion of everything else. The
product of this approach is likely to fail in unexpected ways outside of the
training environment. We believe that, in addition to validation accuracy, the
model development process must give added weight to other performance metrics
such as explainability, resistance to adversarial attacks, and overconfidence
on out-of-distribution data.


http://arxiv.org/abs/1910.02095
Adversarial Examples for Cost-Sensitive Classifiers.
Gavin S. Hartnett; Andrew J. Lohn; Alexander P. Sedlack
  Motivated by safety-critical classification problems, we investigate
adversarial attacks against cost-sensitive classifiers. We use current
state-of-the-art adversarially-resistant neural network classifiers [1] as the
underlying models. Cost-sensitive predictions are then achieved via a final
processing step in the feed-forward evaluation of the network. We evaluate the
effectiveness of cost-sensitive classifiers against a variety of attacks and we
introduce a new cost-sensitive attack which performs better than targeted
attacks in some cases. We also explored the measures a defender can take in
order to limit their vulnerability to these attacks. This attacker/defender
scenario is naturally framed as a two-player zero-sum finite game which we
analyze using game theory.


http://arxiv.org/abs/1910.01329
Perturbations are not Enough: Generating Adversarial Examples with Spatial Distortions.
He Zhao; Trung Le; Paul Montague; Vel Olivier De; Tamas Abraham; Dinh Phung
  Deep neural network image classifiers are reported to be susceptible to
adversarial evasion attacks, which use carefully crafted images created to
mislead a classifier. Recently, various kinds of adversarial attack methods
have been proposed, most of which focus on adding small perturbations to input
images. Despite the success of existing approaches, the way to generate
realistic adversarial images with small perturbations remains a challenging
problem. In this paper, we aim to address this problem by proposing a novel
adversarial method, which generates adversarial examples by imposing not only
perturbations but also spatial distortions on input images, including scaling,
rotation, shear, and translation. As humans are less susceptible to small
spatial distortions, the proposed approach can produce visually more realistic
attacks with smaller perturbations, able to deceive classifiers without
affecting human predictions. We learn our method by amortized techniques with
neural networks and generate adversarial examples efficiently by a forward pass
of the networks. Extensive experiments on attacking different types of
non-robustified classifiers and robust classifiers with defence show that our
method has state-of-the-art performance in comparison with advanced attack
parallels.


http://arxiv.org/abs/1910.02785
BUZz: BUffer Zones for defending adversarial examples in image classification.
Phuong Ha Nguyen; Kaleel Mahmood; Lam M. Nguyen; Thanh Nguyen; Dijk Marten van
  We propose a novel defense against all existing gradient based adversarial
attacks on deep neural networks for image classification problems. Our defense
is based on a combination of deep neural networks and simple image
transformations. While straight forward in implementation, this defense yields
a unique security property which we term buffer zones. In this paper, we
formalize the concept of buffer zones. We argue that our defense based on
buffer zones is secure against state-of-the-art black box attacks. We are able
to achieve this security even when the adversary has access to the {\em entire}
original training data set and unlimited query access to the defense. We verify
our security claims through experimentation using FashionMNIST, CIFAR-10 and
CIFAR-100. We demonstrate $<10\%$ attack success rate -- significantly lower
than what other well-known defenses offer -- at only a price of a 15-20\% drop
in clean accuracy. By using a new intuitive metric we explain why this
trade-off offers a significant improvement over prior work.


http://arxiv.org/abs/1910.01624
Verification of Neural Network Behaviour: Formal Guarantees for Power System Applications.
Andreas Venzke; Spyros Chatzivasileiadis
  This paper presents for the first time, to our knowledge, a framework for
verifying neural network behavior in power system applications. Up to this
moment, neural networks have been applied in power systems as a black-box; this
has presented a major barrier for their adoption in practice. Developing a
rigorous framework based on mixed integer linear programming, our methods can
determine the range of inputs that neural networks classify as safe or unsafe,
and are able to systematically identify adversarial examples. Such methods have
the potential to build the missing trust of power system operators on neural
networks, and unlock a series of new applications in power systems. This paper
presents the framework, methods to assess and improve neural network robustness
in power systems, and addresses concerns related to scalability and accuracy.
We demonstrate our methods on the IEEE 9-bus, 14-bus, and 162-bus systems,
treating both N-1 security and small-signal stability.


http://arxiv.org/abs/1910.01907
Attacking Vision-based Perception in End-to-End Autonomous Driving Models.
Adith Boloor; Karthik Garimella; Xin He; Christopher Gill; Yevgeniy Vorobeychik; Xuan Zhang
  Recent advances in machine learning, especially techniques such as deep
neural networks, are enabling a range of emerging applications. One such
example is autonomous driving, which often relies on deep learning for
perception. However, deep learning-based perception has been shown to be
vulnerable to a host of subtle adversarial manipulations of images.
Nevertheless, the vast majority of such demonstrations focus on perception that
is disembodied from end-to-end control. We present novel end-to-end attacks on
autonomous driving in simulation, using simple physically realizable attacks:
the painting of black lines on the road. These attacks target deep neural
network models for end-to-end autonomous driving control. A systematic
investigation shows that such attacks are easy to engineer, and we describe
scenarios (e.g., right turns) in which they are highly effective. We define
several objective functions that quantify the success of an attack and develop
techniques based on Bayesian Optimization to efficiently traverse the search
space of higher dimensional attacks. Additionally, we define a novel class of
hijacking attacks, where painted lines on the road cause the driver-less car to
follow a target path. Through the use of network deconvolution, we provide
insights into the successful attacks, which appear to work by mimicking
activations of entirely different scenarios. Our code is available at
https://github.com/xz-group/AdverseDrive


http://arxiv.org/abs/1910.00982
Adversarially Robust Few-Shot Learning: A Meta-Learning Approach.
Micah Goldblum; Liam Fowl; Tom Goldstein
  Previous work on adversarially robust neural networks for image
classification requires large training sets and computationally expensive
training procedures. On the other hand, few-shot learning methods are highly
vulnerable to adversarial examples. The goal of our work is to produce networks
which both perform well at few-shot classification tasks and are simultaneously
robust to adversarial examples. We develop an algorithm for producing
adversarially robust meta-learners, and we thoroughly investigate factors which
contribute to adversarial vulnerability. Moreover, our method achieves far
superior robust performance on few-shot image classification tasks, such as
Mini-ImageNet and CIFAR-FS, than robust transfer learning.


http://arxiv.org/abs/1910.00736
Boosting Image Recognition with Non-differentiable Constraints.
Xuan Li; Yuchen Lu; Peng Xu; Jizong Peng; Christian Desrosiers; Xue Liu
  In this paper, we study the problem of image recognition with
non-differentiable constraints. A lot of real-life recognition applications
require a rich output structure with deterministic constraints that are
discrete or modeled by a non-differentiable function. A prime example is
recognizing digit sequences, which are restricted by such rules (e.g.,
\textit{container code detection}, \textit{social insurance number
recognition}, etc.). We investigate the usefulness of adding non-differentiable
constraints in learning for the task of digit sequence recognition. Toward this
goal, we synthesize six different datasets from MNIST and Cropped SVHN, with
three discrete rules inspired by real-life protocols. To deal with the
non-differentiability of these rules, we propose a reinforcement learning
approach based on the policy gradient method. We find that incorporating this
rule-based reinforcement can effectively increase the accuracy for all datasets
and provide a good inductive bias which improves the model even with limited
data. On one of the datasets, MNIST\_Rule2, models trained with rule-based
reinforcement increase the accuracy by 4.7\% for 2000 samples and 23.6\% for
500 samples. We further test our model against synthesized adversarial
examples, e.g., blocking out digits, and observe that adding our rule-based
reinforcement increases the model robustness with a relatively smaller
performance drop.


http://arxiv.org/abs/1910.00727
Generating Semantic Adversarial Examples with Differentiable Rendering.
Lakshya Jain; Wilson Wu; Steven Chen; Uyeong Jang; Varun Chandrasekaran; Sanjit Seshia; Somesh Jha
  Machine learning (ML) algorithms, especially deep neural networks, have
demonstrated success in several domains. However, several types of attacks have
raised concerns about deploying ML in safety-critical domains, such as
autonomous driving and security. An attacker perturbs a data point slightly in
the concrete feature space (e.g., pixel space) and causes the ML algorithm to
produce incorrect output (e.g. a perturbed stop sign is classified as a yield
sign). These perturbed data points are called adversarial examples, and there
are numerous algorithms in the literature for constructing adversarial examples
and defending against them. In this paper we explore semantic adversarial
examples (SAEs) where an attacker creates perturbations in the semantic space
representing the environment that produces input for the ML model. For example,
an attacker can change the background of the image to be cloudier to cause
misclassification. We present an algorithm for constructing SAEs that uses
recent advances in differential rendering and inverse graphics.


http://arxiv.org/abs/1910.00327
Attacking CNN-based anti-spoofing face authentication in the physical domain.
Bowen Zhang; Benedetta Tondi; Mauro Barni
  In this paper, we study the vulnerability of anti-spoofing methods based on
deep learning against adversarial perturbations. We first show that attacking a
CNN-based anti-spoofing face authentication system turns out to be a difficult
task. When a spoofed face image is attacked in the physical world, in fact, the
attack has not only to remove the rebroadcast artefacts present in the image,
but it has also to take into account that the attacked image will be recaptured
again and then compensate for the distortions that will be re-introduced after
the attack by the subsequent rebroadcast process. Subsequently, we propose a
method to craft robust physical domain adversarial images against anti-spoofing
CNN-based face authentication. The attack built in this way can successfully
pass all the steps in the authentication chain (that is, face detection, face
recognition and spoofing detection), by achieving simultaneously the following
goals: i) make the spoofing detection fail; ii) let the facial region be
detected as a face and iii) recognized as belonging to the victim of the
attack. The effectiveness of the proposed attack is validated experimentally
within a realistic setting, by considering the REPLAY-MOBILE database, and by
feeding the adversarial images to a real face authentication system capturing
the input images through a mobile phone camera.


http://arxiv.org/abs/1910.00511
An Efficient and Margin-Approaching Zero-Confidence Adversarial Attack.
Yang Zhang; Shiyu Chang; Mo Yu; Kaizhi Qian
  There are two major paradigms of white-box adversarial attacks that attempt
to impose input perturbations. The first paradigm, called the fix-perturbation
attack, crafts adversarial samples within a given perturbation level. The
second paradigm, called the zero-confidence attack, finds the smallest
perturbation needed to cause mis-classification, also known as the margin of an
input feature. While the former paradigm is well-resolved, the latter is not.
Existing zero-confidence attacks either introduce significant ap-proximation
errors, or are too time-consuming. We therefore propose MARGINATTACK, a
zero-confidence attack framework that is able to compute the margin with
improved accuracy and efficiency. Our experiments show that MARGINATTACK is
able to compute a smaller margin than the state-of-the-art zero-confidence
attacks, and matches the state-of-the-art fix-perturbation at-tacks. In
addition, it runs significantly faster than the Carlini-Wagner attack,
currently the most ac-curate zero-confidence attack algorithm.


http://arxiv.org/abs/1910.01742
Cross-Layer Strategic Ensemble Defense Against Adversarial Examples.
Wenqi Wei; Ling Liu; Margaret Loper; Ka-Ho Chow; Emre Gursoy; Stacey Truex; Yanzhao Wu
  Deep neural network (DNN) has demonstrated its success in multiple domains.
However, DNN models are inherently vulnerable to adversarial examples, which
are generated by adding adversarial perturbations to benign inputs to fool the
DNN model to misclassify. In this paper, we present a cross-layer strategic
ensemble framework and a suite of robust defense algorithms, which are
attack-independent, and capable of auto-repairing and auto-verifying the target
model being attacked. Our strategic ensemble approach makes three original
contributions. First, we employ input-transformation diversity to design the
input-layer strategic transformation ensemble algorithms. Second, we utilize
model-disagreement diversity to develop the output-layer strategic model
ensemble algorithms. Finally, we create an input-output cross-layer strategic
ensemble defense that strengthens the defensibility by combining diverse input
transformation based model ensembles with diverse output verification model
ensembles. Evaluated over 10 attacks on ImageNet dataset, we show that our
strategic ensemble defense algorithms can achieve high defense success rates
and are more robust with high attack prevention success rates and low benign
false negative rates, compared to existing representative defense methods.


http://arxiv.org/abs/1910.00470
Deep Neural Rejection against Adversarial Examples.
Angelo Sotgiu; Ambra Demontis; Marco Melis; Battista Biggio; Giorgio Fumera; Xiaoyi Feng; Fabio Roli
  Despite the impressive performances reported by deep neural networks in
different application domains, they remain largely vulnerable to adversarial
examples, i.e., input samples that are carefully perturbed to cause
misclassification at test time. In this work, we propose a deep neural
rejection mechanism to detect adversarial examples, based on the idea of
rejecting samples that exhibit anomalous feature representations at different
network layers. With respect to competing approaches, our method does not
require generating adversarial examples at training time, and it is less
computationally demanding. To properly evaluate our method, we define an
adaptive white-box attack that is aware of the defense mechanism and aims to
bypass it. Under this worst-case setting, we empirically show that our approach
outperforms previously-proposed methods that detect adversarial examples by
only analyzing the feature representation provided by the output network layer.


http://arxiv.org/abs/1909.13806
Min-Max Optimization without Gradients: Convergence and Applications to Adversarial ML.
Sijia Liu; Songtao Lu; Xiangyi Chen; Yao Feng; Kaidi Xu; Abdullah Al-Dujaili; Minyi Hong; Una-May Obelilly
  In this paper, we study the problem of constrained robust (min-max)
optimization ina black-box setting, where the desired optimizer cannot access
the gradients of the objective function but may query its values. We present a
principled optimization framework, integrating a zeroth-order (ZO) gradient
estimator with an alternating projected stochastic gradient descent-ascent
method, where the former only requires a small number of function queries and
the later needs just one-step descent/ascent update. We show that the proposed
framework, referred to as ZO-Min-Max, has a sub-linear convergence rate under
mild conditions and scales gracefully with problem size. From an application
side, we explore a promising connection between black-box min-max optimization
and black-box evasion and poisoning attacks in adversarial machine learning
(ML). Our empirical evaluations on these use cases demonstrate the
effectiveness of our approach and its scalability to dimensions that prohibit
using recent black-box solvers.


http://arxiv.org/abs/1909.13857
Black-box Adversarial Attacks with Bayesian Optimization.
Satya Narayan Shukla; Anit Kumar Sahu; Devin Willmott; J. Zico Kolter
  We focus on the problem of black-box adversarial attacks, where the aim is to
generate adversarial examples using information limited to loss function
evaluations of input-output pairs. We use Bayesian optimization~(BO) to
specifically cater to scenarios involving low query budgets to develop query
efficient adversarial attacks. We alleviate the issues surrounding BO in
regards to optimizing high dimensional deep learning models by effective
dimension upsampling techniques. Our proposed approach achieves performance
comparable to the state of the art black-box adversarial attacks albeit with a
much lower average query count. In particular, in low query budget regimes, our
proposed method reduces the query count up to $80\%$ with respect to the state
of the art methods.


http://arxiv.org/abs/1910.00068
Role of Spatial Context in Adversarial Robustness for Object Detection.
Aniruddha Saha; Akshayvarun Subramanya; Koninika Patil; Hamed Pirsiavash
  The benefits of utilizing spatial context in fast object detection algorithms
have been studied extensively. Detectors increase inference speed by doing a
single forward pass per image which means they implicitly use contextual
reasoning for their predictions. However, one can show that an adversary can
design adversarial patches which do not overlap with any objects of interest in
the scene and exploit contextual reasoning to fool standard detectors. In this
paper, we examine this problem and design category specific adversarial patches
which make a widely used object detector like YOLO blind to an attacker chosen
object category. We also show that limiting the use of spatial context during
object detector training improves robustness to such adversaries. We believe
the existence of context based adversarial attacks is concerning since the
adversarial patch can affect predictions without being in vicinity of any
objects of interest. Hence, defending against such attacks becomes challenging
and we urge the research community to give attention to this vulnerability.


http://arxiv.org/abs/1910.06907
Techniques for Adversarial Examples Threatening the Safety of Artificial Intelligence Based Systems.
Utku Kose
  Artificial intelligence is known as the most effective technological field
for rapid developments shaping the future of the world. Even today, it is
possible to see intense use of intelligence systems in all fields of the life.
Although advantages of the Artificial Intelligence are widely observed, there
is also a dark side employing efforts to design hacking oriented techniques
against Artificial Intelligence. Thanks to such techniques, it is possible to
trick intelligent systems causing directed results for unsuccessful outputs.
That is critical for also cyber wars of the future as it is predicted that the
wars will be done unmanned, autonomous intelligent systems. Moving from the
explanations, objective of this study is to provide information regarding
adversarial examples threatening the Artificial Intelligence and focus on
details of some techniques, which are used for creating adversarial examples.
Adversarial examples are known as training data, which can trick a Machine
Learning technique to learn incorrectly about the target problem and cause an
unsuccessful or maliciously directed intelligent system at the end. The study
enables the readers to learn enough about details of recent techniques for
creating adversarial examples.


http://arxiv.org/abs/1909.12734
Maximal adversarial perturbations for obfuscation: Hiding certain attributes while preserving rest.
Indu Ilanchezian; Praneeth Vepakomma; Abhishek Singh; Otkrist Gupta; G. N. Srinivasa Prasanna; Ramesh Raskar
  In this paper we investigate the usage of adversarial perturbations for the
purpose of privacy from human perception and model (machine) based detection.
We employ adversarial perturbations for obfuscating certain variables in raw
data while preserving the rest. Current adversarial perturbation methods are
used for data poisoning with minimal perturbations of the raw data such that
the machine learning model's performance is adversely impacted while the human
vision cannot perceive the difference in the poisoned dataset due to minimal
nature of perturbations. We instead apply relatively maximal perturbations of
raw data to conditionally damage model's classification of one attribute while
preserving the model performance over another attribute. In addition, the
maximal nature of perturbation helps adversely impact human perception in
classifying hidden attribute apart from impacting model performance. We
validate our result qualitatively by showing the obfuscated dataset and
quantitatively by showing the inability of models trained on clean data to
predict the hidden attribute from the perturbed dataset while being able to
predict the rest of attributes.


http://arxiv.org/abs/1909.12741
Impact of Low-bitwidth Quantization on the Adversarial Robustness for Embedded Neural Networks.
Rémi Bernhard; Pierre-Alain Moellic; Jean-Max Dutertre
  As the will to deploy neural networks models on embedded systems grows, and
considering the related memory footprint and energy consumption issues, finding
lighter solutions to store neural networks such as weight quantization and more
efficient inference methods become major research topics. Parallel to that,
adversarial machine learning has risen recently with an impressive and
significant attention, unveiling some critical flaws of machine learning
models, especially neural networks. In particular, perturbed inputs called
adversarial examples have been shown to fool a model into making incorrect
predictions. In this article, we investigate the adversarial robustness of
quantized neural networks under different threat models for a classical
supervised image classification task. We show that quantization does not offer
any robust protection, results in severe form of gradient masking and advance
some hypotheses to explain it. However, we experimentally observe poor
transferability capacities which we explain by quantization value shift
phenomenon and gradient misalignment and explore how these results can be
exploited with an ensemble-based defense.


http://arxiv.org/abs/1909.12031
Towards Understanding the Transferability of Deep Representations.
Hong Liu; Mingsheng Long; Jianmin Wang; Michael I. Jordan
  Deep neural networks trained on a wide range of datasets demonstrate
impressive transferability. Deep features appear general in that they are
applicable to many datasets and tasks. Such property is in prevalent use in
real-world applications. A neural network pretrained on large datasets, such as
ImageNet, can significantly boost generalization and accelerate training if
fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort
has been devoted to uncovering the reason of transferability in deep feature
representations. This paper tries to understand transferability from the
perspectives of improved generalization, optimization and the feasibility of
transferability. We demonstrate that 1) Transferred models tend to find flatter
minima, since their weight matrices stay close to the original flat region of
pretrained parameters when transferred to a similar target dataset; 2)
Transferred representations make the loss landscape more favorable with
improved Lipschitzness, which accelerates and stabilizes training
substantially. The improvement largely attributes to the fact that the
principal component of gradient is suppressed in the pretrained parameters,
thus stabilizing the magnitude of gradient in back-propagation. 3) The
feasibility of transferability is related to the similarity of both input and
label. And a surprising discovery is that the feasibility is also impacted by
the training stages in that the transferability first increases during
training, and then declines. We further provide a theoretical analysis to
verify our observations.


http://arxiv.org/abs/1909.12167
Adversarial Machine Learning Attack on Modulation Classification.
Muhammad Usama; Muhammad Asim; Junaid Qadir; Ala Al-Fuqaha; Muhammad Ali Imran
  Modulation classification is an important component of cognitive self-driving
networks. Recently many ML-based modulation classification methods have been
proposed. We have evaluated the robustness of 9 ML-based modulation classifiers
against the powerful Carlini \& Wagner (C-W) attack and showed that the current
ML-based modulation classifiers do not provide any deterrence against
adversarial ML examples. To the best of our knowledge, we are the first to
report the results of the application of the C-W attack for creating
adversarial examples against various ML models for modulation classification.


http://arxiv.org/abs/1909.12161
Adversarial ML Attack on Self Organizing Cellular Networks.
Salah-ud-din Farooq; Muhammad Usama; Junaid Qadir; Muhammad Ali Imran
  Deep Neural Networks (DNN) have been widely adopted in self-organizing
networks (SON) for automating different networking tasks. Recently, it has been
shown that DNN lack robustness against adversarial examples where an adversary
can fool the DNN model into incorrect classification by introducing a small
imperceptible perturbation to the original example. SON is expected to use DNN
for multiple fundamental cellular tasks and many DNN-based solutions for
performing SON tasks have been proposed in the literature have not been tested
against adversarial examples. In this paper, we have tested and explained the
robustness of SON against adversarial example and investigated the performance
of an important SON use case in the face of adversarial attacks. We have also
generated explanations of incorrect classifications by utilizing an explainable
artificial intelligence (AI) technique.


http://arxiv.org/abs/1909.12180
Towards neural networks that provably know when they don't know.
Alexander Meinke; Matthias Hein
  It has recently been shown that ReLU networks produce arbitrarily
over-confident predictions far away from the training data. Thus, ReLU networks
do not know when they don't know. However, this is a highly important property
in safety critical applications. In the context of out-of-distribution
detection (OOD) there have been a number of proposals to mitigate this problem
but none of them are able to make any mathematical guarantees. In this paper we
propose a new approach to OOD which overcomes both problems. Our approach can
be used with ReLU networks and provides provably low confidence predictions far
away from the training data as well as the first certificates for low
confidence predictions in a neighborhood of an out-distribution point. In the
experiments we show that state-of-the-art methods fail in this worst-case
setting whereas our model can guarantee its performance while retaining
state-of-the-art OOD performance.


http://arxiv.org/abs/1909.12272
Lower Bounds on Adversarial Robustness from Optimal Transport.
Arjun Nitin Bhagoji; Daniel Cullina; Prateek Mittal
  While progress has been made in understanding the robustness of machine
learning classifiers to test-time adversaries (evasion attacks), fundamental
questions remain unresolved. In this paper, we use optimal transport to
characterize the minimum possible loss in an adversarial classification
scenario. In this setting, an adversary receives a random labeled example from
one of two classes, perturbs the example subject to a neighborhood constraint,
and presents the modified example to the classifier. We define an appropriate
cost function such that the minimum transportation cost between the
distributions of the two classes determines the minimum $0-1$ loss for any
classifier. When the classifier comes from a restricted hypothesis class, the
optimal transportation cost provides a lower bound. We apply our framework to
the case of Gaussian data with norm-bounded adversaries and explicitly show
matching bounds for the classification and transport problems as well as the
optimality of linear classifiers. We also characterize the sample complexity of
learning in this setting, deriving and extending previously known results as a
special case. Finally, we use our framework to study the gap between the
optimal classification performance possible and that currently achieved by
state-of-the-art robustly trained neural networks for datasets of interest,
namely, MNIST, Fashion MNIST and CIFAR-10.


http://arxiv.org/abs/1909.11786
Probabilistic Modeling of Deep Features for Out-of-Distribution and Adversarial Detection.
Nilesh A. Ahuja; Ibrahima Ndiour; Trushant Kalyanpur; Omesh Tickoo
  We present a principled approach for detecting out-of-distribution (OOD) and
adversarial samples in deep neural networks. Our approach consists in modeling
the outputs of the various layers (deep features) with parametric probability
distributions once training is completed. At inference, the likelihoods of the
deep features w.r.t the previously learnt distributions are calculated and used
to derive uncertainty estimates that can discriminate in-distribution samples
from OOD samples. We explore the use of two classes of multivariate
distributions for modeling the deep features - Gaussian and Gaussian mixture -
and study the trade-off between accuracy and computational complexity. We
demonstrate benefits of our approach on image features by detecting OOD images
and adversarially-generated images, using popular DNN architectures on MNIST
and CIFAR10 datasets. We show that more precise modeling of the feature
distributions result in significantly improved detection of OOD and adversarial
samples; up to 12 percentage points in AUPR and AUROC metrics. We further show
that our approach remains extremely effective when applied to video data and
associated spatio-temporal features by detecting adversarial samples on
activity classification tasks using UCF101 dataset, and the C3D network. To our
knowledge, our methodology is the first one reported for reliably detecting
white-box adversarial framing, a state-of-the-art adversarial attack for video
classifiers.


http://arxiv.org/abs/1909.11515
Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks.
Tianyu Pang; Kun Xu; Jun Zhu
  It has been widely recognized that adversarial examples can be easily crafted
to fool deep networks, which mainly root from the locally non-linear behavior
nearby input examples. Applying mixup in training provides an effective
mechanism to improve generalization performance and model robustness against
adversarial perturbations, which introduces the globally linear behavior
in-between training examples. However, in previous work, the mixup-trained
models only passively defend adversarial attacks in inference by directly
classifying the inputs, where the induced global linearity is not well
exploited. Namely, since the locality of the adversarial perturbations, it
would be more efficient to actively break the locality via the globality of the
model predictions. Inspired by simple geometric intuition, we develop an
inference principle, named mixup inference (MI), for mixup-trained models. MI
mixups the input with other random clean samples, which can shrink and transfer
the equivalent perturbation if the input is adversarial. Our experiments on
CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial
robustness for the models trained by mixup and its variants.


http://arxiv.org/abs/1909.11764
FreeLB: Enhanced Adversarial Training for Natural Language Understanding.
Chen Zhu; Yu Cheng; Zhe Gan; Siqi Sun; Tom Goldstein; Jingjing Liu
  Adversarial training, which minimizes the maximal risk for label-preserving
input perturbations, has proved to be effective for improving the
generalization of language models. In this work, we propose a novel adversarial
training algorithm, FreeLB, that promotes higher invariance in the embedding
space, by adding adversarial perturbations to word embeddings and minimizing
the resultant adversarial risk inside different regions around input samples.
To validate the effectiveness of the proposed approach, we apply it to
Transformer-based models for natural language understanding and commonsense
reasoning tasks. Experiments on the GLUE benchmark show that when applied only
to the finetuning stage, it is able to improve the overall test scores of
BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8.
In addition, the proposed approach achieves state-of-the-art single-model test
accuracies of 85.44\% and 67.75\% on ARC-Easy and ARC-Challenge. Experiments on
CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and
boost the performance of RoBERTa-large model on other tasks as well. Code is
available at \url{https://github.com/zhuchen03/FreeLB .


http://arxiv.org/abs/1909.11202
A Visual Analytics Framework for Adversarial Text Generation.
Brandon Laughlin; Christopher Collins; Karthik Sankaranarayanan; Khalil El-Khatib
  This paper presents a framework which enables a user to more easily make
corrections to adversarial texts. While attack algorithms have been
demonstrated to automatically build adversaries, changes made by the algorithms
can often have poor semantics or syntax. Our framework is designed to
facilitate human intervention by aiding users in making corrections. The
framework extends existing attack algorithms to work within an evolutionary
attack process paired with a visual analytics loop. Using an interactive
dashboard a user is able to review the generation process in real time and
receive suggestions from the system for edits to be made. The adversaries can
be used to both diagnose robustness issues within a single classifier or to
compare various classifier options. With the weaknesses identified, the
framework can also be used as a first step in mitigating adversarial threats.
The framework can be used as part of further research into defense methods in
which the adversarial examples are used to evaluate new countermeasures. We
demonstrate the framework with a word swapping attack for the task of sentiment
classification.


http://arxiv.org/abs/1909.11167
Intelligent image synthesis to attack a segmentation CNN using adversarial learning.
Liang Chen; Paul Bentley; Kensaku Mori; Kazunari Misawa; Michitaka Fujiwara; Daniel Rueckert
  Deep learning approaches based on convolutional neural networks (CNNs) have
been successful in solving a number of problems in medical imaging, including
image segmentation. In recent years, it has been shown that CNNs are vulnerable
to attacks in which the input image is perturbed by relatively small amounts of
noise so that the CNN is no longer able to perform a segmentation of the
perturbed image with sufficient accuracy. Therefore, exploring methods on how
to attack CNN-based models as well as how to defend models against attacks have
become a popular topic as this also provides insights into the performance and
generalization abilities of CNNs. However, most of the existing work assumes
unrealistic attack models, i.e. the resulting attacks were specified in
advance. In this paper, we propose a novel approach for generating adversarial
examples to attack CNN-based segmentation models for medical images. Our
approach has three key features: 1) The generated adversarial examples exhibit
anatomical variations (in form of deformations) as well as appearance
perturbations; 2) The adversarial examples attack segmentation models so that
the Dice scores decrease by a pre-specified amount; 3) The attack is not
required to be specified beforehand. We have evaluated our approach on
CNN-based approaches for the multi-organ segmentation problem in 2D CT images.
We show that the proposed approach can be used to attack different CNN-based
segmentation models.


http://arxiv.org/abs/1909.10773
Sign-OPT: A Query-Efficient Hard-label Adversarial Attack.
Minhao Cheng; Simranjit Singh; Patrick Chen; Pin-Yu Chen; Sijia Liu; Cho-Jui Hsieh
  We study the most practical problem setup for evaluating adversarial
robustness of a machine learning system with limited access: the hard-label
black-box attack setting for generating adversarial examples, where limited
model queries are allowed and only the decision is provided to a queried data
input. Several algorithms have been proposed for this problem but they
typically require huge amount (>20,000) of queries for attacking one example.
Among them, one of the state-of-the-art approaches (Cheng et al., 2019) showed
that hard-label attack can be modeled as an optimization problem where the
objective function can be evaluated by binary search with additional model
queries, thereby a zeroth order optimization algorithm can be applied. In this
paper, we adopt the same optimization formulation but propose to directly
estimate the sign of gradient at any direction instead of the gradient itself,
which enjoys the benefit of single query. Using this single query oracle for
retrieving sign of directional derivative, we develop a novel query-efficient
Sign-OPT approach for hard-label black-box attack. We provide a convergence
analysis of the new algorithm and conduct experiments on several models on
MNIST, CIFAR-10 and ImageNet. We find that Sign-OPT attack consistently
requires 5X to 10X fewer queries when compared to the current state-of-the-art
approaches, and usually converges to an adversarial example with smaller
perturbation.


http://arxiv.org/abs/1909.10480
Adversarial Examples for Deep Learning Cyber Security Analytics.
Alesia Chernikova; Alina Oprea
  As advances in Deep Neural Networks demonstrate unprecedented levels of
performance in many critical applications, their vulnerability to attacks is
still an open question. Adversarial examples are small modifications of
legitimate data points, resulting in mis-classification at testing time. As
Deep Neural Networks found a wide range of applications to cyber security
analytics, it becomes important to study the robustness of these models in this
setting. We consider adversarial testing-time attacks against Deep Learning
models designed for cyber security applications. In security applications,
machine learning models are not typically trained directly on the raw network
traffic or security logs, but on intermediate features defined by domain
experts. Existing attacks applied directly to the intermediate feature
representation result in violation of feature constraints, leading to invalid
adversarial examples. We propose a general framework for crafting adversarial
attacks that takes into consideration the mathematical dependencies between
intermediate features in model input vector, as well as physical constraints
imposed by the applications. We apply our methods on two security applications,
a malicious connection and a malicious domain classifier, to generate feasible
adversarial examples in these domains. We show that with minimal effort (e.g.,
generating 12 network connections), an attacker can change the prediction of a
model from Malicious to Benign. We extensively evaluate the success of our
attacks, and how they depend on several optimization objectives and imbalance
ratios in the training data.


http://arxiv.org/abs/1909.10594
MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples.
Jinyuan Jia; Ahmed Salem; Michael Backes; Yang Zhang; Neil Zhenqiang Gong
  In a membership inference attack, an attacker aims to infer whether a data
sample is in a target classifier's training dataset or not. Specifically, given
a black-box access to the target classifier, the attacker trains a binary
classifier, which takes a data sample's confidence score vector predicted by
the target classifier as an input and predicts the data sample to be a member
or non-member of the target classifier's training dataset. Membership inference
attacks pose severe privacy and security threats to the training dataset. Most
existing defenses leverage differential privacy when training the target
classifier or regularize the training process of the target classifier. These
defenses suffer from two key limitations: 1) they do not have formal
utility-loss guarantees of the confidence score vectors, and 2) they achieve
suboptimal privacy-utility tradeoffs.
  In this work, we propose MemGuard, the first defense with formal utility-loss
guarantees against black-box membership inference attacks. Instead of tampering
the training process of the target classifier, MemGuard adds noise to each
confidence score vector predicted by the target classifier. Our key observation
is that attacker uses a classifier to predict member or non-member and
classifier is vulnerable to adversarial examples. Based on the observation, we
propose to add a carefully crafted noise vector to a confidence score vector to
turn it into an adversarial example that misleads the attacker's classifier.
Our experimental results on three datasets show that MemGuard can effectively
defend against membership inference attacks and achieve better privacy-utility
tradeoffs than existing defenses. Our work is the first one to show that
adversarial examples can be used as defensive mechanisms to defend against
membership inference attacks.


http://arxiv.org/abs/1909.10147
Robust Local Features for Improving the Generalization of Adversarial Training.
Chuanbiao Song; Kun He; Jiadong Lin; Liwei Wang; John E. Hopcroft
  Adversarial training has been demonstrated as one of the most effective
methods for training robust models to defend against adversarial examples.
However, adversarially trained models often lack adversarially robust
generalization on unseen testing data. Recent works show that adversarially
trained models are more biased towards global structure features. Instead, in
this work, we would like to investigate the relationship between the
generalization of adversarial training and the robust local features, as the
robust local features generalize well for unseen shape variation. To learn the
robust local features, we develop a Random Block Shuffle (RBS) transformation
to break up the global structure features on normal adversarial examples. We
continue to propose a new approach called Robust Local Features for Adversarial
Training (RLFAT), which first learns the robust local features by adversarial
training on the RBS-transformed adversarial examples, and then transfers the
robust local features into the training of normal adversarial examples. To
demonstrate the generality of our argument, we implement RLFAT in currently
state-of-the-art adversarial training frameworks. Extensive experiments on
STL-10, CIFAR-10 and CIFAR-100 show that RLFAT significantly improves both the
adversarially robust generalization and the standard generalization of
adversarial training. Additionally, we demonstrate that our models capture more
local features of the object on the images, aligning better with human
perception.


http://arxiv.org/abs/1909.09938
HAWKEYE: Adversarial Example Detector for Deep Neural Networks.
Jinkyu Koo; Michael Roth; Saurabh Bagchi
  Adversarial examples (AEs) are images that can mislead deep neural network
(DNN) classifiers via introducing slight perturbations into original images.
Recent work has shown that detecting AEs can be more effective against AEs than
preventing them from being generated. However, the state-of-the-art AE
detection still shows a high false positive rate, thereby rejecting a
considerable amount of normal images. To address this issue, we propose
HAWKEYE, which is a separate neural network that analyzes the output layer of
the DNN, and detects AEs. HAWKEYE's AE detector utilizes a quantized version of
an input image as a reference, and is trained to distinguish the variation
characteristics of the DNN output on an input image from the DNN output on its
reference image. We also show that cascading our AE detectors that are trained
for different quantization step sizes can drastically reduce a false positive
rate, while keeping a detection rate high.


http://arxiv.org/abs/1909.09481
Adversarial Learning with Margin-based Triplet Embedding Regularization.
Yaoyao Zhong; Weihong Deng
  The Deep neural networks (DNNs) have achieved great success on a variety of
computer vision tasks, however, they are highly vulnerable to adversarial
attacks. To address this problem, we propose to improve the local smoothness of
the representation space, by integrating a margin-based triplet embedding
regularization term into the classification objective, so that the obtained
model learns to resist adversarial examples. The regularization term consists
of two steps optimizations which find potential perturbations and punish them
by a large margin in an iterative way. Experimental results on MNIST,
CASIA-WebFace, VGGFace2 and MS-Celeb-1M reveal that our approach increases the
robustness of the network against both feature and label adversarial attacks in
simple object classification and deep face recognition.


http://arxiv.org/abs/1909.09735
COPYCAT: Practical Adversarial Attacks on Visualization-Based Malware Detection.
Aminollah Khormali; Ahmed Abusnaina; Songqing Chen; DaeHun Nyang; Aziz Mohaisen
  Despite many attempts, the state-of-the-art of adversarial machine learning
on malware detection systems generally yield unexecutable samples. In this
work, we set out to examine the robustness of visualization-based malware
detection system against adversarial examples (AEs) that not only are able to
fool the model, but also maintain the executability of the original input. As
such, we first investigate the application of existing off-the-shelf
adversarial attack approaches on malware detection systems through which we
found that those approaches do not necessarily maintain the functionality of
the original inputs. Therefore, we proposed an approach to generate adversarial
examples, COPYCAT, which is specifically designed for malware detection systems
considering two main goals; achieving a high misclassification rate and
maintaining the executability and functionality of the original input. We
designed two main configurations for COPYCAT, namely AE padding and sample
injection. While the first configuration results in untargeted
misclassification attacks, the sample injection configuration is able to force
the model to generate a targeted output, which is highly desirable in the
malware attribution setting. We evaluate the performance of COPYCAT through an
extensive set of experiments on two malware datasets, and report that we were
able to generate adversarial samples that are misclassified at a rate of 98.9%
and 96.5% with Windows and IoT binary datasets, respectively, outperforming the
misclassification rates in the literature. Most importantly, we report that
those AEs were executable unlike AEs generated by off-the-shelf approaches. Our
transferability study demonstrates that the generated AEs through our proposed
method can be generalized to other models.


http://arxiv.org/abs/1909.09552
Defending Against Physically Realizable Attacks on Image Classification.
Tong Wu; Liang Tong; Yevgeniy Vorobeychik
  We study the problem of defending deep neural network approaches for image
classification from physically realizable attacks. First, we demonstrate that
the two most scalable and effective methods for learning robust models,
adversarial training with PGD attacks and randomized smoothing, exhibit very
limited effectiveness against three of the highest profile physical attacks.
Next, we propose a new abstract adversarial model, rectangular occlusion
attacks, in which an adversary places a small adversarially crafted rectangle
in an image, and develop two approaches for efficiently computing the resulting
adversarial examples. Finally, we demonstrate that adversarial training using
our new attack yields image classification models that exhibit high robustness
against the physically realizable attacks we study, offering the first
effective generic defense against such attacks.


http://arxiv.org/abs/1909.09263
Propagated Perturbation of Adversarial Attack for well-known CNNs: Empirical Study and its Explanation.
Jihyeun Yoon; Kyungyul Kim; Jongseong Jang
  Deep Neural Network based classifiers are known to be vulnerable to
perturbations of inputs constructed by an adversarial attack to force
misclassification. Most studies have focused on how to make vulnerable noise by
gradient based attack methods or to defense model from adversarial attack. The
use of the denoiser model is one of a well-known solution to reduce the
adversarial noise although classification performance had not significantly
improved. In this study, we aim to analyze the propagation of adversarial
attack as an explainable AI(XAI) point of view. Specifically, we examine the
trend of adversarial perturbations through the CNN architectures. To analyze
the propagated perturbation, we measured normalized Euclidean Distance and
cosine distance in each CNN layer between the feature map of the perturbed
image passed through denoiser and the non-perturbed original image. We used
five well-known CNN based classifiers and three gradient-based adversarial
attacks. From the experimental results, we observed that in most cases,
Euclidean Distance explosively increases in the final fully connected layer
while cosine distance fluctuated and disappeared at the last layer. This means
that the use of denoiser can decrease the amount of noise. However, it failed
to defense accuracy degradation.


http://arxiv.org/abs/1909.08864
Adversarial Vulnerability Bounds for Gaussian Process Classification.
Michael Thomas Smith; Kathrin Grosse; Michael Backes; Mauricio A Alvarez
  Machine learning (ML) classification is increasingly used in safety-critical
systems. Protecting ML classifiers from adversarial examples is crucial. We
propose that the main threat is that of an attacker perturbing a confidently
classified input to produce a confident misclassification. To protect against
this we devise an adversarial bound (AB) for a Gaussian process classifier,
that holds for the entire input domain, bounding the potential for any future
adversarial method to cause such misclassification. This is a formal guarantee
of robustness, not just an empirically derived result. We investigate how to
configure the classifier to maximise the bound, including the use of a sparse
approximation, leading to the method producing a practical, useful and provably
robust classifier, which we test using a variety of datasets.


http://arxiv.org/abs/1909.08830
Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks.
Sekitoshi Kanai; Yasutoshi Ida; Yasuhiro Fujiwara; Masanori Yamada; Shuichi Adachi
  We propose Absum, which is a regularization method for improving adversarial
robustness of convolutional neural networks (CNNs). Although CNNs can
accurately recognize images, recent studies have shown that the convolution
operations in CNNs commonly have structural sensitivity to specific noise
composed of Fourier basis functions. By exploiting this sensitivity, they
proposed a simple black-box adversarial attack: Single Fourier attack. To
reduce structural sensitivity, we can use regularization of convolution filter
weights since the sensitivity of linear transform can be assessed by the norm
of the weights. However, standard regularization methods can prevent
minimization of the loss function because they impose a tight constraint for
obtaining high robustness. To solve this problem, Absum imposes a loose
constraint; it penalizes the absolute values of the summation of the parameters
in the convolution layers. Absum can improve robustness against single Fourier
attack while being as simple and efficient as standard regularization methods
(e.g., weight decay and L1 regularization). Our experiments demonstrate that
Absum improves robustness against single Fourier attack more than standard
regularization methods. Furthermore, we reveal that robust CNNs with Absum are
more robust against transferred attacks due to decreasing the common
sensitivity and against high-frequency noise than standard regularization
methods. We also reveal that Absum can improve robustness against
gradient-based attacks (projected gradient descent) when used with adversarial
training.


http://arxiv.org/abs/1909.09034
Training Robust Deep Neural Networks via Adversarial Noise Propagation.
Aishan Liu; Xianglong Liu; Chongzhi Zhang; Hang Yu; Qiang Liu; Junfeng He
  Deep neural networks have been found vulnerable to noises like adversarial
examples and corruption in practice. A number of adversarial defense methods
have been developed, which indeed improve the model robustness towards
adversarial examples in practice. However, only relying on training with the
data mixed with noises, most of them still fail to defend the generalized types
of noises. Motivated by the fact that hidden layers play a very important role
in maintaining a robust model, this paper comes up with a simple yet powerful
training algorithm named Adversarial Noise Propagation (ANP) that injects
diversified noises into the hidden layers in a layer-wise manner. We show that
ANP can be efficiently implemented by exploiting the nature of the popular
backward-forward training style for deep models. To comprehensively understand
the behaviors and contributions of hidden layers, we further explore the
insights from hidden representation insensitivity and human vision perception
alignment. Extensive experiments on MNIST, CIFAR-10, CIFAR-10-C, CIFAR-10-P and
ImageNet demonstrate that ANP enables the strong robustness for deep models
against the generalized noises including both adversarial and corrupted ones,
and significantly outperforms various adversarial defense methods.


http://arxiv.org/abs/1909.12927
Toward Robust Image Classification.
Basemah Alshemali; Alta Graham; Jugal Kalita
  Neural networks are frequently used for image classification, but can be
vulnerable to misclassification caused by adversarial images. Attempts to make
neural network image classification more robust have included variations on
preprocessing (cropping, applying noise, blurring), adversarial training, and
dropout randomization. In this paper, we implemented a model for adversarial
detection based on a combination of two of these techniques: dropout
randomization with preprocessing applied to images within a given Bayesian
uncertainty. We evaluated our model on the MNIST dataset, using adversarial
images generated using Fast Gradient Sign Method (FGSM), Jacobian-based
Saliency Map Attack (JSMA) and Basic Iterative Method (BIM) attacks. Our model
achieved an average adversarial image detection accuracy of 97%, with an
average image classification accuracy, after discarding images flagged as
adversarial, of 99%. Our average detection accuracy exceeded that of recent
papers using similar techniques.


http://arxiv.org/abs/1909.08072
Adversarial Attacks and Defenses in Images, Graphs and Text: A Review.
Han Xu; Yao Ma; Haochen Liu; Debayan Deb; Hui Liu; Jiliang Tang; Anil Jain
  Deep neural networks (DNN) have achieved unprecedented success in numerous
machine learning tasks in various domains. However, the existence of
adversarial examples raises our concerns in adopting deep learning to
safety-critical applications. As a result, we have witnessed increasing
interests in studying attack and defense mechanisms for DNN models on different
data types, such as images, graphs and text. Thus, it is necessary to provide a
systematic and comprehensive overview of the main threats of attacks and the
success of corresponding countermeasures. In this survey, we review the state
of the art algorithms for generating adversarial examples and the
countermeasures against adversarial examples, for three most popular data
types, including images, graphs and text.


http://arxiv.org/abs/1909.07873
Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Model.
Prashanth Vijayaraghavan; Deb Roy
  Recently, generating adversarial examples has become an important means of
measuring robustness of a deep learning model. Adversarial examples help us
identify the susceptibilities of the model and further counter those
vulnerabilities by applying adversarial training techniques. In natural
language domain, small perturbations in the form of misspellings or paraphrases
can drastically change the semantics of the text. We propose a reinforcement
learning based approach towards generating adversarial examples in black-box
settings. We demonstrate that our method is able to fool well-trained models
for (a) IMDB sentiment classification task and (b) AG's news corpus news
categorization task with significantly high success rates. We find that the
adversarial examples generated are semantics-preserving perturbations to the
original text.


http://arxiv.org/abs/1909.08526
Defending against Machine Learning based Inference Attacks via Adversarial Examples: Opportunities and Challenges.
Jinyuan Jia; Neil Zhenqiang Gong
  As machine learning (ML) becomes more and more powerful and easily
accessible, attackers increasingly leverage ML to perform automated large-scale
inference attacks in various domains. In such an ML-equipped inference attack,
an attacker has access to some data (called public data) of an individual, a
software, or a system; and the attacker uses an ML classifier to automatically
infer their private data. Inference attacks pose severe privacy and security
threats to individuals and systems. Inference attacks are successful because
private data are statistically correlated with public data, and ML classifiers
can capture such statistical correlations. In this chapter, we discuss the
opportunities and challenges of defending against ML-equipped inference attacks
via adversarial examples. Our key observation is that attackers rely on ML
classifiers in inference attacks. The adversarial machine learning community
has demonstrated that ML classifiers have various vulnerabilities. Therefore,
we can turn the vulnerabilities of ML into defenses against inference attacks.
For example, ML classifiers are vulnerable to adversarial examples, which add
carefully crafted noise to normal examples such that an ML classifier makes
predictions for the examples as we desire. To defend against inference attacks,
we can add carefully crafted noise into the public data to turn them into
adversarial examples, such that attackers' classifiers make incorrect
predictions for the private data. However, existing methods to construct
adversarial examples are insufficient because they did not consider the unique
challenges and requirements for the crafted noise at defending against
inference attacks. In this chapter, we take defending against inference attacks
in online social networks as an example to illustrate the opportunities and
challenges.


http://arxiv.org/abs/1909.07558
HAD-GAN: A Human-perception Auxiliary Defense GAN model to Defend Adversarial Examples.
Wanting Yu; Hongyi Yu; Lingyun Jiang; Mengli Zhang; Kai Qiao; Linyuan Wang; Bin Yan
  Adversarial examples reveal the vulnerability and unexplained nature of
neural networks. It is of great practical significance to study the defense of
adversarial examples. In fact, most adversarial examples that misclassify
networks are often undetectable by humans. In this paper, we propose a defense
model to train the classifier into a human-perception classification model with
shape preference. The proposed model consisting of a TTN (Texture Transfer
Network) and an auxiliary defense GAN (Generative Adversarial Networks) is
called HAD-GAN (Human-perception Auxiliary Defense GAN). The TTN is used to
extend the texture samples of a clean image and makes classifiers more focused
on its shape. And GAN is utilized to form a training framework for the model
and generate the images we need. A series of experiments conducted on MNIST,
Fashion-MNIST and CIFAR10 show that the proposed model outperforms the
state-of-the-art defense methods for network robust, and have a significant
improvement on defense ability of adversarial examples.


http://arxiv.org/abs/1909.07490
They Might NOT Be Giants: Crafting Black-Box Adversarial Examples with Fewer Queries Using Particle Swarm Optimization.
Rayan Mosli; Matthew Wright; Bo Yuan; Yin Pan
  Machine learning models have been found to be susceptible to adversarial
examples that are often indistinguishable from the original inputs. These
adversarial examples are created by applying adversarial perturbations to input
samples, which would cause them to be misclassified by the target models.
Attacks that search and apply the perturbations to create adversarial examples
are performed in both white-box and black-box settings, depending on the
information available to the attacker about the target. For black-box attacks,
the only capability available to the attacker is the ability to query the
target with specially crafted inputs and observing the labels returned by the
model. Current black-box attacks either have low success rates, requires a high
number of queries, or produce adversarial examples that are easily
distinguishable from their sources. In this paper, we present AdversarialPSO, a
black-box attack that uses fewer queries to create adversarial examples with
high success rates. AdversarialPSO is based on the evolutionary search
algorithm Particle Swarm Optimization, a populationbased gradient-free
optimization algorithm. It is flexible in balancing the number of queries
submitted to the target vs the quality of imperceptible adversarial examples.
The attack has been evaluated using the image classification benchmark datasets
CIFAR-10, MNIST, and Imagenet, achieving success rates of 99.6%, 96.3%, and
82.0%, respectively, while submitting substantially fewer queries than the
state-of-the-art. We also present a black-box method for isolating salient
features used by models when making classifications. This method, called Swarms
with Individual Search Spaces or SWISS, creates adversarial examples by finding
and modifying the most important features in the input.


http://arxiv.org/abs/1909.07283
Towards Quality Assurance of Software Product Lines with Adversarial Configurations.
Paul Temple; Mathieu Acher; Gilles Perrouin; Battista Biggio; Jean-marc Jezequel; Fabio Roli
  Software product line (SPL) engineers put a lot of effort to ensure that,
through the setting of a large number of possible configuration options,
products are acceptable and well-tailored to customers' needs. Unfortunately,
options and their mutual interactions create a huge configuration space which
is intractable to exhaustively explore. Instead of testing all products,
machine learning techniques are increasingly employed to approximate the set of
acceptable products out of a small training sample of configurations. Machine
learning (ML) techniques can refine a software product line through learned
constraints and a priori prevent non-acceptable products to be derived. In this
paper, we use adversarial ML techniques to generate adversarial configurations
fooling ML classifiers and pinpoint incorrect classifications of products
(videos) derived from an industrial video generator. Our attacks yield (up to)
a 100% misclassification rate and a drop in accuracy of 5%. We discuss the
implications these results have on SPL quality assurance.


http://arxiv.org/abs/1909.06978
Interpreting and Improving Adversarial Robustness with Neuron Sensitivity.
Chongzhi Zhang; Aishan Liu; Xianglong Liu; Yitao Xu; Hang Yu; Yuqing Ma; Tianlin Li
  Deep neural networks (DNNs) are vulnerable to adversarial examples where
inputs with imperceptible perturbations mislead DNNs to incorrect results.
Despite the potential risk they bring, adversarial examples are also valuable
for providing insights into the weakness and blind-spots of DNNs. Thus, the
interpretability of a DNN in adversarial setting aims to explain the rationale
behind its decision-making process and makes deeper understanding which results
in better practical applications. To address this issue, we try to explain
adversarial robustness for deep models from a new perspective of neuron
sensitivity which is measured by neuron behavior variation intensity against
benign and adversarial examples. In this paper, we first draw the close
connection between adversarial robustness and neuron sensitivities, as
sensitive neurons make the most non-trivial contributions to model predictions
in adversarial setting. Based on that, we further propose to improve
adversarial robustness by constraining the similarities of sensitive neurons
between benign and adversarial examples which stabilizes the behaviors of
sensitive neurons in adversarial setting. Moreover, we demonstrate that
state-of-the-art adversarial training methods improve model robustness by
reducing neuron sensitivities which in turn confirms the strong connections
between adversarial robustness and neuron sensitivity as well as the
effectiveness of using sensitive neurons to build robust models. Extensive
experiments on various datasets demonstrate that our algorithm effectively
achieve excellent results.


http://arxiv.org/abs/1909.06727
An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms.
Qianyu Guo; Sen Chen; Xiaofei Xie; Lei Ma; Qiang Hu; Hongtao Liu; Yang Liu; Jianjun Zhao; Xiaohong Li
  Deep Learning (DL) has recently achieved tremendous success. A variety of DL
frameworks and platforms play a key role to catalyze such progress. However,
the differences in architecture designs and implementations of existing
frameworks and platforms bring new challenges for DL software development and
deployment. Till now, there is no study on how various mainstream frameworks
and platforms influence both DL software development and deployment in
practice. To fill this gap, we take the first step towards understanding how
the most widely-used DL frameworks and platforms support the DL software
development and deployment. We conduct a systematic study on these frameworks
and platforms by using two types of DNN architectures and three popular
datasets. (1) For development process, we investigate the prediction accuracy
under the same runtime training configuration or same model weights/biases. We
also study the adversarial robustness of trained models by leveraging the
existing adversarial attack techniques. The experimental results show that the
computing differences across frameworks could result in an obvious prediction
accuracy decline, which should draw the attention of DL developers. (2) For
deployment process, we investigate the prediction accuracy and performance
(refers to time cost and memory consumption) when the trained models are
migrated/quantized from PC to real mobile devices and web browsers. The DL
platform study unveils that the migration and quantization still suffer from
compatibility and reliability issues. Meanwhile, we find several DL software
bugs by using the results as a benchmark. We further validate the results
through bug confirmation from stakeholders and industrial positive feedback to
highlight the implications of our study. Through our study, we summarize
practical guidelines, identify challenges and pinpoint new research directions.


http://arxiv.org/abs/1909.06872
Detecting Adversarial Samples Using Influence Functions and Nearest Neighbors.
Gilad Cohen; Guillermo Sapiro; Raja Giryes
  Deep neural networks (DNNs) are notorious for their vulnerability to
adversarial attacks, which are small perturbations added to their input images
to mislead their prediction. Detection of adversarial examples is, therefore, a
fundamental requirement for robust classification frameworks. In this work, we
present a method for detecting such adversarial attacks, which is suitable for
any pre-trained neural network classifier. We use influence functions to
measure the impact of every training sample on the validation set data. From
the influence scores, we find the most supportive training samples for any
given validation example. A k-nearest neighbor (k-NN) model fitted on the DNN's
activation layers is employed to search for the ranking of these supporting
training samples. We observe that these samples are highly correlated with the
nearest neighbors of the normal inputs, while this correlation is much weaker
for adversarial inputs. We train an adversarial detector using the k-NN ranks
and distances and show that it successfully distinguishes adversarial examples,
getting state-of-the-art results on four attack methods with three datasets.


http://arxiv.org/abs/1909.06723
Natural Language Adversarial Attacks and Defenses in Word Level.
Xiaosen Wang; Hao Jin; Kun He
  Up until recent two years, inspired by the big amount of research about
adversarial example in the field of computer vision, there has been a growing
interest in adversarial attacks for Natural Language Processing (NLP). What
followed was a very few works of adversarial defense for NLP. However, there
exists no defense method against the successful synonyms substitution based
attacks that aim to satisfy all the lexical, grammatical, semantic constraints
and thus are hard to perceived by humans. To fill this gap, we postulate the
generalization of the model leads to the existence of adversarial examples, and
propose an adversarial defense method called Synonyms Encoding Method (SEM),
which inserts an encoder before the input layer of the model and then trains
the model to eliminate adversarial perturbations. Extensive experiments
demonstrate that SEM can efficiently defend current best synonym substitution
based adversarial attacks with almost no decay on the accuracy for benign
examples. Besides, to better evaluate SEM, we also propose a strong attack
method called Improved Genetic Algorithm (IGA) that adopts the genetic
metaheuristic against synonyms substitution based attacks. Compared with
existing genetic based adversarial attack, the proposed IGA can achieve higher
attack success rate at the same time maintain the transferability of
adversarial examples.


http://arxiv.org/abs/1909.06500
Adversarial Attack on Skeleton-based Human Action Recognition.
Jian Liu; Naveed Akhtar; Ajmal Mian
  Deep learning models achieve impressive performance for skeleton-based human
action recognition. However, the robustness of these models to adversarial
attacks remains largely unexplored due to their complex spatio-temporal nature
that must represent sparse and discrete skeleton joints. This work presents the
first adversarial attack on skeleton-based action recognition with graph
convolutional networks. The proposed targeted attack, termed Constrained
Iterative Attack for Skeleton Actions (CIASA), perturbs joint locations in an
action sequence such that the resulting adversarial sequence preserves the
temporal coherence, spatial integrity, and the anthropomorphic plausibility of
the skeletons. CIASA achieves this feat by satisfying multiple physical
constraints, and employing spatial skeleton realignments for the perturbed
skeletons along with regularization of the adversarial skeletons with
Generative networks. We also explore the possibility of semantically
imperceptible localized attacks with CIASA, and succeed in fooling the
state-of-the-art skeleton action recognition models with high confidence. CIASA
perturbations show high transferability for black-box attacks. We also show
that the perturbed skeleton sequences are able to induce adversarial behavior
in the RGB videos created with computer graphics. A comprehensive evaluation
with NTU and Kinetics datasets ascertains the effectiveness of CIASA for
graph-based skeleton action recognition and reveals the imminent threat to the
spatio-temporal deep learning tasks in general.


http://arxiv.org/abs/1909.06044
Say What I Want: Towards the Dark Side of Neural Dialogue Models.
Haochen Liu; Tyler Derr; Zitao Liu; Jiliang Tang
  Neural dialogue models have been widely adopted in various chatbot
applications because of their good performance in simulating and generalizing
human conversations. However, there exists a dark side of these models -- due
to the vulnerability of neural networks, a neural dialogue model can be
manipulated by users to say what they want, which brings in concerns about the
security of practical chatbot services. In this work, we investigate whether we
can craft inputs that lead a well-trained black-box neural dialogue model to
generate targeted outputs. We formulate this as a reinforcement learning (RL)
problem and train a Reverse Dialogue Generator which efficiently finds such
inputs for targeted outputs. Experiments conducted on a representative neural
dialogue model show that our proposed model is able to discover such desired
inputs in a considerable portion of cases. Overall, our work reveals this
weakness of neural dialogue models and may prompt further researches of
developing corresponding solutions to avoid it.


http://arxiv.org/abs/1909.06271
White-Box Adversarial Defense via Self-Supervised Data Estimation.
Zudi Lin; Hanspeter Pfister; Ziming Zhang
  In this paper, we study the problem of how to defend classifiers against
adversarial attacks that fool the classifiers using subtly modified input data.
In contrast to previous works, here we focus on the white-box adversarial
defense where the attackers are granted full access to not only the classifiers
but also defenders to produce as strong attacks as possible. In such a context
we propose viewing a defender as a functional, a higher-order function that
takes functions as its argument to represent a function space, rather than
fixed functions conventionally. From this perspective, a defender should be
realized and optimized individually for each adversarial input. To this end, we
propose RIDE, an efficient and provably convergent self-supervised learning
algorithm for individual data estimation to protect the predictions from
adversarial attacks. We demonstrate the significant improvement of adversarial
defense performance on image recognition, eg, 98%, 76%, 43% test accuracy on
MNIST, CIFAR-10, and ImageNet datasets respectively under the state-of-the-art
BPDA attacker.


http://arxiv.org/abs/1909.06137
Defending Against Adversarial Attacks by Suppressing the Largest Eigenvalue of Fisher Information Matrix.
Chaomin Shen; Yaxin Peng; Guixu Zhang; Jinsong Fan
  We propose a scheme for defending against adversarial attacks by suppressing
the largest eigenvalue of the Fisher information matrix (FIM). Our starting
point is one explanation on the rationale of adversarial examples. Based on the
idea of the difference between a benign sample and its adversarial example is
measured by the Euclidean norm, while the difference between their
classification probability densities at the last (softmax) layer of the network
could be measured by the Kullback-Leibler (KL) divergence, the explanation
shows that the output difference is a quadratic form of the input difference.
If the eigenvalue of this quadratic form (a.k.a. FIM) is large, the output
difference becomes large even when the input difference is small, which
explains the adversarial phenomenon. This makes the adversarial defense
possible by controlling the eigenvalues of the FIM. Our solution is adding one
term representing the trace of the FIM to the loss function of the original
network, as the largest eigenvalue is bounded by the trace. Our defensive
scheme is verified by experiments using a variety of common attacking methods
on typical deep neural networks, e.g. LeNet, VGG and ResNet, with datasets
MNIST, CIFAR-10, and German Traffic Sign Recognition Benchmark (GTSRB). Our new
network, after adopting the novel loss function and retraining, has an
effective and robust defensive capability, as it decreases the fooling ratio of
the generated adversarial examples, and remains the classification accuracy of
the original network.


http://arxiv.org/abs/1909.05527
Inspecting adversarial examples using the Fisher information.
Jörg Martin; Clemens Elster
  Adversarial examples are slight perturbations that are designed to fool
artificial neural networks when fed as an input. In this work the usability of
the Fisher information for the detection of such adversarial attacks is
studied. We discuss various quantities whose computation scales well with the
network size, study their behavior on adversarial examples and show how they
can highlight the importance of single input neurons, thereby providing a
visual tool for further analyzing (un-)reasonable behavior of a neural network.
The potential of our methods is demonstrated by applications to the MNIST,
CIFAR10 and Fruits-360 datasets.


http://arxiv.org/abs/1909.05580
An Empirical Investigation of Randomized Defenses against Adversarial Attacks.
Yannik Potdevin; Dirk Nowotka; Vijay Ganesh
  In recent years, Deep Neural Networks (DNNs) have had a dramatic impact on a
variety of problems that were long considered very difficult, e. g., image
classification and automatic language translation to name just a few. The
accuracy of modern DNNs in classification tasks is remarkable indeed. At the
same time, attackers have devised powerful methods to construct
specially-crafted malicious inputs (often referred to as adversarial examples)
that can trick DNNs into mis-classifying them. What is worse is that despite
the many defense mechanisms proposed to protect DNNs against adversarial
attacks, attackers are often able to circumvent these defenses, rendering them
useless. This state of affairs is extremely worrying, especially since machine
learning systems get adopted at scale.
  In this paper, we propose a scientific evaluation methodology aimed at
assessing the quality, efficacy, robustness and efficiency of randomized
defenses to protect DNNs against adversarial examples. Using this methodology,
we evaluate a variety of defense mechanisms. In addition, we also propose a
defense mechanism we call Randomly Perturbed Ensemble Neural Networks (RPENNs).
We provide a thorough and comprehensive evaluation of the considered defense
mechanisms against a white-box attacker model, six different adversarial attack
methods and using the ILSVRC2012 validation data set.


http://arxiv.org/abs/1909.05921
Transferable Adversarial Robustness using Adversarially Trained Autoencoders.
Pratik Vaishnavi; Kevin Eykholt; Atul Prakash; Amir Rahmati
  Machine learning has proven to be an extremely useful tool for solving
complex problems in many application domains. This prevalence makes it an
attractive target for malicious actors. Adversarial machine learning is a
well-studied field of research in which an adversary seeks to cause predicable
errors in a machine learning algorithm through careful manipulation of the
input. In response, numerous techniques have been proposed to harden machine
learning algorithms and mitigate the effect of adversarial attacks. Of these
techniques, adversarial training, which augments the training data with
adversarial inputs, has proven to be an effective defensive technique. However,
adversarial training is computationally expensive and the improvements in
adversarial performance are limited to a single model. In this paper, we
propose Adversarially-Trained Autoencoder Augmentation, the first transferable
adversarial defense that is robust to certain adaptive adversaries. We
disentangle adversarial robustness from the classification pipeline by
adversarially training an autoencoder with respect to the classification loss.
We show that our approach achieves comparable results to state-of-the-art
adversarially trained models on the MNIST, Fashion-MNIST, and CIFAR-10
datasets. Furthermore, we can transfer our approach to other vulnerable models
and improve their adversarial performance without additional training. Finally,
we combine our defense with ensemble methods and parallelize adversarial
training across multiple vulnerable pre-trained models. In a single adversarial
training session, the autoencoder can achieve adversarial performance on the
vulnerable models that is comparable or better than standard adversarial
training.


http://arxiv.org/abs/1909.05443
Feedback Learning for Improving the Robustness of Neural Networks.
Chang Song; Zuoguan Wang; Hai Li
  Recent research studies revealed that neural networks are vulnerable to
adversarial attacks. State-of-the-art defensive techniques add various
adversarial examples in training to improve models' adversarial robustness.
However, these methods are not universal and can't defend unknown or
non-adversarial evasion attacks. In this paper, we analyze the model robustness
in the decision space. A feedback learning method is then proposed, to
understand how well a model learns and to facilitate the retraining process of
remedying the defects. The evaluations according to a set of distance-based
criteria show that our method can significantly improve models' accuracy and
robustness against different types of evasion attacks. Moreover, we observe the
existence of inter-class inequality and propose to compensate it by changing
the proportions of examples generated in different classes.


http://arxiv.org/abs/1909.05040
Sparse and Imperceivable Adversarial Attacks.
Francesco Croce; Matthias Hein
  Neural networks have been proven to be vulnerable to a variety of adversarial
attacks. From a safety perspective, highly sparse adversarial attacks are
particularly dangerous. On the other hand the pixelwise perturbations of sparse
attacks are typically large and thus can be potentially detected. We propose a
new black-box technique to craft adversarial examples aiming at minimizing
$l_0$-distance to the original image. Extensive experiments show that our
attack is better or competitive to the state of the art. Moreover, we can
integrate additional bounds on the componentwise perturbation. Allowing pixels
to change only in region of high variation and avoiding changes along
axis-aligned edges makes our adversarial examples almost non-perceivable.
Moreover, we adapt the Projected Gradient Descent attack to the $l_0$-norm
integrating componentwise constraints. This allows us to do adversarial
training to enhance the robustness of classifiers against sparse and
imperceivable adversarial manipulations.


http://arxiv.org/abs/1909.04779
Localized Adversarial Training for Increased Accuracy and Robustness in Image Classification.
Eitan Rothberg; Tingting Chen; Luo Jie; Hao Ji
  Today's state-of-the-art image classifiers fail to correctly classify
carefully manipulated adversarial images. In this work, we develop a new,
localized adversarial attack that generates adversarial examples by
imperceptibly altering the backgrounds of normal images. We first use this
attack to highlight the unnecessary sensitivity of neural networks to changes
in the background of an image, then use it as part of a new training technique:
localized adversarial training. By including locally adversarial images in the
training set, we are able to create a classifier that suffers less loss than a
non-adversarially trained counterpart model on both natural and adversarial
inputs. The evaluation of our localized adversarial training algorithm on MNIST
and CIFAR-10 datasets shows decreased accuracy loss on natural images, and
increased robustness against adversarial inputs.


http://arxiv.org/abs/1909.04837
Identifying and Resisting Adversarial Videos Using Temporal Consistency.
Xiaojun Jia; Xingxing Wei; Xiaochun Cao
  Video classification is a challenging task in computer vision. Although Deep
Neural Networks (DNNs) have achieved excellent performance in video
classification, recent research shows adding imperceptible perturbations to
clean videos can make the well-trained models output wrong labels with high
confidence. In this paper, we propose an effective defense framework to
characterize and defend adversarial videos. The proposed method contains two
phases: (1) adversarial video detection using temporal consistency between
adjacent frames, and (2) adversarial perturbation reduction via denoisers in
the spatial and temporal domains respectively. Specifically, because of the
linear nature of DNNs, the imperceptible perturbations will enlarge with the
increasing of DNNs depth, which leads to the inconsistency of DNNs output
between adjacent frames. However, the benign video frames often have the same
outputs with their neighbor frames owing to the slight changes. Based on this
observation, we can distinguish between adversarial videos and benign videos.
After that, we utilize different defense strategies against different attacks.
We propose the temporal defense, which reconstructs the polluted frames with
their temporally neighbor clean frames, to deal with the adversarial videos
with sparse polluted frames. For the videos with dense polluted frames, we use
an efficient adversarial denoiser to process each frame in the spatial domain,
and thus purify the perturbations (we call it as spatial defense). A series of
experiments conducted on the UCF-101 dataset demonstrate that the proposed
method significantly improves the robustness of video classifiers against
adversarial attacks.


http://arxiv.org/abs/1909.04778
Effectiveness of Adversarial Examples and Defenses for Malware Classification.
Robert Podschwadt; Hassan Takabi
  Artificial neural networks have been successfully used for many different
classification tasks including malware detection and distinguishing between
malicious and non-malicious programs. Although artificial neural networks
perform very well on these tasks, they are also vulnerable to adversarial
examples. An adversarial example is a sample that has minor modifications made
to it so that the neural network misclassifies it. Many techniques have been
proposed, both for crafting adversarial examples and for hardening neural
networks against them. Most previous work has been done in the image domain.
Some of the attacks have been adopted to work in the malware domain which
typically deals with binary feature vectors. In order to better understand the
space of adversarial examples in malware classification, we study different
approaches of crafting adversarial examples and defense techniques in the
malware domain and compare their effectiveness on multiple datasets.


http://arxiv.org/abs/1909.04839
Towards Noise-Robust Neural Networks via Progressive Adversarial Training.
Hang Yu; Aishan Liu; Xianglong Liu; Jichen Yang; Chongzhi Zhang
  Adversarial examples, intentionally designed inputs tending to mislead deep
neural networks, have attracted great attention in the past few years. Although
a series of defense strategies have been developed and achieved encouraging
model robustness, most of them are still vulnerable to the more commonly
witnessed corruptions, e.g., Gaussian noise, blur, etc., in the real world. In
this paper, we theoretically and empirically discover the fact that there
exists an inherent connection between adversarial robustness and corruption
robustness. Based on the fundamental discovery, this paper further proposes a
more powerful training method named Progressive Adversarial Training (PAT) that
adds diversified adversarial noises progressively during training, and thus
obtains robust model against both adversarial examples and corruptions through
higher training data complexity. Meanwhile, we also theoretically find that PAT
can promise better generalization ability. Experimental evaluation on MNIST,
CIFAR-10 and SVHN show that PAT is able to enhance the robustness and
generalization of the state-of-the-art network structures, performing
comprehensively well compared to various augmentation methods. Moreover, we
also propose Mixed Test to evaluate model generalization ability more fairly.


http://arxiv.org/abs/1909.04326
UPC: Learning Universal Physical Camouflage Attacks on Object Detectors.
Lifeng Huang; Chengying Gao; Yuyin Zhou; Changqing Zou; Cihang Xie; Alan Yuille; Ning Liu
  In this paper, we study physical adversarial attacks on object detectors in
the wild. Prior arts on this matter mostly craft instance-dependent
perturbations only for rigid and planar objects. To this end, we propose to
learn an adversarial pattern to effectively attack all instances belonging to
the same object category (e.g., person, car), referred to as Universal Physical
Camouflage Attack (UPC). Concretely, UPC crafts camouflage by jointly fooling
the region proposal network, as well as misleading the classifier and the
regressor to output errors. In order to make UPC effective for articulated
non-rigid or non-planar objects, we introduce a set of transformations for the
generated camouflage patterns to mimic their deformable properties. We
additionally impose optimization constraint to make generated patterns look
natural for human observers. To fairly evaluate the effectiveness of different
physical-world attacks on object detectors, we present the first standardized
virtual database, AttackScenes, which simulates the real 3D world in a
controllable and reproducible environment. Extensive experiments suggest the
superiority of our proposed UPC compared with existing physical adversarial
attackers not only in virtual environments (AttackScenes), but also in
real-world physical environments. Codes, models, and demos are publicly
available at https://mesunhlf.github.io/index_physical.html.


http://arxiv.org/abs/1909.04385
FDA: Feature Disruptive Attack.
Aditya Ganeshan; B. S. Vivek; R. Venkatesh Babu
  Though Deep Neural Networks (DNN) show excellent performance across various
computer vision tasks, several works show their vulnerability to adversarial
samples, i.e., image samples with imperceptible noise engineered to manipulate
the network's prediction. Adversarial sample generation methods range from
simple to complex optimization techniques. Majority of these methods generate
adversaries through optimization objectives that are tied to the pre-softmax or
softmax output of the network. In this work we, (i) show the drawbacks of such
attacks, (ii) propose two new evaluation metrics: Old Label New Rank (OLNR) and
New Label Old Rank (NLOR) in order to quantify the extent of damage made by an
attack, and (iii) propose a new adversarial attack FDA: Feature Disruptive
Attack, to address the drawbacks of existing attacks. FDA works by generating
image perturbation that disrupt features at each layer of the network and
causes deep-features to be highly corrupt. This allows FDA adversaries to
severely reduce the performance of deep networks. We experimentally validate
that FDA generates stronger adversaries than other state-of-the-art methods for
image classification, even in the presence of various defense measures. More
importantly, we show that FDA disrupts feature-representation based tasks even
without access to the task-specific network or methodology. Code available at:
https://github.com/BardOfCodes/fda


http://arxiv.org/abs/1909.04311
Learning to Disentangle Robust and Vulnerable Features for Adversarial Detection.
Byunggill Joe; Sung Ju Hwang; Insik Shin
  Although deep neural networks have shown promising performances on various
tasks, even achieving human-level performance on some, they are shown to be
susceptible to incorrect predictions even with imperceptibly small
perturbations to an input. There exists a large number of previous works which
proposed to defend against such adversarial attacks either by robust inference
or detection of adversarial inputs. Yet, most of them cannot effectively defend
against whitebox attacks where an adversary has a knowledge of the model and
defense. More importantly, they do not provide a convincing reason why the
generated adversarial inputs successfully fool the target models. To address
these shortcomings of the existing approaches, we hypothesize that the
adversarial inputs are tied to latent features that are susceptible to
adversarial perturbation, which we call vulnerable features. Then based on this
intuition, we propose a minimax game formulation to disentangle the latent
features of each instance into robust and vulnerable ones, using variational
autoencoders with two latent spaces. We thoroughly validate our model for both
blackbox and whitebox attacks on MNIST, Fashion MNIST5, and Cat & Dog datasets,
whose results show that the adversarial inputs cannot bypass our detector
without changing its semantics, in which case the attack has failed.


http://arxiv.org/abs/1909.04288
Toward Finding The Global Optimal of Adversarial Examples.
Zhenxin Xiao; Kai-Wei Chang; Cho-Jui Hsieh
  Current machine learning models are vulnerable to adversarial examples
(Goodfellow et al., 2014), we noticed that current state-of-the-art methods
(Kurakin et al., 2016; Cheng et al., 2018) to attack a well-trained model often
stuck in local optimal values. We conduct series of experiments on both
white-box and black-box settings, and find out that by different
initialization, the attack algorithm will finally converge to very different
local optimals, suggesting the importance of careful and thorough search in the
attack space. In this paper, we propose a general boosting algorithm that can
help current attack to find a more global optimal example. Specifically, we
search for the adversarial examples by starting from different
points/directions, and in certain interval we adopt successive halving
(Jamieson & Talwalkar, 2016) to cut down the searching directions that are not
promising, and use Bayesian Optimization (Pelikan et al., 1999; Bergstra et
al., 2011) to resample from the search space based on the knowledge obtained
from past searches. We demonstrate that by applying our methods to
state-of-the-art attack algorithms in both black-and white box setting, we can
further reduce the distortion between the original image and adversarial sample
about 10%-20%. By adopting dynamic successive halving, we can reduce the
computation cost 5-10 times without harming the final result. We conduct
experiments in models trained on MNIST or ImageNet and also try on decision
tree models, these experiments suggest that our method is a general way to
boost the performance of current adversarial attack methods.


http://arxiv.org/abs/1909.04068
Adversarial Robustness Against the Union of Multiple Perturbation Models.
Pratyush Maini; Eric Wong; J. Zico Kolter
  Owing to the susceptibility of deep learning systems to adversarial attacks,
there has been a great deal of work in developing (both empirically and
certifiably) robust classifiers, but the vast majority has defended against
single types of attacks. Recent work has looked at defending against multiple
attacks, specifically on the MNIST dataset, yet this approach used a relatively
complex architecture, claiming that standard adversarial training can not apply
because it "overfits" to a particular norm. In this work, we show that it is
indeed possible to adversarially train a robust model against a union of
norm-bounded attacks, by using a natural generalization of the standard
PGD-based procedure for adversarial training to multiple threat models. With
this approach, we are able to train standard architectures which are robust
against $\ell_\infty$, $\ell_2$, and $\ell_1$ attacks, outperforming past
approaches on the MNIST dataset and providing the first CIFAR10 network trained
to be simultaneously robust against $(\ell_{\infty}, \ell_{2},\ell_{1})$ threat
models, which achieves adversarial accuracy rates of $(47.6\%, 64.8\%, 53.4\%)$
for $(\ell_{\infty}, \ell_{2},\ell_{1})$ perturbations with radius $\epsilon =
(0.03,0.5,12)$.


http://arxiv.org/abs/1909.03413
STA: Adversarial Attacks on Siamese Trackers.
Xugang Wu; Xiaoping Wang; Xu Zhou; Songlei Jian
  Recently, the majority of visual trackers adopt Convolutional Neural Network
(CNN) as their backbone to achieve high tracking accuracy. However, less
attention has been paid to the potential adversarial threats brought by CNN,
including Siamese network.
  In this paper, we first analyze the existing vulnerabilities in Siamese
trackers and propose the requirements for a successful adversarial attack. On
this basis, we formulate the adversarial generation problem and propose an
end-to-end pipeline to generate a perturbed texture map for the 3D object that
causes the trackers to fail. Finally, we conduct thorough experiments to verify
the effectiveness of our algorithm. Experiment results show that adversarial
examples generated by our algorithm can successfully lower the tracking
accuracy of victim trackers and even make them drift off. To the best of our
knowledge, this is the first work to generate 3D adversarial examples on visual
trackers.


http://arxiv.org/abs/1909.03418
When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures.
Gil Fidel; Ron Bitton; Asaf Shabtai
  State-of-the-art deep neural networks (DNNs) are highly effective in solving
many complex real-world problems. However, these models are vulnerable to
adversarial perturbation attacks, and despite the plethora of research in this
domain, to this day, adversaries still have the upper hand in the cat and mouse
game of adversarial example generation methods vs. detection and prevention
methods. In this research, we present a novel detection method that uses
Shapley Additive Explanations (SHAP) values computed for the internal layers of
a DNN classifier to discriminate between normal and adversarial inputs. We
evaluate our method by building an extensive dataset of adversarial examples
over the popular CIFAR-10 and MNIST datasets, and training a neural
network-based detector to distinguish between normal and adversarial inputs. We
evaluate our detector against adversarial examples generated by diverse
state-of-the-art attacks and demonstrate its high detection accuracy and strong
generalization ability to adversarial inputs generated with different attack
methods.


http://arxiv.org/abs/1909.03084
Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification.
Yichao Zhou; Jyun-Yu Jiang; Kai-Wei Chang; Wei Wang
  Adversarial attacks against machine learning models have threatened various
real-world applications such as spam filtering and sentiment analysis. In this
paper, we propose a novel framework, learning to DIScriminate Perturbations
(DISP), to identify and adjust malicious perturbations, thereby blocking
adversarial attacks for text classification models. To identify adversarial
attacks, a perturbation discriminator validates how likely a token in the text
is perturbed and provides a set of potential perturbations. For each potential
perturbation, an embedding estimator learns to restore the embedding of the
original word based on the context and a replacement token is chosen based on
approximate kNN search. DISP can block adversarial attacks for any NLP model
without modifying the model structure or training procedure. Extensive
experiments on two benchmark datasets demonstrate that DISP significantly
outperforms baseline methods in blocking adversarial attacks for text
classification. In addition, in-depth analysis shows the robustness of DISP
across different situations.


http://arxiv.org/abs/1909.04495
Natural Adversarial Sentence Generation with Gradient-based Perturbation.
Yu-Lun Hsieh; Minhao Cheng; Da-Cheng Juan; Wei Wei; Wen-Lian Hsu; Cho-Jui Hsieh
  This work proposes a novel algorithm to generate natural language adversarial
input for text classification models, in order to investigate the robustness of
these models. It involves applying gradient-based perturbation on the sentence
embeddings that are used as the features for the classifier, and learning a
decoder for generation. We employ this method to a sentiment analysis model and
verify its effectiveness in inducing incorrect predictions by the model. We
also conduct quantitative and qualitative analysis on these examples and
demonstrate that our approach can generate more natural adversaries. In
addition, it can be used to successfully perform black-box attacks, which
involves attacking other existing models whose parameters are not known. On a
public sentiment analysis API, the proposed method introduces a 20% relative
decrease in average accuracy and 74% relative increase in absolute error.


http://arxiv.org/abs/1909.02918
Blackbox Attacks on Reinforcement Learning Agents Using Approximated Temporal Information.
Yiren Zhao; Ilia Shumailov; Han Cui; Xitong Gao; Robert Mullins; Ross Anderson
  Recent research on reinforcement learning has shown that trained agents are
vulnerable to maliciously crafted adversarial samples. In this work, we show
how adversarial samples against RL agents can be generalised from White-box and
Grey-box attacks to a strong Black-box case, namely where the attacker has no
knowledge of the agents and their training methods. We use sequence-to-sequence
models to predict a single action or a sequence of future actions that a
trained agent will make. Our approximation model, based on time-series
information from the agent, successfully predicts agents' future actions with
consistently above 80% accuracy on a wide range of games and training methods.
Second, we find that although such adversarial samples are transferable, they
do not outperform random Gaussian noise as a means of reducing the game scores
of trained RL agents. This highlights a serious methodological deficiency in
previous work on such agents; random jamming should have been taken as the
baseline for evaluation. Third, we do find a novel use for adversarial samples
in this context: they can be used to trigger a trained agent to misbehave after
a specific delay. This appears to be a genuinely new type of attack; it
potentially enables an attacker to use devices controlled by RL agents as time
bombs.


http://arxiv.org/abs/1909.02583
Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents.
Xian Yeow Lee; Sambit Ghadai; Kai Liang Tan; Chinmay Hegde; Soumik Sarkar
  Robustness of Deep Reinforcement Learning (DRL) algorithms towards
adversarial attacks in real world applications such as those deployed in
cyber-physical systems (CPS) are of increasing concern. Numerous studies have
investigated the mechanisms of attacks on the RL agent's state space.
Nonetheless, attacks on the RL agent's action space (AS) (corresponding to
actuators in engineering systems) are equally perverse; such attacks are
relatively less studied in the ML literature. In this work, we first frame the
problem as an optimization problem of minimizing the cumulative reward of an RL
agent with decoupled constraints as the budget of attack. We propose a
white-box Myopic Action Space (MAS) attack algorithm that distributes the
attacks across the action space dimensions. Next, we reformulate the
optimization problem above with the same objective function, but with a
temporally coupled constraint on the attack budget to take into account the
approximated dynamics of the agent. This leads to the white-box Look-ahead
Action Space (LAS) attack algorithm that distributes the attacks across the
action and temporal dimensions. Our results shows that using the same amount of
resources, the LAS attack deteriorates the agent's performance significantly
more than the MAS attack. This reveals the possibility that with limited
resource, an adversary can utilize the agent's dynamics to malevolently craft
attacks that causes the agent to fail. Additionally, we leverage these attack
strategies as a possible tool to gain insights on the potential vulnerabilities
of DRL agents.


http://arxiv.org/abs/1909.02560
Adversarial Examples with Difficult Common Words for Paraphrase Identification.
Zhouxing Shi; Minlie Huang; Ting Yao; Jingfang Xu
  Despite the success of deep models for paraphrase identification on benchmark
datasets, these models are still vulnerable to adversarial examples. In this
paper, we propose a novel algorithm to generate a new type of adversarial
examples to study the robustness of deep paraphrase identification models. We
first sample an original sentence pair from the corpus and then adversarially
replace some word pairs with difficult common words. We take multiple steps and
use beam search to find a modification solution that makes the target model
fail, and thereby obtain an adversarial example. The word replacement is also
constrained by heuristic rules and a language model, to preserve the label and
grammaticality of the example during modification. Experiments show that our
algorithm can generate adversarial examples on which the performance of the
target model drops dramatically. Meanwhile, human annotators are much less
affected, and the generated sentences retain a good grammaticality. We also
show that adversarial training with generated adversarial examples can improve
model robustness.


http://arxiv.org/abs/1909.02436
Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?
Alfred Laugros; Alice Caplier; Matthieu Ospici
  Neural Networks have been shown to be sensitive to common perturbations such
as blur, Gaussian noise, rotations, etc. They are also vulnerable to some
artificial malicious corruptions called adversarial examples. The adversarial
examples study has recently become very popular and it sometimes even reduces
the term "adversarial robustness" to the term "robustness". Yet, we do not know
to what extent the adversarial robustness is related to the global robustness.
Similarly, we do not know if a robustness to various common perturbations such
as translations or contrast losses for instance, could help with adversarial
corruptions. We intend to study the links between the robustnesses of neural
networks to both perturbations. With our experiments, we provide one of the
first benchmark designed to estimate the robustness of neural networks to
common perturbations. We show that increasing the robustness to carefully
selected common perturbations, can make neural networks more robust to unseen
common perturbations. We also prove that adversarial robustness and robustness
to common perturbations are independent. Our results make us believe that
neural network robustness should be addressed in a broader sense.


http://arxiv.org/abs/1909.00986
Certified Robustness to Adversarial Word Substitutions.
Robin Jia; Aditi Raghunathan; Kerem Göksel; Percy Liang
  State-of-the-art NLP models can often be fooled by adversaries that apply
seemingly innocuous label-preserving transformations (e.g., paraphrasing) to
input text. The number of possible transformations scales exponentially with
text length, so data augmentation cannot cover all transformations of an input.
This paper considers one exponentially large family of label-preserving
transformations, in which every word in the input can be replaced with a
similar word. We train the first models that are provably robust to all word
substitutions in this family. Our training procedure uses Interval Bound
Propagation (IBP) to minimize an upper bound on the worst-case loss that any
combination of word substitutions can induce. To evaluate models' robustness to
these transformations, we measure accuracy on adversarially chosen word
substitutions applied to test examples. Our IBP-trained models attain $75\%$
adversarial accuracy on both sentiment analysis on IMDB and natural language
inference on SNLI. In comparison, on IMDB, models trained normally and ones
trained with data augmentation achieve adversarial accuracy of only $8\%$ and
$35\%$, respectively.


http://arxiv.org/abs/1909.01492
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation.
Po-Sen Huang; Robert Stanforth; Johannes Welbl; Chris Dyer; Dani Yogatama; Sven Gowal; Krishnamurthy Dvijotham; Pushmeet Kohli
  Neural networks are part of many contemporary NLP systems, yet their
empirical successes come at the price of vulnerability to adversarial attacks.
Previous work has used adversarial training and data augmentation to partially
mitigate such brittleness, but these are unlikely to find worst-case
adversaries due to the complexity of the search space arising from discrete
text perturbations. In this work, we approach the problem from the opposite
direction: to formally verify a system's robustness against a predefined class
of adversarial attacks. We study text classification under synonym replacements
or character flip perturbations. We propose modeling these input perturbations
as a simplex and then using Interval Bound Propagation -- a formal model
verification method. We modify the conventional log-likelihood training
objective to train models that can be efficiently verified, which would
otherwise come with exponential search complexity. The resulting models show
only little difference in terms of nominal accuracy, but have much improved
verified accuracy under perturbations and come with an efficiently computable
formal guarantee on worst case adversaries.


http://arxiv.org/abs/1909.00900
Metric Learning for Adversarial Robustness.
Chengzhi Mao; Ziyuan Zhong; Junfeng Yang; Carl Vondrick; Baishakhi Ray
  Deep networks are well-known to be fragile to adversarial attacks. Using
several standard image datasets and established attack mechanisms, we conduct
an empirical analysis of deep representations under attack, and find that the
attack causes the internal representation to shift closer to the "false" class.
Motivated by this observation, we propose to regularize the representation
space under attack with metric learning in order to produce more robust
classifiers. By carefully sampling examples for metric learning, our learned
representation not only increases robustness, but also can detect previously
unseen adversarial samples. Quantitative experiments show improvement of
robustness accuracy by up to 4\% and detection efficiency by up to 6\%
according to Area Under Curve (AUC) score over baselines.


http://arxiv.org/abs/1908.11514
Adversarial Training Methods for Network Embedding.
Quanyu Dai; Xiao Shen; Liang Zhang; Qiang Li; Dan Wang
  Network Embedding is the task of learning continuous node representations for
networks, which has been shown effective in a variety of tasks such as link
prediction and node classification. Most of existing works aim to preserve
different network structures and properties in low-dimensional embedding
vectors, while neglecting the existence of noisy information in many real-world
networks and the overfitting issue in the embedding learning process. Most
recently, generative adversarial networks (GANs) based regularization methods
are exploited to regularize embedding learning process, which can encourage a
global smoothness of embedding vectors. These methods have very complicated
architecture and suffer from the well-recognized non-convergence problem of
GANs. In this paper, we aim to introduce a more succinct and effective local
regularization method, namely adversarial training, to network embedding so as
to achieve model robustness and better generalization performance. Firstly, the
adversarial training method is applied by defining adversarial perturbations in
the embedding space with an adaptive $L_2$ norm constraint that depends on the
connectivity pattern of node pairs. Though effective as a regularizer, it
suffers from the interpretability issue which may hinder its application in
certain real-world scenarios. To improve this strategy, we further propose an
interpretable adversarial training method by enforcing the reconstruction of
the adversarial examples in the discrete graph domain. These two regularization
methods can be applied to many existing embedding models, and we take DeepWalk
as the base model for illustration in the paper. Empirical evaluations in both
link prediction and node classification demonstrate the effectiveness of the
proposed methods.


http://arxiv.org/abs/1908.11091
Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness.
Ling Liu; Wenqi Wei; Ka-Ho Chow; Margaret Loper; Emre Gursoy; Stacey Truex; Yanzhao Wu
  Ensemble learning is a methodology that integrates multiple DNN learners for
improving prediction performance of individual learners. Diversity is greater
when the errors of the ensemble prediction is more uniformly distributed.
Greater diversity is highly correlated with the increase in ensemble accuracy.
Another attractive property of diversity optimized ensemble learning is its
robustness against deception: an adversarial perturbation attack can mislead
one DNN model to misclassify but may not fool other ensemble DNN members
consistently. In this paper we first give an overview of the concept of
ensemble diversity and examine the three types of ensemble diversity in the
context of DNN classifiers. We then describe a set of ensemble diversity
measures, a suite of algorithms for creating diversity ensembles and for
performing ensemble consensus (voted or learned) for generating high accuracy
ensemble output by strategically combining outputs of individual members. This
paper concludes with a discussion on a set of open issues in quantifying
ensemble diversity for robust deep learning.


http://arxiv.org/abs/1908.11230
Defending Against Misclassification Attacks in Transfer Learning.
Bang Wu; Xiangwen Yang; Shuo Wang; Xingliang Yuan; Cong Wang; Carsten Rudolph
  Transfer learning accelerates the development of new models (Student Models).
It applies relevant knowledge from a pre-trained model (Teacher Model) to the
new ones with a small amount of training data, yet without affecting the model
accuracy. However, these Teacher Models are normally open in order to
facilitate sharing and reuse, which creates an attack plane in transfer
learning systems. Among others, recent emerging attacks demonstrate that
adversarial inputs can be built with negligible perturbations to the normal
inputs. Such inputs can mimic the internal features of the student models
directly based on the knowledge of the Teacher Models and cause
misclassification in final predictions.
  In this paper, we propose an effective defence against the above
misclassification attacks in transfer learning. First, we propose a distilled
differentiator that can address the targeted attacks, where adversarial inputs
are misclassified to a specific class. Specifically, this dedicated
differentiator is designed with network activation pruning and retraining in a
fine-tuned manner, so as to reach high defence rates and high model accuracy.
To address the non-targeted attacks that misclassify adversarial inputs to
randomly selected classes, we further employ an ensemble structure from the
differentiators to cover all possible misclassification. Our evaluations over
common image recognition tasks confirm that the student models applying our
defence can reject most of the adversarial inputs with a marginal accuracy
loss. We also show that our defence outperforms prior approaches in both
targeted and non-targeted attacks.


http://arxiv.org/abs/1908.11332
Universal, transferable and targeted adversarial attacks.
Junde Wu; Rao Fu
  Deep Neural Network has been found vulnerable recently. A kind of
well-designed inputs, which called adversarial examples, can lead the networks
to make incorrect predictions. Depending on the different scenarios, goals and
capabilities, the difficulty to generate the attack is different. For example,
generating a targeted attack is more difficult than a non-targeted attack, a
universal attack is more difficult than a non-universal attack, a transferable
attack is more difficult than a nontransferable one. The question is: Is there
exist an attack that can survival in the most harsh adversity to meet all these
requirements. Although many cheap and effective attacks have been proposed,
this question is still not completely solved over large models and large scale
dataset. In this paper, we learn a universal mapping from the sources to the
adversarial examples. These examples can fool classification networks into
classifying all of them to one targeted class. Besides, they are also
transferable between different models.


http://arxiv.org/abs/1908.09705
A Statistical Defense Approach for Detecting Adversarial Examples.
Alessandro Cennamo; Ido Freeman; Anton Kummert
  Adversarial examples are maliciously modified inputs created to fool deep
neural networks (DNN). The discovery of such inputs presents a major issue to
the expansion of DNN-based solutions. Many researchers have already contributed
to the topic, providing both cutting edge-attack techniques and various
defensive strategies. In this work, we focus on the development of a system
capable of detecting adversarial samples by exploiting statistical information
from the training-set. Our detector computes several distorted replicas of the
test input, then collects the classifier's prediction vectors to build a
meaningful signature for the detection task. Then, the signature is projected
onto the class-specific statistic vector to infer the input's nature. The
classification output of the original input is used to select the
class-statistic vector. We show that our method reliably detects malicious
inputs, outperforming state-of-the-art approaches in various settings, while
being complementary to other defensive solutions.


http://arxiv.org/abs/1908.09699
Gated Convolutional Networks with Hybrid Connectivity for Image Classification.
Chuanguang Yang; Zhulin An; Hui Zhu; Xiaolong Hu; Kun Zhang; Kaiqiang Xu; Chao Li; Yongjun Xu
  We propose a simple yet effective method to reduce the redundancy of DenseNet
by substantially decreasing the number of stacked modules by replacing the
original bottleneck by our SMG module, which is augmented by local residual.
Furthermore, SMG module is equipped with an efficient two-stage pipeline, which
aims to DenseNet-like architectures that need to integrate all previous
outputs, i.e., squeezing the incoming informative but redundant features
gradually by hierarchical convolutions as a hourglass shape and then exciting
it by multi-kernel depthwise convolutions, the output of which would be compact
and hold more informative multi-scale features. We further develop a forget and
an update gate by introducing the popular attention modules to implement the
effective fusion instead of a simple addition between reused and new features.
Due to the Hybrid Connectivity (nested combination of global dense and local
residual) and Gated mechanisms, we called our network as the HCGNet.
Experimental results on CIFAR and ImageNet datasets show that HCGNet is more
prominently efficient than DenseNet, and can also significantly outperform
state-of-the-art networks with less complexity. Moreover, HCGNet also shows the
remarkable interpretability and robustness by network dissection and
adversarial defense, respectively. On MS-COCO, HCGNet can consistently learn
better features than popular backbones.


http://arxiv.org/abs/1908.09364
Adversarial Edit Attacks for Tree Data.
Benjamin Paaßen
  Many machine learning models can be attacked with adversarial examples, i.e.
inputs close to correctly classified examples that are classified incorrectly.
However, most research on adversarial attacks to date is limited to vectorial
data, in particular image data. In this contribution, we extend the field by
introducing adversarial edit attacks for tree-structured data with potential
applications in medicine and automated program analysis. Our approach solely
relies on the tree edit distance and a logarithmic number of black-box queries
to the attacked classifier without any need for gradient information. We
evaluate our approach on two programming and two biomedical data sets and show
that many established tree classifiers, like tree-kernel-SVMs and recursive
neural networks, can be attacked effectively.


http://arxiv.org/abs/1908.09327
advPattern: Physical-World Attacks on Deep Person Re-Identification via Adversarially Transformable Patterns.
Zhibo Wang; Siyan Zheng; Mengkai Song; Qian Wang; Alireza Rahimpour; Hairong Qi
  Person re-identification (re-ID) is the task of matching person images across
camera views, which plays an important role in surveillance and security
applications. Inspired by great progress of deep learning, deep re-ID models
began to be popular and gained state-of-the-art performance. However, recent
works found that deep neural networks (DNNs) are vulnerable to adversarial
examples, posing potential threats to DNNs based applications. This phenomenon
throws a serious question about whether deep re-ID based systems are vulnerable
to adversarial attacks. In this paper, we take the first attempt to implement
robust physical-world attacks against deep re-ID. We propose a novel attack
algorithm, called advPattern, for generating adversarial patterns on clothes,
which learns the variations of image pairs across cameras to pull closer the
image features from the same camera, while pushing features from different
cameras farther. By wearing our crafted "invisible cloak", an adversary can
evade person search, or impersonate a target person to fool deep re-ID models
in physical world. We evaluate the effectiveness of our transformable patterns
on adversaries'clothes with Market1501 and our established PRCS dataset. The
experimental results show that the rank-1 accuracy of re-ID models for matching
the adversary decreases from 87.9% to 27.1% under Evading Attack. Furthermore,
the adversary can impersonate a target person with 47.1% rank-1 accuracy and
67.9% mAP under Impersonation Attack. The results demonstrate that deep re-ID
systems are vulnerable to our physical attacks.


http://arxiv.org/abs/1908.09163
Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower.
Giorgos Tolias; Filip Radenovic; Ond{ř}ej Chum
  Access to online visual search engines implies sharing of private user
content - the query images. We introduce the concept of targeted mismatch
attack for deep learning based retrieval systems to generate an adversarial
image to conceal the query image. The generated image looks nothing like the
user intended query, but leads to identical or very similar retrieval results.
Transferring attacks to fully unseen networks is challenging. We show
successful attacks to partially unknown systems, by designing various loss
functions for the adversarial image construction. These include loss functions,
for example, for unknown global pooling operation or unknown input resolution
by the retrieval system. We evaluate the attacks on standard retrieval
benchmarks and compare the results retrieved with the original and adversarial
image.


http://arxiv.org/abs/1908.11435
Improving Adversarial Robustness via Attention and Adversarial Logit Pairing.
Dou Goodman; Xingjian Li; Jun Huan; Tao Wei
  Though deep neural networks have achieved the state of the art performance in
visual classification, recent studies have shown that they are all vulnerable
to the attack of adversarial examples. In this paper, we develop improved
techniques for defending against adversarial examples.First, we introduce
enhanced defense using a technique we call \textbf{Attention and Adversarial
Logit Pairing(AT+ALP)}, a method that encourages both attention map and logit
for pairs of examples to be similar. When applied to clean examples and their
adversarial counterparts, \textbf{AT+ALP} improves accuracy on adversarial
examples over adversarial training.Next,We show that our \textbf{AT+ALP} can
effectively increase the average activations of adversarial examples in the key
area and demonstrate that it focuse on more discriminate features to improve
the robustness of the model.Finally,we conducte extensive experiments using a
wide range of datasets and the experiment results show that our \textbf{AT+ALP}
achieves \textbf{the state of the art} defense.For example,on \textbf{17 Flower
Category Database}, under strong 200-iteration \textbf{PGD} gray-box and
black-box attacks where prior art has 34\% and 39\% accuracy, our method
achieves \textbf{50\%} and \textbf{51\%}.Compared with previous work,our work
is evaluated under highly challenging PGD attack:the maximum perturbation
$\epsilon \in \{0.25,0.5\}$ i.e. $L_\infty \in \{0.25,0.5\}$ with 10 to 200
attack iterations.To our knowledge, such a strong attack has not been
previously explored on a wide range of datasets.


http://arxiv.org/abs/1908.08705
AdvHat: Real-world adversarial attack on ArcFace Face ID system.
Stepan Komkov; Aleksandr Petiushko
  In this paper we propose a novel easily reproducible technique to attack the
best public Face ID system ArcFace in different shooting conditions. To create
an attack, we print the rectangular paper sticker on a common color printer and
put it on the hat. The adversarial sticker is prepared with a novel algorithm
for off-plane transformations of the image which imitates sticker location on
the hat. Such an approach confuses the state-of-the-art public Face ID model
LResNet100E-IR, ArcFace@ms1m-refine-v2 and is transferable to other Face ID
models.


http://arxiv.org/abs/1908.08413
Saliency Methods for Explaining Adversarial Attacks.
Jindong Gu; Volker Tresp
  The classification decisions of neural networks can be misled by small
imperceptible perturbations. This work aims to explain the misled
classifications using saliency methods. The idea behind saliency methods is to
explain the classification decisions of neural networks by creating so-called
saliency maps. Unfortunately, a number of recent publications have shown that
many of the proposed saliency methods do not provide insightful explanations. A
prominent example is Guided Backpropagation (GuidedBP), which simply performs
(partial) image recovery. However, our numerical analysis shows the saliency
maps created by GuidedBP do indeed contain class-discriminative information. We
propose a simple and efficient way to enhance the saliency maps. The proposed
enhanced GuidedBP shows the state-of-the-art performance to explain adversary
classifications.


http://arxiv.org/abs/1908.08016
Testing Robustness Against Unforeseen Adversaries.
Daniel Kang; Yi Sun; Dan Hendrycks; Tom Brown; Jacob Steinhardt
  Considerable work on adversarial defense has studied robustness to a fixed,
known family of adversarial distortions, most frequently L_p-bounded
distortions. In reality, the specific form of attack will rarely be known and
adversaries are free to employ distortions outside of any fixed set. The
present work advocates measuring robustness against this much broader range of
unforeseen attacks---attacks whose precise form is not known when designing a
defense.
  We propose a methodology for evaluating a defense against a diverse range of
distortion types together with a summary metric UAR that measures the
Unforeseen Attack Robustness against a distortion. We construct novel JPEG,
Fog, Gabor, and Snow adversarial attacks to simulate unforeseen adversaries and
perform a careful study of adversarial robustness against these and existing
distortion types. We find that evaluation against existing L_p attacks yields
highly correlated information that may not generalize to other attacks and
identify a set of 4 attacks that yields more diverse information. We further
find that adversarial training against either one or multiple distortions,
including our novel ones, does not confer robustness to unforeseen distortions.
These results underscore the need to study robustness against unforeseen
distortions and provide a starting point for doing so.


http://arxiv.org/abs/1908.07899
Evaluating Defensive Distillation For Defending Text Processing Neural Networks Against Adversarial Examples.
Marcus Soll; Tobias Hinz; Sven Magg; Stefan Wermter
  Adversarial examples are artificially modified input samples which lead to
misclassifications, while not being detectable by humans. These adversarial
examples are a challenge for many tasks such as image and text classification,
especially as research shows that many adversarial examples are transferable
between different classifiers. In this work, we evaluate the performance of a
popular defensive strategy for adversarial examples called defensive
distillation, which can be successful in hardening neural networks against
adversarial examples in the image domain. However, instead of applying
defensive distillation to networks for image classification, we examine, for
the first time, its performance on text classification tasks and also evaluate
its effect on the transferability of adversarial text examples. Our results
indicate that defensive distillation only has a minimal impact on text
classifying neural networks and does neither help with increasing their
robustness against adversarial examples nor prevent the transferability of
adversarial examples between neural networks.


http://arxiv.org/abs/1908.07667
Denoising and Verification Cross-Layer Ensemble Against Black-box Adversarial Attacks.
Ka-Ho Chow; Wenqi Wei; Yanzhao Wu; Ling Liu
  Deep neural networks (DNNs) have demonstrated impressive performance on many
challenging machine learning tasks. However, DNNs are vulnerable to adversarial
inputs generated by adding maliciously crafted perturbations to the benign
inputs. As a growing number of attacks have been reported to generate
adversarial inputs of varying sophistication, the defense-attack arms race has
been accelerated. In this paper, we present MODEF, a cross-layer model
diversity ensemble framework. MODEF intelligently combines unsupervised model
denoising ensemble with supervised model verification ensemble by quantifying
model diversity, aiming to boost the robustness of the target model against
adversarial examples. Evaluated using eleven representative attacks on popular
benchmark datasets, we show that MODEF achieves remarkable defense success
rates, compared with existing defense methods, and provides a superior
capability of repairing adversarial inputs and making correct predictions with
high accuracy in the presence of black-box attacks.


http://arxiv.org/abs/1908.07558
Transferring Robustness for Graph Neural Network Against Poisoning Attacks.
Xianfeng Tang; Yandong Li; Yiwei Sun; Huaxiu Yao; Prasenjit Mitra; Suhang Wang
  Graph neural networks (GNNs) are widely used in many applications. However,
their robustness against adversarial attacks is criticized. Prior studies show
that using unnoticeable modifications on graph topology or nodal features can
significantly reduce the performances of GNNs. It is very challenging to design
robust graph neural networks against poisoning attack and several efforts have
been taken. Existing work aims at reducing the negative impact from adversarial
edges only with the poisoned graph, which is sub-optimal since they fail to
discriminate adversarial edges from normal ones. On the other hand, clean
graphs from similar domains as the target poisoned graph are usually available
in the real world. By perturbing these clean graphs, we create supervised
knowledge to train the ability to detect adversarial edges so that the
robustness of GNNs is elevated. However, such potential for clean graphs is
neglected by existing work. To this end, we investigate a novel problem of
improving the robustness of GNNs against poisoning attacks by exploring clean
graphs. Specifically, we propose PA-GNN, which relies on a penalized
aggregation mechanism that directly restrict the negative impact of adversarial
edges by assigning them lower attention coefficients. To optimize PA-GNN for a
poisoned graph, we design a meta-optimization algorithm that trains PA-GNN to
penalize perturbations using clean graphs and their adversarial counterparts,
and transfers such ability to improve the robustness of PA-GNN on the poisoned
graph. Experimental results on four real-world datasets demonstrate the
robustness of PA-GNN against poisoning attacks on graphs. Code and data are
available here: https://github.com/tangxianfeng/PA-GNN.


http://arxiv.org/abs/1908.07125
Universal Adversarial Triggers for NLP.
Eric Wallace; Shi Feng; Nikhil Kandpal; Matt Gardner; Sameer Singh
  Adversarial examples highlight model vulnerabilities and are useful for
evaluation and interpretation. We define universal adversarial triggers:
input-agnostic sequences of tokens that trigger a model to produce a specific
prediction when concatenated to any input from a dataset. We propose a
gradient-guided search over tokens which finds short trigger sequences (e.g.,
one word for classification and four words for language modeling) that
successfully trigger the target prediction. For example, triggers cause SNLI
entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in
SQuAD to be answered "to kill american people", and the GPT-2 language model to
spew racist output even when conditioned on non-racial contexts. Furthermore,
although the triggers are optimized using white-box access to a specific model,
they transfer to other models for all tasks we consider. Finally, since
triggers are input-agnostic, they provide an analysis of global model behavior.
For instance, they confirm that SNLI models exploit dataset biases and help to
diagnose heuristics learned by reading comprehension models.


http://arxiv.org/abs/1908.07116
Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses.
Xiao Wang; Siyue Wang; Pin-Yu Chen; Yanzhi Wang; Brian Kulis; Xue Lin; Peter Chin
  Despite achieving remarkable success in various domains, recent studies have
uncovered the vulnerability of deep neural networks to adversarial
perturbations, creating concerns on model generalizability and new threats such
as prediction-evasive misclassification or stealthy reprogramming. Among
different defense proposals, stochastic network defenses such as random neuron
activation pruning or random perturbation to layer inputs are shown to be
promising for attack mitigation. However, one critical drawback of current
defenses is that the robustness enhancement is at the cost of noticeable
performance degradation on legitimate data, e.g., large drop in test accuracy.
This paper is motivated by pursuing for a better trade-off between adversarial
robustness and test accuracy for stochastic network defenses. We propose
Defense Efficiency Score (DES), a comprehensive metric that measures the gain
in unsuccessful attack attempts at the cost of drop in test accuracy of any
defense. To achieve a better DES, we propose hierarchical random switching
(HRS), which protects neural networks through a novel randomization scheme. A
HRS-protected model contains several blocks of randomly switching channels to
prevent adversaries from exploiting fixed model structures and parameters for
their malicious purposes. Extensive experiments show that HRS is superior in
defending against state-of-the-art white-box and adaptive adversarial
misclassification attacks. We also demonstrate the effectiveness of HRS in
defending adversarial reprogramming, which is the first defense against
adversarial programs. Moreover, in most settings the average DES of HRS is at
least 5X higher than current stochastic network defenses, validating its
significantly improved robustness-accuracy trade-off.


http://arxiv.org/abs/1908.07000
Hybrid Batch Attacks: Finding Black-box Adversarial Examples with Limited Queries.
Fnu Suya; Jianfeng Chi; David Evans; Yuan Tian
  We study adversarial examples in a black-box setting where the adversary only
has API access to the target model and each query is expensive. Prior work on
black-box adversarial examples follows one of two main strategies: (1) transfer
attacks use white-box attacks on local models to find candidate adversarial
examples that transfer to the target model, and (2) optimization-based attacks
use queries to the target model and apply optimization techniques to search for
adversarial examples. We propose hybrid attacks that combine both strategies,
using candidate adversarial examples from local models as starting points for
optimization-based attacks and using labels learned in optimization-based
attacks to tune local models for finding transfer candidates. We empirically
demonstrate on the MNIST, CIFAR10, and ImageNet datasets that our hybrid attack
strategy reduces cost and improves success rates. We also introduce a seed
prioritization strategy which enables attackers to focus their resources on the
most promising seeds. Combining hybrid attacks with our seed prioritization
strategy enables batch attacks that can reliably find adversarial examples with
only a handful of queries.


http://arxiv.org/abs/1908.06401
On the Robustness of Human Pose Estimation.
Sahil Shah; Naman jain; Abhishek Sharma; Arjun Jain
  This paper provides, to the best of our knowledge, the first comprehensive
and exhaustive study of adversarial attacks on human pose estimation. Besides
highlighting the important differences between well-studied classification and
human pose-estimation systems w.r.t. adversarial attacks, we also provide deep
insights into the design choices of pose-estimation systems to shape future
work. We compare the robustness of several pose-estimation architectures
trained on the standard datasets, MPII and COCO. In doing so, we also explore
the problem of attacking non-classification based networks including regression
based networks, which has been virtually unexplored in the past.
  We find that compared to clas