http://arxiv.org/abs/1907.07640
Robustness properties of Facebook's ResNeXt WSL models. (92%)
A. Emin Orhan
  We investigate the robustness properties of ResNeXt image recognition models
trained with billion scale weakly-supervised data (ResNeXt WSL models). These
models, recently made public by Facebook AI, were trained on ~1B images from
Instagram and fine-tuned on ImageNet. We show that these models display an
unprecedented degree of robustness against common image corruptions and
perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. The
largest of the released models, in particular, achieves state-of-the-art
results on both ImageNet-C and ImageNet-P by a large margin. The gains on
ImageNet-C and ImageNet-P far outpace the gains on ImageNet validation
accuracy, suggesting the former as more useful benchmarks to measure further
progress in image recognition. Remarkably, the ResNeXt WSL models even achieve
a limited degree of adversarial robustness against state-of-the-art white-box
attacks (10-step PGD attacks). However, in contrast to adversarially trained
models, the robustness of the ResNeXt WSL models rapidly declines with the
number of PGD steps, suggesting that these models do not achieve genuine
adversarial robustness. Visualization of the learned features also confirms
this conclusion. Finally, we show that although the ResNeXt WSL models are more
shape-biased in their predictions than comparable ImageNet-trained models, they
still remain much more texture-biased than humans.


http://arxiv.org/abs/1907.07487
Real-time Evasion Attacks with Physical Constraints on Deep Learning-based Anomaly Detectors in Industrial Control Systems. (87%)
Alessandro Erba; Riccardo Taormina; Stefano Galelli; Marcello Pogliani; Michele Carminati; Stefano Zanero; Nils Ole Tippenhauer
  Recently, a number of deep learning-based anomaly detection algorithms were
proposed to detect attacks in dynamic industrial control systems. The detectors
operate on measured sensor data, leveraging physical process models learned a
priori. Evading detection by such systems is challenging, as an attacker needs
to manipulate a constrained number of sensor readings in real-time with
realistic perturbations according to the current state of the system. In this
work, we propose a number of evasion attacks (with different assumptions on the
attacker's knowledge), and compare the attacks' cost and efficiency against
replay attacks. In particular, we show that a replay attack on a subset of
sensor values can be detected easily as it violates physical constraints. In
contrast, our proposed attacks leverage manipulated sensor readings that
observe learned physical constraints of the system. Our proposed white box
attacker uses an optimization approach with a detection oracle, while our black
box attacker uses an autoencoder (or a convolutional neural network) to
translate anomalous data into normal data. Our proposed approaches are
implemented and evaluated on two different datasets pertaining to the domain of
water distribution networks. We then demonstrated the efficacy of the real-time
attack on a realistic testbed. Results show that the accuracy of the detection
algorithms can be significantly reduced through real-time adversarial actions:
for the BATADAL dataset, the attacker can reduce the detection accuracy from
0.6 to 0.14. In addition, we discuss and implement an Availability attack, in
which the attacker introduces detection events with minimal changes of the
reported data, in order to reduce confidence in the detector.


http://arxiv.org/abs/1907.07732
Connecting Lyapunov Control Theory to Adversarial Attacks. (16%)
Arash Rahnama; Andre T. Nguyen; Edward Raff
  Significant work is being done to develop the math and tools necessary to
build provable defenses, or at least bounds, against adversarial attacks of
neural networks. In this work, we argue that tools from control theory could be
leveraged to aid in defending against such attacks. We do this by example,
building a provable defense against a weaker adversary. This is done so we can
focus on the mechanisms of control theory, and illuminate its intrinsic value.


http://arxiv.org/abs/1907.07001
Latent Adversarial Defence with Boundary-guided Generation. (99%)
Xiaowei Zhou; Ivor W. Tsang; Jie Yin
  Deep Neural Networks (DNNs) have recently achieved great success in many
tasks, which encourages DNNs to be widely used as a machine learning service in
model sharing scenarios. However, attackers can easily generate adversarial
examples with a small perturbation to fool the DNN models to predict wrong
labels. To improve the robustness of shared DNN models against adversarial
attacks, we propose a novel method called Latent Adversarial Defence (LAD). The
proposed LAD method improves the robustness of a DNN model through adversarial
training on generated adversarial examples. Different from popular attack
methods which are carried in the input space and only generate adversarial
examples of repeating patterns, LAD generates myriad of adversarial examples
through adding perturbations to latent features along the normal of the
decision boundary which is constructed by an SVM with an attention mechanism.
Once adversarial examples are generated, we adversarially train the model
through augmenting the training data with generated adversarial examples.
Extensive experiments on the MNIST, SVHN, and CelebA dataset demonstrate the
effectiveness of our model in defending against different types of adversarial
attacks.


http://arxiv.org/abs/1907.07174
Natural Adversarial Examples. (81%)
Dan Hendrycks; Kevin Zhao; Steven Basart; Jacob Steinhardt; Dawn Song
  We introduce natural adversarial examples -- real-world, unmodified, and
naturally occurring examples that cause classifier accuracy to significantly
degrade. We curate 7,500 natural adversarial examples and release them in an
ImageNet classifier test set that we call ImageNet-A. This dataset serves as a
new way to measure classifier robustness. Like l_p adversarial examples,
ImageNet-A examples successfully transfer to unseen or black-box classifiers.
For example, on ImageNet-A a DenseNet-121 obtains around 2% accuracy, an
accuracy drop of approximately 90%. Recovering this accuracy is not simple
because ImageNet-A examples exploit deep flaws in current classifiers including
their over-reliance on color, texture, and background cues. We observe that
popular training techniques for improving robustness have little effect, but we
show that some architectural changes can enhance robustness to natural
adversarial examples. Future research is required to enable robust
generalization to this hard ImageNet test set.


http://arxiv.org/abs/1907.06826
Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving. (76%)
Yulong Cao; Chaowei Xiao; Benjamin Cyr; Yimeng Zhou; Won Park; Sara Rampazzi; Qi Alfred Chen; Kevin Fu; Z. Morley Mao
  In Autonomous Vehicles (AVs), one fundamental pillar is perception, which
leverages sensors like cameras and LiDARs (Light Detection and Ranging) to
understand the driving environment. Due to its direct impact on road safety,
multiple prior efforts have been made to study its the security of perception
systems. In contrast to prior work that concentrates on camera-based
perception, in this work we perform the first security study of LiDAR-based
perception in AV settings, which is highly important but unexplored. We
consider LiDAR spoofing attacks as the threat model and set the attack goal as
spoofing obstacles close to the front of a victim AV. We find that blindly
applying LiDAR spoofing is insufficient to achieve this goal due to the machine
learning-based object detection process. Thus, we then explore the possibility
of strategically controlling the spoofed attack to fool the machine learning
model. We formulate this task as an optimization problem and design modeling
methods for the input perturbation function and the objective function. We also
identify the inherent limitations of directly solving the problem using
optimization and design an algorithm that combines optimization and global
sampling, which improves the attack success rates to around 75%. As a case
study to understand the attack impact at the AV driving decision level, we
construct and evaluate two attack scenarios that may damage road safety and
mobility. We also discuss defense directions at the AV system, sensor, and
machine learning model levels.


http://arxiv.org/abs/1907.07296
Explaining Vulnerabilities to Adversarial Machine Learning through Visual Analytics. (64%)
Yuxin Ma; Tiankai Xie; Jundong Li; Ross Maciejewski
  Machine learning models are currently being deployed in a variety of
real-world applications where model predictions are used to make decisions
about healthcare, bank loans, and numerous other critical tasks. As the
deployment of artificial intelligence technologies becomes ubiquitous, it is
unsurprising that adversaries have begun developing methods to manipulate
machine learning models to their advantage. While the visual analytics
community has developed methods for opening the black box of machine learning
models, little work has focused on helping the user understand their model
vulnerabilities in the context of adversarial attacks. In this paper, we
present a visual analytics framework for explaining and exploring model
vulnerabilities to adversarial attacks. Our framework employs a multi-faceted
visualization scheme designed to support the analysis of data poisoning attacks
from the perspective of models, data instances, features, and local structures.
We demonstrate our framework through two case studies on binary classifiers and
illustrate model vulnerabilities with respect to varying attack strategies.


http://arxiv.org/abs/1907.07291
Adversarial Security Attacks and Perturbations on Machine Learning and Deep Learning Methods. (8%)
Arif Siddiqi
  The ever-growing big data and emerging artificial intelligence (AI) demand
the use of machine learning (ML) and deep learning (DL) methods. Cybersecurity
also benefits from ML and DL methods for various types of applications. These
methods however are susceptible to security attacks. The adversaries can
exploit the training and testing data of the learning models or can explore the
workings of those models for launching advanced future attacks. The topic of
adversarial security attacks and perturbations within the ML and DL domains is
a recent exploration and a great interest is expressed by the security
researchers and practitioners. The literature covers different adversarial
security attacks and perturbations on ML and DL methods and those have their
own presentation styles and merits. A need to review and consolidate knowledge
that is comprehending of this increasingly focused and growing topic of
research; however, is the current demand of the research communities. In this
review paper, we specifically aim to target new researchers in the
cybersecurity domain who may seek to acquire some basic knowledge on the
machine learning and deep learning models and algorithms, as well as some of
the relevant adversarial security attacks and perturbations.


http://arxiv.org/abs/1907.06890
A General Framework for Uncertainty Estimation in Deep Learning. (3%)
Mattia Segù; Antonio Loquercio; Davide Scaramuzza
  End-to-end learning has recently emerged as a promising technique to tackle
the problem of autonomous driving. Existing works show that learning a
navigation policy from raw sensor data may reduce the system's reliance on
external sensing systems, (e.g. GPS), and/or outperform traditional methods
based on state estimation and planning. However, existing end-to-end methods
generally trade off performance for safety, hindering their diffusion to
real-life applications. For example, when confronted with an input which is
radically different from the training data, end-to-end autonomous driving
systems are likely to fail, compromising the safety of the vehicle. To detect
such failure cases, this work proposes a general framework for uncertainty
estimation which enables a policy trained end-to-end to predict not only action
commands, but also a confidence about its own predictions. In contrast to
previous works, our framework can be applied to any existing neural network and
task, without the need to change the network's architecture or loss, or to
train the network. In order to do so, we generate confidence levels by forward
propagation of input and model uncertainties using Bayesian inference. We test
our framework on the task of steering angle regression for an autonomous car,
and compare our approach to existing methods with both qualitative and
quantitative results on a real dataset. Finally, we show an interesting
by-product of our framework: robustness against adversarial attacks.


http://arxiv.org/abs/1907.06565
Recovery Guarantees for Compressible Signals with Adversarial Noise. (99%)
Jasjeet Dhaliwal; Kyle Hambrook
  We provide recovery guarantees for compressible signals that have been
corrupted with noise and extend the framework introduced in [1] to defend
neural networks against $\ell_0$-norm and $\ell_2$-norm attacks. Concretely,
for a signal that is approximately sparse in some transform domain and has been
perturbed with noise, we provide guarantees for accurately recovering the
signal in the transform domain. We can then use the recovered signal to
reconstruct the signal in its original domain while largely removing the noise.
Our results are general as they can be directly applied to most unitary
transforms used in practice and hold for both $\ell_0$-norm bounded noise and
$\ell_2$-norm bounded noise. In the case of $\ell_0$-norm bounded noise, we
prove recovery guarantees for Iterative Hard Thresholding (IHT) and Basis
Pursuit (BP). For the case of $\ell_2$-norm bounded noise, we provide recovery
guarantees for BP. These guarantees theoretically bolster the defense framework
introduced in [1] for defending neural networks against adversarial inputs.
Finally, we experimentally demonstrate this defense framework using both IHT
and BP against the One Pixel Attack [21], Carlini-Wagner $\ell_0$ and $\ell_2$
attacks [3], Jacobian Saliency Based attack [18], and the DeepFool attack [17]
on CIFAR-10 [12], MNIST [13], and Fashion-MNIST [27] datasets. This expands
beyond the experimental demonstrations of [1].


http://arxiv.org/abs/1907.06800
Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning. (96%)
Bao Wang; Stanley J. Osher
  Improving the accuracy and robustness of deep neural nets (DNNs) and adapting
them to small training data are primary tasks in deep learning research. In
this paper, we replace the output activation function of DNNs, typically the
data-agnostic softmax function, with a graph Laplacian-based high dimensional
interpolating function which, in the continuum limit, converges to the solution
of a Laplace-Beltrami equation on a high dimensional manifold. Furthermore, we
propose end-to-end training and testing algorithms for this new architecture.
The proposed DNN with graph interpolating activation integrates the advantages
of both deep learning and manifold learning. Compared to the conventional DNNs
with the softmax function as output activation, the new framework demonstrates
the following major advantages: First, it is better applicable to
data-efficient learning in which we train high capacity DNNs without using a
large number of training data. Second, it remarkably improves both natural
accuracy on the clean images and robust accuracy on the adversarial images
crafted by both white-box and black-box adversarial attacks. Third, it is a
natural choice for semi-supervised learning. For reproducibility, the code is
available at \url{https://github.com/BaoWangMath/DNN-DataDependentActivation}.


http://arxiv.org/abs/1907.06291
Measuring the Transferability of Adversarial Examples. (99%)
Deyan Petrov; Timothy M. Hospedales
  Adversarial examples are of wide concern due to their impact on the
reliability of contemporary machine learning systems. Effective adversarial
examples are mostly found via white-box attacks. However, in some cases they
can be transferred across models, thus enabling them to attack black-box
models. In this work we evaluate the transferability of three adversarial
attacks - the Fast Gradient Sign Method, the Basic Iterative Method, and the
Carlini & Wagner method, across two classes of models - the VGG class(using
VGG16, VGG19 and an ensemble of VGG16 and VGG19), and the Inception
class(Inception V3, Xception, Inception Resnet V2, and an ensemble of the
three). We also outline the problems with the assessment of transferability in
the current body of research and attempt to amend them by picking specific
"strong" parameters for the attacks, and by using a L-Infinity clipping
technique and the SSIM metric for the final evaluation of the attack
transferability.


http://arxiv.org/abs/1907.05587
Stateful Detection of Black-Box Adversarial Attacks. (99%)
Steven Chen; Nicholas Carlini; David Wagner
  The problem of adversarial examples, evasion attacks on machine learning
classifiers, has proven extremely difficult to solve. This is true even when,
as is the case in many practical settings, the classifier is hosted as a remote
service and so the adversary does not have direct access to the model
parameters.
  This paper argues that in such settings, defenders have a much larger space
of actions than have been previously explored. Specifically, we deviate from
the implicit assumption made by prior work that a defense must be a stateless
function that operates on individual examples, and explore the possibility for
stateful defenses.
  To begin, we develop a defense designed to detect the process of adversarial
example generation. By keeping a history of the past queries, a defender can
try to identify when a sequence of queries appears to be for the purpose of
generating an adversarial example. We then introduce query blinding, a new
class of attacks designed to bypass defenses that rely on such a defense
approach.
  We believe that expanding the study of adversarial examples from stateless
classifiers to stateful systems is not only more realistic for many black-box
settings, but also gives the defender a much-needed advantage in responding to
the adversary.


http://arxiv.org/abs/1907.05793
Unsupervised Adversarial Attacks on Deep Feature-based Retrieval with GAN. (96%)
Guoping Zhao; Mingyu Zhang; Jiajun Liu; Ji-Rong Wen
  Studies show that Deep Neural Network (DNN)-based image classification models
are vulnerable to maliciously constructed adversarial examples. However, little
effort has been made to investigate how DNN-based image retrieval models are
affected by such attacks. In this paper, we introduce Unsupervised Adversarial
Attacks with Generative Adversarial Networks (UAA-GAN) to attack deep
feature-based image retrieval systems. UAA-GAN is an unsupervised learning
model that requires only a small amount of unlabeled data for training. Once
trained, it produces query-specific perturbations for query images to form
adversarial queries. The core idea is to ensure that the attached perturbation
is barely perceptible to human yet effective in pushing the query away from its
original position in the deep feature space. UAA-GAN works with various
application scenarios that are based on deep features, including image
retrieval, person Re-ID and face search. Empirical results show that UAA-GAN
cripples retrieval performance without significant visual changes in the query
images. UAA-GAN generated adversarial examples are less distinguishable because
they tend to incorporate subtle perturbations in textured or salient areas of
the images, such as key body parts of human, dominant structural
patterns/textures or edges, rather than in visually insignificant areas (e.g.,
background and sky). Such tendency indicates that the model indeed learned how
to toy with both image retrieval systems and human eyes.


http://arxiv.org/abs/1907.05600
Generative Modeling by Estimating Gradients of the Data Distribution. (10%)
Yang Song; Stefano Ermon
  We introduce a new generative model where samples are produced via Langevin
dynamics using gradients of the data distribution estimated with score
matching. Because gradients might be ill-defined when the data resides on
low-dimensional manifolds, we perturb the data with different levels of
Gaussian noise and jointly estimate the corresponding scores, i.e., the vector
fields of gradients of the perturbed data distribution for all noise levels.
For sampling, we propose an annealed Langevin dynamics where we use gradients
corresponding to gradually decreasing noise levels as the sampling process gets
closer to the data manifold. Our framework allows flexible model architectures,
requires no sampling during training or the use of adversarial methods, and
provides a learning objective that can be used for principled model
comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and
CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.91 on
CIFAR-10. Additionally, we demonstrate that our models learn effective
representations via image inpainting experiments.


http://arxiv.org/abs/1907.05718
Why Blocking Targeted Adversarial Perturbations Impairs the Ability to Learn. (99%)
Ziv Katzir; Yuval Elovici
  Despite their accuracy, neural network-based classifiers are still prone to
manipulation through adversarial perturbations. Those perturbations are
designed to be misclassified by the neural network, while being perceptually
identical to some valid input. The vast majority of attack methods rely on
white-box conditions, where the attacker has full knowledge of the attacked
network's parameters. This allows the attacker to calculate the network's loss
gradient with respect to some valid input and use this gradient in order to
create an adversarial example. The task of blocking white-box attacks has
proven difficult to solve. While a large number of defense methods have been
suggested, they have had limited success. In this work we examine this
difficulty and try to understand it. We systematically explore the abilities
and limitations of defensive distillation, one of the most promising defense
mechanisms against adversarial perturbations suggested so far in order to
understand the defense challenge. We show that contrary to commonly held
belief, the ability to bypass defensive distillation is not dependent on an
attack's level of sophistication. In fact, simple approaches, such as the
Targeted Gradient Sign Method, are capable of effectively bypassing defensive
distillation. We prove that defensive distillation is highly effective against
non-targeted attacks but is unsuitable for targeted attacks. This discovery
leads us to realize that targeted attacks leverage the same input gradient that
allows a network to be trained. This implies that blocking them will require
losing the network's ability to learn, presenting an impossible tradeoff to the
research community.


http://arxiv.org/abs/1907.05418
Adversarial Objects Against LiDAR-Based Autonomous Driving Systems. (99%)
Yulong Cao; Chaowei Xiao; Dawei Yang; Jing Fang; Ruigang Yang; Mingyan Liu; Bo Li
  Deep neural networks (DNNs) are found to be vulnerable against adversarial
examples, which are carefully crafted inputs with a small magnitude of
perturbation aiming to induce arbitrarily incorrect predictions. Recent studies
show that adversarial examples can pose a threat to real-world
security-critical applications: a "physical adversarial Stop Sign" can be
synthesized such that the autonomous driving cars will misrecognize it as
others (e.g., a speed limit sign). However, these image-space adversarial
examples cannot easily alter 3D scans of widely equipped LiDAR or radar on
autonomous vehicles. In this paper, we reveal the potential vulnerabilities of
LiDAR-based autonomous driving detection systems, by proposing an optimization
based approach LiDAR-Adv to generate adversarial objects that can evade the
LiDAR-based detection system under various conditions. We first show the
vulnerabilities using a blackbox evolution-based algorithm, and then explore
how much a strong adversary can do, using our gradient-based approach
LiDAR-Adv. We test the generated adversarial objects on the Baidu Apollo
autonomous driving platform and show that such physical systems are indeed
vulnerable to the proposed attacks. We also 3D-print our adversarial objects
and perform physical experiments to illustrate that such vulnerability exists
in the real world. Please find more visualizations and results on the anonymous
website: https://sites.google.com/view/lidar-adv.


http://arxiv.org/abs/1907.05401
Computational Concentration of Measure: Optimal Bounds, Reductions, and More. (3%)
Omid Etesami; Saeed Mahloujifar; Mohammad Mahmoody
  Product measures of dimension $n$ are known to be concentrated in Hamming
distance: for any set $S$ in the product space of probability $\epsilon$, a
random point in the space, with probability $1-\delta$, has a neighbor in $S$
that is different from the original point in only
$O(\sqrt{n\ln(1/(\epsilon\delta))})$ coordinates. We obtain the tight
computational version of this result, showing how given a random point and
access to an $S$-membership oracle, we can find such a close point in
polynomial time. This resolves an open question of [Mahloujifar and Mahmoody,
ALT 2019]. As corollaries, we obtain polynomial-time poisoning and (in certain
settings) evasion attacks against learning algorithms when the original
vulnerabilities have any cryptographically non-negligible probability.
  We call our algorithm MUCIO ("MUltiplicative Conditional Influence
Optimizer") since proceeding through the coordinates, it decides to change each
coordinate of the given point based on a multiplicative version of the
influence of that coordinate, where influence is computed conditioned on
previously updated coordinates.
  We also define a new notion of algorithmic reduction between computational
concentration of measure in different metric probability spaces. As an
application, we get computational concentration of measure for high-dimensional
Gaussian distributions under the $\ell_1$ metric.
  We prove several extensions to the results above: (1) Our computational
concentration result is also true when the Hamming distance is weighted. (2) We
obtain an algorithmic version of concentration around mean, more specifically,
McDiarmid's inequality. (3) Our result generalizes to discrete random
processes, and this leads to new tampering algorithms for collective coin
tossing protocols. (4) We prove exponential lower bounds on the average running
time of non-adaptive query algorithms.


http://arxiv.org/abs/1907.04774
Metamorphic Detection of Adversarial Examples in Deep Learning Models With Affine Transformations.
Rohan Reddy Mekala; Gudjon Einar Magnusson; Adam Porter; Mikael Lindvall; Madeline Diep
  Adversarial attacks are small, carefully crafted perturbations, imperceptible
to the naked eye; that when added to an image cause deep learning models to
misclassify the image with potentially detrimental outcomes. With the rise of
artificial intelligence models in consumer safety and security intensive
industries such as self-driving cars, camera surveillance and face recognition,
there is a growing need for guarding against adversarial attacks. In this
paper, we present an approach that uses metamorphic testing principles to
automatically detect such adversarial attacks. The approach can detect image
manipulations that are so small, that they are impossible to detect by a human
through visual inspection. By applying metamorphic relations based on distance
ratio preserving affine image transformations which compare the behavior of the
original and transformed image; we show that our proposed approach can
determine whether or not the input image is adversarial with a high degree of
accuracy.


http://arxiv.org/abs/1907.04449
Generating Adversarial Fragments with Adversarial Networks for Physical-world Implementation.
Zelun Kong; Cong Liu
  Although deep neural networks have been widely applied in many application
domains, they are found to be vulnerable to adversarial attacks. A recent
promising set of attacking techniques have been proposed, which mainly focus on
generating adversarial examples under digital-world settings. Such strategies
are unfortunately not implementable for any physical-world scenarios such as
autonomous driving. In this paper, we present FragGAN, a new GAN-based
framework which is capable of generating an adversarial image which differs
from the original input image only through replacing a targeted fragment within
the image using a corresponding visually indistinguishable adversarial
fragment. FragGAN ensures that the resulting entire image is effective in
attacking. For any physical-world implementation, an attacker could physically
print out the adversarial fragment and then paste it onto the original fragment
(e.g., a roadside sign for autonomous driving scenarios). FragGAN also enables
clean-label attacks against image classification, as the resulting attacks may
succeed even without modifying any essential content of an image. Extensive
experiments including physical-world case studies on state-of-the-art
autonomous steering and image classification models demonstrate that FragGAN is
highly effective and superior to simple extensions of existing approaches. To
the best of our knowledge, FragGAN is the first approach that can implement
effective and clean-label physical-world attacks.


http://arxiv.org/abs/1907.05274
Affine Disentangled GAN for Interpretable and Robust AV Perception. (92%)
Letao Liu; Martin Saerbeck; Justin Dauwels
  Autonomous vehicles (AV) have progressed rapidly with the advancements in
computer vision algorithms. The deep convolutional neural network as the main
contributor to this advancement has boosted the classification accuracy
dramatically. However, the discovery of adversarial examples reveals the
generalization gap between dataset and the real world. Furthermore, affine
transformations may also confuse computer vision based object detectors. The
degradation of the perception system is undesirable for safety critical systems
such as autonomous vehicles. In this paper, a deep learning system is proposed:
Affine Disentangled GAN (ADIS-GAN), which is robust against affine
transformations and adversarial attacks. It is demonstrated that conventional
data augmentation for affine transformation and adversarial attacks are
orthogonal, while ADIS-GAN can handle both attacks at the same time. Useful
information such as image rotation angle and scaling factor are also generated
in ADIS-GAN. On MNIST dataset, ADIS-GAN can achieve over 98 percent
classification accuracy within 30 degrees rotation, and over 90 percent
classification accuracy against FGSM and PGD adversarial attack.


http://arxiv.org/abs/1907.02957
Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions.
Yao Qin; Nicholas Frosst; Sara Sabour; Colin Raffel; Garrison Cottrell; Geoffrey Hinton
  Adversarial examples raise questions about whether neural network models are
sensitive to the same visual features as humans. Most of the proposed methods
for mitigating adversarial examples have subsequently been defeated by stronger
attacks. Motivated by these issues, we take a different approach and propose to
instead detect adversarial examples based on class-conditional reconstructions
of the input. Our method uses the reconstruction network proposed as part of
Capsule Networks (CapsNets), but is general enough to be applied to standard
convolutional networks. We find that adversarial or otherwise corrupted images
result in much larger reconstruction errors than normal inputs, prompting a
simple detection method by thresholding the reconstruction error. Based on
these findings, we propose the Reconstructive Attack which seeks both to cause
a misclassification and a low reconstruction error. While this attack produces
undetected adversarial examples, we find that for CapsNets the resulting
perturbations can cause the images to appear visually more like the target
class. This suggests that CapsNets utilize features that are more aligned with
human perception and address the central issue raised by adversarial examples.


http://arxiv.org/abs/1907.02610
Adversarial Robustness through Local Linearization.
Chongli Dj Qin; James Dj Martens; Sven Dj Gowal; Dilip Dj Krishnan; Dj Krishnamurthy; Dvijotham; Alhussein Fawzi; Soham De; Robert Stanforth; Pushmeet Kohli
  Adversarial training is an effective methodology for training deep neural
networks that are robust against adversarial, norm-bounded perturbations.
However, the computational cost of adversarial training grows prohibitively as
the size of the model and number of input dimensions increase. Further,
training against less expensive and therefore weaker adversaries produces
models that are robust against weak attacks but break down under attacks that
are stronger. This is often attributed to the phenomenon of gradient
obfuscation; such models have a highly non-linear loss surface in the vicinity
of training examples, making it hard for gradient-based attacks to succeed even
though adversarial examples still exist. In this work, we introduce a novel
regularizer that encourages the loss to behave linearly in the vicinity of the
training data, thereby penalizing gradient obfuscation while encouraging
robustness. We show via extensive experiments on CIFAR-10 and ImageNet, that
models trained with our regularizer avoid gradient obfuscation and can be
trained significantly faster than adversarial training. Using this regularizer,
we exceed current state of the art and achieve 47% adversarial accuracy for
ImageNet with l-infinity adversarial perturbations of radius 4/255 under an
untargeted, strong, white-box attack. Additionally, we match state of the art
results for CIFAR-10 at 8/255.


http://arxiv.org/abs/1907.02477
Adversarial Attacks in Sound Event Classification.
Vinod Subramanian; Emmanouil Benetos; Ning Xu; SKoT McDonald; Mark Sandler
  Adversarial attacks refer to a set of methods that perturb the input to a
classification model in order to fool the classifier. In this paper we apply
different gradient based adversarial attack algorithms on five deep learning
models trained for sound event classification. Four of the models use
mel-spectrogram input and one model uses raw audio input. The models represent
standard architectures such as convolutional, recurrent and dense networks. The
dataset used for training is the Freesound dataset released for task 2 of the
DCASE 2018 challenge and the models used are from participants of the challenge
who open sourced their code. Our experiments show that adversarial attacks can
be generated with high confidence and low perturbation. In addition, we show
that the adversarial attacks are very effective across the different models.


http://arxiv.org/abs/1907.01996
Robust Synthesis of Adversarial Visual Examples Using a Deep Image Prior.
Thomas Gittings; Steve Schneider; John Collomosse
  We present a novel method for generating robust adversarial image examples
building upon the recent `deep image prior' (DIP) that exploits convolutional
network architectures to enforce plausible texture in image synthesis.
Adversarial images are commonly generated by perturbing images to introduce
high frequency noise that induces image misclassification, but that is fragile
to subsequent digital manipulation of the image. We show that using DIP to
reconstruct an image under adversarial constraint induces perturbations that
are more robust to affine deformation, whilst remaining visually imperceptible.
Furthermore we show that our DIP approach can also be adapted to produce local
adversarial patches (`adversarial stickers'). We demonstrate robust adversarial
examples over a broad gamut of images and object classes drawn from the
ImageNet dataset.


http://arxiv.org/abs/1907.02044
Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack.
Francesco Croce; Matthias Hein
  The evaluation of robustness against adversarial manipulation of neural
networks-based classifiers is mainly tested with empirical attacks as the
methods for the exact computation, even when available, do not scale to large
networks. We propose in this paper a new white-box adversarial attack wrt the
$l_p$-norms for $p \in \{1,2,\infty\}$ aiming at finding the minimal
perturbation necessary to change the class of a given input. It has an
intuitive geometric meaning, yields high quality results already with one
restart, minimizes the size of the perturbation, so that the robust accuracy
can be evaluated at all possible thresholds with a single run, and comes with
almost no free parameters except number of iterations and restarts. It achieves
better or similar robust test accuracy compared to state-of-the-art attacks
which are partially specialized to one $l_p$-norm.


http://arxiv.org/abs/1907.01197
Treant: Training Evasion-Aware Decision Trees.
Stefano Calzavara; Claudio Lucchese; Gabriele Tolomei; Seyum Assefa Abebe; Salvatore Orlando
  Despite its success and popularity, machine learning is now recognized as
vulnerable to evasion attacks, i.e., carefully crafted perturbations of test
inputs designed to force prediction errors. In this paper we focus on evasion
attacks against decision tree ensembles, which are among the most successful
predictive models for dealing with non-perceptual problems. Even though they
are powerful and interpretable, decision tree ensembles have received only
limited attention by the security and machine learning communities so far,
leading to a sub-optimal state of the art for adversarial learning techniques.
We thus propose Treant, a novel decision tree learning algorithm that, on the
basis of a formal threat model, minimizes an evasion-aware loss function at
each step of the tree construction. Treant is based on two key technical
ingredients: robust splitting and attack invariance, which jointly guarantee
the soundness of the learning process. Experimental results on three publicly
available datasets show that Treant is able to generate decision tree ensembles
that are at the same time accurate and nearly insensitive to evasion attacks,
outperforming state-of-the-art adversarial learning techniques.


http://arxiv.org/abs/1907.00895
Comment on "Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network".
Roland S. Zimmermann
  A recent paper by Liu et al. combines the topics of adversarial training and
Bayesian Neural Networks (BNN) and suggests that adversarially trained BNNs are
more robust against adversarial attacks than their non-Bayesian counterparts.
Here, I analyze the proposed defense and suggest that one needs to adjust the
adversarial attack to incorporate the stochastic nature of a Bayesian network
to perform an accurate evaluation of its robustness. Using this new type of
attack I show that there appears to be no strong evidence for higher robustness
of the adversarially trained BNNs.


http://arxiv.org/abs/1907.01023
Diminishing the Effect of Adversarial Perturbations via Refining Feature Representation.
Nader Asadi; AmirMohammad Sarfi; Sahba Tahsini; Mahdi Eftekhari
  Deep neural networks are highly vulnerable to adversarial examples, which
imposes severe security issues for these state-of-the-art models. Many defense
methods have been proposed to mitigate this problem. However, a lot of them
depend on modification or additional training of the target model. In this
work, we analytically investigate each layer representation of non-perturbed
and perturbed images and show the effect of perturbations on each of these
representations. Accordingly, a method based on whitening coloring transform is
proposed in order to diminish the misrepresentation of any desirable layer
caused by adversaries. Our method can be applied to any layer of any arbitrary
model without the need of any modification or additional training. Due to the
fact that full whitening of the layer representation is not easily
differentiable, our proposed method is superbly robust against white-box
attacks. Furthermore, we demonstrate the strength of our method against some
state-of-the-art black-box attacks such as Carlini-Wagner L2 attack and we show
that our method is able to defend against some non-constrained attacks.


http://arxiv.org/abs/1907.01003
Accurate, reliable and fast robustness evaluation.
Wieland Brendel; Jonas Rauber; Matthias Kümmerer; Ivan Ustyuzhaninov; Matthias Bethge
  Throughout the past five years, the susceptibility of neural networks to
minimal adversarial perturbations has moved from a peculiar phenomenon to a
core issue in Deep Learning. Despite much attention, however, progress towards
more robust models is significantly impaired by the difficulty of evaluating
the robustness of neural network models. Today's methods are either fast but
brittle (gradient-based attacks), or they are fairly reliable but slow (score-
and decision-based attacks). We here develop a new set of gradient-based
adversarial attacks which (a) are more reliable in the face of gradient-masking
than other gradient-based attacks, (b) perform better and are more query
efficient than current state-of-the-art gradient-based attacks, (c) can be
flexibly adapted to a wide range of adversarial criteria and (d) require
virtually no hyperparameter tuning. These findings are carefully validated
across a diverse set of six different models and hold for L2 and L_infinity in
both targeted as well as untargeted scenarios. Implementations will be made
available in all major toolboxes (Foolbox, CleverHans and ART). Furthermore, we
will soon add additional content and experiments, including L0 and L1 versions
of our attack as well as additional comparisons to other L2 and L_infinity
attacks. We hope that this class of attacks will make robustness evaluations
easier and more reliable, thus contributing to more signal in the search for
more robust machine learning models.


http://arxiv.org/abs/1907.00374
Fooling a Real Car with Adversarial Traffic Signs.
Nir Morgulis; Alexander Kreines; Shachar Mendelowitz; Yuval Weisglass
  The attacks on the neural-network-based classifiers using adversarial images
have gained a lot of attention recently. An adversary can purposely generate an
image that is indistinguishable from a innocent image for a human being but is
incorrectly classified by the neural networks. The adversarial images do not
need to be tuned to a particular architecture of the classifier - an image that
fools one network can fool another one with a certain success rate.The
published works mostly concentrate on the use of modified image files for
attacks against the classifiers trained on the model databases. Although there
exists a general understanding that such attacks can be carried in the real
world as well, the works considering the real-world attacks are scarce.
Moreover, to the best of our knowledge, there have been no reports on the
attacks against real production-grade image classification systems.In our work
we present a robust pipeline for reproducible production of adversarial traffic
signs that can fool a wide range of classifiers, both open-source and
production-grade in the real world. The efficiency of the attacks was checked
both with the neural-network-based classifiers and legacy computer vision
systems. Most of the attacks have been performed in the black-box mode, e.g.
the adversarial signs produced for a particular classifier were used to attack
a variety of other classifiers. The efficiency was confirmed in drive-by
experiments with a production-grade traffic sign recognition systems of a real
car.


http://arxiv.org/abs/1906.12340
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.
Dan Hendrycks; Mantas Mazeika; Saurav Kadavath; Dawn Song
  Self-supervision provides effective representations for downstream tasks
without requiring labels. However, existing approaches lag behind fully
supervised training and are often not thought beneficial beyond obviating the
need for annotations. We find that self-supervision can benefit robustness in a
variety of ways, including robustness to adversarial examples, label
corruption, and common input corruptions. Additionally, self-supervision
greatly benefits out-of-distribution detection on difficult, near-distribution
outliers, so much so that it exceeds the performance of fully supervised
methods. These results demonstrate the promise of self-supervision for
improving robustness and uncertainty estimation and establish these tasks as
new axes of evaluation for future self-supervised learning research.


http://arxiv.org/abs/1906.12269
Certifiable Robustness and Robust Training for Graph Convolutional Networks.
Daniel Zügner; Stephan Günnemann
  Recent works show that Graph Neural Networks (GNNs) are highly non-robust
with respect to adversarial attacks on both the graph structure and the node
attributes, making their outcomes unreliable. We propose the first method for
certifiable (non-)robustness of graph convolutional networks with respect to
perturbations of the node attributes. We consider the case of binary node
attributes (e.g. bag-of-words) and perturbations that are L_0-bounded. If a
node has been certified with our method, it is guaranteed to be robust under
any possible perturbation given the attack model. Likewise, we can certify
non-robustness. Finally, we propose a robust semi-supervised training procedure
that treats the labeled and unlabeled nodes jointly. As shown in our
experimental evaluation, our method significantly improves the robustness of
the GNN with only minimal effect on the predictive accuracy.


http://arxiv.org/abs/1906.12061
Learning to Cope with Adversarial Attacks.
Xian Yeow Lee; Aaron Havens; Girish Chowdhary; Soumik Sarkar
  The security of Deep Reinforcement Learning (Deep RL) algorithms deployed in
real life applications are of a primary concern. In particular, the robustness
of RL agents in cyber-physical systems against adversarial attacks are
especially vital since the cost of a malevolent intrusions can be extremely
high. Studies have shown Deep Neural Networks (DNN), which forms the core
decision-making unit in most modern RL algorithms, are easily subjected to
adversarial attacks. Hence, it is imperative that RL agents deployed in
real-life applications have the capability to detect and mitigate adversarial
attacks in an online fashion. An example of such a framework is the
Meta-Learned Advantage Hierarchy (MLAH) agent that utilizes a meta-learning
framework to learn policies robustly online. Since the mechanism of this
framework are still not fully explored, we conducted multiple experiments to
better understand the framework's capabilities and limitations. Our results
shows that the MLAH agent exhibits interesting coping behaviors when subjected
to different adversarial attacks to maintain a nominal reward. Additionally,
the framework exhibits a hierarchical coping capability, based on the
adaptability of the Master policy and sub-policies themselves. From empirical
results, we also observed that as the interval of adversarial attacks increase,
the MLAH agent can maintain a higher distribution of rewards, though at the
cost of higher instabilities.


http://arxiv.org/abs/1907.00098
Robustness Guarantees for Deep Neural Networks on Videos.
Min Wu; Marta Kwiatkowska
  The widespread adoption of deep learning models places demands on their
robustness. In this paper, we consider the robustness of deep neural networks
on videos, which comprise both the spatial features of individual frames
extracted by a convolutional neural network and the temporal dynamics between
adjacent frames captured by a recurrent neural network. To measure robustness,
we study the maximum safe radius problem, which computes the minimum distance
from the optical flow set obtained from a given input to that of an adversarial
example in the norm ball. We demonstrate that, under the assumption of
Lipschitz continuity, the problem can be approximated using finite optimisation
via discretising the optical flow space, and the approximation has provable
guarantees. We then show that the finite optimisation problem can be solved by
utilising a two-player turn-based game in a cooperative setting, where the
first player selects the optical flows and the second player determines the
dimensions to be manipulated in the chosen flow. We employ an anytime approach
to solve the game, in the sense of approximating the value of the game by
monotonically improving its upper and lower bounds. We exploit a gradient-based
search algorithm to compute the upper bounds, and the admissible A* algorithm
to update the lower bounds. Finally, we evaluate our framework on the UCF101
video dataset.


http://arxiv.org/abs/1906.11567
Adversarial Robustness via Adversarial Label-Smoothing.
Morgane Goibert; Elvis Dohmatob
  We study Label-Smoothing as a means for improving adversarial robustness of
supervised deep-learning models. After establishing a thorough and unified
framework, we propose several novel Label-Smoothing methods: adversarial,
Boltzmann and second-best Label-Smoothing methods. On various datasets (MNIST,
CIFAR10, SVHN) and models (linear models, MLPs, LeNet, ResNet), we show that
these methods improve adversarial robustness against a variety of attacks
(FGSM, BIM, DeepFool, Carlini-Wagner) by better taking account of the dataset
geometry. These proposed Label-Smoothing methods have two main advantages: they
can be implemented as a modified cross-entropy loss, thus do not require any
modifications of the network architecture nor do they lead to increased
training times, and they improve both standard and adversarial accuracy.


http://arxiv.org/abs/1906.11729
Using Intuition from Empirical Properties to Simplify Adversarial Training Defense.
Guanxiong Liu; Issa Khalil; Abdallah Khreishah
  Due to the surprisingly good representation power of complex distributions,
neural network (NN) classifiers are widely used in many tasks which include
natural language processing, computer vision and cyber security. In recent
works, people noticed the existence of adversarial examples. These adversarial
examples break the NN classifiers' underlying assumption that the environment
is attack free and can easily mislead fully trained NN classifier without
noticeable changes. Among defensive methods, adversarial training is a popular
choice. However, original adversarial training with single-step adversarial
examples (Single-Adv) can not defend against iterative adversarial examples.
Although adversarial training with iterative adversarial examples (Iter-Adv)
can defend against iterative adversarial examples, it consumes too much
computational power and hence is not scalable. In this paper, we analyze
Iter-Adv techniques and identify two of their empirical properties. Based on
these properties, we propose modifications which enhance Single-Adv to perform
competitively as Iter-Adv. Through preliminary evaluation, we show that the
proposed method enhances the test accuracy of state-of-the-art (SOTA)
Single-Adv defensive method against iterative adversarial examples by up to
16.93% while reducing its training cost by 28.75%.


http://arxiv.org/abs/1906.11667
Evolving Robust Neural Architectures to Defend from Adversarial Attacks.
Danilo Vasconcellos Vargas; Shashank Kotyan
  Deep neural networks were shown to misclassify slightly modified input
images. Recently, many defenses have been proposed but none have improved
consistently the robustness of neural networks. Here, we propose to use attacks
as a function evaluation to automatically search for architectures that can
resist such attacks. Experiments on neural architecture search algorithms from
the literature show that although their accurate results, they are not able to
find robust architectures. Most of the reason for this lies in their limited
search space. By creating a novel neural architecture search with options for
dense layers to connect with convolution layers and vice-versa as well as the
addition of multiplication, addition and concatenation layers in the search
space, we were able to evolve an architecture that is $58\%$ accurate on
adversarial samples. Interestingly, this inherent robustness of the evolved
architecture rivals state-of-the-art defenses such as adversarial training
while being trained only on the training dataset. Moreover, the evolved
architecture makes use of some peculiar traits which might be useful for
developing even more robust ones. Thus, the results here demonstrate that more
robust architectures exist as well as opens up a new range of possibilities for
the development and exploration of deep neural networks using automatic
architecture search.
  Code available at http://bit.ly/RobustArchitectureSearch.


http://arxiv.org/abs/1906.11327
The Adversarial Robustness of Sampling.
Omri Ben-Eliezer; Eylon Yogev
  Random sampling is a fundamental primitive in modern algorithms, statistics,
and machine learning, used as a generic method to obtain a small yet
"representative" subset of the data. In this work, we investigate the
robustness of sampling against adaptive adversarial attacks in a streaming
setting: An adversary sends a stream of elements from a universe $U$ to a
sampling algorithm (e.g., Bernoulli sampling or reservoir sampling), with the
goal of making the sample "very unrepresentative" of the underlying data
stream. The adversary is fully adaptive in the sense that it knows the exact
content of the sample at any given point along the stream, and can choose which
element to send next accordingly, in an online manner.
  Well-known results in the static setting indicate that if the full stream is
chosen in advance (non-adaptively), then a random sample of size $\Omega(d /
\varepsilon^2)$ is an $\varepsilon$-approximation of the full data with good
probability, where $d$ is the VC-dimension of the underlying set system
$(U,R)$. Does this sample size suffice for robustness against an adaptive
adversary? The simplistic answer is \emph{negative}: We demonstrate a set
system where a constant sample size (corresponding to VC-dimension $1$)
suffices in the static setting, yet an adaptive adversary can make the sample
very unrepresentative, as long as the sample size is (strongly) sublinear in
the stream length, using a simple and easy-to-implement attack.
  However, this attack is "theoretical only", requiring the set system size to
(essentially) be exponential in the stream length. This is not a coincidence:
We show that to make Bernoulli or reservoir sampling robust against adaptive
adversaries, the modification required is solely to replace the VC-dimension
term $d$ in the sample size with the cardinality term $\log |R|$. This nearly
matches the bound imposed by the attack.


http://arxiv.org/abs/1906.10973
Defending Adversarial Attacks by Correcting logits.
Yifeng Li; Lingxi Xie; Ya Zhang; Rui Zhang; Yanfeng Wang; Qi Tian
  Generating and eliminating adversarial examples has been an intriguing topic
in the field of deep learning. While previous research verified that
adversarial attacks are often fragile and can be defended via image-level
processing, it remains unclear how high-level features are perturbed by such
attacks. We investigate this issue from a new perspective, which purely relies
on logits, the class scores before softmax, to detect and defend adversarial
attacks. Our defender is a two-layer network trained on a mixed set of clean
and perturbed logits, with the goal being recovering the original prediction.
Upon a wide range of adversarial attacks, our simple approach shows promising
results with relatively high accuracy in defense, and the defender can transfer
across attackers with similar properties. More importantly, our defender can
work in the scenarios that image data are unavailable, and enjoys high
interpretability especially at the semantic level.


http://arxiv.org/abs/1906.10395
Quantitative Verification of Neural Networks And its Security Applications.
Teodora Baluta; Shiqi Shen; Shweta Shinde; Kuldeep S. Meel; Prateek Saxena
  Neural networks are increasingly employed in safety-critical domains. This
has prompted interest in verifying or certifying logically encoded properties
of neural networks. Prior work has largely focused on checking existential
properties, wherein the goal is to check whether there exists any input that
violates a given property of interest. However, neural network training is a
stochastic process, and many questions arising in their analysis require
probabilistic and quantitative reasoning, i.e., estimating how many inputs
satisfy a given property. To this end, our paper proposes a novel and
principled framework to quantitative verification of logical properties
specified over neural networks. Our framework is the first to provide PAC-style
soundness guarantees, in that its quantitative estimates are within a
controllable and bounded error from the true count. We instantiate our
algorithmic framework by building a prototype tool called NPAQ that enables
checking rich properties over binarized neural networks. We show how emerging
security analyses can utilize our framework in 3 concrete point applications:
quantifying robustness to adversarial inputs, efficacy of trojan attacks, and
fairness/bias of given neural networks.


http://arxiv.org/abs/1906.10773
Are Adversarial Perturbations a Showstopper for ML-Based CAD? A Case Study on CNN-Based Lithographic Hotspot Detection.
Kang Liu; Haoyu Yang; Yuzhe Ma; Benjamin Tan; Bei Yu; Evangeline F. Y. Young; Ramesh Karri; Siddharth Garg
  There is substantial interest in the use of machine learning (ML) based
techniques throughout the electronic computer-aided design (CAD) flow,
particularly those based on deep learning. However, while deep learning methods
have surpassed state-of-the-art performance in several applications, they have
exhibited intrinsic susceptibility to adversarial perturbations --- small but
deliberate alterations to the input of a neural network, precipitating
incorrect predictions. In this paper, we seek to investigate whether
adversarial perturbations pose risks to ML-based CAD tools, and if so, how
these risks can be mitigated. To this end, we use a motivating case study of
lithographic hotspot detection, for which convolutional neural networks (CNN)
have shown great promise. In this context, we show the first adversarial
perturbation attacks on state-of-the-art CNN-based hotspot detectors;
specifically, we show that small (on average 0.5% modified area), functionality
preserving and design-constraint satisfying changes to a layout can nonetheless
trick a CNN-based hotspot detector into predicting the modified layout as
hotspot free (with up to 99.7% success). We propose an adversarial retraining
strategy to improve the robustness of CNN-based hotspot detection and show that
this strategy significantly improves robustness (by a factor of ~3) against
adversarial attacks without compromising classification accuracy.


http://arxiv.org/abs/1906.10571
Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals.
Yunhan Huang; Quanyan Zhu
  This paper studies reinforcement learning (RL) under malicious falsification
on cost signals and introduces a quantitative framework of attack models to
understand the vulnerabilities of RL. Focusing on $Q$-learning, we show that
$Q$-learning algorithms converge under stealthy attacks and bounded
falsifications on cost signals. We characterize the relation between the
falsified cost and the $Q$-factors as well as the policy learned by the
learning agent which provides fundamental limits for feasible offensive and
defensive moves. We propose a robust region in terms of the cost within which
the adversary can never achieve the targeted policy. We provide conditions on
the falsified cost which can mislead the agent to learn an adversary's favored
policy. A numerical case study of water reservoir control is provided to show
the potential hazards of RL in learning-based control systems and corroborate
the results.


http://arxiv.org/abs/1906.09525
Defending Against Adversarial Examples with K-Nearest Neighbor.
Chawin Sitawarin; David Wagner
  Robustness is an increasingly important property of machine learning models
as they become more and more prevalent. We propose a defense against
adversarial examples based on a k-nearest neighbor (kNN) on the intermediate
activation of neural networks. Our scheme surpasses state-of-the-art defenses
on MNIST and CIFAR-10 against l2-perturbation by a significant margin. With our
models, the mean perturbation norm required to fool our MNIST model is 3.07 and
2.30 on CIFAR-10. Additionally, we propose a simple certifiable lower bound on
the l2-norm of the adversarial perturbation using a more specific version of
our scheme, a 1-NN on representations learned by a Lipschitz network. Our model
provides a nontrivial average lower bound of the perturbation norm, comparable
to other schemes on MNIST with similar clean accuracy.


http://arxiv.org/abs/1906.09288
Hiding Faces in Plain Sight: Disrupting AI Face Synthesis with Adversarial Perturbations.
Yuezun Li; Xin Yang; Baoyuan Wu; Siwei Lyu
  Recent years have seen fast development in synthesizing realistic human faces
using AI technologies. Such fake faces can be weaponized to cause negative
personal and social impact. In this work, we develop technologies to defend
individuals from becoming victims of recent AI synthesized fake videos by
sabotaging would-be training data. This is achieved by disrupting deep neural
network (DNN) based face detection method with specially designed imperceptible
adversarial perturbations to reduce the quality of the detected faces. We
describe attacking schemes under white-box, gray-box and black-box settings,
each with decreasing information about the DNN based face detectors. We
empirically show the effectiveness of our methods in disrupting
state-of-the-art DNN based face detectors on several datasets.


http://arxiv.org/abs/1906.09300
Adversarial Examples to Fool Iris Recognition Systems.
Sobhan Soleymani; Ali Dabouei; Jeremy Dawson; Nasser M. Nasrabadi
  Adversarial examples have recently proven to be able to fool deep learning
methods by adding carefully crafted small perturbation to the input space
image. In this paper, we study the possibility of generating adversarial
examples for code-based iris recognition systems. Since generating adversarial
examples requires back-propagation of the adversarial loss, conventional filter
bank-based iris-code generation frameworks cannot be employed in such a setup.
Therefore, to compensate for this shortcoming, we propose to train a deep
auto-encoder surrogate network to mimic the conventional iris code generation
procedure. This trained surrogate network is then deployed to generate the
adversarial examples using the iterative gradient sign method algorithm. We
consider non-targeted and targeted attacks through three attack scenarios.
Considering these attacks, we study the possibility of fooling an iris
recognition system in white-box and black-box frameworks.


http://arxiv.org/abs/1906.08988
A Fourier Perspective on Model Robustness in Computer Vision.
Dong Yin; Raphael Gontijo Lopes; Jonathon Shlens; Ekin D. Cubuk; Justin Gilmer
  Achieving robustness to distributional shift is a longstanding and
challenging goal of computer vision. Data augmentation is a commonly used
approach for improving robustness, however robustness gains are typically not
uniform across corruption types. Indeed increasing performance in the presence
of random noise is often met with reduced performance on other corruptions such
as contrast change. Understanding when and why these sorts of trade-offs occur
is a crucial step towards mitigating them. Towards this end, we investigate
recently observed trade-offs caused by Gaussian data augmentation and
adversarial training. We find that both methods improve robustness to
corruptions that are concentrated in the high frequency domain while reducing
robustness to corruptions that are concentrated in the low frequency domain.
This suggests that one way to mitigate these trade-offs via data augmentation
is to use a more diverse set of augmentations. Towards this end we observe that
AutoAugment, a recently proposed data augmentation policy optimized for clean
accuracy, achieves state-of-the-art robustness on the CIFAR-10-C and ImageNet-C
benchmarks.


http://arxiv.org/abs/1906.09072
Evolution Attack On Neural Networks.
YiGui Luo; RuiJia Yang; Wei Sha; WeiYi Ding; YouTeng Sun; YiSi Wang
  Many studies have been done to prove the vulnerability of neural networks to
adversarial example. A trained and well-behaved model can be fooled by a
visually imperceptible perturbation, i.e., an originally correctly classified
image could be misclassified after a slight perturbation. In this paper, we
propose a black-box strategy to attack such networks using an evolution
algorithm. First, we formalize the generation of an adversarial example into
the optimization problem of perturbations that represent the noise added to an
original image at each pixel. To solve this optimization problem in a black-box
way, we find that an evolution algorithm perfectly meets our requirement since
it can work without any gradient information. Therefore, we test various
evolution algorithms, including a simple genetic algorithm, a
parameter-exploring policy gradient, an OpenAI evolution strategy, and a
covariance matrix adaptive evolution strategy. Experimental results show that a
covariance matrix adaptive evolution Strategy performs best in this
optimization problem. Additionally, we also perform several experiments to
explore the effect of different regularizations on improving the quality of an
adversarial example.


http://arxiv.org/abs/1906.11897
On Physical Adversarial Patches for Object Detection.
Mark Lee; Zico Kolter
  In this paper, we demonstrate a physical adversarial patch attack against
object detectors, notably the YOLOv3 detector. Unlike previous work on physical
object detection attacks, which required the patch to overlap with the objects
being misclassified or avoiding detection, we show that a properly designed
patch can suppress virtually all the detected objects in the image. That is, we
can place the patch anywhere in the image, causing all existing objects in the
image to be missed entirely by the detector, even those far away from the patch
itself. This in turn opens up new lines of physical attacks against object
detection systems, which require no modification of the objects in a scene. A
demo of the system can be found at https://youtu.be/WXnQjbZ1e7Y.


http://arxiv.org/abs/1907.03720
Catfish Effect Between Internal and External Attackers:Being Semi-honest is Helpful.
Hanqing Liu; Na Ruan; Joseph K. Liu
  The consensus protocol named proof of work (PoW) is widely applied by
cryptocurrencies like Bitcoin. Although security of a PoW cryptocurrency is
always the top priority, it is threatened by mining attacks like selfish
mining. Researchers have proposed many mining attack models with one attacker,
and optimized the attacker's strategy. During these mining attacks, an attacker
pursues a higher relative revenue (RR) by wasting a large amount of
computational power of the honest miners at the cost of a small amount of
computational power of himself. In this paper, we propose a mining attack model
with two phases: the original system and the multi-attacker system. It is the
first model to provide both theoretical and quantitative analysis of mining
attacks with two attackers. We explain how the original system turns into the
multi-attacker system by introducing two attackers: the internal attacker and
the external attacker. If both attackers take the attacking strategy selfish
mining, the RR of the internal attacker in multi-attacker system will drop by
up to 31.9% compared with his RR in original system. The external attacker will
overestimate his RR by up to 44.6% in multiattacker system. Unexpected
competitions, auctions between attackers and overestimation of attackers'
influence factor are three main causes of both attackers' dropping RR. We
propose a mining strategy named Partial Initiative Release (PIR) which is a
semi-honest mining strategy in multi-attacker system. In some specific
situations, PIR allows the attacker to get more block reward by launching an
attack in multi-attacker system.


http://arxiv.org/abs/1906.08416
Improving the robustness of ImageNet classifiers using elements of human visual cognition.
A. Emin Orhan; Brenden M. Lake
  We investigate the robustness properties of image recognition models equipped
with two features inspired by human vision, an explicit episodic memory and a
shape bias, at the ImageNet scale. As reported in previous work, we show that
an explicit episodic memory improves the robustness of image recognition models
against small-norm adversarial perturbations under some threat models. It does
not, however, improve the robustness against more natural, and typically
larger, perturbations. Learning more robust features during training appears to
be necessary for robustness in this second sense. We show that features derived
from a model that was encouraged to learn global, shape-based representations
(Geirhos et al., 2019) do not only improve the robustness against natural
perturbations, but when used in conjunction with an episodic memory, they also
provide additional robustness against adversarial perturbations. Finally, we
address three important design choices for the episodic memory: memory size,
dimensionality of the memories and the retrieval method. We show that to make
the episodic memory more compact, it is preferable to reduce the number of
memories by clustering them, instead of reducing their dimensionality.


http://arxiv.org/abs/1906.07982
A unified view on differential privacy and robustness to adversarial examples.
Rafael Pinot; Florian Yger; Cédric Gouy-Pailler; Jamal Atif
  This short note highlights some links between two lines of research within
the emerging topic of trustworthy machine learning: differential privacy and
robustness to adversarial examples. By abstracting the definitions of both
notions, we show that they build upon the same theoretical ground and hence
results obtained so far in one domain can be transferred to the other. More
precisely, our analysis is based on two key elements: probabilistic mappings
(also called randomized algorithms in the differential privacy community), and
the Renyi divergence which subsumes a large family of divergences. We first
generalize the definition of robustness against adversarial examples to
encompass probabilistic mappings. Then we observe that Renyi-differential
privacy (a generalization of differential privacy recently proposed
in~\cite{Mironov2017RenyiDP}) and our definition of robustness share several
similarities. We finally discuss how can both communities benefit from this
connection to transfer technical tools from one research field to the other.


http://arxiv.org/abs/1906.07916
Convergence of Adversarial Training in Overparametrized Networks.
Ruiqi Gao; Tianle Cai; Haochuan Li; Liwei Wang; Cho-Jui Hsieh; Jason D. Lee
  Neural networks are vulnerable to adversarial examples, i.e. inputs that are
imperceptibly perturbed from natural data and yet incorrectly classified by the
network. Adversarial training, a heuristic form of robust optimization that
alternates between minimization and maximization steps, has proven to be among
the most successful methods to train networks that are robust against a
pre-defined family of perturbations. This paper provides a partial answer to
the success of adversarial training. When the inner maximization problem can be
solved to optimality, we prove that adversarial training finds a network of
small robust train loss. When the maximization problem is solved by a heuristic
algorithm, we prove that adversarial training finds a network of small robust
surrogate train loss. The analysis technique leverages recent work on the
analysis of neural networks via Neural Tangent Kernel (NTK), combined with
online-learning when the maximization is solved by a heuristic, and the
expressiveness of the NTK kernel in the $\ell_\infty$-norm.


http://arxiv.org/abs/1906.07920
Global Adversarial Attacks for Assessing Deep Learning Robustness.
Hanbin Hu; Mit Shah; Jianhua Z. Huang; Peng Li
  It has been shown that deep neural networks (DNNs) may be vulnerable to
adversarial attacks, raising the concern on their robustness particularly for
safety-critical applications. Recognizing the local nature and limitations of
existing adversarial attacks, we present a new type of global adversarial
attacks for assessing global DNN robustness. More specifically, we propose a
novel concept of global adversarial example pairs in which each pair of two
examples are close to each other but have different class labels predicted by
the DNN. We further propose two families of global attack methods and show that
our methods are able to generate diverse and intriguing adversarial example
pairs at locations far from the training or testing data. Moreover, we
demonstrate that DNNs hardened using the strong projected gradient descent
(PGD) based (local) adversarial training are vulnerable to the proposed global
adversarial example pairs, suggesting that global robustness must be considered
while training robust deep learning networks.


http://arxiv.org/abs/1906.07997
Cloud-based Image Classification Service Is Not Robust To Simple Transformations: A Forgotten Battlefield.
Dou Goodman; Tao Wei
  Many recent works demonstrated that Deep Learning models are vulnerable to
adversarial examples.Fortunately, generating adversarial examples usually
requires white-box access to the victim model, and the attacker can only access
the APIs opened by cloud platforms. Thus, keeping models in the cloud can
usually give a (false) sense of security.Unfortunately, cloud-based image
classification service is not robust to simple transformations such as Gaussian
Noise, Salt-and-Pepper Noise, Rotation and Monochromatization. In this
paper,(1) we propose one novel attack method called Image Fusion(IF) attack,
which achieve a high bypass rate,can be implemented only with OpenCV and is
difficult to defend; and (2) we make the first attempt to conduct an extensive
empirical study of Simple Transformation (ST) attacks against real-world
cloud-based classification services. Through evaluations on four popular cloud
platforms including Amazon, Google, Microsoft, Clarifai, we demonstrate that ST
attack has a success rate of approximately 100% except Amazon approximately
50%, IF attack have a success rate over 98% among different classification
services. (3) We discuss the possible defenses to address these security
challenges.Experiments show that our defense technology can effectively defend
known ST attacks.


http://arxiv.org/abs/1906.07927
SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing.
Haonan Qiu; Chaowei Xiao; Lei Yang; Xinchen Yan; Honglak Lee; Bo Li
  Deep neural networks (DNNs) have achieved great success in various
applications due to their strong expressive power. However, recent studies have
shown that DNNs are vulnerable to adversarial examples which are manipulated
instances targeting to mislead DNNs to make incorrect predictions. Currently,
most such adversarial examples try to guarantee "subtle perturbation" by
limiting its $L_p$ norm. In this paper, we aim to explore the impact of
semantic manipulation on DNNs predictions by manipulating the semantic
attributes of images and generate "unrestricted adversarial examples". Such
semantic based perturbation is more practical compared with pixel level
manipulation. In particular, we propose an algorithm SemanticAdv which
leverages disentangled semantic factors to generate adversarial perturbation
via altering either single or a combination of semantic attributes. We conduct
extensive experiments to show that the semantic based adversarial examples can
not only fool different learning tasks such as face verification and landmark
detection, but also achieve high attack success rate against real-world
black-box services such as Azure face verification service. Such structured
adversarial examples with controlled semantic manipulation can shed light on
further understanding about vulnerabilities of DNNs as well as potential
defensive approaches.


http://arxiv.org/abs/1906.07153
Adversarial attacks on Copyright Detection Systems.
Parsa Saadatpanah; Ali Shafahi; Tom Goldstein
  It is well-known that many machine learning models are susceptible to
so-called "adversarial attacks," in which an attacker evades a classifier by
making small perturbations to inputs. This paper discusses how industrial
copyright detection tools, which serve a central role on the web, are
susceptible to adversarial attacks. We discuss a range of copyright detection
systems, and why they are particularly vulnerable to attacks. These
vulnerabilities are especially apparent for neural network based systems. As a
proof of concept, we describe a well-known music identification method, and
implement this system in the form of a neural net. We then attack this system
using simple gradient methods. Adversarial music created this way successfully
fools industrial systems, including the AudioTag copyright detector and
YouTube's Content ID system. Our goal is to raise awareness of the threats
posed by adversarial examples in this space, and to highlight the importance of
hardening copyright detection systems to attacks.


http://arxiv.org/abs/1906.06919
Improving Black-box Adversarial Attacks with a Transfer-based Prior.
Shuyu Cheng; Yinpeng Dong; Tianyu Pang; Hang Su; Jun Zhu
  We consider the black-box adversarial setting, where the adversary has to
generate adversarial perturbations without access to the target models to
compute gradients. Previous methods tried to approximate the gradient either by
using a transfer gradient of a surrogate white-box model, or based on the query
feedback. However, these methods often suffer from low attack success rates or
poor query efficiency since it is non-trivial to estimate the gradient in a
high-dimensional space with limited information. To address these problems, we
propose a prior-guided random gradient-free (P-RGF) method to improve black-box
adversarial attacks, which takes the advantage of a transfer-based prior and
the query information simultaneously. The transfer-based prior given by the
gradient of a surrogate model is appropriately integrated into our algorithm by
an optimal coefficient derived by a theoretical analysis. Extensive experiments
demonstrate that our method requires much fewer queries to attack black-box
models with higher success rates compared with the alternative state-of-the-art
methods.


http://arxiv.org/abs/1906.07077
The Attack Generator: A Systematic Approach Towards Constructing Adversarial Attacks.
Felix Assion; Peter Schlicht; Florens Greßner; Wiebke Günther; Fabian Hüger; Nico Schmidt; Umair Rasheed
  Most state-of-the-art machine learning (ML) classification systems are
vulnerable to adversarial perturbations. As a consequence, adversarial
robustness poses a significant challenge for the deployment of ML-based systems
in safety- and security-critical environments like autonomous driving, disease
detection or unmanned aerial vehicles. In the past years we have seen an
impressive amount of publications presenting more and more new adversarial
attacks. However, the attack research seems to be rather unstructured and new
attacks often appear to be random selections from the unlimited set of possible
adversarial attacks. With this publication, we present a structured analysis of
the adversarial attack creation process. By detecting different building blocks
of adversarial attacks, we outline the road to new sets of adversarial attacks.
We call this the "attack generator". In the pursuit of this objective, we
summarize and extend existing adversarial perturbation taxonomies. The
resulting taxonomy is then linked to the application context of computer vision
systems for autonomous vehicles, i.e. semantic segmentation and object
detection. Finally, in order to prove the usefulness of the attack generator,
we investigate existing semantic segmentation attacks with respect to the
detected defining components of adversarial attacks.


http://arxiv.org/abs/1906.06784
Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Accuracy.
Alex Lamb; Vikas Verma; Juho Kannala; Yoshua Bengio
  Adversarial robustness has become a central goal in deep learning, both in
theory and practice. However, successful methods to improve adversarial
robustness (such as adversarial training) greatly hurt generalization
performance on the clean data. This could have a major impact on how
adversarial robustness affects real world systems (i.e. many may opt to forego
robustness if it can improve performance on the clean data). We propose
Interpolated Adversarial Training, which employs recently proposed
interpolation based training methods in the framework of adversarial training.
On CIFAR-10, adversarial training increases clean test error from 5.8% to
16.7%, whereas with our Interpolated adversarial training we retain adversarial
robustness while achieving a clean test error of only 6.5%. With our technique,
the relative error increase for the robust model is reduced from 187.9% to just
12.1%


http://arxiv.org/abs/1906.06765
Defending Against Adversarial Attacks Using Random Forests.
Yifan Ding; Liqiang Wang; Huan Zhang; Jinfeng Yi; Deliang Fan; Boqing Gong
  As deep neural networks (DNNs) have become increasingly important and
popular, the robustness of DNNs is the key to the safety of both the Internet
and the physical world. Unfortunately, some recent studies show that
adversarial examples, which are hard to be distinguished from real examples,
can easily fool DNNs and manipulate their predictions. Upon observing that
adversarial examples are mostly generated by gradient-based methods, in this
paper, we first propose to use a simple yet very effective non-differentiable
hybrid model that combines DNNs and random forests, rather than hide gradients
from attackers, to defend against the attacks. Our experiments show that our
model can successfully and completely defend the white-box attacks, has a lower
transferability, and is quite resistant to three representative types of
black-box attacks; while at the same time, our model achieves similar
classification accuracy as the original DNNs. Finally, we investigate and
suggest a criterion to define where to grow random forests in DNNs.


http://arxiv.org/abs/1906.06627
Uncovering Why Deep Neural Networks Lack Robustness: Representation Metrics that Link to Adversarial Attacks.
Danilo Vasconcellos Vargas; Shashank Kotyan; Moe Matsuki
  Neural networks have been shown vulnerable to adversarial samples. Slightly
perturbed input images are able to change the classification of accurate
models, showing that the representation learned is not as good as previously
thought.To aid the development of better neural networks, it would be important
to evaluate to what extent are current neural networks' representations
capturing the existing features.Here we propose a test that can evaluate neural
networks using a new type of zero-shot test, entitled Raw Zero-Shot.This test
is based on the principle that some features are present on unknown classes and
that unknown classes can be defined as a combination of previous learned
features without learning bias. To evaluate the soft-labels of unknown classes,
two metrics are proposed.One is based on clustering validation techniques
(Davies-Bouldin Index) and the other is based on soft-label distance of a given
correct soft-label.Experiments show that such metrics are in accordance with
the robustness to adversarial attacks and might serve as a guidance to build
better models as well as be used in loss functions to improve the models
directly.Interestingly, the results suggests that dynamic routing networks such
as CapsNet have better representation while some DNNs might be trading off
representation quality for accuracy.
  Code available at \url{http://bit.ly/RepresentationMetrics}.


http://arxiv.org/abs/1906.06032
Adversarial Training Can Hurt Generalization.
Aditi Raghunathan; Sang Michael Xie; Fanny Yang; John C. Duchi; Percy Liang
  While adversarial training can improve robust accuracy (against an
adversary), it sometimes hurts standard accuracy (when there is no adversary).
Previous work has studied this tradeoff between standard and robust accuracy,
but only in the setting where no predictor performs well on both objectives in
the infinite data limit. In this paper, we show that even when the optimal
predictor with infinite data performs well on both objectives, a tradeoff can
still manifest itself with finite data. Furthermore, since our construction is
based on a convex learning problem, we rule out optimization concerns, thus
laying bare a fundamental tension between robustness and generalization.
Finally, we show that robust self-training mostly eliminates this tradeoff by
leveraging unlabeled data.


http://arxiv.org/abs/1906.06110
Towards Compact and Robust Deep Neural Networks.
Vikash Sehwag; Shiqi Wang; Prateek Mittal; Suman Jana
  Deep neural networks have achieved impressive performance in many
applications but their large number of parameters lead to significant
computational and storage overheads. Several recent works attempt to mitigate
these overheads by designing compact networks using pruning of connections.
However, we observe that most of the existing strategies to design compact
networks fail to preserve network robustness against adversarial examples. In
this work, we rigorously study the extension of network pruning strategies to
preserve both benign accuracy and robustness of a network. Starting with a
formal definition of the pruning procedure, including pre-training, weights
pruning, and fine-tuning, we propose a new pruning method that can create
compact networks while preserving both benign accuracy and robustness. Our
method is based on two main insights: (1) we ensure that the training
objectives of the pre-training and fine-tuning steps match the training
objective of the desired robust model (e.g., adversarial robustness/verifiable
robustness), and (2) we keep the pruning strategy agnostic to pre-training and
fine-tuning objectives. We evaluate our method on four different networks on
the CIFAR-10 dataset and measure benign accuracy, empirical robust accuracy,
and verifiable robust accuracy. We demonstrate that our pruning method can
preserve on average 93\% benign accuracy, 92.5\% empirical robust accuracy, and
85.0\% verifiable robust accuracy while compressing the tested network by
10$\times$.


http://arxiv.org/abs/1906.06355
Perceptual Based Adversarial Audio Attacks.
Joseph Szurley; J. Zico Kolter
  Recent work has shown the possibility of adversarial attacks on automatic
speechrecognition (ASR) systems. However, in the vast majority of work in this
area, theattacks have been executed only in the digital space, or have involved
short phrasesand static room settings. In this paper, we demonstrate a
physically realizableaudio adversarial attack. We base our approach
specifically on a psychoacoustic-property-based loss function, and automated
generation of room impulse responses, to create adversarial attacks that are
robust when played over a speaker in multiple environments. We show that such
attacks are possible even while being virtually imperceptible to listeners.


http://arxiv.org/abs/1906.06086
Copy and Paste: A Simple But Effective Initialization Method for Black-Box Adversarial Attacks.
Thomas Brunner; Frederik Diehl; Alois Knoll
  Many optimization methods for generating black-box adversarial examples have
been proposed, but the aspect of initializing said optimizers has not been
considered in much detail. We show that the choice of starting points is indeed
crucial, and that the performance of state-of-the-art attacks depends on it.
First, we discuss desirable properties of starting points for attacking image
classifiers, and how they can be chosen to increase query efficiency. Notably,
we find that simply copying small patches from other images is a valid
strategy. In an evaluation on ImageNet, we show that this initialization
reduces the number of queries required for a state-of-the-art Boundary Attack
by 81%, significantly outperforming previous results reported for targeted
black-box adversarial examples.


http://arxiv.org/abs/1906.06449
Robust or Private? Adversarial Training Makes Models More Vulnerable to Privacy Attacks.
Felipe A. Mejia; Paul Gamble; Zigfried Hampel-Arias; Michael Lomnitz; Nina Lopatina; Lucas Tindall; Maria Alejandra Barrios
  Adversarial training was introduced as a way to improve the robustness of
deep learning models to adversarial attacks. This training method improves
robustness against adversarial attacks, but increases the models vulnerability
to privacy attacks. In this work we demonstrate how model inversion attacks,
extracting training data directly from the model, previously thought to be
intractable become feasible when attacking a robustly trained model. The input
space for a traditionally trained model is dominated by adversarial examples -
data points that strongly activate a certain class but lack semantic meaning -
this makes it difficult to successfully conduct model inversion attacks. We
demonstrate this effect using the CIFAR-10 dataset under three different model
inversion attacks, a vanilla gradient descent method, gradient based method at
different scales, and a generative adversarial network base attacks.


http://arxiv.org/abs/1906.06316
Towards Stable and Efficient Training of Verifiably Robust Neural Networks.
Huan Zhang; Hongge Chen; Chaowei Xiao; Bo Li; Duane Boning; Cho-Jui Hsieh
  Training neural networks with verifiable robustness guarantees is
challenging. Several existing successful approaches utilize relatively tight
linear relaxation based bounds of neural network outputs, but they can slow
down training by a factor of hundreds and over-regularize the network.
Meanwhile, interval bound propagation (IBP) based training is efficient and
significantly outperform linear relaxation based methods on some tasks, yet it
suffers from stability issues since the bounds are much looser. In this paper,
we first interpret IBP training as training an augmented network which computes
non-linear bounds, thus explaining its good performance. We then propose a new
certified adversarial training method, CROWN-IBP, by combining the fast IBP
bounds in the forward pass and a tight linear relaxation based bound, CROWN, in
the backward pass. The proposed method is computationally efficient and
consistently outperforms IBP baselines on training verifiably robust neural
networks. We conduct large scale experiments using 53 models on MNIST,
Fashion-MNIST and CIFAR datasets. On MNIST with $\epsilon=0.3$ and
$\epsilon=0.4$ ($\ell_\infty$ norm distortion) we achieve 7.46\% and 12.96\%
verified error on test set, respectively, outperforming previous certified
defense methods.


http://arxiv.org/abs/1906.06026
Model Agnostic Dual Quality Assessment for Adversarial Machine Learning and an Analysis of Current Neural Networks and Defenses.
Danilo Vasconcellos Vargas; Shashank Kotyan
  In adversarial machine learning, there are a huge number of attacks of
various types which makes the evaluation of robustness for new models and
defenses a daunting task. To make matters worse, there is an inherent bias in
attacks and defenses. Here, we organize the problems faced (model dependence,
insufficient evaluation, unreliable adversarial samples and perturbation
dependent results) and propose a dual quality assessment method together with
the concept of robustness levels to tackle them. We validate the dual quality
assessment on state-of-the-art models (WideResNet, ResNet, AllConv, DenseNet,
NIN, LeNet and CapsNet) as well as the current hardest defenses proposed at
ICLR 2018 as well as the widely known adversarial training, showing that
current models and defenses are vulnerable in all levels of robustness.
Moreover, we show that robustness to $L_0$ and $L_\infty$ attacks differ
greatly and therefore duality should be taken into account for a correct
assessment. Interestingly, a by-product of the assessment proposed is a novel
$L_\infty$ black-box method which requires even less perturbation than the
One-Pixel Attack (only 12\% of One-Pixel Attack's amount of perturbation) to
achieve similar results. Thus, this paper elucidates the problems of robustness
evaluation, proposes a dual quality assessment to tackle them as well as
analyze the robustness of current models and defenses. Hopefully, the current
analysis and proposed methods would aid the development of more robust deep
neural networks and hybrids alike.
  Code available at: http://bit.ly/DualQualityAssessment


http://arxiv.org/abs/1906.05599
A Computationally Efficient Method for Defending Adversarial Deep Learning Attacks.
Rajeev Sahay; Rehana Mahfuz; Aly El Gamal
  The reliance on deep learning algorithms has grown significantly in recent
years. Yet, these models are highly vulnerable to adversarial attacks, which
introduce visually imperceptible perturbations into testing data to induce
misclassifications. The literature has proposed several methods to combat such
adversarial attacks, but each method either fails at high perturbation values,
requires excessive computing power, or both. This letter proposes a
computationally efficient method for defending the Fast Gradient Sign (FGS)
adversarial attack by simultaneously denoising and compressing data.
Specifically, our proposed defense relies on training a fully connected
multi-layer Denoising Autoencoder (DAE) and using its encoder as a defense
against the adversarial attack. Our results show that using this dimensionality
reduction scheme is not only highly effective in mitigating the effect of the
FGS attack in multiple threat models, but it also provides a 2.43x speedup in
comparison to defense strategies providing similar robustness against the same
attack.


http://arxiv.org/abs/1906.05815
Lower Bounds for Adversarially Robust PAC Learning.
Dimitrios I. Diochnos; Saeed Mahloujifar; Mohammad Mahmoody
  In this work, we initiate a formal study of probably approximately correct
(PAC) learning under evasion attacks, where the adversary's goal is to
\emph{misclassify} the adversarially perturbed sample point $\widetilde{x}$,
i.e., $h(\widetilde{x})\neq c(\widetilde{x})$, where $c$ is the ground truth
concept and $h$ is the learned hypothesis. Previous work on PAC learning of
adversarial examples have all modeled adversarial examples as corrupted inputs
in which the goal of the adversary is to achieve $h(\widetilde{x}) \neq c(x)$,
where $x$ is the original untampered instance. These two definitions of
adversarial risk coincide for many natural distributions, such as images, but
are incomparable in general.
  We first prove that for many theoretically natural input spaces of high
dimension $n$ (e.g., isotropic Gaussian in dimension $n$ under $\ell_2$
perturbations), if the adversary is allowed to apply up to a sublinear
$o(||x||)$ amount of perturbations on the test instances, PAC learning requires
sample complexity that is exponential in $n$. This is in contrast with results
proved using the corrupted-input framework, in which the sample complexity of
robust learning is only polynomially more.
  We then formalize hybrid attacks in which the evasion attack is preceded by a
poisoning attack. This is perhaps reminiscent of "trapdoor attacks" in which a
poisoning phase is involved as well, but the evasion phase here uses the
error-region definition of risk that aims at misclassifying the perturbed
instances. In this case, we show PAC learning is sometimes impossible all
together, even when it is possible without the attack (e.g., due to the bounded
VC dimension).


http://arxiv.org/abs/1906.04606
Mimic and Fool: A Task Agnostic Adversarial Attack.
Akshay Chaturvedi; Utpal Garain
  At present, adversarial attacks are designed in a task-specific fashion.
However, for downstream computer vision tasks such as image captioning, image
segmentation etc., the current deep learning systems use an image classifier
like VGG16, ResNet50, Inception-v3 etc. as a feature extractor. Keeping this in
mind, we propose Mimic and Fool, a task agnostic adversarial attack. Given a
feature extractor, the proposed attack finds an adversarial image which can
mimic the image feature of the original image. This ensures that the two images
give the same (or similar) output regardless of the task. We randomly select
1000 MSCOCO validation images for experimentation. We perform experiments on
two image captioning models, Show and Tell, Show Attend and Tell and one VQA
model, namely, end-to-end neural module network (N2NMN). The proposed attack
achieves success rate of 74.0%, 81.0% and 89.6% for Show and Tell, Show Attend
and Tell and N2NMN respectively. We also propose a slight modification to our
attack to generate natural-looking adversarial images. In addition, it is shown
that the proposed attack also works for invertible architecture. Since Mimic
and Fool only requires information about the feature extractor of the model, it
can be considered as a gray-box attack.


http://arxiv.org/abs/1906.04392
Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks.
Ziang Yan; Yiwen Guo; Changshui Zhang
  Unlike the white-box counterparts that are widely studied and readily
accessible, adversarial examples in black-box settings are generally more
Herculean on account of the difficulty of estimating gradients. Many methods
achieve the task by issuing numerous queries to target classification systems,
which makes the whole procedure costly and suspicious to the systems. In this
paper, we aim at reducing the query complexity of black-box attacks in this
category. We propose to exploit gradients of a few reference models which
arguably span some promising search subspaces. Experimental results show that,
in comparison with the state-of-the-arts, our method can gain up to 2x and 4x
reductions in the requisite mean and medium numbers of queries with much lower
failure rates even if the reference models are trained on a small and
inadequate dataset disjoint to the one for training the victim model. Code and
models for reproducing our results will be made publicly available.


http://arxiv.org/abs/1906.04893
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks.
Mahyar Fazlyab; Alexander Robey; Hamed Hassani; Manfred Morari; George J. Pappas
  Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is
useful in many applications ranging from robustness certification of
classifiers to stability analysis of closed-loop systems with reinforcement
learning controllers. Existing methods in the literature for estimating the
Lipschitz constant suffer from either lack of accuracy or poor scalability. In
this paper, we present a convex optimization framework to compute guaranteed
upper bounds on the Lipschitz constant of DNNs both accurately and efficiently.
Our main idea is to interpret activation functions as gradients of convex
potential functions. Hence, they satisfy certain properties that can be
described by quadratic constraints. This particular description allows us to
pose the Lipschitz constant estimation problem as a semidefinite program (SDP).
The resulting SDP can be adapted to increase either the estimation accuracy (by
capturing the interaction between activation functions of different layers) or
scalability (by decomposition and parallel implementation). We illustrate the
utility of our approach with a variety of experiments on randomly generated
networks and on classifiers trained on the MNIST and Iris datasets. In
particular, we experimentally demonstrate that our Lipschitz bounds are the
most accurate compared to those in the literature. We also study the impact of
adversarial training methods on the Lipschitz bounds of the resulting
classifiers and show that our bounds can be used to efficiently provide
robustness guarantees.


http://arxiv.org/abs/1906.04214
Topology Attack and Defense for Graph Neural Networks: An Optimization Perspective.
Kaidi Xu; Hongge Chen; Sijia Liu; Pin-Yu Chen; Tsui-Wei Weng; Mingyi Hong; Xue Lin
  Graph neural networks (GNNs) which apply the deep neural networks to graph
data have achieved significant performance for the task of semi-supervised node
classification. However, only few work has addressed the adversarial robustness
of GNNs. In this paper, we first present a novel gradient-based attack method
that facilitates the difficulty of tackling discrete graph data. When comparing
to current adversarial attacks on GNNs, the results show that by only
perturbing a small number of edge perturbations, including addition and
deletion, our optimization-based attack can lead to a noticeable decrease in
classification performance. Moreover, leveraging our gradient-based attack, we
propose the first optimization-based adversarial training for GNNs. Our method
yields higher robustness against both different gradient based and greedy
attack methods without sacrificing classification accuracy on original graph.


http://arxiv.org/abs/1906.03973
E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles.
Markus Kettunen; Erik Härkönen; Jaakko Lehtinen
  It has been recently shown that the hidden variables of convolutional neural
networks make for an efficient perceptual similarity metric that accurately
predicts human judgment on relative image similarity assessment. First, we show
that such learned perceptual similarity metrics (LPIPS) are susceptible to
adversarial attacks that dramatically contradict human visual similarity
judgment. While this is not surprising in light of neural networks' well-known
weakness to adversarial perturbations, we proceed to show that self-ensembling
with an infinite family of random transformations of the input --- a technique
known not to render classification networks robust --- is enough to turn the
metric robust against attack, while retaining predictive power on human
judgments. Finally, we study the geometry imposed by our our novel
self-ensembled metric (E-LPIPS) on the space of natural images. We find
evidence of "perceptual convexity" by showing that convex combinations of
similar-looking images retain appearance, and that discrete geodesics yield
meaningful frame interpolation and texture morphing, all without explicit
correspondences.


http://arxiv.org/abs/1906.03972
Evaluating the Robustness of Nearest Neighbor Classifiers: A Primal-Dual Perspective.
Lu Wang; Xuanqing Liu; Jinfeng Yi; Zhi-Hua Zhou; Cho-Jui Hsieh
  We study the problem of computing the minimum adversarial perturbation of the
Nearest Neighbor (NN) classifiers. Previous attempts either conduct attacks on
continuous approximations of NN models or search for the perturbation by some
heuristic methods. In this paper, we propose the first algorithm that is able
to compute the minimum adversarial perturbation. The main idea is to formulate
the problem as a list of convex quadratic programming (QP) problems that can be
efficiently solved by the proposed algorithms for 1-NN models. Furthermore, we
show that dual solutions for these QP problems could give us a valid lower
bound of the adversarial perturbation that can be used for formal robustness
verification, giving us a nice view of attack/verification for NN models. For
$K$-NN models with larger $K$, we show that the same formulation can help us
efficiently compute the upper and lower bounds of the minimum adversarial
perturbation, which can be used for attack and verification.


http://arxiv.org/abs/1906.03849
Robustness Verification of Tree-based Models.
Hongge Chen; Huan Zhang; Si Si; Yang Li; Duane Boning; Cho-Jui Hsieh
  We study the robustness verification problem for tree-based models, including
decision trees, random forests (RFs) and gradient boosted decision trees
(GBDTs). Formal robustness verification of decision tree ensembles involves
finding the exact minimal adversarial perturbation or a guaranteed lower bound
of it. Existing approaches find the minimal adversarial perturbation by a mixed
integer linear programming (MILP) problem, which takes exponential time so is
impractical for large ensembles. Although this verification problem is
NP-complete in general, we give a more precise complexity characterization. We
show that there is a simple linear time algorithm for verifying a single tree,
and for tree ensembles, the verification problem can be cast as a max-clique
problem on a multi-partite graph with bounded boxicity. For low dimensional
problems when boxicity can be viewed as constant, this reformulation leads to a
polynomial time algorithm. For general problems, by exploiting the boxicity of
the graph, we develop an efficient multi-level verification algorithm that can
give tight lower bounds on the robustness of decision tree ensembles, while
allowing iterative improvement and any-time termination. OnRF/GBDT models
trained on 10 datasets, our algorithm is hundreds of times faster than the
previous approach that requires solving MILPs, and is able to give tight
robustness verification bounds on large GBDTs with hundreds of deep trees.


http://arxiv.org/abs/1906.04584
Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers.
Hadi Salman; Greg Yang; Jerry Li; Pengchuan Zhang; Huan Zhang; Ilya Razenshteyn; Sebastien Bubeck
  Recent works have shown the effectiveness of randomized smoothing as a
scalable technique for building neural network-based classifiers that are
provably robust to $\ell_2$-norm adversarial perturbations. In this paper, we
employ adversarial training to improve the performance of randomized smoothing.
We design an adapted attack for smoothed classifiers, and we show how this
attack can be used in an adversarial training setting to boost the provable
robustness of smoothed classifiers. We demonstrate through extensive
experimentation that our method consistently outperforms all existing provably
$\ell_2$-robust classifiers by a significant margin on ImageNet and CIFAR-10,
establishing the state-of-the-art for provable $\ell_2$-defenses. Our code and
trained models are available at
http://github.com/Hadisalman/smoothing-adversarial .


http://arxiv.org/abs/1906.03787
Intriguing properties of adversarial training.
Cihang Xie; Alan Yuille
  Adversarial training is one of the main defenses against adversarial attacks.
In this paper, we provide the first rigorous study on diagnosing elements of
adversarial training, which reveals two intriguing properties.
  First, we study the role of normalization. Batch normalization (BN) is a
crucial element for achieving state-of-the-art performance on many vision
tasks, but we show it may prevent networks from obtaining strong robustness in
adversarial training. One unexpected observation is that, for models trained
with BN, simply removing clean images from training data largely boosts
adversarial robustness, i.e., 18.3%. We relate this phenomenon to the
hypothesis that clean images and adversarial images are drawn from two
different domains. This two-domain hypothesis may explain the issue of BN when
training with a mixture of clean and adversarial images, as estimating
normalization statistics of this mixture distribution is challenging. Guided by
this two-domain hypothesis, we show disentangling the mixture distribution for
normalization, i.e., applying separate BNs to clean and adversarial images for
statistics estimation, achieves much stronger robustness. Additionally, we find
that enforcing BNs to behave consistently at training and testing can further
enhance robustness.
  Second, we study the role of network capacity. We find our so-called "deep"
networks are still shallow for the task of adversarial learning. Unlike
traditional classification tasks where accuracy is only marginally improved by
adding more layers to "deep" networks (e.g., ResNet-152), adversarial training
exhibits a much stronger demand on deeper networks to achieve higher
adversarial robustness. This robustness improvement can be observed
substantially and consistently even by pushing the network capacity to an
unprecedented scale, i.e., ResNet-638.


http://arxiv.org/abs/1906.03750
Attacking Graph Convolutional Networks via Rewiring.
Yao Ma; Suhang Wang; Lingfei Wu; Jiliang Tang
  Graph Neural Networks (GNNs) have boosted the performance of many graph
related tasks such as node classification and graph classification. Recent
researches show that graph neural networks are vulnerable to adversarial
attacks, which deliberately add carefully created unnoticeable perturbation to
the graph structure. The perturbation is usually created by adding/deleting a
few edges, which might be noticeable even when the number of edges modified is
small. In this paper, we propose a graph rewiring operation which affects the
graph in a less noticeable way compared to adding/deleting edges. We then use
reinforcement learning to learn the attack strategy based on the proposed
rewiring operation. Experiments on real world graphs demonstrate the
effectiveness of the proposed framework. To understand the proposed framework,
we further analyze how its generated perturbation to the graph structure
affects the output of the target model.


http://arxiv.org/abs/1906.03749
Improved Adversarial Robustness via Logit Regularization Methods.
Cecilia Summers; Michael J. Dinneen
  While great progress has been made at making neural networks effective across
a wide range of visual tasks, most models are surprisingly vulnerable. This
frailness takes the form of small, carefully chosen perturbations of their
input, known as adversarial examples, which represent a security threat for
learned vision models in the wild -- a threat which should be responsibly
defended against in safety-critical applications of computer vision. In this
paper, we advocate for and experimentally investigate the use of a family of
logit regularization techniques as an adversarial defense, which can be used in
conjunction with other methods for creating adversarial robustness at little to
no marginal cost. We also demonstrate that much of the effectiveness of one
recent adversarial defense mechanism can in fact be attributed to logit
regularization, and show how to improve its defense against both white-box and
black-box attacks, in the process creating a stronger black-box attack against
PGD-based models. We validate our methods on three datasets and include results
on both gradient-free attacks and strong gradient-based iterative attacks with
as many as 1,000 steps.


http://arxiv.org/abs/1906.03612
On the Vulnerability of Capsule Networks to Adversarial Attacks.
Felix Michels; Tobias Uelwer; Eric Upschulte; Stefan Harmeling
  This paper extensively evaluates the vulnerability of capsule networks to
different adversarial attacks. Recent work suggests that these architectures
are more robust towards adversarial attacks than other neural networks.
However, our experiments show that capsule networks can be fooled as easily as
convolutional neural networks.


http://arxiv.org/abs/1906.03563
Beyond Adversarial Training: Min-Max Optimization in Adversarial Attack and Defense.
Jingkang Wang; Tianyun Zhang; Sijia Liu; Pin-Yu Chen; Jiacen Xu; Makan Fardad; Bo Li
  The worst-case training principle that minimizes the maximal adversarial
loss, also known as adversarial training (AT), has shown to be a
state-of-the-art approach for enhancing adversarial robustness against
norm-ball bounded input perturbations. Nonetheless, min-max optimization beyond
the purpose of AT has not been rigorously explored in the research of
adversarial attack and defense. In particular, given a set of risk sources
(domains), minimizing the maximal loss induced from the domain set can be
reformulated as a general min-max problem that is different from AT, since the
maximization is taken over the probability simplex of the domain set. Examples
of this general formulation include attacking model ensembles, devising
universal perturbation to input samples or data transformations, and
generalized AT over multiple norm-ball threat models. We show that these
problems can be solved under a unified and theoretically principled min-max
optimization framework. Our proposed approach leads to substantial performance
improvement over the uniform averaging strategy in four different tasks.
Moreover, we show how the self-adjusted weighting factors of the probability
simplex from our proposed algorithms can be used to explain the importance of
different attack and defense models.


http://arxiv.org/abs/1906.03526
Provably Robust Boosted Decision Stumps and Trees against Adversarial Attacks.
Maksym Andriushchenko; Matthias Hein
  The problem of adversarial samples has been studied extensively for neural
networks. However, for boosting, in particular boosted decision trees and
decision stumps there are almost no results, even though boosted decision
trees, as e.g. XGBoost, are quite popular due to their interpretability and
good prediction performance. We show in this paper that for boosted decision
stumps the exact min-max optimal robust loss and test error for an
$l_\infty$-attack can be computed in $O(n\,T\log T)$, where $T$ is the number
of decision stumps and $n$ the number of data points, as well as an optimal
update of the ensemble in $O(n^2\,T\log T)$. While not exact, we show how to
optimize an upper bound on the robust loss for boosted trees. Up to our
knowledge, these are the first algorithms directly optimizing provable
robustness guarantees in the area of boosting. We make the code of all our
experiments publicly available at
https://github.com/max-andr/provably-robust-boosting


http://arxiv.org/abs/1906.03499
ML-LOO: Detecting Adversarial Examples with Feature Attribution.
Puyudi Yang; Jianbo Chen; Cho-Jui Hsieh; Jane-Ling Wang; Michael I. Jordan
  Deep neural networks obtain state-of-the-art performance on a series of
tasks. However, they are easily fooled by adding a small adversarial
perturbation to input. The perturbation is often human imperceptible on image
data. We observe a significant difference in feature attributions of
adversarially crafted examples from those of original ones. Based on this
observation, we introduce a new framework to detect adversarial examples
through thresholding a scale estimate of feature attribution scores.
Furthermore, we extend our method to include multi-layer feature attributions
in order to tackle the attacks with mixed confidence levels. Through vast
experiments, our method achieves superior performances in distinguishing
adversarial examples from popular attack methods on a variety of real data sets
among state-of-the-art detection methods. In particular, our method is able to
detect adversarial examples of mixed confidence levels, and transfer between
different attacking methods.


http://arxiv.org/abs/1906.03466
Strategies to architect AI Safety: Defense to guard AI from Adversaries.
Rajagopal. A; Nirmala. V
  The impact of designing for security of AI is critical for humanity in the AI
era. With humans increasingly becoming dependent upon AI, there is a need for
neural networks that work reliably, inspite of Adversarial attacks. The vision
for Safe and secure AI for popular use is achievable. To achieve safety of AI,
this paper explores strategies and a novel deep learning architecture. To guard
AI from adversaries, paper explores combination of 3 strategies:
  1. Introduce randomness at inference time to hide the representation learning
from adversaries.
  2. Detect presence of adversaries by analyzing the sequence of inferences.
  3. Exploit visual similarity.
  To realize these strategies, this paper designs a novel architecture, Dynamic
Neural Defense, DND. This defense has 3 deep learning architectural features:
  1. By hiding the way a neural network learns from exploratory attacks using a
random computation graph, DND evades attack.
  2. By analyzing input sequence to cloud AI inference engine with LSTM, DND
detects attack sequence.
  3. By inferring with visual similar inputs generated by VAE, any AI defended
by DND approach does not succumb to hackers.
  Thus, a roadmap to develop reliable, safe and secure AI is presented.


http://arxiv.org/abs/1906.03455
Sensitivity of Deep Convolutional Networks to Gabor Noise.
Kenneth T. Co; Luis Muñoz-González; Emil C. Lupu
  Deep Convolutional Networks (DCNs) have been shown to be sensitive to
Universal Adversarial Perturbations (UAPs): input-agnostic perturbations that
fool a model on large portions of a dataset. These UAPs exhibit interesting
visual patterns, but this phenomena is, as yet, poorly understood. Our work
shows that visually similar procedural noise patterns also act as UAPs. In
particular, we demonstrate that different DCN architectures are sensitive to
Gabor noise patterns. This behaviour, its causes, and implications deserve
further in-depth study.


http://arxiv.org/abs/1906.03444
Defending against Adversarial Attacks through Resilient Feature Regeneration.
Tejas Borkar; Felix Heide; Lina Karam
  Deep neural network (DNN) predictions have been shown to be vulnerable to
carefully crafted adversarial perturbations. Specifically, so-called universal
adversarial perturbations are image-agnostic perturbations that can be added to
any image and can fool a target network into making erroneous predictions.
Departing from existing adversarial defense strategies, which work in the image
domain, we present a novel defense which operates in the DNN feature domain and
effectively defends against such universal adversarial attacks. Our approach
identifies pre-trained convolutional features that are most vulnerable to
adversarial noise and deploys defender units which transform (regenerate) these
DNN filter activations into noise-resilient features, guarding against unseen
adversarial perturbations. The proposed defender units are trained using a
target loss on synthetic adversarial perturbations, which we generate with a
novel efficient synthesis method. We validate the proposed method for different
DNN architectures, and demonstrate that it outperforms existing defense
strategies across network architectures by more than 10% in restored accuracy.
Moreover, we demonstrate that the approach also improves resilience of DNNs to
other unseen adversarial attacks.


http://arxiv.org/abs/1906.03397
Making targeted black-box evasion attacks effective and efficient.
Mika Juuti; Buse Gul Atli; N. Asokan
  We investigate how an adversary can optimally use its query budget for
targeted evasion attacks against deep neural networks in a black-box setting.
We formalize the problem setting and systematically evaluate what benefits the
adversary can gain by using substitute models. We show that there is an
exploration-exploitation tradeoff in that query efficiency comes at the cost of
effectiveness. We present two new attack strategies for using substitute models
and show that they are as effective as previous query-only techniques but
require significantly fewer queries, by up to three orders of magnitude. We
also show that an agile adversary capable of switching through different attack
techniques can achieve pareto-optimal efficiency. We demonstrate our attack
against Google Cloud Vision showing that the difficulty of black-box attacks
against real-world prediction APIs is significantly easier than previously
thought (requiring approximately 500 queries instead of approximately 20,000 as
in previous works).


http://arxiv.org/abs/1906.03367
Using learned optimizers to make models robust to input noise.
Luke Metz; Niru Maheswaranathan; Jonathon Shlens; Jascha Sohl-Dickstein; Ekin D. Cubuk
  State-of-the art vision models can achieve superhuman performance on image
classification tasks when testing and training data come from the same
distribution. However, when models are tested on corrupted images (e.g. due to
scale changes, translations, or shifts in brightness or contrast), performance
degrades significantly. Here, we explore the possibility of meta-training a
learned optimizer that can train image classification models such that they are
robust to common image corruptions. Specifically, we are interested training
models that are more robust to noise distributions not present in the training
data. We find that a learned optimizer meta-trained to produce models which are
robust to Gaussian noise trains models that are more robust to Gaussian noise
at other scales compared to traditional optimizers like Adam. The effect of
meta-training is more complicated when targeting a more general set of noise
distributions, but led to improved performance on half of held-out corruption
tasks. Our results suggest that meta-learning provides a novel approach for
studying and improving the robustness of deep learning models.


http://arxiv.org/abs/1906.03333
Efficient Project Gradient Descent for Ensemble Adversarial Attack.
Fanyou Wu; Rado Gazo; Eva Haviarova; Bedrich Benes
  Recent advances show that deep neural networks are not robust to deliberately
crafted adversarial examples which many are generated by adding human
imperceptible perturbation to clear input. Consider $l_2$ norms attacks,
Project Gradient Descent (PGD) and the Carlini and Wagner (C\&W) attacks are
the two main methods, where PGD control max perturbation for adversarial
examples while C\&W approach treats perturbation as a regularization term
optimized it with loss function together. If we carefully set parameters for
any individual input, both methods become similar. In general, PGD attacks
perform faster but obtains larger perturbation to find adversarial examples
than the C\&W when fixing the parameters for all inputs. In this report, we
propose an efficient modified PGD method for attacking ensemble models by
automatically changing ensemble weights and step size per iteration per input.
This method generates smaller perturbation adversarial examples than PGD method
while remains efficient as compared to C\&W method. Our method won the first
place in IJCAI19 Targeted Adversarial Attack competition.


http://arxiv.org/abs/1906.03310
Adversarial Examples for Non-Parametric Methods: Attacks, Defenses and Large Sample Limits.
Yao-Yuan Yang; Cyrus Rashtchian; Yizhen Wang; Kamalika Chaudhuri
  Adversarial examples have received a great deal of recent attention because
of their potential to uncover security flaws in machine learning systems.
However, most prior work on adversarial examples has been on parametric
classifiers, for which generic attack and defense methods are known;
non-parametric methods have been only considered on an ad-hoc or
classifier-specific basis. In this work, we take a holistic look at adversarial
examples for non-parametric methods. We first provide a general region-based
attack that applies to a wide range of classifiers, including nearest
neighbors, decision trees, and random forests. Motivated by the close
connection between non-parametric methods and the Bayes Optimal classifier, we
next exhibit a robust analogue to the Bayes Optimal, and we use it to motivate
a novel and generic defense that we call adversarial pruning. We empirically
show that the region-based attack and adversarial pruning defense are either
better than or competitive with existing attacks and defenses for
non-parametric methods, while being considerably more generally applicable.


http://arxiv.org/abs/1906.03231
A cryptographic approach to black box adversarial machine learning.
Kevin Shi; Daniel Hsu; Allison Bishop
  We propose an ensemble technique for converting any classifier into a
computationally secure classifier. We define a simpler security problem for
random binary classifiers and prove a reduction from this model to the security
of the overall ensemble classifier. We provide experimental evidence of the
security of our random binary classifiers, as well as empirical results of the
adversarial accuracy of the overall ensemble to black-box attacks. Our
construction crucially leverages hidden randomness in the multiclass-to-binary
reduction.


http://arxiv.org/abs/1906.02931
Inductive Bias of Gradient Descent based Adversarial Training on Separable Data.
Yan Li; Ethan X. Fang; Huan Xu; Tuo Zhao
  Adversarial training is a principled approach for training robust neural
networks. Despite of tremendous successes in practice, its theoretical
properties still remain largely unexplored. In this paper, we provide new
theoretical insights of gradient descent based adversarial training by studying
its computational properties, specifically on its inductive bias. We take the
binary classification task on linearly separable data as an illustrative
example, where the loss asymptotically attains its infimum as the parameter
diverges to infinity along certain directions. Specifically, we show that when
the adversarial perturbation during training has bounded $\ell_2$-norm, the
classifier learned by gradient descent based adversarial training converges in
direction to the maximum $\ell_2$-norm margin classifier at the rate of
$\tilde{\mathcal{O}}(1/\sqrt{T})$, significantly faster than the rate
$\mathcal{O}(1/\log T)$ of training with clean data. In addition, when the
adversarial perturbation during training has bounded $\ell_q$-norm for some
$q\ge 1$, the resulting classifier converges in direction to a maximum
mixed-norm margin classifier, which has a natural interpretation of robustness,
as being the maximum $\ell_2$-norm margin classifier under worst-case
$\ell_q$-norm perturbation to the data. Our findings provide theoretical
backups for adversarial training that it indeed promotes robustness against
adversarial perturbation.


http://arxiv.org/abs/1906.02896
Reliable Classification Explanations via Adversarial Attacks on Robust Networks.
Walt Woods; Jack Chen; Christof Teuscher
  Neural Networks (NNs) have been found vulnerable to a class of imperceptible
attacks, called adversarial examples, which arbitrarily alter the output of the
network. These attacks have called the validity of NNs into question,
particularly on sensitive problems such as medical imaging or fraud detection.
We further argue that the fields of explainable AI and Human-In-The-Loop (HITL)
algorithms are impacted by adversarial attacks, as attacks result in
perturbations outside of the salient regions highlighted by state-of-the-art
techniques such as LIME or Grad-CAM. This work accomplishes three things which
greatly reduce the impact of adversarial examples, and pave the way for future
HITL workflows: we propose a novel regularization technique inspired by the
Lipschitz constraint which greatly improves an NN's resistance to adversarial
examples; we propose a collection of novel network and training changes to
complement the proposed regularization technique, including a Half-Huber
activation function and an integrator-based controller for regularization
strength; and we demonstrate that networks trained with this technique may be
deliberately attacked to generate rich explanations. Our techniques led to
networks more robust than the previous state of the art: using the
Accuracy-Robustness Area (ARA), our most robust ImageNet classification network
scored 42.2% top-1 accuracy on unmodified images and demonstrated an attack ARA
of 0.0053, an ARA 2.4x greater than the previous state-of-the-art at the same
level of accuracy on clean data, achieved with a network one-third the size. A
far-reaching benefit of this technique is its ability to intuitively
demonstrate decision boundaries to a human observer, allowing for improved
debugging of NN decisions, and providing a means for improving the underlying
model.


http://arxiv.org/abs/1906.09453
Computer Vision with a Single (Robust) Classifier.
Shibani Santurkar; Dimitris Tsipras; Brandon Tran; Andrew Ilyas; Logan Engstrom; Aleksander Madry
  We show that the basic classification framework alone can be used to tackle
some of the most challenging computer vision tasks. In contrast to other
state-of-the-art approaches, the toolkit we develop is rather minimal: it uses
a single, off-the-shelf classifier for all these tasks. The crux of our
approach is that we train this classifier to be adversarially robust. It turns
out that adversarial robustness is precisely what we need to directly
manipulate salient features of the input. Overall, our findings demonstrate the
utility of robustness in the broader machine learning context. Code and models
for our experiments can be found at https://git.io/robust-apps.


http://arxiv.org/abs/1906.02816
Robust Attacks against Multiple Classifiers.
Juan C. Perdomo; Yaron Singer
  We address the challenge of designing optimal adversarial noise algorithms
for settings where a learner has access to multiple classifiers. We demonstrate
how this problem can be framed as finding strategies at equilibrium in a
two-player, zero-sum game between a learner and an adversary. In doing so, we
illustrate the need for randomization in adversarial attacks. In order to
compute Nash equilibrium, our main technical focus is on the design of best
response oracles that can then be implemented within a Multiplicative Weights
Update framework to boost deterministic perturbations against a set of models
into optimal mixed strategies. We demonstrate the practical effectiveness of
our approach on a series of image classification tasks using both linear
classifiers and deep neural networks.


http://arxiv.org/abs/1906.02611
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation.
Raphael Gontijo Lopes; Dong Yin; Ben Poole; Justin Gilmer; Ekin D. Cubuk
  Deploying machine learning systems in the real world requires both high
accuracy on clean data and robustness to naturally occurring corruptions. While
architectural advances have led to improved accuracy, building robust models
remains challenging. Prior work has argued that there is an inherent trade-off
between robustness and accuracy, which is exemplified by standard data augment
techniques such as Cutout, which improves clean accuracy but not robustness,
and additive Gaussian noise, which improves robustness but hurts accuracy. To
overcome this trade-off, we introduce Patch Gaussian, a simple augmentation
scheme that adds noise to randomly selected patches in an input image. Models
trained with Patch Gaussian achieve state of the art on the CIFAR-10 and
ImageNetCommon Corruptions benchmarks while also improving accuracy on clean
data. We find that this augmentation leads to reduced sensitivity to high
frequency noise(similar to Gaussian) while retaining the ability to take
advantage of relevant high frequency information in the image (similar to
Cutout). Finally, we show that Patch Gaussian can be used in conjunction with
other regularization methods and data augmentation policies such as
AutoAugment, and improves performance on the COCO object detection benchmark.


http://arxiv.org/abs/1906.02494
Understanding Adversarial Behavior of DNNs by Disentangling Non-Robust and Robust Components in Performance Metric.
Yujun Shi; Benben Liao; Guangyong Chen; Yun Liu; Ming-Ming Cheng; Jiashi Feng
  The vulnerability to slight input perturbations is a worrying yet intriguing
property of deep neural networks (DNNs). Despite many previous works studying
the reason behind such adversarial behavior, the relationship between the
generalization performance and adversarial behavior of DNNs is still little
understood. In this work, we reveal such relation by introducing a metric
characterizing the generalization performance of a DNN. The metric can be
disentangled into an information-theoretic non-robust component, responsible
for adversarial behavior, and a robust component. Then, we show by experiments
that current DNNs rely heavily on optimizing the non-robust component in
achieving decent performance. We also demonstrate that current state-of-the-art
adversarial training algorithms indeed try to robustify the DNNs by preventing
them from using the non-robust component to distinguish samples from different
categories. Also, based on our findings, we take a step forward and point out
the possible direction for achieving decent standard performance and
adversarial robustness simultaneously. We believe that our theory could further
inspire the community to make more interesting discoveries about the
relationship between standard generalization and adversarial generalization of
deep learning models.


http://arxiv.org/abs/1906.02439
Should Adversarial Attacks Use Pixel p-Norm?.
Ayon Sen; Xiaojin Zhu; Liam Marshall; Robert Nowak
  Adversarial attacks aim to confound machine learning systems, while remaining
virtually imperceptible to humans. Attacks on image classification systems are
typically gauged in terms of $p$-norm distortions in the pixel feature space.
We perform a behavioral study, demonstrating that the pixel $p$-norm for any
$0\le p \le \infty$, and several alternative measures including earth mover's
distance, structural similarity index, and deep net embedding, do not fit human
perception. Our result has the potential to improve the understanding of
adversarial attack and defense strategies.


http://arxiv.org/abs/1906.02398
Query-efficient Meta Attack to Deep Neural Networks.
Jiawei Du; Hu Zhang; Joey Tianyi Zhou; Yi Yang; Jiashi Feng
  Recently, several adversarial attack methods to black-box deep neural
networks have been proposed and they serve as an excellent testing bed for
investigating safety issues with DNNs. These methods generally take in the
query and corresponding feedback from the targeted DNN model and infer suitable
attack patterns accordingly. However, due to lacking prior and inefficiency in
leveraging the query information, these methods are mostly query-intensive. In
this work, we propose a meta attack strategy which is capable of attacking the
target black-box model with much fewer queries. Its high query-efficiency comes
from prior abstraction on training a meta attacker which can speed up the
search for adversarial examples significantly. Extensive experiments on MNIST,
CIFAR10 and tiny-Imagenet demonstrate that, our meta-attack method can
remarkably reduce the number of model queries without sacrificing the attack
performance. Moreover, the obtained meta attacker is not restricted to a
particular model but can be reused easily with fast adaptive ability to attack
a variety of models.


http://arxiv.org/abs/1906.02337
MNIST-C: A Robustness Benchmark for Computer Vision.
Norman Mu; Justin Gilmer
  We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions
applied to the MNIST test set, for benchmarking out-of-distribution robustness
in computer vision. Through several experiments and visualizations we
demonstrate that our corruptions significantly degrade performance of
state-of-the-art computer vision models while preserving the semantic content
of the test images. In contrast to the popular notion of adversarial
robustness, our model-agnostic corruptions do not seek worst-case performance
but are instead designed to be broad and diverse, capturing multiple failure
modes of modern models. In fact, we find that several previously published
adversarial defenses significantly degrade robustness as measured by MNIST-C.
We hope that our benchmark serves as a useful tool for future work in designing
systems that are able to learn robust feature representations that capture the
underlying semantics of the input.


http://arxiv.org/abs/1906.02282
Enhancing Gradient-based Attacks with Symbolic Intervals.
Shiqi Wang; Yizheng Chen; Ahmed Abdou; Suman Jana
  Recent breakthroughs in defenses against adversarial examples, like
adversarial training, make the neural networks robust against various classes
of attackers (e.g., first-order gradient-based attacks). However, it is an open
question whether the adversarially trained networks are truly robust under
unknown attacks. In this paper, we present interval attacks, a new technique to
find adversarial examples to evaluate the robustness of neural networks.
Interval attacks leverage symbolic interval propagation, a bound propagation
technique that can exploit a broader view around the current input to locate
promising areas containing adversarial instances, which in turn can be searched
with existing gradient-guided attacks. We can obtain such a broader view using
sound bound propagation methods to track and over-approximate the errors of the
network within given input ranges. Our results show that, on state-of-the-art
adversarially trained networks, interval attack can find on average 47%
relatively more violations than the state-of-the-art gradient-guided PGD
attack.


http://arxiv.org/abs/1906.02033
Multi-way Encoding for Robustness.
Donghyun Kim; Sarah Adel Bargal; Jianming Zhang; Stan Sclaroff
  Deep models are state-of-the-art for many computer vision tasks including
image classification and object detection. However, it has been shown that deep
models are vulnerable to adversarial examples. We highlight how one-hot
encoding directly contributes to this vulnerability and propose breaking away
from this widely-used, but highly-vulnerable mapping. We demonstrate that by
leveraging a different output encoding, multi-way encoding, we decorrelate
source and target models, making target models more secure. Our approach makes
it more difficult for adversaries to find useful gradients for generating
adversarial attacks of the target model. We present robustness for black-box
and white-box attacks on four benchmark datasets. The strength of our approach
is also presented in the form of an attack for model watermarking by
decorrelating a target model from a source model.


http://arxiv.org/abs/1906.01527
Adversarial Training Generalizes Data-dependent Spectral Norm Regularization.
Kevin Roth; Yannic Kilcher; Thomas Hofmann
  We establish a theoretical link between adversarial training and operator
norm regularization for deep neural networks. Specifically, we show that
adversarial training is a data-dependent generalization of spectral norm
regularization. This intriguing connection provides fundamental insights into
the origin of adversarial vulnerability and hints at novel ways to robustify
and defend against adversarial attacks. We provide extensive empirical evidence
to support our theoretical results.


http://arxiv.org/abs/1906.01171
Conditional Generative Models are not Robust.
Ethan Fetaya; Jörn-Henrik Jacobsen; Richard Zemel
  Class-conditional generative models are an increasingly popular approach to
achieve robust classification. They are a natural choice to solve
discriminative tasks in a robust manner as they jointly optimize for predictive
performance and accurate modeling of the input distribution. In this work, we
investigate robust classification with likelihood-based conditional generative
models from a theoretical and practical perspective. Our theoretical result
reveals that it is impossible to guarantee detectability of adversarial
examples even for near-optimal generative classifiers. Experimentally, we show
that naively trained conditional generative models have poor discriminative
performance, making them unsuitable for classification. This is related to
overlooked issues with training conditional generative models and we show
methods to improve performance. Finally, we analyze the robustness of our
proposed conditional generative models on MNIST and CIFAR10. While we are able
to train robust models for MNIST, robustness completely breaks down on CIFAR10.
This lack of robustness is related to various undesirable model properties
maximum likelihood fails to penalize. Our results indicate that likelihood may
fundamentally be at odds with robust classification on challenging problems.


http://arxiv.org/abs/1906.01121
Adversarial Exploitation of Policy Imitation.
Vahid Behzadan; William Hsu
  This paper investigates a class of attacks targeting the confidentiality
aspect of security in Deep Reinforcement Learning (DRL) policies. Recent
research have established the vulnerability of supervised machine learning
models (e.g., classifiers) to model extraction attacks. Such attacks leverage
the loosely-restricted ability of the attacker to iteratively query the model
for labels, thereby allowing for the forging of a labeled dataset which can be
used to train a replica of the original model. In this work, we demonstrate the
feasibility of exploiting imitation learning techniques in launching model
extraction attacks on DRL agents. Furthermore, we develop proof-of-concept
attacks that leverage such techniques for black-box attacks against the
integrity of DRL policies. We also present a discussion on potential solution
concepts for mitigation techniques.


http://arxiv.org/abs/1906.01110
RL-Based Method for Benchmarking the Adversarial Resilience and Robustness of Deep Reinforcement Learning Policies.
Vahid Behzadan; William Hsu
  This paper investigates the resilience and robustness of Deep Reinforcement
Learning (DRL) policies to adversarial perturbations in the state space. We
first present an approach for the disentanglement of vulnerabilities caused by
representation learning of DRL agents from those that stem from the sensitivity
of the DRL policies to distributional shifts in state transitions. Building on
this approach, we propose two RL-based techniques for quantitative benchmarking
of adversarial resilience and robustness in DRL policies against perturbations
of state transitions. We demonstrate the feasibility of our proposals through
experimental evaluation of resilience and robustness in DQN, A2C, and PPO2
policies trained in the Cartpole environment.


http://arxiv.org/abs/1906.01040
A Surprising Density of Illusionable Natural Speech.
Melody Y. Guan; Gregory Valiant
  Recent work on adversarial examples has demonstrated that most natural inputs
can be perturbed to fool even state-of-the-art machine learning systems. But
does this happen for humans as well? In this work, we investigate: what
fraction of natural instances of speech can be turned into "illusions" which
either alter humans' perception or result in different people having
significantly different perceptions? We first consider the McGurk effect, the
phenomenon by which adding a carefully chosen video clip to the audio channel
affects the viewer's perception of what is said (McGurk and MacDonald, 1976).
We obtain empirical estimates that a significant fraction of both words and
sentences occurring in natural speech have some susceptibility to this effect.
We also learn models for predicting McGurk illusionability. Finally we
demonstrate that the Yanny or Laurel auditory illusion (Pressnitzer et al.,
2018) is not an isolated occurrence by generating several very different new
instances. We believe that the surprising density of illusionable natural
speech warrants further investigation, from the perspectives of both security
and cognitive science. Supplementary videos are available at:
https://www.youtube.com/playlist?list=PLaX7t1K-e_fF2iaenoKznCatm0RC37B_k.


http://arxiv.org/abs/1906.00945
Learning Perceptually-Aligned Representations via Adversarial Robustness.
Logan Engstrom; Andrew Ilyas; Shibani Santurkar; Dimitris Tsipras; Brandon Tran; Aleksander Madry
  Many applications of machine learning require models that are human-aligned,
i.e., that make decisions based on human-meaningful information about the
input. We identify the pervasive brittleness of deep networks' learned
representations as a fundamental barrier to attaining this goal. We then
re-cast robust optimization as a tool for enforcing human priors on the
features learned by deep neural networks. The resulting robust feature
representations turn out to be significantly more aligned with human
perception. We leverage these representations to perform input interpolation,
feature manipulation, and sensitivity mapping, without any post-processing or
human intervention after model training. Our code and models for reproducing
these results is available at https://git.io/robust-reps.


http://arxiv.org/abs/1906.00735
Achieving Generalizable Robustness of Deep Neural Networks by Stability Training.
Jan Laermann; Wojciech Samek; Nils Strodthoff
  We study the recently introduced stability training as a general-purpose
method to increase the robustness of deep neural networks against input
perturbations. In particular, we explore its use as an alternative to data
augmentation and validate its performance against a number of distortion types
and transformations including adversarial examples. In our ImageNet-scale image
classification experiments stability training performs on a par or even
outperforms data augmentation for specific transformations, while consistently
offering improved robustness against a broader range of distortion strengths
and types unseen during training, a considerably smaller hyperparameter
dependence and less potentially negative side effects compared to data
augmentation.


http://arxiv.org/abs/1906.00698
Adversarial Risk Bounds for Neural Networks through Sparsity based Compression.
Emilio Rafael Balda; Arash Behboodi; Niklas Koep; Rudolf Mathar
  Neural networks have been shown to be vulnerable against minor adversarial
perturbations of their inputs, especially for high dimensional data under
$\ell_\infty$ attacks. To combat this problem, techniques like adversarial
training have been employed to obtain models which are robust on the training
set. However, the robustness of such models against adversarial perturbations
may not generalize to unseen data. To study how robustness generalizes, recent
works assume that the inputs have bounded $\ell_2$-norm in order to bound the
adversarial risk for $\ell_\infty$ attacks with no explicit dimension
dependence. In this work we focus on $\ell_\infty$ attacks on $\ell_\infty$
bounded inputs and prove margin-based bounds. Specifically, we use a
compression based approach that relies on efficiently compressing the set of
tunable parameters without distorting the adversarial risk. To achieve this, we
apply the concept of effective sparsity and effective joint sparsity on the
weight matrices of neural networks. This leads to bounds with no explicit
dependence on the input dimension, neither on the number of classes. Our
results show that neural networks with approximately sparse weight matrices not
only enjoy enhanced robustness, but also better generalization.


http://arxiv.org/abs/1906.00679
The Adversarial Machine Learning Conundrum: Can The Insecurity of ML Become The Achilles' Heel of Cognitive Networks?.
Muhammad Usama; Junaid Qadir; Ala Al-Fuqaha; Mounir Hamdi
  The holy grail of networking is to create \textit{cognitive networks} that
organize, manage, and drive themselves. Such a vision now seems attainable
thanks in large part to the progress in the field of machine learning (ML),
which has now already disrupted a number of industries and revolutionized
practically all fields of research. But are the ML models foolproof and robust
to security attacks to be in charge of managing the network? Unfortunately,
many modern ML models are easily misled by simple and easily-crafted
adversarial perturbations, which does not bode well for the future of ML-based
cognitive networks unless ML vulnerabilities for the cognitive networking
environment are identified, addressed, and fixed. The purpose of this article
is to highlight the problem of insecure ML and to sensitize the readers to the
danger of adversarial ML by showing how an easily-crafted adversarial ML
example can compromise the operations of the cognitive self-driving network. In
this paper, we demonstrate adversarial attacks on two simple yet representative
cognitive networking applications (namely, intrusion detection and network
traffic classification). We also provide some guidelines to design secure ML
models for cognitive networks that are robust to adversarial attacks on the ML
pipeline of cognitive networks.


http://arxiv.org/abs/1906.00628
Fast and Stable Interval Bounds Propagation for Training Verifiably Robust Models.
Paweł Morawiecki; Przemysław Spurek; Marek Śmieja; Jacek Tabor
  We present an efficient technique, which allows to train classification
networks which are verifiably robust against norm-bounded adversarial attacks.
This framework is built upon the work of Gowal et al., who applies the interval
arithmetic to bound the activations at each layer and keeps the prediction
invariant to the input perturbation. While that method is faster than
competitive approaches, it requires careful tuning of hyper-parameters and a
large number of epochs to converge. To speed up and stabilize training, we
supply the cost function with an additional term, which encourages the model to
keep the interval bounds at hidden layers small. Experimental results
demonstrate that we can achieve comparable (or even better) results using a
smaller number of training iterations, in a more stable fashion. Moreover, the
proposed model is not so sensitive to the exact specification of the training
process, which makes it easier to use by practitioners.


http://arxiv.org/abs/1906.00555
Adversarially Robust Generalization Just Requires More Unlabeled Data.
Runtian Zhai; Tianle Cai; Di He; Chen Dan; Kun He; John Hopcroft; Liwei Wang
  Neural network robustness has recently been highlighted by the existence of
adversarial examples. Many previous works show that the learned networks do not
perform well on perturbed test data, and significantly more labeled data is
required to achieve adversarially robust generalization. In this paper, we
theoretically and empirically show that with just more unlabeled data, we can
learn a model with better adversarially robust generalization. The key insight
of our results is based on a risk decomposition theorem, in which the expected
robust risk is separated into two parts: the stability part which measures the
prediction stability in the presence of perturbations, and the accuracy part
which evaluates the standard classification accuracy. As the stability part
does not depend on any label information, we can optimize this part using
unlabeled data. We further prove that for a specific Gaussian mixture problem
illustrated by \cite{schmidt2018adversarially}, adversarially robust
generalization can be almost as easy as the standard generalization in
supervised learning if a sufficiently large amount of unlabeled data is
provided. Inspired by the theoretical findings, we propose a new algorithm
called PASS by leveraging unlabeled data during adversarial training. We show
that in the transductive and semi-supervised settings, PASS achieves higher
robust accuracy and defense success rate on the Cifar-10 task.


http://arxiv.org/abs/1906.00335
Adversarial Examples for Edge Detection: They Exist, and They Transfer.
Christian Cosgrove; Alan L. Yuille
  Convolutional neural networks have recently advanced the state of the art in
many tasks including edge and object boundary detection. However, in this
paper, we demonstrate that these edge detectors inherit a troubling property of
neural networks: they can be fooled by adversarial examples. We show that
adding small perturbations to an image causes HED, a CNN-based edge detection
model, to fail to locate edges, to detect nonexistent edges, and even to
hallucinate arbitrary configurations of edges. More surprisingly, we find that
these adversarial examples transfer to other CNN-based vision models. In
particular, attacks on edge detection result in significant drops in accuracy
in models trained to perform unrelated, high-level tasks like image
classification and semantic segmentation. Our code will be made public.


http://arxiv.org/abs/1906.00258
Enhancing Transformation-based Defenses using a Distribution Classifier.
Connie Kou; Hwee Kuan Lee; Teck Khim Ng; Ee-Chien Chang
  Adversarial attacks on convolutional neural networks (CNN) have gained
significant attention and research efforts have focused on defense methods that
make the classifiers more robust. Stochastic input transformation methods have
been proposed, where the idea is to randomly transform the input images to try
to recover from the adversarial attacks. While these transformation-based
methods have shown considerable success at recovering from adversarial images,
the performance on clean images deteriorates as the magnitude of the
transformation increases. In this paper, we propose a defense mechanism that
can be integrated with existing transformation-based defenses and reduce the
deterioration of performance on clean images. Exploiting the fact that the
transformation methods are stochastic, our method samples a population of
transformed images and performs the final classification on distributions of
softmax probabilities. We train a separate compact distribution classifier to
recognize distinctive features in the distributions of softmax probabilities of
transformed clean images. Without retraining the original CNN, our distribution
classifier improves the performance of transformation-based defenses on both
clean and adversarial images, even though the distribution classifier was never
trained on distributions obtained from the adversarial images. Our method is
generic and can be integrated with existing transformation-based methods.


http://arxiv.org/abs/1906.00204
Perceptual Evaluation of Adversarial Attacks for CNN-based Image Classification.
Sid Ahmed Fezza; Yassine Bakhti; Wassim Hamidouche; Olivier Déforges
  Deep neural networks (DNNs) have recently achieved state-of-the-art
performance and provide significant progress in many machine learning tasks,
such as image classification, speech processing, natural language processing,
etc. However, recent studies have shown that DNNs are vulnerable to adversarial
attacks. For instance, in the image classification domain, adding small
imperceptible perturbations to the input image is sufficient to fool the DNN
and to cause misclassification. The perturbed image, called \textit{adversarial
example}, should be visually as close as possible to the original image.
However, all the works proposed in the literature for generating adversarial
examples have used the $L_{p}$ norms ($L_{0}$, $L_{2}$ and $L_{\infty}$) as
distance metrics to quantify the similarity between the original image and the
adversarial example. Nonetheless, the $L_{p}$ norms do not correlate with human
judgment, making them not suitable to reliably assess the perceptual
similarity/fidelity of adversarial examples. In this paper, we present a
database for visual fidelity assessment of adversarial examples. We describe
the creation of the database and evaluate the performance of fifteen
state-of-the-art full-reference (FR) image fidelity assessment metrics that
could substitute $L_{p}$ norms. The database as well as subjective scores are
publicly available to help designing new metrics for adversarial examples and
to facilitate future research works.


http://arxiv.org/abs/1905.13736
Unlabeled Data Improves Adversarial Robustness.
Yair Carmon; Aditi Raghunathan; Ludwig Schmidt; Percy Liang; John C. Duchi
  We demonstrate, theoretically and empirically, that adversarial robustness
can significantly benefit from semisupervised learning. Theoretically, we
revisit the simple Gaussian model of Schmidt et al. that shows a sample
complexity gap between standard and robust classification. We prove that this
gap does not pertain to labels: a simple semisupervised learning procedure
(self-training) achieves robust accuracy using the same number of labels
required for standard accuracy. Empirically, we augment CIFAR-10 with 500K
unlabeled images sourced from 80 Million Tiny Images and use robust
self-training to outperform state-of-the-art robust accuracies by over 5 points
in (i) $\ell_\infty$ robustness against several strong attacks via adversarial
training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via
randomized smoothing. On SVHN, adding the dataset's own extra training set with
the labels removed provides gains of 4 to 10 points, within 1 point of the gain
from using the extra labels as well.


http://arxiv.org/abs/1905.13472
Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness.
Andrey Malinin; Mark Gales
  Ensemble approaches for uncertainty estimation have recently been applied to
the tasks of misclassification detection, out-of-distribution input detection
and adversarial attack detection. Prior Networks have been proposed as an
approach to efficiently emulating an ensemble of models by parameterising a
Dirichlet prior distribution over output distributions. These models have been
shown to outperform ensemble approaches, such as Monte-Carlo Dropout, on the
task of out-of-distribution input detection. However, scaling Prior Networks to
complex datasets with many classes is difficult using the training criteria
originally proposed. This paper makes two contributions. Firstly, we show that
the appropriate training criterion for Prior Networks is the reverse
KL-divergence between Dirichlet distributions. Using this loss we successfully
train Prior Networks on image classification datasets with up to 200 classes
and improve out-of-distribution detection performance. Secondly, taking
advantage of the new training criterion, this paper investigates using Prior
Networks to detect adversarial attacks. It is shown that the construction of
successful adaptive whitebox attacks, which affect the prediction and evade
detection, against Prior Networks trained on CIFAR-10 and CIFAR-100 takes a
greater amount of computational effort than against standard neural networks,
adversarially trained neural networks and dropout-defended networks.


http://arxiv.org/abs/1905.13725
Are Labels Required for Improving Adversarial Robustness?.
Jonathan Uesato; Jean-Baptiste Alayrac; Po-Sen Huang; Robert Stanforth; Alhussein Fawzi; Pushmeet Kohli
  Recent work has uncovered the interesting (and somewhat surprising) finding
that training models to be invariant to adversarial perturbations requires
substantially larger datasets than those required for standard classification.
This result is a key hurdle in the deployment of robust machine learning models
in many real world applications where labeled data is expensive. Our main
insight is that unlabeled data can be a competitive alternative to labeled data
for training adversarially robust models. Theoretically, we show that in a
simple statistical setting, the sample complexity for learning an adversarially
robust model from unlabeled data matches the fully supervised case up to
constant factors. On standard datasets like CIFAR-10, a simple Unsupervised
Adversarial Training (UAT) approach using unlabeled data improves robust
accuracy by 21.7% over using 4K supervised examples alone, and captures over
95% of the improvement from the same number of labeled examples. Finally, we
report an improvement of 4% over the previous state-of-the-art on CIFAR-10
against the strongest known attack by using additional unlabeled data from the
uncurated 80 Million Tiny Images dataset. This demonstrates that our finding
extends as well to the more realistic case where unlabeled data is also
uncurated, therefore opening a new avenue for improving adversarial training.


http://arxiv.org/abs/1905.13399
Real-Time Adversarial Attacks.
Yuan Gong; Boyang Li; Christian Poellabauer; Yiyu Shi
  In recent years, many efforts have demonstrated that modern machine learning
algorithms are vulnerable to adversarial attacks, where small, but carefully
crafted, perturbations on the input can make them fail. While these attack
methods are very effective, they only focus on scenarios where the target model
takes static input, i.e., an attacker can observe the entire original sample
and then add a perturbation at any point of the sample. These attack approaches
are not applicable to situations where the target model takes streaming input,
i.e., an attacker is only able to observe past data points and add
perturbations to the remaining (unobserved) data points of the input. In this
paper, we propose a real-time adversarial attack scheme for machine learning
models with streaming inputs.


http://arxiv.org/abs/1905.13386
Residual Networks as Nonlinear Systems: Stability Analysis using Linearization.
Kai Rothauge; Zhewei Yao; Zixi Hu; Michael W. Mahoney
  We regard pre-trained residual networks (ResNets) as nonlinear systems and
use linearization, a common method used in the qualitative analysis of
nonlinear systems, to understand the behavior of the networks under small
perturbations of the input images. We work with ResNet-56 and ResNet-110
trained on the CIFAR-10 data set. We linearize these networks at the level of
residual units and network stages, and the singular value decomposition is used
in the stability analysis of these components. It is found that most of the
singular values of the linearizations of residual units are 1 and, in spite of
the fact that the linearizations depend directly on the activation maps, the
singular values differ only slightly for different input images. However,
adjusting the scaling of the skip connection or the values of the weights in a
residual unit has a significant impact on the singular value distributions.
Inspection of how random and adversarial perturbations of input images
propagate through the network reveals that there is a dramatic jump in the
magnitude of adversarial perturbations towards the end of the final stage of
the network that is not present in the case of random perturbations. We attempt
to gain a better understanding of this phenomenon by projecting the
perturbations onto singular vectors of the linearizations of the residual
units.


http://arxiv.org/abs/1905.13284
Identifying Classes Susceptible to Adversarial Attacks.
Rangeet Pan; Md Johirul Islam; Shibbir Ahmed; Hridesh Rajan
  Despite numerous attempts to defend deep learning based image classifiers,
they remain susceptible to the adversarial attacks. This paper proposes a
technique to identify susceptible classes, those classes that are more easily
subverted. To identify the susceptible classes we use distance-based measures
and apply them on a trained model. Based on the distance among original
classes, we create mapping among original classes and adversarial classes that
helps to reduce the randomness of a model to a significant amount in an
adversarial setting. We analyze the high dimensional geometry among the feature
classes and identify the k most susceptible target classes in an adversarial
attack. We conduct experiments using MNIST, Fashion MNIST, CIFAR-10 (ImageNet
and ResNet-32) datasets. Finally, we evaluate our techniques in order to
determine which distance-based measure works best and how the randomness of a
model changes with perturbation.


http://arxiv.org/abs/1905.13074
Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness.
Adnan Siraj Rakin; Zhezhi He; Li Yang; Yanzhi Wang; Liqiang Wang; Deliang Fan
  Deep Neural Network (DNN) trained by the gradient descent method is known to
be vulnerable to maliciously perturbed adversarial input, aka. adversarial
attack. As one of the countermeasures against adversarial attack, increasing
the model capacity for DNN robustness enhancement was discussed and reported as
an effective approach by many recent works. In this work, we show that
shrinking the model size through proper weight pruning can even be helpful to
improve the DNN robustness under adversarial attack. For obtaining a
simultaneously robust and compact DNN model, we propose a multi-objective
training method called Robust Sparse Regularization (RSR), through the fusion
of various regularization techniques, including channel-wise noise injection,
lasso weight penalty, and adversarial training. We conduct extensive
experiments across popular ResNet-20, ResNet-18 and VGG-16 DNN architectures to
demonstrate the effectiveness of RSR against popular white-box (i.e., PGD and
FGSM) and black-box attacks. Thanks to RSR, 85% weight connections of ResNet-18
can be pruned while still achieving 0.68% and 8.72% improvement in clean- and
perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its
PGD adversarial training baseline.


http://arxiv.org/abs/1905.12864
Interpretable Adversarial Training for Text.
Samuel Barham; Soheil Feizi
  Generating high-quality and interpretable adversarial examples in the text
domain is a much more daunting task than it is in the image domain. This is due
partly to the discrete nature of text, partly to the problem of ensuring that
the adversarial examples are still probable and interpretable, and partly to
the problem of maintaining label invariance under input perturbations. In order
to address some of these challenges, we introduce sparse projected gradient
descent (SPGD), a new approach to crafting interpretable adversarial examples
for text. SPGD imposes a directional regularization constraint on input
perturbations by projecting them onto the directions to nearby word embeddings
with highest cosine similarities. This constraint ensures that perturbations
move each word embedding in an interpretable direction (i.e., towards another
nearby word embedding). Moreover, SPGD imposes a sparsity constraint on
perturbations at the sentence level by ignoring word-embedding perturbations
whose norms are below a certain threshold. This constraint ensures that our
method changes only a few words per sequence, leading to higher quality
adversarial examples. Our experiments with the IMDB movie review dataset show
that the proposed SPGD method improves adversarial example interpretability and
likelihood (evaluated by average per-word perplexity) compared to
state-of-the-art methods, while suffering little to no loss in training
performance.


http://arxiv.org/abs/1906.00001
Functional Adversarial Attacks.
Cassidy Laidlaw; Soheil Feizi
  We propose functional adversarial attacks, a novel class of threat models for
crafting adversarial examples to fool machine learning models. Unlike a
standard $\ell_p$-ball threat model, a functional adversarial threat model
allows only a single function to be used to perturb input features to produce
an adversarial example. For example, a functional adversarial attack applied on
colors of an image can change all red pixels simultaneously to light red. Such
global uniform changes in images can be less perceptible than perturbing pixels
of the image individually. For simplicity, we refer to functional adversarial
attacks on image colors as ReColorAdv, which is the main focus of our
experiments. We show that functional threat models can be combined with
existing additive ($\ell_p$) threat models to generate stronger threat models
that allow both small, individual perturbations and large, uniform changes to
an input. Moreover, we prove that such combinations encompass perturbations
that would not be allowed in either constituent threat model. In practice,
ReColorAdv can significantly reduce the accuracy of a ResNet-32 trained on
CIFAR-10. Furthermore, to the best of our knowledge, combining ReColorAdv with
other attacks leads to the strongest existing attack even after adversarial
training.


http://arxiv.org/abs/1905.12797
Bandlimiting Neural Networks Against Adversarial Attacks.
Yuping Lin; Kasra Ahmadi K. A.; Hui Jiang
  In this paper, we study the adversarial attack and defence problem in deep
learning from the perspective of Fourier analysis. We first explicitly compute
the Fourier transform of deep ReLU neural networks and show that there exist
decaying but non-zero high frequency components in the Fourier spectrum of
neural networks. We demonstrate that the vulnerability of neural networks
towards adversarial samples can be attributed to these insignificant but
non-zero high frequency components. Based on this analysis, we propose to use a
simple post-averaging technique to smooth out these high frequency components
to improve the robustness of neural networks against adversarial attacks.
Experimental results on the ImageNet dataset have shown that our proposed
method is universally effective to defend many existing adversarial attacking
methods proposed in the literature, including FGSM, PGD, DeepFool and C&W
attacks. Our post-averaging method is simple since it does not require any
re-training, and meanwhile it can successfully defend over 95% of the
adversarial samples generated by these methods without introducing any
significant performance degradation (less than 1%) on the original clean
images.


http://arxiv.org/abs/1905.12762
Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and The Way Forward.
Adnan Qayyum; Muhammad Usama; Junaid Qadir; Ala Al-Fuqaha
  Connected and autonomous vehicles (CAVs) will form the backbone of future
next-generation intelligent transportation systems (ITS) providing travel
comfort, road safety, along with a number of value-added services. Such a
transformation---which will be fuelled by concomitant advances in technologies
for machine learning (ML) and wireless communications---will enable a future
vehicular ecosystem that is better featured and more efficient. However, there
are lurking security problems related to the use of ML in such a critical
setting where an incorrect ML decision may not only be a nuisance but can lead
to loss of precious lives. In this paper, we present an in-depth overview of
the various challenges associated with the application of ML in vehicular
networks. In addition, we formulate the ML pipeline of CAVs and present various
potential security issues associated with the adoption of ML methods. In
particular, we focus on the perspective of adversarial ML attacks on CAVs and
outline a solution to defend against adversarial attacks in multiple settings.


http://arxiv.org/abs/1905.12386
Misleading Authorship Attribution of Source Code using Adversarial Learning.
Erwin Quiring; Alwin Maier; Konrad Rieck
  In this paper, we present a novel attack against authorship attribution of
source code. We exploit that recent attribution methods rest on machine
learning and thus can be deceived by adversarial examples of source code. Our
attack performs a series of semantics-preserving code transformations that
mislead learning-based attribution but appear plausible to a developer. The
attack is guided by Monte-Carlo tree search that enables us to operate in the
discrete domain of source code. In an empirical evaluation with source code
from 204 programmers, we demonstrate that our attack has a substantial effect
on two recent attribution methods, whose accuracy drops from over 88% to 1%
under attack. Furthermore, we show that our attack can imitate the coding style
of developers with high accuracy and thereby induce false attributions. We
conclude that current approaches for authorship attribution are inappropriate
for practical application and there is a need for resilient analysis
techniques.


http://arxiv.org/abs/1905.12282
Targeted Attacks on Deep Reinforcement Learning Agents through Adversarial Observations.
Léonard Hussenot; Matthieu Geist; Olivier Pietquin
  This paper deals with adversarial attacks on perceptions of neural network
policies in the Reinforcement Learning (RL) context. While previous approaches
perform untargeted attacks on the state of the agent, we propose a method to
perform targeted attacks to lure an agent into consistently following a desired
policy. We place ourselves in a realistic setting, where attacks are performed
on observations of the environment rather than the internal state of the agent
and develop constant attacks instead of per-observation ones. We illustrate our
method by attacking deep RL agents playing Atari games and show that universal
additive masks can be applied not only to degrade performance but to take
control of an agent.


http://arxiv.org/abs/1905.13545
High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks.
Haohan Wang; Xindi Wu; Pengcheng Yin; Eric P. Xing
  We investigate the relationship between the frequency spectrum of image data
and the generalization behavior of convolutional neural networks (CNN). We
first notice CNN's ability in capturing the high-frequency components of
images. These high-frequency components are almost imperceptible to a human.
Thus the observation can serve as one of the explanations of the existence of
adversarial examples, and can also help verify CNN's trade-off between
robustness and accuracy. Our observation also immediately leads to methods that
can improve the adversarial robustness of trained CNN. Finally, we also utilize
this observation to design a (semi) black-box adversarial attack method.


http://arxiv.org/abs/1905.12202
Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness.
Saeed Mahloujifar; Xiao Zhang; Mohammad Mahmoody; David Evans
  Many recent works have shown that adversarial examples that fool classifiers
can be found by minimally perturbing a normal input. Recent theoretical
results, starting with Gilmer et al. (2018), show that if the inputs are drawn
from a concentrated metric probability space, then adversarial examples with
small perturbation are inevitable. A concentrated space has the property that
any subset with $\Omega(1)$ (e.g., 1/100) measure, according to the imposed
distribution, has small distance to almost all (e.g., 99/100) of the points in
the space. It is not clear, however, whether these theoretical results apply to
actual distributions such as images. This paper presents a method for
empirically measuring and bounding the concentration of a concrete dataset
which is proven to converge to the actual concentration. We use it to
empirically estimate the intrinsic robustness to $\ell_\infty$ and $\ell_2$
perturbations of several image classification benchmarks.


http://arxiv.org/abs/1905.12105
Certifiably Robust Interpretation in Deep Learning.
Alexander Levine; Sahil Singla; Soheil Feizi
  Although gradient-based saliency maps are popular methods for deep learning
interpretation, they can be extremely vulnerable to adversarial attacks. This
is worrisome especially due to the lack of practical defenses for protecting
deep learning interpretations against attacks. In this paper, we address this
problem and provide two defense methods for deep learning interpretation.
First, we show that a sparsified version of the popular SmoothGrad method,
which computes the average saliency maps over random perturbations of the
input, is certifiably robust against adversarial perturbations. We obtain this
result by extending recent bounds for certifiably robust smooth classifiers to
the interpretation setting. Experiments on ImageNet samples validate our
theory. Second, we introduce an adversarial training approach to further
robustify deep learning interpretation by adding a regularization term to
penalize the inconsistency of saliency maps between normal and crafted
adversarial samples. Empirically, we observe that this approach not only
improves the robustness of deep learning interpretation to adversarial attacks,
but it also improves the quality of the gradient-based saliency maps.


http://arxiv.org/abs/1905.11971
ME-Net: Towards Effective Adversarial Robustness with Matrix Estimation.
Yuzhe Yang; Guo Zhang; Dina Katabi; Zhi Xu
  Deep neural networks are vulnerable to adversarial attacks. The literature is
rich with algorithms that can easily craft successful adversarial examples. In
contrast, the performance of defense techniques still lags behind. This paper
proposes ME-Net, a defense method that leverages matrix estimation (ME). In
ME-Net, images are preprocessed using two steps: first pixels are randomly
dropped from the image; then, the image is reconstructed using ME. We show that
this process destroys the adversarial structure of the noise, while
re-enforcing the global structure in the original image. Since humans typically
rely on such global structures in classifying images, the process makes the
network mode compatible with human perception. We conduct comprehensive
experiments on prevailing benchmarks such as MNIST, CIFAR-10, SVHN, and
Tiny-ImageNet. Comparing ME-Net with state-of-the-art defense mechanisms shows
that ME-Net consistently outperforms prior techniques, improving robustness
against both black-box and white-box attacks.


http://arxiv.org/abs/1905.11832
Snooping Attacks on Deep Reinforcement Learning.
Matthew Inkawhich; Yiran Chen; Hai Li
  Adversarial attacks have exposed a significant security vulnerability in
state-of-the-art machine learning models. Among these models include deep
reinforcement learning agents. The existing methods for attacking reinforcement
learning agents assume the adversary either has access to the target agent's
learned parameters or the environment that the agent interacts with. In this
work, we propose a new class of threat models, called snooping threat models,
that are unique to reinforcement learning. In these snooping threat models, the
adversary does not have the ability to personally interact with the
environment, and can only eavesdrop on the action and reward signals being
exchanged between agent and environment. We show that adversaries operating in
these highly constrained threat models can still launch devastating attacks
against the target agent by training proxy models on related tasks and
leveraging the transferability of adversarial examples.


http://arxiv.org/abs/1905.11831
Adversarial Attacks on Remote User Authentication Using Behavioural Mouse Dynamics.
Yi Xiang Marcus Tan; Alfonso Iacovazzi; Ivan Homoliak; Yuval Elovici; Alexander Binder
  Mouse dynamics is a potential means of authenticating users. Typically, the
authentication process is based on classical machine learning techniques, but
recently, deep learning techniques have been introduced for this purpose.
Although prior research has demonstrated how machine learning and deep learning
algorithms can be bypassed by carefully crafted adversarial samples, there has
been very little research performed on the topic of behavioural biometrics in
the adversarial domain. In an attempt to address this gap, we built a set of
attacks, which are applications of several generative approaches, to construct
adversarial mouse trajectories that bypass authentication models. These
generated mouse sequences will serve as the adversarial samples in the context
of our experiments. We also present an analysis of the attack approaches we
explored, explaining their limitations. In contrast to previous work, we
consider the attacks in a more realistic and challenging setting in which an
attacker has access to recorded user data but does not have access to the
authentication model or its outputs. We explore three different attack
strategies: 1) statistics-based, 2) imitation-based, and 3) surrogate-based; we
show that they are able to evade the functionality of the authentication
models, thereby impacting their robustness adversely. We show that
imitation-based attacks often perform better than surrogate-based attacks,
unless, however, the attacker can guess the architecture of the authentication
model. In such cases, we propose a potential detection mechanism against
surrogate-based attacks.


http://arxiv.org/abs/1905.11736
Cross-Domain Transferability of Adversarial Perturbations.
Muzammal Naseer; Salman H. Khan; Harris Khan; Fahad Shahbaz Khan; Fatih Porikli
  Adversarial examples reveal the blind spots of deep neural networks (DNNs)
and represent a major concern for security-critical applications. The
transferability of adversarial examples makes real-world attacks possible in
black-box settings, where the attacker is forbidden to access the internal
parameters of the model. The underlying assumption in most adversary generation
methods, whether learning an instance-specific or an instance-agnostic
perturbation, is the direct or indirect reliance on the original
domain-specific data distribution. In this work, for the first time, we
demonstrate the existence of domain-invariant adversaries, thereby showing
common adversarial space among different datasets and models. To this end, we
propose a framework capable of launching highly transferable attacks that
crafts adversarial patterns to mislead networks trained on wholly different
domains. For instance, an adversarial function learned on Paintings, Cartoons
or Medical images can successfully perturb ImageNet samples to fool the
classifier, with success rates as high as $\sim$99\% ($\ell_{\infty} \le 10$).
The core of our proposed adversarial function is a generative network that is
trained using a relativistic supervisory signal that enables domain-invariant
perturbations. Our approach sets the new state-of-the-art for fooling rates,
both under the white-box and black-box scenarios. Furthermore, despite being an
instance-agnostic perturbation function, our attack outperforms the
conventionally much stronger instance-specific attack methods.


http://arxiv.org/abs/1905.11713
Improving the Robustness of Deep Neural Networks via Adversarial Training with Triplet Loss.
Pengcheng Li; Jinfeng Yi; Bowen Zhou; Lijun Zhang
  Recent studies have highlighted that deep neural networks (DNNs) are
vulnerable to adversarial examples. In this paper, we improve the robustness of
DNNs by utilizing techniques of Distance Metric Learning. Specifically, we
incorporate Triplet Loss, one of the most popular Distance Metric Learning
methods, into the framework of adversarial training. Our proposed algorithm,
Adversarial Training with Triplet Loss (AT$^2$L), substitutes the adversarial
example against the current model for the anchor of triplet loss to effectively
smooth the classification boundary. Furthermore, we propose an ensemble version
of AT$^2$L, which aggregates different attack methods and model structures for
better defense effects. Our empirical studies verify that the proposed approach
can significantly improve the robustness of DNNs without sacrificing accuracy.
Finally, we demonstrate that our specially designed triplet loss can also be
used as a regularization term to enhance other defense methods.


http://arxiv.org/abs/1905.12418
Probabilistically True and Tight Bounds for Robust Deep Neural Network Training.
Salman Alsubaihi; Adel Bibi; Modar Alfadly; Bernard Ghanem
  Training Deep Neural Networks (DNNs) that are robust to norm bounded
adversarial attacks remains an elusive problem. While verification based
methods are generally too expensive to robustly train large networks, it was
demonstrated in Gowal et. al. that bounded input intervals can be inexpensively
propagated per layer through large networks. This interval bound propagation
(IBP) approach led to high robustness and was the first to be employed on large
networks. However, due to the very loose nature of the IBP bounds, particularly
for large networks, the required training procedure is complex and involved. In
this paper, we closely examine the bounds of a block of layers composed of an
affine layer followed by a ReLU nonlinearity followed by another affine layer.
In doing so, we propose probabilistic bounds, true bounds in expectation, that
are provably tighter than IBP bounds in expectation. We then extend this result
to deeper networks through blockwise propagation and show that we can achieve
orders of magnitudes tighter bounds compared to IBP. With such tight bounds, we
demonstrate that a simple standard training procedure can achieve the best
robustness-accuracy trade-off across several architectures on both MNIST and
CIFAR10.


http://arxiv.org/abs/1905.12171
Brain-inspired reverse adversarial examples.
Shaokai Ye; Sia Huat Tan; Kaidi Xu; Yanzhi Wang; Chenglong Bao; Kaisheng Ma
  A human does not have to see all elephants to recognize an animal as an
elephant. On contrast, current state-of-the-art deep learning approaches
heavily depend on the variety of training samples and the capacity of the
network. In practice, the size of network is always limited and it is
impossible to access all the data samples. Under this circumstance, deep
learning models are extremely fragile to human-imperceivable adversarial
examples, which impose threats to all safety critical systems. Inspired by the
association and attention mechanisms of the human brain, we propose reverse
adversarial examples method that can greatly improve models' robustness on
unseen data. Experiments show that our reverse adversarial method can improve
accuracy on average 19.02% on ResNet18, MobileNet, and VGG16 on unseen data
transformation. Besides, the proposed method is also applicable to compressed
models and shows potential to compensate the robustness drop brought by model
quantization - an absolute 30.78% accuracy improvement.


http://arxiv.org/abs/1905.11564
Adversarially Robust Learning Could Leverage Computational Hardness.
Sanjam Garg; Somesh Jha; Saeed Mahloujifar; Mohammad Mahmoody
  Over recent years, devising classification algorithms that are robust to
adversarial perturbations has emerged as a challenging problem. In particular,
deep neural nets (DNNs) seem to be susceptible to small imperceptible changes
over test instances. In this work, we study whether there is any learning task
for which it is possible to design classifiers that are only robust against
polynomial-time adversaries. Indeed, numerous cryptographic tasks (e.g.
encryption of long messages) are only be secure against computationally bounded
adversaries, and are indeed mpossible for computationally unbounded attackers.
Thus, it is natural to ask if the same strategy could help robust learning.
  We show that computational limitation of attackers can indeed be useful in
robust learning by demonstrating a classifier for a learning task in which
computational and information theoretic adversaries of bounded perturbations
have very different power. Namely, while computationally unbounded adversaries
can attack successfully and find adversarial examples with small perturbation,
polynomial time adversaries are unable to do so unless they can break standard
cryptographic hardness assumptions. Our results, therefore, indicate that
perhaps a similar approach to cryptography (relying on computational hardness)
holds promise for achieving computationally robust machine learning. We also
show that the existence of such learning task in which computational robustness
beats information theoretic robustness implies (average case) hard problems in
$\mathbf{NP}$.


http://arxiv.org/abs/1905.11544
Label Universal Targeted Attack.
Naveed Akhtar; Mohammad A. A. K. Jalwana; Mohammed Bennamoun; Ajmal Mian
  We introduce Label Universal Targeted Attack (LUTA) that makes a deep model
predict a label of attacker's choice for `any' sample of a given source class
with high probability. Our attack stochastically maximizes the log-probability
of the target label for the source class with first order gradient
optimization, while accounting for the gradient moments. It also suppresses the
leakage of attack information to the non-source classes for avoiding the attack
suspicions. The perturbations resulting from our attack achieve high fooling
ratios on the large-scale ImageNet and VGGFace models, and transfer well to the
Physical World. Given full control over the perturbation scope in LUTA, we also
demonstrate it as a tool for deep model autopsy. The proposed attack reveals
interesting perturbation patterns and observations regarding the deep models.


http://arxiv.org/abs/1905.11475
Divide-and-Conquer Adversarial Detection.
Xuwang Yin; Soheil Kolouri; Gustavo K. Rohde
  The vulnerabilities of deep neural networks against adversarial examples have
become a major concern for deploying these models in sensitive domains.
Devising a definitive defense against such attacks is proven to be challenging,
and the methods relying on detecting adversarial samples have been shown to be
only effective when the attacker is oblivious to the detection mechanism, i.e.,
in non-adaptive attacks. In this paper, we propose an effective and practical
method for detecting adaptive/dynamic adversaries. In short, we train
adversary-robust auxiliary detectors to discriminate in-class natural examples
from adversarially crafted out-of-class examples. To identify a potential
adversary, we first obtain the estimated class of the input using the
classification system, and then use the corresponding detector to verify
whether the input is a natural example of that class, or is an adversarially
manipulated example. Experimental results on MNIST and CIFAR10 dataset show
that our method could withstand adaptive PGD attacks. Furthermore, we
demonstrate that with our novel training scheme our models learn significant
more robust representation than ordinary adversarial training.


http://arxiv.org/abs/1905.11468
Scaleable input gradient regularization for adversarial robustness.
Chris Finlay; Adam M Oberman
  Input gradient regularization is not thought to be an effective means for
promoting adversarial robustness. In this work we revisit this regularization
scheme with some new ingredients. First, we derive new per-image theoretical
robustness bounds based on local gradient information, and curvature
information when available. These bounds strongly motivate input gradient
regularization. Second, we implement a scaleable version of input gradient
regularization which avoids double backpropagation: adversarially robust
ImageNet models are trained in 33 hours on four consumer grade GPUs. Finally,
we show experimentally that input gradient regularization is competitive with
adversarial training.


http://arxiv.org/abs/1905.11268
Combating Adversarial Misspellings with Robust Word Recognition.
Danish Pruthi; Bhuwan Dhingra; Zachary C. Lipton
  To combat adversarial spelling mistakes, we propose placing a word
recognition model in front of the downstream classifier. Our word recognition
models build upon the RNN semi-character architecture, introducing several new
backoff strategies for handling rare and unseen words. Trained to recognize
words corrupted by random adds, drops, swaps, and keyboard mistakes, our method
achieves 32% relative (and 3.3% absolute) error reduction over the vanilla
semi-character model. Notably, our pipeline confers robustness on the
downstream classifier, outperforming both adversarial training and
off-the-shelf spell checkers. Against a BERT model fine-tuned for sentiment
analysis, a single adversarially-chosen character attack lowers accuracy from
90.3% to 45.8%. Our defense restores accuracy to 75%. Surprisingly, better word
recognition does not always entail greater robustness. Our analysis reveals
that robustness also depends upon a quantity that we denote the sensitivity.


http://arxiv.org/abs/1905.11213
Provable robustness against all adversarial $l_p$-perturbations for $p\geq 1$.
Francesco Croce; Matthias Hein
  In recent years several adversarial attacks and defenses have been proposed.
Often seemingly robust models turn out to be non-robust when more sophisticated
attacks are used. One way out of this dilemma are provable robustness
guarantees. While provably robust models for specific $l_p$-perturbation models
have been developed, they are still vulnerable to other $l_q$-perturbations. We
propose a new regularization scheme, MMR-Universal, for ReLU networks which
enforces robustness wrt $l_1$- and $l_\infty$-perturbations and show how that
leads to provably robust models wrt any $l_p$-norm for $p\geq 1$.


http://arxiv.org/abs/1905.11026
Fooling Detection Alone is Not Enough: First Adversarial Attack against Multiple Object Tracking.
Yunhan Jia; Yantao Lu; Junjie Shen; Qi Alfred Chen; Zhenyu Zhong; Tao Wei
  Recent work in adversarial machine learning started to focus on the visual
perception in autonomous driving and studied Adversarial Examples (AEs) for
object detection models. However, in such visual perception pipeline the
detected objects must also be tracked, in a process called Multiple Object
Tracking (MOT), to build the moving trajectories of surrounding obstacles.
Since MOT is designed to be robust against errors in object detection, it poses
a general challenge to existing attack techniques that blindly target objection
detection: we find that a success rate of over 98% is needed for them to
actually affect the tracking results, a requirement that no existing attack
technique can satisfy. In this paper, we are the first to study adversarial
machine learning attacks against the complete visual perception pipeline in
autonomous driving, and discover a novel attack technique, tracker hijacking,
that can effectively fool MOT using AEs on object detection. Using our
technique, successful AEs on as few as one single frame can move an existing
object in to or out of the headway of an autonomous vehicle to cause potential
safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and
find that on average when 3 frames are attacked, our attack can have a nearly
100% success rate while attacks that blindly target object detection only have
up to 25%.


http://arxiv.org/abs/1905.11015
Unsupervised Euclidean Distance Attack on Network Embedding.
Qi Xuan; Jun Zheng; Lihong Chen; Shanqing Yu; Jinyin Chen; Dan Zhang; Qingpeng Zhang Member
  Considering the wide application of network embedding methods in graph data
mining, inspired by the adversarial attack in deep learning, this paper
proposes a Genetic Algorithm (GA) based Euclidean Distance Attack strategy
(EDA) to attack the network embedding, so as to prevent certain structural
information from being discovered. EDA focuses on disturbing the Euclidean
distance between a pair of nodes in the embedding space as much as possible
through minimal modifications of the network structure. Since a large number of
downstream network algorithms, such as community detection and node
classification, rely on the Euclidean distance between nodes to evaluate the
similarity between them in the embedding space, EDA can be considered as a
universal attack on a variety of network algorithms. Different from traditional
supervised attack strategies, EDA does not need labeling information, and, to
the best of our knowledge, is the first unsupervised network embedding attack
method. We take DeepWalk as the base embedding method to develop the EDA.
Experiments with a set of real networks demonstrate that the proposed EDA
method can significantly reduce the performance of DeepWalk-based networking
algorithms, i.e., community detection and node classification, outperforming
several heuristic attack strategies. We also show that EDA also works well on
attacking the network algorithms based on other common network embedding
methods such as High-Order Proximity preserved Embedding (HOPE) and
non-embedding-based network algorithms such as Label Propagation Algorithm
(LPA) and Eigenvectors of Matrices (EM). The results indicate a strong
transferability of the EDA method.


http://arxiv.org/abs/1905.12429
Analyzing the Interpretability Robustness of Self-Explaining Models.
Haizhong Zheng; Earlence Fernandes; Atul Prakash
  Recently, interpretable models called self-explaining models (SEMs) have been
proposed with the goal of providing interpretability robustness. We evaluate
the interpretability robustness of SEMs and show that explanations provided by
SEMs as currently proposed are not robust to adversarial inputs. Specifically,
we successfully created adversarial inputs that do not change the model outputs
but cause significant changes in the explanations. We find that even though
current SEMs use stable co-efficients for mapping explanations to output
labels, they do not consider the robustness of the first stage of the model
that creates interpretable basis concepts from the input, leading to non-robust
explanations. Our work makes a case for future work to start examining how to
generate interpretable basis concepts in a robust way.


http://arxiv.org/abs/1905.11382
State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations.
Alex Lamb; Jonathan Binas; Anirudh Goyal; Sandeep Subramanian; Ioannis Mitliagkas; Denis Kazakov; Yoshua Bengio; Michael C. Mozer
  Machine learning promises methods that generalize well from finite labeled
data. However, the brittleness of existing neural net approaches is revealed by
notable failures, such as the existence of adversarial examples that are
misclassified despite being nearly identical to a training example, or the
inability of recurrent sequence-processing nets to stay on track without
teacher forcing. We introduce a method, which we refer to as \emph{state
reification}, that involves modeling the distribution of hidden states over the
training data and then projecting hidden states observed during testing toward
this distribution. Our intuition is that if the network can remain in a
familiar manifold of hidden space, subsequent layers of the net should be well
trained to respond appropriately. We show that this state-reification method
helps neural nets to generalize better, especially when labeled data are
sparse, and also helps overcome the challenge of achieving robust
generalization with adversarial training.


http://arxiv.org/abs/1905.10906
Non-Determinism in Neural Networks for Adversarial Robustness.
Daanish Ali Khan; Linhong Li; Ninghao Sha; Zhuoran Liu; Abelino Jimenez; Bhiksha Raj; Rita Singh
  Recent breakthroughs in the field of deep learning have led to advancements
in a broad spectrum of tasks in computer vision, audio processing, natural
language processing and other areas. In most instances where these tasks are
deployed in real-world scenarios, the models used in them have been shown to be
susceptible to adversarial attacks, making it imperative for us to address the
challenge of their adversarial robustness. Existing techniques for adversarial
robustness fall into three broad categories: defensive distillation techniques,
adversarial training techniques, and randomized or non-deterministic model
based techniques. In this paper, we propose a novel neural network paradigm
that falls under the category of randomized models for adversarial robustness,
but differs from all existing techniques under this category in that it models
each parameter of the network as a statistical distribution with learnable
parameters. We show experimentally that this framework is highly robust to a
variety of white-box and black-box adversarial attacks, while preserving the
task-specific performance of the traditional neural network model.


http://arxiv.org/abs/1905.10904
Robust Classification using Robust Feature Augmentation.
Kevin Eykholt; Swati Gupta; Atul Prakash; Haizhong Zheng
  Existing deep neural networks, say for image classification, have been shown
to be vulnerable to adversarial images that can cause a DNN misclassification,
without any perceptible change to an image. In this work, we propose shock
absorbing robust features such as binarization, e.g., rounding, and group
extraction, e.g., color or shape, to augment the classification pipeline,
resulting in more robust classifiers. Experimentally, we show that augmenting
ML models with these techniques leads to improved overall robustness on
adversarial inputs as well as significant improvements in training time. On the
MNIST dataset, we achieved 14x speedup in training time to obtain 90%
adversarial accuracy com-pared to the state-of-the-art adversarial training
method of Madry et al., as well as retained higher adversarial accuracy over a
broader range of attacks. We also find robustness improvements on traffic sign
classification using robust feature augmentation. Finally, we give theoretical
insights for why one can expect robust feature augmentation to reduce
adversarial input space


http://arxiv.org/abs/1905.10900
Enhancing ML Robustness Using Physical-World Constraints.
Varun Chandrasekaran; Brian Tang; Varsha Pendyala; Kassem Fawaz; Somesh Jha; Xi Wu
  Recent advances in Machine Learning (ML) have demonstrated that neural
networks can exceed human performance in many tasks. While generalizing well
over natural inputs, neural networks are vulnerable to adversarial inputs -an
input that is ``similar'' to the original input, but misclassified by the
model.
  Existing defenses focus on Lp-norm bounded adversaries that perturb ML inputs
in the digital space. In the real world, however, attackers can generate
adversarial perturbations that have a large Lp-norm in the digital space.
Additionally, these defenses also come at a cost to accuracy, making their
applicability questionable in the real world.
  To defend models against such a powerful adversary, we leverage one
constraint on its power: the perturbation should not change the human's
perception of the physical information; the physical world places some
constraints on the space of possible attacks. Two questions follow: how to
extract and model these constraints? and how to design a classification
paradigm that leverages these constraints to improve robustness accuracy
trade-off?
  We observe that an ML model is typically a part of a larger system with
access to different input modalities. Utilizing these modalities, we introduce
invariants that limit the attacker's action space. We design a hierarchical
classification paradigm that enforces these invariants at inference time.
  As a case study, we implement and evaluate our proposal in the context of the
real-world application of road sign classification because of its applicability
to autonomous driving. With access to different input modalities, such as
LiDAR, camera, and location we show how to extract invariants and develop a
hierarchical classifier. Our results on the KITTI and GTSRB datasets show that
we can improve the robustness against physical attacks at minimal harm to
accuracy.


http://arxiv.org/abs/1905.10729
Purifying Adversarial Perturbation with Adversarially Trained Auto-encoders.
Hebi Li; Qi Xiao; Shixin Tian; Jin Tian
  Machine learning models are vulnerable to adversarial examples. Iterative
adversarial training has shown promising results against strong white-box
attacks. However, adversarial training is very expensive, and every time a
model needs to be protected, such expensive training scheme needs to be
performed. In this paper, we propose to apply iterative adversarial training
scheme to an external auto-encoder, which once trained can be used to protect
other models directly. We empirically show that our model outperforms other
purifying-based methods against white-box attacks, and transfers well to
directly protect other base models with different architectures.


http://arxiv.org/abs/1905.10864
Generalizable Adversarial Attacks Using Generative Models.
Avishek Joey Bose; Andre Cianflone; William L. Hamilton
  Adversarial attacks on deep neural networks traditionally rely on a
constrained optimization paradigm, where an optimization procedure is used to
obtain a single adversarial perturbation for a given input example. Here, we
instead view adversarial attacks as a generative modelling problem, with the
goal of producing entire distributions of adversarial examples given an
unperturbed input. We show that this generative perspective can be used to
design a unified encoder-decoder framework, which is domain-agnostic in that
the same framework can be employed to attack different domains with minimal
modification. Across three diverse domains---images, text, and graphs---our
approach generates whitebox attacks with success rates that are competitive
with or superior to existing approaches, with a new state-of-the-art achieved
in the graph domain. Finally, we demonstrate that our generative framework can
efficiently generate a diverse set of attacks for a single given input, and is
even capable of attacking unseen test instances in a zero-shot manner,
exhibiting attack generalization.


http://arxiv.org/abs/1905.11381
Trust but Verify: An Information-Theoretic Explanation for the Adversarial Fragility of Machine Learning Systems, and a General Defense against Adversarial Attacks.
Jirong Yi; Hui Xie; Leixin Zhou; Xiaodong Wu; Weiyu Xu; Raghuraman Mudumbai
  Deep-learning based classification algorithms have been shown to be
susceptible to adversarial attacks: minor changes to the input of classifiers
can dramatically change their outputs, while being imperceptible to humans. In
this paper, we present a simple hypothesis about a feature compression property
of artificial intelligence (AI) classifiers and present theoretical arguments
to show that this hypothesis successfully accounts for the observed fragility
of AI classifiers to small adversarial perturbations. Drawing on ideas from
information and coding theory, we propose a general class of defenses for
detecting classifier errors caused by abnormally small input perturbations. We
further show theoretical guarantees for the performance of this detection
method. We present experimental results with (a) a voice recognition system,
and (b) a digit recognition system using the MNIST database, to demonstrate the
effectiveness of the proposed defense methods. The ideas in this paper are
motivated by a simple analogy between AI classifiers and the standard Shannon
model of a communication system.


http://arxiv.org/abs/1905.10695
Adversarial Distillation for Ordered Top-k Attacks.
Zekun Zhang; Tianfu Wu
  Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, especially
white-box targeted attacks. One scheme of learning attacks is to design a
proper adversarial objective function that leads to the imperceptible
perturbation for any test image (e.g., the Carlini-Wagner (C&W) method). Most
methods address targeted attacks in the Top-1 manner. In this paper, we propose
to learn ordered Top-k attacks (k>= 1) for image classification tasks, that is
to enforce the Top-k predicted labels of an adversarial example to be the k
(randomly) selected and ordered labels (the ground-truth label is exclusive).
To this end, we present an adversarial distillation framework: First, we
compute an adversarial probability distribution for any given ordered Top-k
targeted labels with respect to the ground-truth of a test image. Then, we
learn adversarial examples by minimizing the Kullback-Leibler (KL) divergence
together with the perturbation energy penalty, similar in spirit to the network
distillation method. We explore how to leverage label semantic similarities in
computing the targeted distributions, leading to knowledge-oriented attacks. In
experiments, we thoroughly test Top-1 and Top-5 attacks in the ImageNet-1000
validation dataset using two popular DNNs trained with clean ImageNet-1000
train dataset, ResNet-50 and DenseNet-121. For both models, our proposed
adversarial distillation approach outperforms the C&W method in the Top-1
setting, as well as other baseline methods. Our approach shows significant
improvement in the Top-5 setting against a strong modified C&W method.


http://arxiv.org/abs/1905.10626
Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness.
Tianyu Pang; Kun Xu; Yinpeng Dong; Chao Du; Ning Chen; Jun Zhu
  Previous work shows that adversarially robust generalization requires larger
sample complexity, and the same dataset, e.g., CIFAR-10, which enables good
standard accuracy may not suffice to train robust models. Since collecting new
training data could be costly, we instead focus on inducing locally dense
sample distribution, i.e., high sample density in the feature space which could
lead to locally sufficient samples for robust learning. We first formally show
that the softmax cross-entropy (SCE) loss and its variants induce inappropriate
sample density distributions in the feature space, which inspires us to design
appropriate training objectives. Specifically, we propose the Max-Mahalanobis
center (MMC) loss to create high-density regions for better robustness. It
encourages the learned features to gather around the preset class centers with
optimal inter-class dispersion. Comparing to the SCE loss and its variants, we
empirically demonstrate that applying the MMC loss can significantly improve
robustness even under strong adaptive attacks, while keeping state-of-the-art
accuracy on clean inputs with little extra computation.


http://arxiv.org/abs/1905.10615
Adversarial Policies: Attacking Deep Reinforcement Learning.
Adam Gleave; Michael Dennis; Neel Kant; Cody Wild; Sergey Levine; Stuart Russell
  Deep reinforcement learning (RL) policies are known to be vulnerable to
adversarial perturbations to their observations, similar to adversarial
examples for classifiers. However, an attacker is not usually able to directly
modify another agent's observations. This might lead one to wonder: is it
possible to attack an RL agent simply by choosing an adversarial policy acting
in a multi-agent environment so as to create natural observations that are
adversarial? We demonstrate the existence of adversarial policies in zero-sum
games between simulated humanoid robots with proprioceptive observations,
against state-of-the-art victims trained via self-play to be robust to
opponents. The adversarial policies reliably win against the victims but
generate seemingly random and uncoordinated behavior. We find that these
policies are more successful in high-dimensional environments, and induce
substantially different activations in the victim policy network than when the
victim plays against a normal opponent. Videos are available at
http://adversarialpolicies.github.io.


http://arxiv.org/abs/1905.13021
Robustness to Adversarial Perturbations in Learning from Incomplete Data.
Amir Najafi; Shin-ichi Maeda; Masanori Koyama; Takeru Miyato
  What is the role of unlabeled data in an inference problem, when the presumed
underlying distribution is adversarially perturbed? To provide a concrete
answer to this question, this paper unifies two major learning frameworks:
Semi-Supervised Learning (SSL) and Distributionally Robust Learning (DRL). We
develop a generalization theory for our framework based on a number of novel
complexity measures, such as an adversarial extension of Rademacher complexity
and its semi-supervised analogue. Moreover, our analysis is able to quantify
the role of unlabeled data in the generalization under a more general condition
compared to the existing theoretical works in SSL. Based on our framework, we
also present a hybrid of DRL and EM algorithms that has a guaranteed
convergence rate. When implemented with deep neural networks, our method shows
a comparable performance to those of the state-of-the-art on a number of
real-world benchmark datasets.


http://arxiv.org/abs/1905.10510
Resisting Adversarial Attacks by $k$-Winners-Take-All.
Chang Xiao; Peilin Zhong; Changxi Zheng
  We propose a simple change to the current neural network structure for
defending against gradient-based adversarial attacks. Instead of using popular
activation functions (such as ReLU), we advocate the use of
$k$-Winners-Take-All ($k$-WTA) activation, a $C^0$ discontinuous function that
purposely invalidates the neural network model's gradient at densely
distributed input data points. Our proposal is theoretically rationalized. We
show why the discontinuities in $k$-WTA networks can largely prevent
gradient-based search of adversarial examples and why they at the same time
remain innocuous to the network training. This understanding is also
empirically backed. Even without notoriously expensive adversarial training,
the robustness performance of our networks is comparable to conventional ReLU
networks optimized by adversarial training. Furthermore, after also optimized
through adversarial training, our networks outperform the state-of-the-art
methods under white-box attacks on various datasets that we experimented with.


http://arxiv.org/abs/1905.10029
Power up! Robust Graph Convolutional Network against Evasion Attacks based on Graph Powering.
Ming Jin; Heng Chang; Wenwu Zhu; Somayeh Sojoudi
  Graph convolutional networks (GCNs) are powerful tools for graph-structured
data. However, they have been recently shown to be prone to topological
attacks. Despite substantial efforts to search for new architectures, it still
remains a challenge to improve performance in both benign and adversarial
situations simultaneously. In this paper, we re-examine the fundamental
building block of GCN---the Laplacian operator---and highlight some basic flaws
in the spatial and spectral domains. As an alternative, we propose an operator
based on graph powering, and prove that it enjoys a desirable property of
"spectral separation." Based on the operator, we propose a robust learning
paradigm, where the network is trained on a family of "'smoothed" graphs that
span a spatial and spectral range for generalizability. We also use the new
operator in replacement of the classical Laplacian to construct an architecture
with improved spectral robustness, expressivity and interpretability. The
enhanced performance and robustness are demonstrated in extensive experiments.


http://arxiv.org/abs/1905.09591
A Direct Approach to Robust Deep Learning Using Adversarial Networks.
Huaxia Wang; Chun-Nam Yu
  Deep neural networks have been shown to perform well in many classical
machine learning problems, especially in image classification tasks. However,
researchers have found that neural networks can be easily fooled, and they are
surprisingly sensitive to small perturbations imperceptible to humans.
Carefully crafted input images (adversarial examples) can force a well-trained
neural network to provide arbitrary outputs. Including adversarial examples
during training is a popular defense mechanism against adversarial attacks. In
this paper we propose a new defensive mechanism under the generative
adversarial network (GAN) framework. We model the adversarial noise using a
generative network, trained jointly with a classification discriminative
network as a minimax game. We show empirically that our adversarial network
approach works well against black box attacks, with performance on par with
state-of-art methods such as ensemble adversarial training and adversarial
training with projected gradient descent.


http://arxiv.org/abs/1905.09894
PHom-GeM: Persistent Homology for Generative Models.
Jeremy Charlier; Radu State; Jean Hilger
  Generative neural network models, including Generative Adversarial Network
(GAN) and Auto-Encoders (AE), are among the most popular neural network models
to generate adversarial data. The GAN model is composed of a generator that
produces synthetic data and of a discriminator that discriminates between the
generator's output and the true data. AE consist of an encoder which maps the
model distribution to a latent manifold and of a decoder which maps the latent
manifold to a reconstructed distribution. However, generative models are known
to provoke chaotically scattered reconstructed distribution during their
training, and consequently, incomplete generated adversarial distributions.
Current distance measures fail to address this problem because they are not
able to acknowledge the shape of the data manifold, i.e. its topological
features, and the scale at which the manifold should be analyzed. We propose
Persistent Homology for Generative Models, PHom-GeM, a new methodology to
assess and measure the distribution of a generative model. PHom-GeM minimizes
an objective function between the true and the reconstructed distributions and
uses persistent homology, the study of the topological features of a space at
different spatial resolutions, to compare the nature of the true and the
generated distributions. Our experiments underline the potential of persistent
homology for Wasserstein GAN in comparison to Wasserstein AE and Variational
AE. The experiments are conducted on a real-world data set particularly
challenging for traditional distance measures and generative neural network
models. PHom-GeM is the first methodology to propose a topological distance
measure, the bottleneck distance, for generative models used to compare
adversarial samples in the context of credit card transactions.


http://arxiv.org/abs/1905.09871
Thwarting finite difference adversarial attacks with output randomization.
Haidar Khan; Daniel Park; Azer Khan; Bülent Yener
  Adversarial examples pose a threat to deep neural network models in a variety
of scenarios, from settings where the adversary has complete knowledge of the
model and to the opposite "black box" setting. Black box attacks are
particularly threatening as the adversary only needs access to the input and
output of the model. Defending against black box adversarial example generation
attacks is paramount as currently proposed defenses are not effective. Since
these types of attacks rely on repeated queries to the model to estimate
gradients over input dimensions, we investigate the use of randomization to
thwart such adversaries from successfully creating adversarial examples.
Randomization applied to the output of the deep neural network model has the
potential to confuse potential attackers, however this introduces a tradeoff
between accuracy and robustness. We show that for certain types of
randomization, we can bound the probability of introducing errors by carefully
setting distributional parameters. For the particular case of finite difference
black box attacks, we quantify the error introduced by the defense in the
finite difference estimate of the gradient. Lastly, we show empirically that
the defense can thwart two adaptive black box adversarial attack algorithms.


http://arxiv.org/abs/1905.09797
Interpreting Adversarially Trained Convolutional Neural Networks.
Tianyuan Zhang; Zhanxing Zhu
  We attempt to interpret how adversarially trained convolutional neural
networks (AT-CNNs) recognize objects. We design systematic approaches to
interpret AT-CNNs in both qualitative and quantitative ways and compare them
with normally trained models. Surprisingly, we find that adversarial training
alleviates the texture bias of standard CNNs when trained on object recognition
tasks, and helps CNNs learn a more shape-biased representation. We validate our
hypothesis from two aspects. First, we compare the salience maps of AT-CNNs and
standard CNNs on clean images and images under different transformations. The
comparison could visually show that the prediction of the two types of CNNs is
sensitive to dramatically different types of features. Second, to achieve
quantitative verification, we construct additional test datasets that destroy
either textures or shapes, such as style-transferred version of clean data,
saturated images and patch-shuffled ones, and then evaluate the classification
accuracy of AT-CNNs and normal CNNs on these datasets. Our findings shed some
light on why AT-CNNs are more robust than those normally trained ones and
contribute to a better understanding of adversarial training over CNNs from an
interpretation perspective.


http://arxiv.org/abs/1905.09747
Adversarially Robust Distillation.
Micah Goldblum; Liam Fowl; Soheil Feizi; Tom Goldstein
  Knowledge distillation is effective for producing small high-performance
neural networks for classification, but these small networks are vulnerable to
adversarial attacks. We first study how robustness transfers from robust
teacher to student network during knowledge distillation. We find that a large
amount of robustness may be inherited by the student even when distilled on
only clean images. Second, we introduce Adversarially Robust Distillation (ARD)
for distilling robustness onto small student networks. ARD is an analogue of
adversarial training but for distillation. In addition to producing small
models with high test accuracy like conventional distillation, ARD also passes
the superior robustness of large networks onto the student. In our experiments,
we find that ARD student models decisively outperform adversarially trained
networks of identical architecture on robust accuracy. Finally, we adapt recent
fast adversarial training methods to ARD for accelerated robust distillation.


http://arxiv.org/abs/1905.09209
Convergence and Margin of Adversarial Training on Separable Data.
Zachary Charles; Shashank Rajput; Stephen Wright; Dimitris Papailiopoulos
  Adversarial training is a technique for training robust machine learning
models. To encourage robustness, it iteratively computes adversarial examples
for the model, and then re-trains on these examples via some update rule. This
work analyzes the performance of adversarial training on linearly separable
data, and provides bounds on the number of iterations required for large
margin. We show that when the update rule is given by an arbitrary empirical
risk minimizer, adversarial training may require exponentially many iterations
to obtain large margin. However, if gradient or stochastic gradient update
rules are used, only polynomially many iterations are required to find a
large-margin separator. By contrast, without the use of adversarial examples,
gradient methods may require exponentially many iterations to achieve large
margin. Our results are derived by showing that adversarial training with
gradient updates minimizes a robust version of the empirical risk at a
$\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness. We corroborate our
theory empirically.


http://arxiv.org/abs/1905.09186
Detecting Adversarial Examples and Other Misclassifications in Neural Networks by Introspection.
Jonathan Aigrain; Marcin Detyniecki
  Despite having excellent performances for a wide variety of tasks, modern
neural networks are unable to provide a reliable confidence value allowing to
detect misclassifications. This limitation is at the heart of what is known as
an adversarial example, where the network provides a wrong prediction
associated with a strong confidence to a slightly modified image. Moreover,
this overconfidence issue has also been observed for regular errors and
out-of-distribution data. We tackle this problem by what we call introspection,
i.e. using the information provided by the logits of an already pretrained
neural network. We show that by training a simple 3-layers neural network on
top of the logit activations, we are able to detect misclassifications at a
competitive level.


http://arxiv.org/abs/1905.08790
DoPa: A Fast and Comprehensive CNN Defense Methodology against Physical Adversarial Attacks.
Zirui Xu; Fuxun Yu; Xiang Chen
  Recently, Convolutional Neural Networks (CNNs) demonstrate a considerable
vulnerability to adversarial attacks, which can be easily misled by adversarial
perturbations. With more aggressive methods proposed, adversarial attacks can
be also applied to the physical world, causing practical issues to various CNN
powered applications. Most existing defense works for physical adversarial
attacks only focus on eliminating explicit perturbation patterns from inputs,
ignoring interpretation and solution to CNN's intrinsic vulnerability.
Therefore, most of them depend on considerable data processing costs and lack
the expected versatility to different attacks. In this paper, we propose DoPa -
a fast and comprehensive CNN defense methodology against physical adversarial
attacks. By interpreting the CNN's vulnerability, we find that non-semantic
adversarial perturbations can activate CNN with significantly abnormal
activations and even overwhelm other semantic input patterns' activations. We
improve the CNN recognition process by adding a self-verification stage to
analyze the semantics of distinguished activation patterns with only one CNN
inference involved. Based on the detection result, we further propose a data
recovery methodology to defend the physical adversarial attacks. We apply such
detection and defense methodology into both image and audio CNN recognition
process. Experiments show that our methodology can achieve an average rate of
90% success for attack detection and 81% accuracy recovery for image physical
adversarial attacks. Also, the proposed defense method can achieve a 92%
detection successful rate and 77.5% accuracy recovery for audio recognition
applications. Moreover, the proposed defense methods are at most 2.3x faster
compared to the state-of-the-art defense methods, making them feasible to
resource-constrained platforms, such as mobile devices.


http://arxiv.org/abs/1905.08232
Adversarially robust transfer learning.
Ali Shafahi; Parsa Saadatpanah; Chen Zhu; Amin Ghiasi; Christoph Studer; David Jacobs; Tom Goldstein
  Transfer learning, in which a network is trained on one task and re-purposed
on another, is often used to produce neural network classifiers when data is
scarce or full-scale training is too costly. When the goal is to produce a
model that is not only accurate but also adversarially robust, data scarcity
and computational limitations become even more cumbersome.
  We consider robust transfer learning, in which we transfer not only
performance but also robustness from a source model to a target domain. We
start by observing that robust networks contain robust feature extractors. By
training classifiers on top of these feature extractors, we produce new models
that inherit the robustness of their parent networks. We then consider the case
of fine-tuning a network by re-training end-to-end in the target domain. When
using lifelong learning strategies, this process preserves the robustness of
the source network while achieving high accuracy. By using such strategies, it
is possible to produce accurate and robust models with little data, and without
the cost of adversarial training.


http://arxiv.org/abs/1905.07831
Testing Deep Neural Network based Image Classifiers.
Yuchi Tian; Ziyuan Zhong; Vicente Ordonez; Baishakhi Ray
  Image classification is an important task in today's world with many
applications from socio-technical to safety-critical domains. The recent advent
of Deep Neural Network (DNN) is the key behind such a wide-spread success.
However, such wide adoption comes with the concerns about the reliability of
these systems, as several erroneous behaviors have already been reported in
many sensitive and critical circumstances. Thus, it has become crucial to
rigorously test the image classifiers to ensure high reliability. Many reported
erroneous cases in popular neural image classifiers appear because the models
often confuse one class with another, or show biases towards some classes over
others. These errors usually violate some group properties. Most existing DNN
testing and verification techniques focus on per image violations and thus fail
to detect such group-level confusions or biases. In this paper, we design,
implement and evaluate DeepInspect, a white box testing tool, for automatically
detecting confusion and bias of DNN-driven image classification applications.
We evaluate DeepInspect using popular DNN-based image classifiers and detect
hundreds of classification mistakes. Some of these cases are able to expose
potential biases of the network towards certain populations. DeepInspect
further reports many classification errors in state-of-the-art robust models.


http://arxiv.org/abs/1905.07672
Things You May Not Know About Adversarial Example: A Black-box Adversarial Image Attack.
Yuchao Duan; Zhe Zhao; Lei Bu; Fu Song
  Numerous methods for crafting adversarial examples were proposed recently
with high success rate. Most existing works normalize images into a continuous,
real vector, domain firstly, and then craft adversarial examples in this
domain. However, "adversarial" examples may become benign after de-normalizing
them back into the discrete integer domain, known as the discretization
problem. The discretization problem was mentioned in some work, but was
underestimated and has received relatively little attention. In this work, we
conduct the first comprehensive study of the discretization problem. We
theoretically analyze 34 representative methods and empirically study 20
representative open source tools for crafting adversarial images. Our study
reveals that almost all existing works suffer from the discretization problem
and it is far more serious than originally thought. For instance, most
black-box methods downgrade to white-box ones and methods having higher success
rates drop down to lower high success rates, e.g., from 100% to 10%. This
suggests that the discretization problem should be taken into account when
crafting adversarial examples. As a first step towards addressing this problem,
we propose a black-box method which reduces the adversarial example searching
problem to a derivative-free optimization problem. Our method is able to craft
`real' adversarial images by derivative-free search on the discrete integer
domain. Experimental results show that our method achieves significantly higher
success rate in terms of adversarial examples in the discrete integer domain
than most other methods, no matter white-box or black-box. Moreover, our method
is able to handle models that is non-differentiable and we successfully break
the winner of NIPS 17 competition on defense with 95% success rate.


http://arxiv.org/abs/1905.07666
What Do Adversarially Robust Models Look At?.
Takahiro Itazuri; Yoshihiro Fukuhara; Hirokatsu Kataoka; Shigeo Morishima
  In this paper, we address the open question: "What do adversarially robust
models look at?" Recently, it has been reported in many works that there exists
the trade-off between standard accuracy and adversarial robustness. According
to prior works, this trade-off is rooted in the fact that adversarially robust
and standard accurate models might depend on very different sets of features.
However, it has not been well studied what kind of difference actually exists.
In this paper, we analyze this difference through various experiments visually
and quantitatively. Experimental results show that adversarially robust models
look at things at a larger scale than standard models and pay less attention to
fine textures. Furthermore, although it has been claimed that adversarially
robust features are not compatible with standard accuracy, there is even a
positive effect by using them as pre-trained models particularly in low
resolution datasets.


http://arxiv.org/abs/1905.07387
POPQORN: Quantifying Robustness of Recurrent Neural Networks.
Ching-Yun Ko; Zhaoyang Lyu; Tsui-Wei Weng; Luca Daniel; Ngai Wong; Dahua Lin
  The vulnerability to adversarial attacks has been a critical issue for deep
neural networks. Addressing this issue requires a reliable way to evaluate the
robustness of a network. Recently, several methods have been developed to
compute $\textit{robustness quantification}$ for neural networks, namely,
certified lower bounds of the minimum adversarial perturbation. Such methods,
however, were devised for feed-forward networks, e.g. multi-layer perceptron or
convolutional networks. It remains an open problem to quantify robustness for
recurrent networks, especially LSTM and GRU. For such networks, there exist
additional challenges in computing the robustness quantification, such as
handling the inputs at multiple steps and the interaction between gates and
states. In this work, we propose $\textit{POPQORN}$
($\textbf{P}$ropagated-$\textbf{o}$ut$\textbf{p}$ut $\textbf{Q}$uantified
R$\textbf{o}$bustness for $\textbf{RN}$Ns), a general algorithm to quantify
robustness of RNNs, including vanilla RNNs, LSTMs, and GRUs. We demonstrate its
effectiveness on different network architectures and show that the robustness
quantification on individual steps can lead to new insights.


http://arxiv.org/abs/1905.07121
Simple Black-box Adversarial Attacks.
Chuan Guo; Jacob R. Gardner; Yurong You; Andrew Gordon Wilson; Kilian Q. Weinberger
  We propose an intriguingly simple method for the construction of adversarial
images in the black-box setting. In constrast to the white-box scenario,
constructing black-box adversarial images has the additional constraint on
query budget, and efficient attacks remain an open problem to date. With only
the mild assumption of continuous-valued confidence scores, our highly
query-efficient algorithm utilizes the following simple iterative principle: we
randomly sample a vector from a predefined orthonormal basis and either add or
subtract it to the target image. Despite its simplicity, the proposed method
can be used for both untargeted and targeted attacks -- resulting in previously
unprecedented query efficiency in both settings. We demonstrate the efficacy
and efficiency of our algorithm on several real world settings including the
Google Cloud Vision API. We argue that our proposed algorithm should serve as a
strong baseline for future black-box attacks, in particular because it is
extremely fast and its implementation requires less than 20 lines of PyTorch
code.


http://arxiv.org/abs/1905.07112
A critique of the DeepSec Platform for Security Analysis of Deep Learning Models.
Nicholas Carlini
  At IEEE S&P 2019, the paper "DeepSec: A Uniform Platform for Security
Analysis of Deep Learning Model" aims to to "systematically evaluate the
existing adversarial attack and defense methods." While the paper's goals are
laudable, it fails to achieve them and presents results that are fundamentally
flawed and misleading. We explain the flaws in the DeepSec work, along with how
its analysis fails to meaningfully evaluate the various attacks and defenses.
Specifically, DeepSec (1) evaluates each defense obliviously, using attacks
crafted against undefended models; (2) evaluates attacks and defenses using
incorrect implementations that greatly under-estimate their effectiveness; (3)
evaluates the robustness of each defense as an average, not based on the most
effective attack against that defense; (4) performs several statistical
analyses incorrectly and fails to report variance; and, (5) as a result of
these errors draws invalid conclusions and makes sweeping generalizations.


http://arxiv.org/abs/1905.06635
Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization.
Seungyong Moon; Gaon An; Hyun Oh Song
  Solving for adversarial examples with projected gradient descent has been
demonstrated to be highly effective in fooling the neural network based
classifiers. However, in the black-box setting, the attacker is limited only to
the query access to the network and solving for a successful adversarial
example becomes much more difficult. To this end, recent methods aim at
estimating the true gradient signal based on the input queries but at the cost
of excessive queries. We propose an efficient discrete surrogate to the
optimization problem which does not require estimating the gradient and
consequently becomes free of the first order update hyperparameters to tune.
Our experiments on Cifar-10 and ImageNet show the state of the art black-box
attack performance with significant reduction in the required queries compared
to a number of recently proposed methods. The source code is available at
https://github.com/snu-mllab/parsimonious-blackbox-attack.


http://arxiv.org/abs/1905.08614
War: Detecting adversarial examples by pre-processing input data.
Hua Wang; Jie Wang; Zhaoxia Yin
  Deep neural networks (DNNs) have demonstrated their outstanding performance
in many fields such as image classification and speech recognition. However,
DNNs image classifiers are susceptible to interference from adversarial
examples, which ultimately leads to incorrect classification output of neural
network models. Based on this, this paper proposes a method based on War
(WebPcompression and resize) to detect adversarial examples. The method takes
WebP compression as the core, firstly performs WebP compression on the input
image, and then appropriately resizes the compressed image, so that the label
of the adversarial example changes, thereby detecting the existence of the
adversarial image. The experimental results show that the proposed method can
effectively resist IFGSM, DeepFool and C&W attacks, and the recognition
accuracy is improved by more than 10% compared with the HGD method, the
detection success rate of adversarial examples is 5% higher than that of the
Feature Squeezing method. The method in this paper can effectively reduce the
small noise disturbance in the adversarial image, and accurately detect the
adversarial example according to the change of the
samplelabelwhileensuringtheaccuracyoftheoriginalsampleidentification.


http://arxiv.org/abs/1905.06455
On Norm-Agnostic Robustness of Adversarial Training.
Bai Li; Changyou Chen; Wenlin Wang; Lawrence Carin
  Adversarial examples are carefully perturbed in-puts for fooling machine
learning models. A well-acknowledged defense method against such examples is
adversarial training, where adversarial examples are injected into training
data to increase robustness. In this paper, we propose a new attack to unveil
an undesired property of the state-of-the-art adversarial training, that is it
fails to obtain robustness against perturbations in $\ell_2$ and $\ell_\infty$
norms simultaneously. We discuss a possible solution to this issue and its
limitations as well.


http://arxiv.org/abs/1905.05454
Robustification of deep net classifiers by key based diversified aggregation with pre-filtering.
Olga Taran; Shideh Rezaeifar; Taras Holotyak; Slava Voloshynovskiy
  In this paper, we address a problem of machine learning system vulnerability
to adversarial attacks. We propose and investigate a Key based Diversified
Aggregation (KDA) mechanism as a defense strategy. The KDA assumes that the
attacker (i) knows the architecture of classifier and the used defense
strategy, (ii) has an access to the training data set but (iii) does not know
the secret key. The robustness of the system is achieved by a specially
designed key based randomization. The proposed randomization prevents the
gradients' back propagation or the creating of a "bypass" system. The
randomization is performed simultaneously in several channels and a
multi-channel aggregation stabilizes the results of randomization by
aggregating soft outputs from each classifier in multi-channel system. The
performed experimental evaluation demonstrates a high robustness and
universality of the KDA against the most efficient gradient based attacks like
those proposed by N. Carlini and D. Wagner and the non-gradient based sparse
adversarial perturbations like OnePixel attacks.


http://arxiv.org/abs/1905.05163
Adversarial Examples for Electrocardiograms.
Xintian Han; Yuxuan Hu; Luca Foschini; Larry Chinitz; Lior Jankelson; Rajesh Ranganath
  In recent years, the electrocardiogram (ECG) has seen a large diffusion in
both medical and commercial applications, fueled by the rise of single-lead
versions. Single-lead ECG can be embedded in medical devices and wearable
products such as the injectable Medtronic Linq monitor, the iRhythm Ziopatch
wearable monitor, and the Apple Watch Series 4. Recently, deep neural networks
have been used to automatically analyze ECG tracings, outperforming even
physicians specialized in cardiac electrophysiology in detecting certain rhythm
irregularities. However, deep learning classifiers have been shown to be
brittle to adversarial examples, which are examples created to look
incontrovertibly belonging to a certain class to a human eye but contain subtle
features that fool the classifier into misclassifying them into the wrong
class. Very recently, adversarial examples have also been created for
medical-related tasks. Yet, traditional attack methods to create adversarial
examples, such as projected gradient descent (PGD) do not extend directly to
ECG signals, as they generate examples that introduce square wave artifacts
that are not physiologically plausible. Here, we developed a method to
construct smoothed adversarial examples for single-lead ECG. First, we
implemented a neural network model achieving state-of-the-art performance on
the data from the 2017 PhysioNet/Computing-in-Cardiology Challenge for
arrhythmia detection from single lead ECG classification. For this model, we
utilized a new technique to generate smoothed examples to produce signals that
are 1) indistinguishable to cardiologists from the original examples and 2)
incorrectly classified by the neural network. Finally, we show that adversarial
examples are not unique and provide a general technique to collate and perturb
known adversarial examples to create new ones.


http://arxiv.org/abs/1905.05137
Analyzing Adversarial Attacks Against Deep Learning for Intrusion Detection in IoT Networks.
Olakunle Ibitoye; Omair Shafiq; Ashraf Matrawy
  Adversarial attacks have been widely studied in the field of computer vision
but their impact on network security applications remains an area of open
research. As IoT, 5G and AI continue to converge to realize the promise of the
fourth industrial revolution (Industry 4.0), security incidents and events on
IoT networks have increased. Deep learning techniques are being applied to
detect and mitigate many of such security threats against IoT networks.
Feedforward Neural Networks (FNN) have been widely used for classifying
intrusion attacks in IoT networks. In this paper, we consider a variant of the
FNN known as the Self-normalizing Neural Network (SNN) and compare its
performance with the FNN for classifying intrusion attacks in an IoT network.
Our analysis is performed using the BoT-IoT dataset from the Cyber Range Lab of
the center of UNSW Canberra Cyber. In our experimental results, the FNN
outperforms the SNN for intrusion detection in IoT networks based on multiple
performance metrics such as accuracy, precision, and recall as well as
multi-classification metrics such as Cohen's Kappa score. However, when tested
for adversarial robustness, the SNN demonstrates better resilience against the
adversarial samples from the IoT dataset, presenting a promising future in the
quest for safer and more secure deep learning in IoT networks.


http://arxiv.org/abs/1905.05186
Harnessing the Vulnerability of Latent Layers in Adversarially Trained Models.
Mayank Singh; Abhishek Sinha; Nupur Kumari; Harshitha Machiraju; Balaji Krishnamurthy; Vineeth N Balasubramanian
  Neural networks are vulnerable to adversarial attacks -- small visually
imperceptible crafted noise which when added to the input drastically changes
the output. The most effective method of defending against these adversarial
attacks is to use the methodology of adversarial training. We analyze the
adversarially trained robust models to study their vulnerability against
adversarial attacks at the level of the latent layers. Our analysis reveals
that contrary to the input layer which is robust to adversarial attack, the
latent layer of these robust models are highly susceptible to adversarial
perturbations of small magnitude. Leveraging this information, we introduce a
new technique Latent Adversarial Training (LAT) which comprises of fine-tuning
the adversarially trained models to ensure the robustness at the feature
layers. We also propose Latent Attack (LA), a novel algorithm for construction
of adversarial examples. LAT results in minor improvement in test accuracy and
leads to a state-of-the-art adversarial accuracy against the universal
first-order adversarial PGD attack which is shown for the MNIST, CIFAR-10,
CIFAR-100 datasets.


http://arxiv.org/abs/1905.13148
Moving Target Defense for Deep Visual Sensing against Adversarial Examples.
Qun Song; Zhenyu Yan; Rui Tan
  Deep learning based visual sensing has achieved attractive accuracy but is
shown vulnerable to adversarial example attacks. Specifically, once the
attackers obtain the deep model, they can construct adversarial examples to
mislead the model to yield wrong classification results. Deployable adversarial
examples such as small stickers pasted on the road signs and lanes have been
shown effective in misleading advanced driver-assistance systems. Many existing
countermeasures against adversarial examples build their security on the
attackers' ignorance of the defense mechanisms. Thus, they fall short of
following Kerckhoffs's principle and can be subverted once the attackers know
the details of the defense. This paper applies the strategy of moving target
defense (MTD) to generate multiple new deep models after system deployment,
that will collaboratively detect and thwart adversarial examples. Our MTD
design is based on the adversarial examples' minor transferability to models
differing from the one (e.g., the factory-designed model) used for attack
construction. The post-deployment quasi-secret deep models significantly
increase the bar for the attackers to construct effective adversarial examples.
We also apply the technique of serial data fusion with early stopping to reduce
the inference time by a factor of up to 5 while maintaining the sensing and
defense performance. Extensive evaluation based on three datasets including a
road sign image database and a GPU-equipped Jetson embedded computing board
shows the effectiveness of our approach.


http://arxiv.org/abs/1905.04270
Interpreting and Evaluating Neural Network Robustness.
Fuxun Yu; Zhuwei Qin; Chenchen Liu; Liang Zhao; Yanzhi Wang; Xiang Chen
  Recently, adversarial deception becomes one of the most considerable threats
to deep neural networks. However, compared to extensive research in new designs
of various adversarial attacks and defenses, the neural networks' intrinsic
robustness property is still lack of thorough investigation. This work aims to
qualitatively interpret the adversarial attack and defense mechanism through
loss visualization, and establish a quantitative metric to evaluate the neural
network model's intrinsic robustness. The proposed robustness metric identifies
the upper bound of a model's prediction divergence in the given domain and thus
indicates whether the model can maintain a stable prediction. With extensive
experiments, our metric demonstrates several advantages over conventional
adversarial testing accuracy based robustness estimation: (1) it provides a
uniformed evaluation to models with different structures and parameter scales;
(2) it over-performs conventional accuracy based robustness estimation and
provides a more reliable evaluation that is invariant to different test
settings; (3) it can be fast generated without considerable testing cost.


http://arxiv.org/abs/1905.04172
On the Connection Between Adversarial Robustness and Saliency Map Interpretability.
Christian Etmann; Sebastian Lunz; Peter Maass; Carola-Bibiane Schönlieb
  Recent studies on the adversarial vulnerability of neural networks have shown
that models trained to be more robust to adversarial attacks exhibit more
interpretable saliency maps than their non-robust counterparts. We aim to
quantify this behavior by considering the alignment between input image and
saliency map. We hypothesize that as the distance to the decision boundary
grows,so does the alignment. This connection is strictly true in the case of
linear models. We confirm these theoretical findings with experiments based on
models trained with a local Lipschitz regularization and identify where the
non-linear nature of neural networks weakens the relation.


http://arxiv.org/abs/1905.04016
Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables.
Yan Xu; Baoyuan Wu; Fumin Shen; Yanbo Fan; Yong Zhang; Heng Tao Shen; Wei Liu
  In this work, we study the robustness of a CNN+RNN based image captioning
system being subjected to adversarial noises. We propose to fool an image
captioning system to generate some targeted partial captions for an image
polluted by adversarial noises, even the targeted captions are totally
irrelevant to the image content. A partial caption indicates that the words at
some locations in this caption are observed, while words at other locations are
not restricted.It is the first work to study exact adversarial attacks of
targeted partial captions. Due to the sequential dependencies among words in a
caption, we formulate the generation of adversarial noises for targeted partial
captions as a structured output learning problem with latent variables. Both
the generalized expectation maximization algorithm and structural SVMs with
latent variables are then adopted to optimize the problem. The proposed methods
generate very successful at-tacks to three popular CNN+RNN based image
captioning models. Furthermore, the proposed attack methods are used to
understand the inner mechanism of image captioning systems, providing the
guidance to further improve automatic image captioning systems towards human
captioning.


http://arxiv.org/abs/1905.03828
Universal Adversarial Perturbations for Speech Recognition Systems.
Paarth Neekhara; Shehzeen Hussain; Prakhar Pandey; Shlomo Dubnov; Julian McAuley; Farinaz Koushanfar
  In this work, we demonstrate the existence of universal adversarial audio
perturbations that cause mis-transcription of audio signals by automatic speech
recognition (ASR) systems. We propose an algorithm to find a single
quasi-imperceptible perturbation, which when added to any arbitrary speech
signal, will most likely fool the victim speech recognition model. Our
experiments demonstrate the application of our proposed technique by crafting
audio-agnostic universal perturbations for the state-of-the-art ASR system --
Mozilla DeepSpeech. Additionally, we show that such perturbations generalize to
a significant extent across models that are not available during training, by
performing a transferability test on a WaveNet based ASR system.


http://arxiv.org/abs/1905.03767
Learning Interpretable Features via Adversarially Robust Optimization.
Ashkan Khakzar; Shadi Albarqouni; Nassir Navab
  Neural networks are proven to be remarkably successful for classification and
diagnosis in medical applications. However, the ambiguity in the
decision-making process and the interpretability of the learned features is a
matter of concern. In this work, we propose a method for improving the feature
interpretability of neural network classifiers. Initially, we propose a
baseline convolutional neural network with state of the art performance in
terms of accuracy and weakly supervised localization. Subsequently, the loss is
modified to integrate robustness to adversarial examples into the training
process. In this work, feature interpretability is quantified via evaluating
the weakly supervised localization using the ground truth bounding boxes.
Interpretability is also visually assessed using class activation maps and
saliency maps. The method is applied to NIH ChestX-ray14, the largest publicly
available chest x-rays dataset. We demonstrate that the adversarially robust
optimization paradigm improves feature interpretability both quantitatively and
visually.


http://arxiv.org/abs/1905.03679
Adversarial Defense Framework for Graph Neural Network.
Shen Wang; Zhengzhang Chen; Jingchao Ni; Xiao Yu; Zhichun Li; Haifeng Chen; Philip S. Yu
  Graph neural network (GNN), as a powerful representation learning model on
graph data, attracts much attention across various disciplines. However, recent
studies show that GNN is vulnerable to adversarial attacks. How to make GNN
more robust? What are the key vulnerabilities in GNN? How to address the
vulnerabilities and defense GNN against the adversarial attacks? In this paper,
we propose DefNet, an effective adversarial defense framework for GNNs. In
particular, we first investigate the latent vulnerabilities in every layer of
GNNs and propose corresponding strategies including dual-stage aggregation and
bottleneck perceptron. Then, to cope with the scarcity of training data, we
propose an adversarial contrastive learning method to train the GNN in a
conditional GAN manner by leveraging the high-level graph representation.
Extensive experiments on three public datasets demonstrate the effectiveness of
DefNet in improving the robustness of popular GNN variants, such as Graph
Convolutional Network and GraphSAGE, under various types of adversarial
attacks.


http://arxiv.org/abs/1905.03517
Mitigating Deep Learning Vulnerabilities from Adversarial Examples Attack in the Cybersecurity Domain.
Chris Einar San Agustin
  Deep learning models are known to solve classification and regression
problems by employing a number of epoch and training samples on a large dataset
with optimal accuracy. However, that doesn't mean they are attack-proof or
unexposed to vulnerabilities. Newly deployed systems particularly on a public
environment (i.e public networks) are vulnerable to attacks from various
entities. Moreover, published research on deep learning systems (Goodfellow et
al., 2014) have determined a significant number of attacks points and a wide
array of attack surface that has evidence of exploitation from adversarial
examples. Successful exploit on these systems could lead to critical real world
repercussions. For instance, (1) an adversarial attack on a self-driving car
running a deep reinforcement learning system yields a direct misclassification
on humans causing untoward accidents.(2) a self-driving vehicle misreading a
red light signal may cause the car to crash to another car (3)
misclassification of a pedestrian lane as an intersection lane that could lead
to car crashes. This is just the tip of the iceberg, computer vision deployment
are not entirely focused on self-driving cars but on many other areas as well -
that would have definitive impact on the real-world. These vulnerabilities must
be mitigated at an early stage of development. It is imperative to develop and
implement baseline security standards at a global level prior to real-world
deployment.


http://arxiv.org/abs/1905.03837
Exploring the Hyperparameter Landscape of Adversarial Robustness.
Evelyn Duesterwald; Anupama Murthi; Ganesh Venkataraman; Mathieu Sinn; Deepak Vijaykeerthy
  Adversarial training shows promise as an approach for training models that
are robust towards adversarial perturbation. In this paper, we explore some of
the practical challenges of adversarial training. We present a sensitivity
analysis that illustrates that the effectiveness of adversarial training hinges
on the settings of a few salient hyperparameters. We show that the robustness
surface that emerges across these salient parameters can be surprisingly
complex and that therefore no effective one-size-fits-all parameter settings
exist. We then demonstrate that we can use the same salient hyperparameters as
tuning knob to navigate the tension that can arise between robustness and
accuracy. Based on these findings, we present a practical approach that
leverages hyperparameter optimization techniques for tuning adversarial
training to maximize robustness while keeping the loss in accuracy within a
defined budget.


http://arxiv.org/abs/1905.03434
ROSA: Robust Salient Object Detection against Adversarial Attacks.
Haofeng Li; Guanbin Li; Yizhou Yu
  Recently salient object detection has witnessed remarkable improvement owing
to the deep convolutional neural networks which can harvest powerful features
for images. In particular, state-of-the-art salient object detection methods
enjoy high accuracy and efficiency from fully convolutional network (FCN) based
frameworks which are trained from end to end and predict pixel-wise labels.
However, such framework suffers from adversarial attacks which confuse neural
networks via adding quasi-imperceptible noises to input images without changing
the ground truth annotated by human subjects. To our knowledge, this paper is
the first one that mounts successful adversarial attacks on salient object
detection models and verifies that adversarial samples are effective on a wide
range of existing methods. Furthermore, this paper proposes a novel end-to-end
trainable framework to enhance the robustness for arbitrary FCN-based salient
object detection models against adversarial attacks. The proposed framework
adopts a novel idea that first introduces some new generic noise to destroy
adversarial perturbations, and then learns to predict saliency maps for input
images with the introduced noise. Specifically, our proposed method consists of
a segment-wise shielding component, which preserves boundaries and destroys
delicate adversarial noise patterns and a context-aware restoration component,
which refines saliency maps through global contrast modeling. Experimental
results suggest that our proposed framework improves the performance
significantly for state-of-the-art models on a series of datasets.


http://arxiv.org/abs/1905.03421
Adversarial Image Translation: Unrestricted Adversarial Examples in Face Recognition Systems.
Kazuya Kakizaki; Kosuke Yoshida
  Thanks to recent advances in Deep Neural Networks (DNNs), face recognition
systems have achieved high accuracy in classification of a large number of face
images. However, recent works demonstrate that DNNs could be vulnerable to
adversarial examples and raise concerns about robustness of face recognition
systems. In particular adversarial examples that are not restricted to small
perturbations could be more serious risks since conventional certified defenses
might be ineffective against them. To shed light on the vulnerability to this
type of adversarial examples, we propose a flexible and efficient method to
generate unrestricted adversarial examples using image translation techniques.
Our method enables us to translate a source image into any desired facial
appearance with large perturbations so that target face recognition systems
could be deceived. Through our experiments, we demonstrate that our method
achieves about 90% and 30% attack success rates under a white- and black-box
setting, respectively. We also illustrate that our translated images are
perceptually realistic and maintain personal identity while the perturbations
are large enough to bypass certified defenses.


http://arxiv.org/abs/1905.03333
Enhancing Cross-task Transferability of Adversarial Examples with Dispersion Reduction.
Yunhan Jia; Yantao Lu; Senem Velipasalar; Zhenyu Zhong; Tao Wei
  Neural networks are known to be vulnerable to carefully crafted adversarial
examples, and these malicious samples often transfer, i.e., they maintain their
effectiveness even against other models. With great efforts delved into the
transferability of adversarial examples, surprisingly, less attention has been
paid to its impact on real-world deep learning deployment. In this paper, we
investigate the transferability of adversarial examples across a wide range of
real-world computer vision tasks, including image classification, explicit
content detection, optical character recognition (OCR), and object detection.
It represents the cybercriminal's situation where an ensemble of different
detection mechanisms need to be evaded all at once. We propose practical attack
that overcomes existing attacks' limitation of requiring task-specific loss
functions by targeting on the `dispersion' of internal feature map. We report
evaluation on four different computer vision tasks provided by Google Cloud
Vision APIs to show how our approach outperforms existing attacks by degrading
performance of multiple CV tasks by a large margin with only modest
perturbations.


http://arxiv.org/abs/1905.02704
A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks.
Saima Sharmin; Priyadarshini Panda; Syed Shakib Sarwar; Chankyu Lee; Wachirawit Ponghiran; Kaushik Roy
  In this era of machine learning models, their functionality is being
threatened by adversarial attacks. In the face of this struggle for making
artificial neural networks robust, finding a model, resilient to these attacks,
is very important. In this work, we present, for the first time, a
comprehensive analysis of the behavior of more bio-plausible networks, namely
Spiking Neural Network (SNN) under state-of-the-art adversarial tests. We
perform a comparative study of the accuracy degradation between conventional
VGG-9 Artificial Neural Network (ANN) and equivalent spiking network with
CIFAR-10 dataset in both whitebox and blackbox setting for different types of
single-step and multi-step FGSM (Fast Gradient Sign Method) attacks. We
demonstrate that SNNs tend to show more resiliency compared to ANN under
black-box attack scenario. Additionally, we find that SNN robustness is largely
dependent on the corresponding training mechanism. We observe that SNNs trained
by spike-based backpropagation are more adversarially robust than the ones
obtained by ANN-to-SNN conversion rules in several whitebox and blackbox
scenarios. Finally, we also propose a simple, yet, effective framework for
crafting adversarial attacks from SNNs. Our results suggest that attacks
crafted from SNNs following our proposed method are much stronger than those
crafted from ANNs.


http://arxiv.org/abs/1905.02675
An Empirical Evaluation of Adversarial Robustness under Transfer Learning.
Todor Davchev; Timos Korres; Stathi Fotiadis; Nick Antonopoulos; Subramanian Ramamoorthy
  In this work, we evaluate adversarial robustness in the context of transfer
learning from a source trained on CIFAR 100 to a target network trained on
CIFAR 10. Specifically, we study the effects of using robust optimisation in
the source and target networks. This allows us to identify transfer learning
strategies under which adversarial defences are successfully retained, in
addition to revealing potential vulnerabilities. We study the extent to which
features learnt by a fast gradient sign method (FGSM) and its iterative
alternative (PGD) can preserve their defence properties against black and
white-box attacks under three different transfer learning strategies. We find
that using PGD examples during training on the source task leads to more
general robust features that are easier to transfer. Furthermore, under
successful transfer, it achieves 5.2% more accuracy against white-box PGD
attacks than suitable baselines. Overall, our empirical evaluations give
insights on how well adversarial robustness under transfer learning can
generalise.


http://arxiv.org/abs/1905.02463
Generating Realistic Unrestricted Adversarial Inputs using Dual-Objective GAN Training.
Isaac Dunn; Tom Melham; Daniel Kroening
  The correctness of deep neural networks is well-known to be vulnerable to
small, 'adversarial' perturbations of their inputs. Although studying these
attacks is valuable, they do not necessarily conform to any real-world threat
model. This has led to interest in the generation of (and robustness to)
unrestricted adversarial inputs, which are not constructed as small
perturbations of correctly-classified ground-truth inputs. We introduce a novel
algorithm to generate realistic unrestricted adversarial inputs, in the sense
that they cannot reliably be distinguished from the training dataset by a
human. This is achieved by modifying generative adversarial networks: a
generator neural network is trained to construct examples that deceive a fixed
target network (so they are adversarial) while also deceiving the usual
co-training discriminator network (so they are realistic). Our approach is
demonstrated by the generation of unrestricted adversarial inputs for a trained
image classifier that is robust to perturbation-based attacks. We find that
human judges are unable to identify which image out of ten was generated by our
method about 50 percent of the time, providing evidence that they are
moderately realistic.


http://arxiv.org/abs/1905.02422
Representation of White- and Black-Box Adversarial Examples in Deep Neural Networks and Humans: A Functional Magnetic Resonance Imaging Study.
Chihye Han; Wonjun Yoon; Gihyun Kwon; Seungkyu Nam; Daeshik Kim
  The recent success of brain-inspired deep neural networks (DNNs) in solving
complex, high-level visual tasks has led to rising expectations for their
potential to match the human visual system. However, DNNs exhibit
idiosyncrasies that suggest their visual representation and processing might be
substantially different from human vision. One limitation of DNNs is that they
are vulnerable to adversarial examples, input images on which subtle, carefully
designed noises are added to fool a machine classifier. The robustness of the
human visual system against adversarial examples is potentially of great
importance as it could uncover a key mechanistic feature that machine vision is
yet to incorporate. In this study, we compare the visual representations of
white- and black-box adversarial examples in DNNs and humans by leveraging
functional magnetic resonance imaging (fMRI). We find a small but significant
difference in representation patterns for different (i.e. white- versus black-
box) types of adversarial examples for both humans and DNNs. However, human
performance on categorical judgment is not degraded by noise regardless of the
type unlike DNN. These results suggest that adversarial examples may be
differentially represented in the human visual system, but unable to affect the
perceptual experience.


http://arxiv.org/abs/1905.02161
Batch Normalization is a Cause of Adversarial Vulnerability.
Angus Galloway; Anna Golubeva; Thomas Tanay; Medhat Moussa; Graham W. Taylor
  Batch normalization (batch norm) is often used in an attempt to stabilize and
accelerate training in deep neural networks. In many cases it indeed decreases
the number of parameter updates required to achieve low training error.
However, it also reduces robustness to small adversarial input perturbations
and noise by double-digit percentages, as we show on five standard datasets.
Furthermore, substituting weight decay for batch norm is sufficient to nullify
the relationship between adversarial vulnerability and the input dimension. Our
work is consistent with a mean-field analysis that found that batch norm causes
exploding gradients.


http://arxiv.org/abs/1905.02175
Adversarial Examples Are Not Bugs, They Are Features.
Andrew Ilyas; Shibani Santurkar; Dimitris Tsipras; Logan Engstrom; Brandon Tran; Aleksander Madry
  Adversarial examples have attracted significant attention in machine
learning, but the reasons for their existence and pervasiveness remain unclear.
We demonstrate that adversarial examples can be directly attributed to the
presence of non-robust features: features derived from patterns in the data
distribution that are highly predictive, yet brittle and incomprehensible to
humans. After capturing these features within a theoretical framework, we
establish their widespread existence in standard datasets. Finally, we present
a simple setting where we can rigorously tie the phenomena we observe in
practice to a misalignment between the (human-specified) notion of robustness
and the inherent geometry of the data.


http://arxiv.org/abs/1905.01726
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples.
Vikash Sehwag; Arjun Nitin Bhagoji; Liwei Song; Chawin Sitawarin; Daniel Cullina; Mung Chiang; Prateek Mittal
  A large body of recent work has investigated the phenomenon of evasion
attacks using adversarial examples for deep learning systems, where the
addition of norm-bounded perturbations to the test inputs leads to incorrect
output classification. Previous work has investigated this phenomenon in
closed-world systems where training and test inputs follow a pre-specified
distribution. However, real-world implementations of deep learning
applications, such as autonomous driving and content classification are likely
to operate in the open-world environment. In this paper, we demonstrate the
success of open-world evasion attacks, where adversarial examples are generated
from out-of-distribution inputs (OOD adversarial examples). In our study, we
use 11 state-of-the-art neural network models trained on 3 image datasets of
varying complexity. We first demonstrate that state-of-the-art detectors for
out-of-distribution data are not robust against OOD adversarial examples. We
then consider 5 known defenses for adversarial examples, including
state-of-the-art robust training methods, and show that against these defenses,
OOD adversarial examples can achieve up to 4$\times$ higher target success
rates compared to adversarial examples generated from in-distribution data. We
also take a quantitative look at how open-world evasion attacks may affect
real-world systems. Finally, we present the first steps towards a robust
open-world machine learning system.


http://arxiv.org/abs/1905.01034
Transfer of Adversarial Robustness Between Perturbation Types.
Daniel Kang; Yi Sun; Tom Brown; Dan Hendrycks; Jacob Steinhardt
  We study the transfer of adversarial robustness of deep neural networks
between different perturbation types. While most work on adversarial examples
has focused on $L_\infty$ and $L_2$-bounded perturbations, these do not capture
all types of perturbations available to an adversary. The present work
evaluates 32 attacks of 5 different types against models adversarially trained
on a 100-class subset of ImageNet. Our empirical results suggest that
evaluating on a wide range of perturbation sizes is necessary to understand
whether adversarial robustness transfers between perturbation types. We further
demonstrate that robustness against one perturbation type may not always imply
and may sometimes hurt robustness against other perturbation types. In light of
these results, we recommend evaluation of adversarial defenses take place on a
diverse range of perturbation types and sizes.


http://arxiv.org/abs/1905.01019
Adversarial Training with Voronoi Constraints.
Marc Khoury; Dylan Hadfield-Menell
  Adversarial examples are a pervasive phenomenon of machine learning models
where seemingly imperceptible perturbations to the input lead to
misclassifications for otherwise statistically accurate models. We propose a
geometric framework, drawing on tools from the manifold reconstruction
literature, to analyze the high-dimensional geometry of adversarial examples.
In particular, we highlight the importance of codimension: for low-dimensional
data manifolds embedded in high-dimensional space there are many directions off
the manifold in which an adversary could construct adversarial examples.
Adversarial examples are a natural consequence of learning a decision boundary
that classifies the low-dimensional data manifold well, but classifies points
near the manifold incorrectly. Using our geometric framework we prove that
adversarial training is sample inefficient, and show sufficient sampling
conditions under which nearest neighbor classifiers and ball-based adversarial
training are robust. Finally we introduce adversarial training with Voronoi
constraints, which replaces the norm ball constraint with the Voronoi cell for
each point in the training set. We show that adversarial training with Voronoi
constraints produces robust models which significantly improve over the
state-of-the-art on MNIST and are competitive on CIFAR-10.


http://arxiv.org/abs/1905.00568
Weight Map Layer for Noise and Adversarial Attack Robustness.
Mohammed Amer; Tomás Maul
  Convolutional neural networks (CNNs) are known for their good performance and
generalization in vision-related tasks and have become state-of-the-art in both
application and research-based domains. However, just like other neural network
models, they suffer from a susceptibility to noise and adversarial attacks. An
adversarial defence aims at reducing a neural network's susceptibility to
adversarial attacks through learning or architectural modifications. We propose
a weight map layer (WM) as a generic architectural addition to CNNs and show
that it can increase their robustness to noise and adversarial attacks. We
further explain the enhanced robustness of the two WM variants introduced via
an adaptive noise-variance amplification (ANVA) hypothesis and provide evidence
and insights in support of it. We show that the WM layer can be integrated into
scaled up models to increase their noise and adversarial attack robustness,
while achieving the same or similar accuracy levels.


http://arxiv.org/abs/1905.00877
You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle.
Dinghuai Zhang; Tianyuan Zhang; Yiping Lu; Zhanxing Zhu; Bin Dong
  Deep learning achieves state-of-the-art results in many tasks in computer
vision and natural language processing. However, recent works have shown that
deep networks can be vulnerable to adversarial perturbations, which raised a
serious robustness issue of deep networks. Adversarial training, typically
formulated as a robust optimization problem, is an effective way of improving
the robustness of deep networks. A major drawback of existing adversarial
training algorithms is the computational overhead of the generation of
adversarial examples, typically far greater than that of the network training.
This leads to the unbearable overall computational cost of adversarial
training. In this paper, we show that adversarial training can be cast as a
discrete time differential game. Through analyzing the Pontryagin's Maximal
Principle (PMP) of the problem, we observe that the adversary update is only
coupled with the parameters of the first layer of the network. This inspires us
to restrict most of the forward and back propagation within the first layer of
the network during adversary updates. This effectively reduces the total number
of full forward and backward propagation to only one for each group of
adversary updates. Therefore, we refer to this algorithm YOPO (You Only
Propagate Once). Numerical experiments demonstrate that YOPO can achieve
comparable defense accuracy with approximately 1/5 ~ 1/4 GPU time of the
projected gradient descent (PGD) algorithm. Our codes are available at
https://https://github.com/a1600012888/YOPO-You-Only-Propagate-Once.


http://arxiv.org/abs/1906.03181
POBA-GA: Perturbation Optimized Black-Box Adversarial Attacks via Genetic Algorithm.
Jinyin Chen; Mengmeng Su; Shijing Shen; Hui Xiong; Haibin Zheng
  Most deep learning models are easily vulnerable to adversarial attacks.
Various adversarial attacks are designed to evaluate the robustness of models
and develop defense model. Currently, adversarial attacks are brought up to
attack their own target model with their own evaluation metrics. And most of
the black-box adversarial attack algorithms cannot achieve the expected success
rate compared with white-box attacks. In this paper, comprehensive evaluation
metrics are brought up for different adversarial attack methods. A novel
perturbation optimized black-box adversarial attack based on genetic algorithm
(POBA-GA) is proposed for achieving white-box comparable attack performances.
Approximate optimal adversarial examples are evolved through evolutionary
operations including initialization, selection, crossover and mutation. Fitness
function is specifically designed to evaluate the example individual in both
aspects of attack ability and perturbation control. Population diversity
strategy is brought up in evolutionary process to promise the approximate
optimal perturbations obtained. Comprehensive experiments are carried out to
testify POBA-GA's performances. Both simulation and application results prove
that our method is better than current state-of-art black-box attack methods in
aspects of attack capability and perturbation control.


http://arxiv.org/abs/1905.00441
NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks.
Yandong Li; Lijun Li; Liqiang Wang; Tong Zhang; Boqing Gong
  Powerful adversarial attack methods are vital for understanding how to
construct robust deep neural networks (DNNs) and for thoroughly testing defense
techniques. In this paper, we propose a black-box adversarial attack algorithm
that can defeat both vanilla DNNs and those generated by various defense
techniques developed recently. Instead of searching for an "optimal"
adversarial example for a benign input to a targeted DNN, our algorithm finds a
probability density distribution over a small region centered around the input,
such that a sample drawn from this distribution is likely an adversarial
example, without the need of accessing the DNN's internal layers or weights.
Our approach is universal as it can successfully attack different neural
networks by a single algorithm. It is also strong; according to the testing
against 2 vanilla DNNs and 13 defended ones, it outperforms state-of-the-art
black-box or white-box attack methods for most test cases. Additionally, our
results reveal that adversarial training remains one of the best defense
techniques, and the adversarial examples are not as transferable across
defended DNNs as them across vanilla DNNs.


http://arxiv.org/abs/1905.00180
Dropping Pixels for Adversarial Robustness.
Hossein Hosseini; Sreeram Kannan; Radha Poovendran
  Deep neural networks are vulnerable against adversarial examples. In this
paper, we propose to train and test the networks with randomly subsampled
images with high drop rates. We show that this approach significantly improves
robustness against adversarial examples in all cases of bounded L0, L2 and
L_inf perturbations, while reducing the standard accuracy by a small value. We
argue that subsampling pixels can be thought to provide a set of robust
features for the input image and, thus, improves robustness without performing
adversarial training.


http://arxiv.org/abs/1904.13195
Test Selection for Deep Learning Systems.
Wei Ma; Mike Papadakis; Anestis Tsakmalis; Maxime Cordy; Yves Le Traon
  Testing of deep learning models is challenging due to the excessive number
and complexity of computations involved. As a result, test data selection is
performed manually and in an ad hoc way. This raises the question of how we can
automatically select candidate test data to test deep learning models. Recent
research has focused on adapting test selection metrics from code-based
software testing (such as coverage) to deep learning. However, deep learning
models have different attributes from code such as spread of computations
across the entire network reflecting training data properties, balance of
neuron weights and redundancy (use of many more neurons than needed). Such
differences make code-based metrics inappropriate to select data that can
challenge the models (can trigger misclassification). We thus propose a set of
test selection metrics based on the notion of model uncertainty (model
confidence on specific inputs). Intuitively, the more uncertain we are about a
candidate sample, the more likely it is that this sample triggers a
misclassification. Similarly, the samples for which we are the most uncertain,
are the most informative and should be used to improve the model by retraining.
We evaluate these metrics on two widely-used image classification problems
involving real and artificial (adversarial) data. We show that
uncertainty-based metrics have a strong ability to select data that are
misclassified and lead to major improvement in classification accuracy during
retraining: up to 80% more gain than random selection and other
state-of-the-art metrics on one dataset and up to 29% on the other.


http://arxiv.org/abs/1904.13094
Detecting Adversarial Examples through Nonlinear Dimensionality Reduction.
Francesco Crecchi; Davide Bacciu; Battista Biggio
  Deep neural networks are vulnerable to adversarial examples, i.e.,
carefully-perturbed inputs aimed to mislead classification. This work proposes
a detection method based on combining non-linear dimensionality reduction and
density estimation techniques. Our empirical findings show that the proposed
approach is able to effectively detect adversarial examples crafted by
non-adaptive attackers, i.e., not specifically tuned to bypass the detection
method. Given our promising results, we plan to extend our analysis to adaptive
attackers in future work.


http://arxiv.org/abs/1904.13000
Adversarial Training and Robustness for Multiple Perturbations.
Florian Tramèr; Dan Boneh
  Defenses against adversarial examples, such as adversarial training, are
typically tailored to a single perturbation type (e.g., small
$\ell_\infty$-noise). For other perturbations, these defenses offer no
guarantees and, at times, even increase the model's vulnerability. Our aim is
to understand the reasons underlying this robustness trade-off, and to train
models that are simultaneously robust to multiple perturbation types. We prove
that a trade-off in robustness to different types of $\ell_p$-bounded and
spatial perturbations must exist in a natural and simple statistical setting.
We corroborate our formal analysis by demonstrating similar robustness
trade-offs on MNIST and CIFAR10. Building upon new multi-perturbation
adversarial training schemes, and a novel efficient attack for finding
$\ell_1$-bounded adversarial examples, we show that no model trained against
multiple attacks achieves robustness competitive with that of models trained on
each attack individually. In particular, we uncover a pernicious
gradient-masking phenomenon on MNIST, which causes adversarial training with
first-order $\ell_\infty, \ell_1$ and $\ell_2$ adversaries to achieve merely
$50\%$ accuracy. Our results question the viability and computational
scalability of extending adversarial robustness, and adversarial training, to
multiple perturbation types.


http://arxiv.org/abs/1904.12843
Adversarial Training for Free!.
Ali Shafahi; Mahyar Najibi; Amin Ghiasi; Zheng Xu; John Dickerson; Christoph Studer; Larry S. Davis; Gavin Taylor; Tom Goldstein
  Adversarial training, in which a network is trained on adversarial examples,
is one of the few defenses against adversarial attacks that withstands strong
attacks. Unfortunately, the high cost of generating strong adversarial examples
makes standard adversarial training impractical on large-scale problems like
ImageNet. We present an algorithm that eliminates the overhead cost of
generating adversarial examples by recycling the gradient information computed
when updating model parameters. Our "free" adversarial training algorithm
achieves state-of-the-art robustness on CIFAR-10 and CIFAR-100 datasets at
negligible additional cost compared to natural training, and can be 7 to 30
times faster than other strong adversarial training methods. Using a single
workstation with 4 P100 GPUs and 2 days of runtime, we can train a robust model
for the large-scale ImageNet classification task that maintains 40% accuracy
against PGD attacks.


http://arxiv.org/abs/1904.12181
Non-Local Context Encoder: Robust Biomedical Image Segmentation against Adversarial Attacks.
Xiang He; Sibei Yang; Guanbin Li?; Haofeng Li; Huiyou Chang; Yizhou Yu
  Recent progress in biomedical image segmentation based on deep convolutional
neural networks (CNNs) has drawn much attention. However, its vulnerability
towards adversarial samples cannot be overlooked. This paper is the first one
that discovers that all the CNN-based state-of-the-art biomedical image
segmentation models are sensitive to adversarial perturbations. This limits the
deployment of these methods in safety-critical biomedical fields. In this
paper, we discover that global spatial dependencies and global contextual
information in a biomedical image can be exploited to defend against
adversarial attacks. To this end, non-local context encoder (NLCE) is proposed
to model short- and long range spatial dependencies and encode global contexts
for strengthening feature activations by channel-wise attention. The NLCE
modules enhance the robustness and accuracy of the non-local context encoding
network (NLCEN), which learns robust enhanced pyramid feature representations
with NLCE modules, and then integrates the information across different levels.
Experiments on both lung and skin lesion segmentation datasets have
demonstrated that NLCEN outperforms any other state-of-the-art biomedical image
segmentation methods against adversarial attacks. In addition, NLCE modules can
be applied to improve the robustness of other CNN-based biomedical image
segmentation methods.


http://arxiv.org/abs/1904.11803
Robustness Verification of Support Vector Machines.
Francesco Ranzato; Marco Zanella
  We study the problem of formally verifying the robustness to adversarial
examples of support vector machines (SVMs), a major machine learning model for
classification and regression tasks. Following a recent stream of works on
formal robustness verification of (deep) neural networks, our approach relies
on a sound abstract version of a given SVM classifier to be used for checking
its robustness. This methodology is parametric on a given numerical abstraction
of real values and, analogously to the case of neural networks, needs neither
abstract least upper bounds nor widening operators on this abstraction. The
standard interval domain provides a simple instantiation of our abstraction
technique, which is enhanced with the domain of reduced affine forms, which is
an efficient abstraction of the zonotope abstract domain. This robustness
verification technique has been fully implemented and experimentally evaluated
on SVMs based on linear and nonlinear (polynomial and radial basis function)
kernels, which have been trained on the popular MNIST dataset of images and on
the recent and more challenging Fashion-MNIST dataset. The experimental results
of our prototype SVM robustness verifier appear to be encouraging: this
automated verification is fast, scalable and shows significantly high
percentages of provable robustness on the test set of MNIST, in particular
compared to the analogous provable robustness of neural networks.


http://arxiv.org/abs/1904.11042
Physical Adversarial Textures that Fool Visual Object Tracking.
Rey Reza Wiyatno; Anqi Xu
  We present a system for generating inconspicuous-looking textures that, when
displayed in the physical world as digital or printed posters, cause visual
object tracking systems to become confused. For instance, as a target being
tracked by a robot's camera moves in front of such a poster, our generated
texture makes the tracker lock onto it and allows the target to evade. This
work aims to fool seldom-targeted regression tasks, and in particular compares
diverse optimization strategies: non-targeted, targeted, and a new family of
guided adversarial losses. While we use the Expectation Over Transformation
(EOT) algorithm to generate physical adversaries that fool tracking models when
imaged under diverse conditions, we compare the impacts of different
conditioning variables, including viewpoint, lighting, and appearances, to find
practical attack setups with high resulting adversarial strength and
convergence speed. We further showcase textures optimized solely using
simulated scenes can confuse real-world tracking systems.


http://arxiv.org/abs/1904.10990
A Robust Approach for Securing Audio Classification Against Adversarial Attacks.
Mohammad Esmaeilpour; Patrick Cardinal; Alessandro Lameiras Koerich
  Adversarial audio attacks can be considered as a small perturbation
unperceptive to human ears that is intentionally added to the audio signal and
causes a machine learning model to make mistakes. This poses a security concern
about the safety of machine learning models since the adversarial attacks can
fool such models toward the wrong predictions. In this paper we first review
some strong adversarial attacks that may affect both audio signals and their 2D
representations and evaluate the resiliency of the most common machine learning
model, namely deep learning models and support vector machines (SVM) trained on
2D audio representations such as short time Fourier transform (STFT), discrete
wavelet transform (DWT) and cross recurrent plot (CRP) against several
state-of-the-art adversarial attacks. Next, we propose a novel approach based
on pre-processed DWT representation of audio signals and SVM to secure audio
systems against adversarial attacks. The proposed architecture has several
preprocessing modules for generating and enhancing spectrograms including
dimension reduction and smoothing. We extract features from small patches of
the spectrograms using speeded up robust feature (SURF) algorithm which are
further used to generate a codebook using the K-Means++ algorithm. Finally,
codewords are used to train a SVM on the codebook of the SURF-generated
vectors. All these steps yield to a novel approach for audio classification
that provides a good trade-off between accuracy and resilience. Experimental
results on three environmental sound datasets show the competitive performance
of proposed approach compared to the deep neural networks both in terms of
accuracy and robustness against strong adversarial attacks.


http://arxiv.org/abs/1904.10390
Minimizing Perceived Image Quality Loss Through Adversarial Attack Scoping.
Kostiantyn Khabarlak; Larysa Koriashkina
  Neural networks are now actively being used for computer vision tasks in
security critical areas such as robotics, face recognition, autonomous vehicles
yet their safety is under question after the discovery of adversarial attacks.
In this paper we develop simplified adversarial attack algorithms based on a
scoping idea, which enables execution of fast adversarial attacks that minimize
structural image quality (SSIM) loss, allows performing efficient transfer
attacks with low target inference network call count and opens a possibility of
an attack using pen-only drawings on a paper for the MNIST handwritten digit
dataset. The presented adversarial attack analysis and the idea of attack
scoping can be easily expanded to different datasets, thus making the paper's
results applicable to a wide range of practical tasks.


http://arxiv.org/abs/1904.10076
Using Videos to Evaluate Image Model Robustness.
Keren Gu; Brandon Yang; Jiquan Ngiam; Quoc Le; Jonathon Shlens
  Human visual systems are robust to a wide range of image transformations that
are challenging for artificial networks. We present the first study of image
model robustness to the minute transformations found across video frames, which
we term "natural robustness". Compared to previous studies on adversarial
examples and synthetic distortions, natural robustness captures a more diverse
set of common image transformations that occur in the natural environment. Our
study across a dozen model architectures shows that more accurate models are
more robust to natural transformations, and that robustness to synthetic color
distortions is a good proxy for natural robustness. In examining brittleness in
videos, we find that majority of the brittleness found in videos lies outside
the typical definition of adversarial examples (99.9\%). Finally, we
investigate training techniques to reduce brittleness and find that no single
technique systematically improves natural robustness across twelve tested
architectures.


http://arxiv.org/abs/1904.09804
blessing in disguise: Designing Robust Turing Test by Employing Algorithm Unrobustness.
Jiaming Zhang; Jitao Sang; Kaiyuan Xu; Shangxi Wu; Yongli Hu; Yanfeng Sun; Jian Yu
  Turing test was originally proposed to examine whether machine's behavior is
indistinguishable from a human. The most popular and practical Turing test is
CAPTCHA, which is to discriminate algorithm from human by offering
recognition-alike questions. The recent development of deep learning has
significantly advanced the capability of algorithm in solving CAPTCHA
questions, forcing CAPTCHA designers to increase question complexity. Instead
of designing questions difficult for both algorithm and human, this study
attempts to employ the limitations of algorithm to design robust CAPTCHA
questions easily solvable to human. Specifically, our data analysis observes
that human and algorithm demonstrates different vulnerability to visual
distortions: adversarial perturbation is significantly annoying to algorithm
yet friendly to human. We are motivated to employ adversarially perturbed
images for robust CAPTCHA design in the context of character-based questions.
Three modules of multi-target attack, ensemble adversarial training, and image
preprocessing differentiable approximation are proposed to address the
characteristics of character-based CAPTCHA cracking. Qualitative and
quantitative experimental results demonstrate the effectiveness of the proposed
solution. We hope this study can lead to the discussions around adversarial
attack/defense in CAPTCHA design and also inspire the future attempts in
employing algorithm limitation for practical usage.


http://arxiv.org/abs/1904.09633
Beyond Explainability: Leveraging Interpretability for Improved Adversarial Learning.
Devinder Kumar; Ibrahim Ben-Daya; Kanav Vats; Jeffery Feng; Graham Taylor and; Alexander Wong
  In this study, we propose the leveraging of interpretability for tasks beyond
purely the purpose of explainability. In particular, this study puts forward a
novel strategy for leveraging gradient-based interpretability in the realm of
adversarial examples, where we use insights gained to aid adversarial learning.
More specifically, we introduce the concept of spatially constrained one-pixel
adversarial perturbations, where we guide the learning of such adversarial
perturbations towards more susceptible areas identified via gradient-based
interpretability. Experimental results using different benchmark datasets show
that such a spatially constrained one-pixel adversarial perturbation strategy
can noticeably improve the speed of convergence as well as produce successful
attacks that were also visually difficult to perceive, thus illustrating an
effective use of interpretability methods for tasks outside of the purpose of
purely explainability.


http://arxiv.org/abs/1904.09433
Can Machine Learning Model with Static Features be Fooled: an Adversarial Machine Learning Approach.
Rahim Taheri; Reza Javidan; Mohammad Shojafar; Vinod P; Mauro Conti
  The widespread adoption of smartphones dramatically increases the risk of
attacks and the spread of mobile malware, especially on the Android platform.
Machine learning based solutions have been already used as a tool to supersede
signature based anti-malware systems. However, malware authors leverage
attributes from malicious and legitimate samples to estimate statistical
difference in-order to create adversarial examples. Hence, to evaluate the
vulnerability of machine learning algorithms in malware detection, we propose
five different attack scenarios to perturb malicious applications (apps). By
doing this, the classification algorithm inappropriately fits discriminant
function on the set of data points, eventually yielding a higher
misclassification rate. Further, to distinguish the adversarial examples from
benign samples, we propose two defense mechanisms to counter attacks. To
validate our attacks and solutions, we test our model on three different
benchmark datasets. We also test our methods using various classifier
algorithms and compare them with the state-of-the-art data poisoning method
using the Jacobian matrix. Promising results show that generated adversarial
samples can evade detection with a very high probability. Additionally, evasive
variants generated by our attacks models when used to harden the developed
anti-malware system improves the detection rate.


http://arxiv.org/abs/1904.09146
Salient Object Detection in the Deep Learning Era: An In-Depth Survey.
Wenguan Wang; Qiuxia Lai; Huazhu Fu; Jianbing Shen; Haibin Ling
  As an important problem in computer vision, salient object detection (SOD)
from images has been attracting an increasing amount of research effort over
the years. Recent advances in SOD, not surprisingly, are dominantly led by deep
learning-based solutions (named deep SOD) and reflected by hundreds of
published papers. To facilitate the in-depth understanding of deep SODs, in
this paper we provide a comprehensive survey covering various aspects ranging
from algorithm taxonomy to unsolved open issues. In particular, we first review
deep SOD algorithms from different perspectives including network architecture,
level of supervision, learning paradigm and object/instance level detection.
Following that, we summarize existing SOD evaluation datasets and metrics.
Then, we carefully compile a thorough benchmark results of SOD methods based on
previous work, and provide detailed analysis of the comparison results.
Moreover, we study the performance of SOD algorithms under different
attributes, which have been barely explored previously, by constructing a novel
SOD dataset with rich attribute annotations. We further analyze, for the first
time in the field, the robustness and transferability of deep SOD models w.r.t.
adversarial attacks. We also look into the influence of input perturbations,
and the generalization and hardness of existing SOD datasets. Finally, we
discuss several open issues and challenges of SOD, and point out possible
research directions in future. All the saliency prediction maps, our
constructed dataset with annotations, and codes for evaluation are made
publicly available at https://github.com/wenguanwang/SODsurvey.


http://arxiv.org/abs/1904.08653
Fooling automated surveillance cameras: adversarial patches to attack person detection.
Simen Thys; Ranst Wiebe Van; Toon Goedemé
  Adversarial attacks on machine learning models have seen increasing interest
in the past years. By making only subtle changes to the input of a
convolutional neural network, the output of the network can be swayed to output
a completely different result. The first attacks did this by changing pixel
values of an input image slightly to fool a classifier to output the wrong
class. Other approaches have tried to learn "patches" that can be applied to an
object to fool detectors and classifiers. Some of these approaches have also
shown that these attacks are feasible in the real-world, i.e. by modifying an
object and filming it with a video camera. However, all of these approaches
target classes that contain almost no intra-class variety (e.g. stop signs).
The known structure of the object is then used to generate an adversarial patch
on top of it.
  In this paper, we present an approach to generate adversarial patches to
targets with lots of intra-class variety, namely persons. The goal is to
generate a patch that is able successfully hide a person from a person
detector. An attack that could for instance be used maliciously to circumvent
surveillance systems, intruders can sneak around undetected by holding a small
cardboard plate in front of their body aimed towards the surveillance camera.
  From our results we can see that our system is able significantly lower the
accuracy of a person detector. Our approach also functions well in real-life
scenarios where the patch is filmed by a camera. To the best of our knowledge
we are the first to attempt this kind of attack on targets with a high level of
intra-class variety like persons.


http://arxiv.org/abs/1904.08554
Gotta Catch 'Em All: Using Concealed Trapdoors to Detect Adversarial Attacks on Neural Networks.
Shawn Shan; Emily Willson; Bolun Wang; Bo Li; Haitao Zheng; Ben Y. Zhao
  Deep neural networks are vulnerable to adversarial attacks. Numerous efforts
have focused on defenses that either try to patch `holes' in trained models or
try to make it difficult or costly to compute adversarial examples exploiting
these holes. In our work, we explore a counter-intuitive approach of
constructing "adversarial trapdoors. Unlike prior works that try to patch or
disguise vulnerable points in the manifold, we intentionally inject
`trapdoors,' artificial weaknesses in the manifold that attract optimized
perturbation into certain pre-embedded local optima. As a result, the
adversarial generation functions naturally gravitate towards our trapdoors,
producing adversarial examples that the model owner can recognize through a
known neuron activation signature.
  In this paper, we introduce trapdoors and describe an implementation of
trapdoors using similar strategies to backdoor/Trojan attacks. We show that by
proactively injecting trapdoors into the models (and extracting their neuron
activation signature), we can detect adversarial examples generated by the
state of the art attacks (Projected Gradient Descent, Optimization based CW,
and Elastic Net) with high detection success rate and negligible impact on
normal inputs. These results also generalize across multiple classification
domains (image recognition, face recognition and traffic sign recognition). We
explore different properties of trapdoors, and discuss potential
countermeasures (adaptive attacks) and mitigations.


http://arxiv.org/abs/1904.08516
ZK-GanDef: A GAN based Zero Knowledge Adversarial Training Defense for Neural Networks.
Guanxiong Liu; Issa Khalil; Abdallah Khreishah
  Neural Network classifiers have been used successfully in a wide range of
applications. However, their underlying assumption of attack free environment
has been defied by adversarial examples. Researchers tried to develop defenses;
however, existing approaches are still far from providing effective solutions
to this evolving problem. In this paper, we design a generative adversarial net
(GAN) based zero knowledge adversarial training defense, dubbed ZK-GanDef,
which does not consume adversarial examples during training. Therefore,
ZK-GanDef is not only efficient in training but also adaptive to new
adversarial examples. This advantage comes at the cost of small degradation in
test accuracy compared to full knowledge approaches. Our experiments show that
ZK-GanDef enhances test accuracy on adversarial examples by up-to 49.17%
compared to zero knowledge approaches. More importantly, its test accuracy is
close to that of the state-of-the-art full knowledge approaches (maximum
degradation of 8.46%), while taking much less training time.


http://arxiv.org/abs/1904.08489
Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers.
Ameya Joshi; Amitangshu Mukherjee; Soumik Sarkar; Chinmay Hegde
  Deep neural networks have been shown to exhibit an intriguing vulnerability
to adversarial input images corrupted with imperceptible perturbations.
However, the majority of adversarial attacks assume global, fine-grained
control over the image pixel space. In this paper, we consider a different
setting: what happens if the adversary could only alter specific attributes of
the input image? These would generate inputs that might be perceptibly
different, but still natural-looking and enough to fool a classifier. We
propose a novel approach to generate such `semantic' adversarial examples by
optimizing a particular adversarial loss over the range-space of a parametric
conditional generative model. We demonstrate implementations of our attacks on
binary classifiers trained on face images, and show that such natural-looking
semantic adversarial examples exist. We evaluate the effectiveness of our
attack on synthetic and real data, and present detailed comparisons with
existing attack methods. We supplement our empirical results with theoretical
bounds that demonstrate the existence of such parametric adversarial examples.


http://arxiv.org/abs/1904.08444
Defensive Quantization: When Efficiency Meets Robustness.
Ji Lin; Chuang Gan; Song Han
  Neural network quantization is becoming an industry standard to efficiently
deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and
FPGAs. However, we observe that the conventional quantization approaches are
vulnerable to adversarial attacks. This paper aims to raise people's awareness
about the security of the quantized models, and we designed a novel
quantization methodology to jointly optimize the efficiency and robustness of
deep learning models. We first conduct an empirical study to show that vanilla
quantization suffers more from adversarial attacks. We observe that the
inferior robustness comes from the error amplification effect, where the
quantization operation further enlarges the distance caused by amplified noise.
Then we propose a novel Defensive Quantization (DQ) method by controlling the
Lipschitz constant of the network during quantization, such that the magnitude
of the adversarial noise remains non-expansive during inference. Extensive
experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization
method can defend neural networks against adversarial examples, and even
achieves superior robustness than their full-precision counterparts while
maintaining the same hardware efficiency as vanilla quantization approaches. As
a by-product, DQ can also improve the accuracy of quantized models without
adversarial attack.


http://arxiv.org/abs/1904.08279
Interpreting Adversarial Examples with Attributes.
Sadaf Gulshad; Jan Hendrik Metzen; Arnold Smeulders; Zeynep Akata
  Deep computer vision systems being vulnerable to imperceptible and carefully
crafted noise have raised questions regarding the robustness of their
decisions. We take a step back and approach this problem from an orthogonal
direction. We propose to enable black-box neural networks to justify their
reasoning both for clean and for adversarial examples by leveraging attributes,
i.e. visually discriminative properties of objects. We rank attributes based on
their class relevance, i.e. how the classification decision changes when the
input is visually slightly perturbed, as well as image relevance, i.e. how well
the attributes can be localized on both clean and perturbed images. We present
comprehensive experiments for attribute prediction, adversarial example
generation, adversarially robust learning, and their qualitative and
quantitative analysis using predicted attributes on three benchmark datasets.


http://arxiv.org/abs/1904.08089
Adversarial Defense Through Network Profiling Based Path Extraction.
Yuxian Qiu; Jingwen Leng; Cong Guo; Quan Chen; Chao Li; Minyi Guo; Yuhao Zhu
  Recently, researchers have started decomposing deep neural network models
according to their semantics or functions. Recent work has shown the
effectiveness of decomposed functional blocks for defending adversarial
attacks, which add small input perturbation to the input image to fool the DNN
models. This work proposes a profiling-based method to decompose the DNN models
to different functional blocks, which lead to the effective path as a new
approach to exploring DNNs' internal organization. Specifically, the per-image
effective path can be aggregated to the class-level effective path, through
which we observe that adversarial images activate effective path different from
normal images. We propose an effective path similarity-based method to detect
adversarial images with an interpretable model, which achieve better accuracy
and broader applicability than the state-of-the-art technique.


http://arxiv.org/abs/1904.07980
Reducing Adversarial Example Transferability Using Gradient Regularization.
George Adam; Petr Smirnov; Benjamin Haibe-Kains; Anna Goldenberg
  Deep learning algorithms have increasingly been shown to lack robustness to
simple adversarial examples (AdvX). An equally troubling observation is that
these adversarial examples transfer between different architectures trained on
different datasets. We investigate the transferability of adversarial examples
between models using the angle between the input-output Jacobians of different
models. To demonstrate the relevance of this approach, we perform case studies
that involve jointly training pairs of models. These case studies empirically
justify the theoretical intuitions for why the angle between gradients is a
fundamental quantity in AdvX transferability. Furthermore, we consider the
asymmetry of AdvX transferability between two models of the same architecture
and explain it in terms of differences in gradient norms between the models.
Lastly, we provide a simple modification to existing training setups that
reduces transferability of adversarial examples between pairs of models.


http://arxiv.org/abs/1904.07793
AT-GAN: A Generative Attack Model for Adversarial Transferring on Generative Adversarial Nets.
Xiaosen Wang; Kun He; Chuan Guo; Kilian Q. Weinberger; John E. Hopcroft
  Recent studies have discovered the vulnerability of Deep Neural Networks
(DNNs) to adversarial examples, which are imperceptible to humans but can
easily fool DNNs. Existing methods for crafting adversarial examples are mainly
based on adding small-magnitude perturbations to the original images so that
the generated adversarial examples are constrained by the benign examples
within a small matrix norm. In this work, we propose a new attack method called
AT-GAN that directly generates the adversarial examples from random noise using
generative adversarial nets (GANs). The key idea is to transfer a pre-trained
GAN to generate adversarial examples for the target classifier to be attacked.
Once the model is transferred for attack, AT-GAN can generate diverse
adversarial examples efficiently, making it helpful to potentially accelerate
the adversarial training on defenses. We evaluate AT-GAN in both semi-whitebox
and black-box settings under typical defense methods on the MNIST handwritten
digit database. Empirical comparisons with existing attack baselines
demonstrate that AT-GAN can achieve a higher attack success rate.


http://arxiv.org/abs/1904.07370
Are Self-Driving Cars Secure? Evasion Attacks against Deep Neural Networks for Steering Angle Prediction.
Alesia Chernikova; Alina Oprea; Cristina Nita-Rotaru; BaekGyu Kim
  Deep Neural Networks (DNNs) have tremendous potential in advancing the vision
for self-driving cars. However, the security of DNN models in this context
leads to major safety implications and needs to be better understood. We
consider the case study of steering angle prediction from camera images, using
the dataset from the 2014 Udacity challenge. We demonstrate for the first time
adversarial testing-time attacks for this application for both classification
and regression settings. We show that minor modifications to the camera image
(an L2 distance of 0.82 for one of the considered models) result in
mis-classification of an image to any class of attacker's choice. Furthermore,
our regression attack results in a significant increase in Mean Square Error
(MSE) by a factor of 69 in the worst case.


http://arxiv.org/abs/1904.06964
Influence of Control Parameters and the Size of Biomedical Image Datasets on the Success of Adversarial Attacks.
Vassili Kovalev; Dmitry Voynov
  In this paper, we study dependence of the success rate of adversarial attacks
to the Deep Neural Networks on the biomedical image type, control parameters,
and image dataset size. With this work, we are going to contribute towards
accumulation of experimental results on adversarial attacks for the community
dealing with biomedical images. The white-box Projected Gradient Descent
attacks were examined based on 8 classification tasks and 13 image datasets
containing a total of 605,080 chest X-ray and 317,000 histology images of
malignant tumors. We concluded that: (1) An increase of the amplitude of
perturbation in generating malicious adversarial images leads to a growth of
the fraction of successful attacks for the majority of image types examined in
this study. (2) Histology images tend to be less sensitive to the growth of
amplitude of adversarial perturbations. (3) Percentage of successful attacks is
growing with an increase of the number of iterations of the algorithm of
generating adversarial perturbations with an asymptotic stabilization. (4) It
was found that the success of attacks dropping dramatically when the original
confidence of predicting image class exceeds 0.95. (5) The expected dependence
of the percentage of successful attacks on the size of image training set was
not confirmed.


http://arxiv.org/abs/1904.06606
Exploiting Vulnerabilities of Load Forecasting Through Adversarial Attacks.
Yize Chen; Yushi Tan; Baosen Zhang
  Load forecasting plays a critical role in the operation and planning of power
systems. By using input features such as historical loads and weather
forecasts, system operators and utilities build forecast models to guide
decision making in commitment and dispatch. As the forecasting techniques
becomes more sophisticated, however, they also become more vulnerable to
cybersecurity threats. In this paper, we study the vulnerability of a class of
load forecasting algorithms and analyze the potential impact on the power
system operations, such as load shedding and increased dispatch costs.
Specifically, we propose data injection attack algorithms that require minimal
assumptions on the ability of the adversary. The attacker does not need to have
knowledge about the load forecasting model or the underlying power system.
Surprisingly, our results indicate that standard load forecasting algorithms
are quite vulnerable to the designed black-box attacks. By only injecting
malicious data in temperature from online weather forecast APIs, an attacker
could manipulate load forecasts in arbitrary directions and cause significant
and targeted damages to system operations.


http://arxiv.org/abs/1904.06347
Big but Imperceptible Adversarial Perturbations via Semantic Manipulation.
Anand Bhattad; Min Jin Chong; Kaizhao Liang; Bo Li; David A. Forsyth
  Machine learning, especially deep learning, is widely applied to a range of
applications including computer vision, robotics and natural language
processing. However, it has been shown that machine learning models are
vulnerable to adversarial examples, carefully crafted samples that deceive
learning models. In-depth studies on adversarial examples can help better
understand potential vulnerabilities and therefore improve model robustness.
Recent works have introduced various methods which generate adversarial
examples. However, all require the perturbation to be of small magnitude
($\mathcal{L}_p$ norm) for them to be imperceptible to humans, which is hard to
deploy in practice. In this paper we propose two novel methods, tAdv and cAdv,
which leverage texture transfer and colorization to generate natural
perturbation with a large $\mathcal{L}_p$ norm. We conduct extensive
experiments to show that the proposed methods are general enough to attack both
image classification and image captioning tasks on ImageNet and MSCOCO dataset.
In addition, we conduct comprehensive user studies under various conditions to
show that our generated adversarial examples are imperceptible to humans even
when the perturbations are large. We also evaluate the transferability and
robustness of the proposed attacks against several state-of-the-art defenses.


http://arxiv.org/abs/1904.06097
Evaluating Robustness of Deep Image Super-Resolution against Adversarial Attacks.
Jun-Ho Choi; Huan Zhang; Jun-Hyuk Kim; Cho-Jui Hsieh; Jong-Seok Lee
  Single-image super-resolution aims to generate a high-resolution version of a
low-resolution image, which serves as an essential component in many computer
vision applications. This paper investigates the robustness of deep
learning-based super-resolution methods against adversarial attacks, which can
significantly deteriorate the super-resolved images without noticeable
distortion in the attacked low-resolution images. It is demonstrated that
state-of-the-art deep super-resolution methods are highly vulnerable to
adversarial attacks. Different levels of robustness of different methods are
analyzed theoretically and experimentally. We also present analysis on
transferability of attacks, and feasibility of targeted attacks and universal
attacks.


http://arxiv.org/abs/1904.06026
Cycle-Consistent Adversarial GAN: the integration of adversarial attack and defense.
Lingyun Jiang; Kai Qiao; Ruoxi Qin; Linyuan Wang; Jian Chen; Haibing Bu; Bin Yan
  In image classification of deep learning, adversarial examples where inputs
intended to add small magnitude perturbations may mislead deep neural networks
(DNNs) to incorrect results, which means DNNs are vulnerable to them. Different
attack and defense strategies have been proposed to better research the
mechanism of deep learning. However, those research in these networks are only
for one aspect, either an attack or a defense, not considering that attacks and
defenses should be interdependent and mutually reinforcing, just like the
relationship between spears and shields. In this paper, we propose
Cycle-Consistent Adversarial GAN (CycleAdvGAN) to generate adversarial
examples, which can learn and approximate the distribution of original
instances and adversarial examples. For CycleAdvGAN, once the Generator and are
trained, can generate adversarial perturbations efficiently for any instance,
so as to make DNNs predict wrong, and recovery adversarial examples to clean
instances, so as to make DNNs predict correct. We apply CycleAdvGAN under
semi-white box and black-box settings on two public datasets MNIST and CIFAR10.
Using the extensive experiments, we show that our method has achieved the
state-of-the-art adversarial attack method and also efficiently improve the
defense ability, which make the integration of adversarial attack and defense
come true. In additional, it has improved attack effect only trained on the
adversarial dataset generated by any kind of adversarial attack.


http://arxiv.org/abs/1904.06292
Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks.
David J. Miller; Zhen Xiang; George Kesidis
  There is great potential for damage from adversarial learning (AL) attacks on
machine-learning based systems. In this paper, we provide a contemporary survey
of AL, focused particularly on defenses against attacks on statistical
classifiers. After introducing relevant terminology and the goals and range of
possible knowledge of both attackers and defenders, we survey recent work on
test-time evasion (TTE), data poisoning (DP), and reverse engineering (RE)
attacks and particularly defenses against same. In so doing, we distinguish
robust classification from anomaly detection (AD), unsupervised from
supervised, and statistical hypothesis-based defenses from ones that do not
have an explicit null (no attack) hypothesis; we identify the hyperparameters a
particular method requires, its computational complexity, as well as the
performance measures on which it was evaluated and the obtained quality. We
then dig deeper, providing novel insights that challenge conventional AL wisdom
and that target unresolved issues, including: 1) robust classification versus
AD as a defense strategy; 2) the belief that attack success increases with
attack strength, which ignores susceptibility to AD; 3) small perturbations for
test-time evasion attacks: a fallacy or a requirement?; 4) validity of the
universal assumption that a TTE attacker knows the ground-truth class for the
example to be attacked; 5) black, grey, or white box attacks as the standard
for defense evaluation; 6) susceptibility of query-based RE to an AD defense.
We also discuss attacks on the privacy of training data. We then present
benchmark comparisons of several defenses against TTE, RE, and backdoor DP
attacks on images. The paper concludes with a discussion of future work.


http://arxiv.org/abs/1904.06186
Generating Minimal Adversarial Perturbations with Integrated Adaptive Gradients.
Yatie Xiao; Chi-Man Pun
  Deep neural networks are easily fooled high confidence predictions for
adversarial samples


http://arxiv.org/abs/1904.05586
Black-Box Decision based Adversarial Attack with Symmetric $\alpha$-stable Distribution.
Vignesh Srinivasan; Ercan E. Kuruoglu; Klaus-Robert Müller; Wojciech Samek; Shinichi Nakajima
  Developing techniques for adversarial attack and defense is an important
research field for establishing reliable machine learning and its applications.
Many existing methods employ Gaussian random variables for exploring the data
space to find the most adversarial (for attacking) or least adversarial (for
defense) point. However, the Gaussian distribution is not necessarily the
optimal choice when the exploration is required to follow the complicated
structure that most real-world data distributions exhibit. In this paper, we
investigate how statistics of random variables affect such random walk
exploration. Specifically, we generalize the Boundary Attack, a
state-of-the-art black-box decision based attacking strategy, and propose the
L\'evy-Attack, where the random walk is driven by symmetric $\alpha$-stable
random variables. Our experiments on MNIST and CIFAR10 datasets show that the
L\'evy-Attack explores the image data space more efficiently, and significantly
improves the performance. Our results also give an insight into the recently
found fact in the whitebox attacking scenario that the choice of the norm for
measuring the amplitude of the adversarial patterns is essential.


http://arxiv.org/abs/1904.05475
Learning to Generate Synthetic Data via Compositing.
Shashank Tripathi; Siddhartha Chandra; Amit Agrawal; Ambrish Tyagi; James M. Rehg; Visesh Chari
  We present a task-aware approach to synthetic data generation. Our framework
employs a trainable synthesizer network that is optimized to produce meaningful
training samples by assessing the strengths and weaknesses of a `target'
network. The synthesizer and target networks are trained in an adversarial
manner wherein each network is updated with a goal to outdo the other.
Additionally, we ensure the synthesizer generates realistic data by pairing it
with a discriminator trained on real-world images. Further, to make the target
classifier invariant to blending artefacts, we introduce these artefacts to
background regions of the training images so the target does not over-fit to
them.
  We demonstrate the efficacy of our approach by applying it to different
target networks including a classification network on AffNIST, and two object
detection networks (SSD, Faster-RCNN) on different datasets. On the AffNIST
benchmark, our approach is able to surpass the baseline results with just half
the training examples. On the VOC person detection benchmark, we show
improvements of up to 2.7% as a result of our data augmentation. Similarly on
the GMU detection benchmark, we report a performance boost of 3.5% in mAP over
the baseline method, outperforming the previous state of the art approaches by
up to 7.5% on specific categories.


http://arxiv.org/abs/1904.05181
Black-box Adversarial Attacks on Video Recognition Models.
Linxi Jiang; Xingjun Ma; Shaoxiang Chen; James Bailey; Yu-Gang Jiang
  Deep neural networks (DNNs) are known for their vulnerability to adversarial
examples. These are examples that have undergone small, carefully crafted
perturbations, and which can easily fool a DNN into making misclassifications
at test time. Thus far, the field of adversarial research has mainly focused on
image models, under either a white-box setting, where an adversary has full
access to model parameters, or a black-box setting where an adversary can only
query the target model for probabilities or labels. Whilst several white-box
attacks have been proposed for video models, black-box video attacks are still
unexplored. To close this gap, we propose the first black-box video attack
framework, called V-BAD. V-BAD utilizes tentative perturbations transferred
from image models, and partition-based rectifications found by the NES on
partitions (patches) of tentative perturbations, to obtain good adversarial
gradient estimates with fewer queries to the target model. V-BAD is equivalent
to estimating the projection of an adversarial gradient on a selected subspace.
Using three benchmark video datasets, we demonstrate that V-BAD can craft both
untargeted and targeted attacks to fool two state-of-the-art deep video
recognition models. For the targeted attack, it achieves $>$93\% success rate
using only an average of $3.4 \sim 8.4 \times 10^4$ queries, a similar number
of queries to state-of-the-art black-box image attacks. This is despite the
fact that videos often have two orders of magnitude higher dimensionality than
static images. We believe that V-BAD is a promising new tool to evaluate and
improve the robustness of video recognition models to black-box adversarial
attacks.


http://arxiv.org/abs/1904.04802
Short Paper: Creating Adversarial Malware Examples using Code Insertion.
Daniel Park; Haidar Khan; Bülent Yener
  There has been an increased interest in the application of convolutional
neural networks for image based malware classification, but the susceptibility
of neural networks to adversarial examples allows malicious actors to evade
classifiers. We shed light on the definition of an adversarial example in the
malware domain. Then, we propose a method to obfuscate malware using patterns
found in adversarial examples such that the newly obfuscated malware evades
classification while maintaining executability and the original program logic.


http://arxiv.org/abs/1904.04433
Efficient Decision-based Black-box Adversarial Attacks on Face Recognition.
Yinpeng Dong; Hang Su; Baoyuan Wu; Zhifeng Li; Wei Liu; Tong Zhang; Jun Zhu
  Face recognition has obtained remarkable progress in recent years due to the
great improvement of deep convolutional neural networks (CNNs). However, deep
CNNs are vulnerable to adversarial examples, which can cause fateful
consequences in real-world face recognition applications with
security-sensitive purposes. Adversarial attacks are widely studied as they can
identify the vulnerability of the models before they are deployed. In this
paper, we evaluate the robustness of state-of-the-art face recognition models
in the decision-based black-box attack setting, where the attackers have no
access to the model parameters and gradients, but can only acquire hard-label
predictions by sending queries to the target model. This attack setting is more
practical in real-world face recognition systems. To improve the efficiency of
previous methods, we propose an evolutionary attack algorithm, which can model
the local geometries of the search directions and reduce the dimension of the
search space. Extensive experiments demonstrate the effectiveness of the
proposed method that induces a minimum perturbation to an input face image with
fewer queries. We also apply the proposed method to attack a real-world face
recognition system successfully.


http://arxiv.org/abs/1904.04334
A Target-Agnostic Attack on Deep Models: Exploiting Security Vulnerabilities of Transfer Learning.
Shahbaz Rezaei; Xin Liu
  Due to the lack of enough training data and high computational cost to train
a deep neural network from scratch, transfer learning has been extensively used
in many deep-neural-network-based applications, such as face recognition, image
classification, speech recognition, etc. A commonly-used transfer learning
approach involves taking a part of a pre-trained model, adding a few layers at
the end, and re-training the new layers with a small dataset. This approach,
while efficient and widely used, imposes a security vulnerability because the
pre-trained models used in transfer learning are usually available publicly to
everyone, including potential attackers. In this paper, we show that without
any additional knowledge other than the pre-trained model, an attacker can
launch an effective and efficient brute force attack that can craft instances
of input to trigger each target class with high confidence. Note that we assume
that the attacker does not have access to any target-specific information,
including samples from target classes, re-trained model, and probabilities
assigned by Softmax to each class, and thus called target-agnostic attack.
These assumptions render all previous attacks impractical, to the best of our
knowledge. To evaluate the proposed attack, we perform a set of experiments on
face recognition and speech recognition tasks and show the effectiveness of the
attack. Our work sheds light on a fundamental security challenge of transfer
learning in deep neural networks.


http://arxiv.org/abs/1904.03750
JumpReLU: A Retrofit Defense Strategy for Adversarial Attacks.
N. Benjamin Erichson; Zhewei Yao; Michael W. Mahoney
  It has been demonstrated that very simple attacks can fool
highly-sophisticated neural network architectures. In particular, so-called
adversarial examples, constructed from perturbations of input data that are
small or imperceptible to humans but lead to different predictions, may lead to
an enormous risk in certain critical applications. In light of this, there has
been a great deal of work on developing adversarial training strategies to
improve model robustness. These training strategies are very expensive, in both
human and computational time. To complement these approaches, we propose a very
simple and inexpensive strategy which can be used to ``retrofit'' a
previously-trained network to improve its resilience to adversarial attacks.
More concretely, we propose a new activation function---the JumpReLU---which,
when used in place of a ReLU in an already-trained model, leads to a trade-off
between predictive accuracy and robustness. This trade-off is controlled by the
jump size, a hyper-parameter which can be tuned during the validation stage.
Our empirical results demonstrate that this increases model robustness,
protecting against adversarial attacks with substantially increased levels of
perturbations. This is accomplished simply by retrofitting existing networks
with our JumpReLU activation function, without the need for retraining the
model. Additionally, we demonstrate that adversarially trained (robust) models
can greatly benefit from retrofitting.


http://arxiv.org/abs/1904.05747
Malware Evasion Attack and Defense.
Yonghong Huang; Utkarsh Verma; Celeste Fralick; Gabriel Infante-Lopez; Brajesh Kumarz; Carl Woodward
  Machine learning (ML) classifiers are vulnerable to adversarial examples. An
adversarial example is an input sample which is slightly modified to induce
misclassification in an ML classifier. In this work, we investigate white-box
and grey-box evasion attacks to an ML-based malware detector and conduct
performance evaluations in a real-world setting. We compare the defense
approaches in mitigating the attacks. We propose a framework for deploying
grey-box and black-box attacks to malware detection systems.


http://arxiv.org/abs/1904.03542
On Training Robust PDF Malware Classifiers.
Yizheng Chen; Shiqi Wang; Dongdong She; Suman Jana
  Although state-of-the-art PDF malware classifiers can be trained with almost
perfect test accuracy (99%) and extremely low false positive rate (under 0.1%),
it has been shown that even a simple adversary can evade them. A practically
useful malware classifier must be robust against evasion attacks. However,
achieving such robustness is an extremely challenging task.
  In this paper, we take the first steps towards training robust PDF malware
classifiers with verifiable robustness properties. For instance, a robustness
property can enforce that no matter how many pages from benign documents are
inserted into a PDF malware, the classifier must still classify it as
malicious. We demonstrate how the worst-case behavior of a malware classifier
with respect to specific robustness properties can be formally verified.
Furthermore, we find that training classifiers that satisfy formally verified
robustness properties can increase the computation cost of unbounded (i.e., not
bounded by the robustness properties) attackers by eliminating simple evasion
attacks.
  Specifically, we propose a new distance metric that operates on the PDF tree
structure and specify two classes of robustness properties including subtree
insertions and deletions. We utilize state-of-the-art verifiably robust
training method to build robust PDF malware classifiers. Our results show that,
we can achieve 99% verified robust accuracy, while maintaining 99.80% accuracy
and 0.41% false positive rate. With simple robustness properties, the
state-of-the-art unbounded attacker found no successful evasion on the robust
classifier in 6 hours. Even for a new unbounded adaptive attacker we have
designed, the number of successful evasions within a fixed time budget is cut
down by 4x.


http://arxiv.org/abs/1904.02884
Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks.
Yinpeng Dong; Tianyu Pang; Hang Su; Jun Zhu
  Deep neural networks are vulnerable to adversarial examples, which can
mislead classifiers by adding imperceptible perturbations. An intriguing
property of adversarial examples is their good transferability, making
black-box attacks feasible in real-world applications. Due to the threat of
adversarial attacks, many methods have been proposed to improve the robustness.
Several state-of-the-art defenses are shown to be robust against transferable
adversarial examples. In this paper, we propose a translation-invariant attack
method to generate more transferable adversarial examples against the defense
models. By optimizing a perturbation over an ensemble of translated images, the
generated adversarial example is less sensitive to the white-box model being
attacked and has better transferability. To improve the efficiency of attacks,
we further show that our method can be implemented by convolving the gradient
at the untranslated image with a pre-defined kernel. Our method is generally
applicable to any gradient-based attack method. Extensive experiments on the
ImageNet dataset validate the effectiveness of the proposed method. Our best
attack fools eight state-of-the-art defenses at an 82% success rate on average
based only on the transferability, demonstrating the insecurity of the current
defense techniques.


http://arxiv.org/abs/1904.02841
Minimum Uncertainty Based Detection of Adversaries in Deep Neural Networks.
Fatemeh Sheikholeslami; Swayambhoo Jain; Georgios B. Giannakis
  Despite their unprecedented performance in various domains, utilization of
Deep Neural Networks (DNNs) in safety-critical environments is severely limited
in the presence of even small adversarial perturbations. The present work
develops a randomized approach to detecting such perturbations based on minimum
uncertainty metrics that rely on sampling at the hidden layers during the DNN
inference stage. The sampling probabilities are designed for effective
detection of the adversarially corrupted inputs. Being modular, the novel
detector of adversaries can be conveniently employed by any pre-trained DNN at
no extra training overhead. Selecting which units to sample per hidden layer
entails quantifying the amount of DNN output uncertainty from the viewpoint of
Bayesian neural networks, where the overall uncertainty is expressed in terms
of its layer-wise components - what also promotes scalability. Sampling
probabilities are then sought by minimizing uncertainty measures
layer-by-layer, leading to a novel convex optimization problem that admits an
exact solver with superlinear convergence rate. By simplifying the objective
function, low-complexity approximate solvers are also developed. In addition to
valuable insights, these approximations link the novel approach with
state-of-the-art randomized adversarial detectors. The effectiveness of the
novel detectors in the context of competing alternatives is highlighted through
extensive tests for various types of adversarial attacks with variable levels
of strength.


http://arxiv.org/abs/1904.02405
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks.
Yotam Gil; Yoav Chai; Or Gorodissky; Jonathan Berant
  Adversarial examples are important for understanding the behavior of neural
models, and can improve their robustness through adversarial training. Recent
work in natural language processing generated adversarial examples by assuming
white-box access to the attacked model, and optimizing the input directly
against it (Ebrahimi et al., 2018). In this work, we show that the knowledge
implicit in the optimization procedure can be distilled into another more
efficient neural network. We train a model to emulate the behavior of a
white-box attack and show that it generalizes well across examples. Moreover,
it reduces adversarial example generation time by 19x-39x. We also show that
our approach transfers to a black-box setting, by attacking The Google
Perspective API and exposing its vulnerability. Our attack flips the
API-predicted label in 42\% of the generated examples, while humans maintain
high-accuracy in predicting the gold label.


http://arxiv.org/abs/1904.02057
Interpreting Adversarial Examples by Activation Promotion and Suppression.
Kaidi Xu; Sijia Liu; Gaoyuan Zhang; Mengshu Sun; Pu Zhao; Quanfu Fan; Chuang Gan; Xue Lin
  It is widely known that convolutional neural networks (CNNs) are vulnerable
to adversarial examples: crafted images with imperceptible perturbations.
However, interpretability of these perturbations is less explored in the
literature. This work aims to better understand the roles of adversarial
perturbations and provide visual explanations from pixel, image and network
perspectives. We show that adversaries make a promotion and suppression effect
(PSE) on neurons' activation and can be primarily categorized into three types:
1)suppression-dominated perturbations that mainly reduce the classification
score of the true label, 2)promotion-dominated perturbations that focus on
boosting the confidence of the target label, and 3)balanced perturbations that
play a dual role on suppression and promotion. Further, we provide the
image-level interpretability of adversarial examples, which links PSE of
pixel-level perturbations to class-specific discriminative image regions
localized by class activation mapping. Lastly, we analyze the effect of
adversarial examples through network dissection, which offers concept-level
interpretability of hidden units. We show that there exists a tight connection
between the sensitivity (against attacks) of internal response of units with
their interpretability on semantic concepts.


http://arxiv.org/abs/1904.10504
Understanding the efficacy, reliability and resiliency of computer vision techniques for malware detection and future research directions.
Li Chen
  My research lies in the intersection of security and machine learning. This
overview summarizes one component of my research: combining computer vision
with malware exploit detection for enhanced security solutions. I will present
the perspectives of efficacy, reliability and resiliency to formulate threat
detection as computer vision problems and develop state-of-the-art image-based
malware classification. Representing malware binary as images provides a direct
visualization of data samples, reduces the efforts for feature extraction, and
consumes the whole binary for holistic structural analysis. Employing transfer
learning of deep neural networks effective for large scale image classification
to malware classification demonstrates superior classification efficacy
compared with classical machine learning algorithms. To enhance reliability of
these vision-based malware detectors, interpretation frameworks can be
constructed on the malware visual representations and useful for extracting
faithful explanation, so that security practitioners have confidence in the
model before deployment. In cyber-security applications, we should always
assume that a malware writer constantly modifies code to bypass detection.
Addressing the resiliency of the malware detectors is equivalently important as
efficacy and reliability. Via understanding the attack surfaces of machine
learning models used for malware detection, we can greatly improve the
robustness of the algorithms to combat malware adversaries in the wild. Finally
I will discuss future research directions worth pursuing in this research
community.


http://arxiv.org/abs/1904.02144
HopSkipJumpAttack: A Query-Efficient Decision-Based Attack.
Jianbo Chen; Michael I. Jordan; Martin J. Wainwright
  The goal of a decision-based adversarial attack on a trained model is to
generate adversarial examples based solely on observing output labels returned
by the targeted model. We develop HopSkipJumpAttack, a family of algorithms
based on a novel estimate of the gradient direction using binary information at
the decision boundary. The proposed family includes both untargeted and
targeted attacks optimized for $\ell_2$ and $\ell_\infty$ similarity metrics
respectively. Theoretical analysis is provided for the proposed algorithms and
the gradient direction estimate. Experiments show HopSkipJumpAttack requires
significantly fewer model queries than Boundary Attack, a powerful existing
decision-based attack. It also achieves competitive performance in attacking
adversarially trained models on MNIST. (HopSkipJumpAttack was named Boundary
Attack++ in a previous version of the preprint.)


http://arxiv.org/abs/1904.01231
Adversarial Attacks against Deep Saliency Models.
Zhaohui Che; Ali Borji; Guangtao Zhai; Suiyi Ling; Guodong Guo; Patrick Le Callet
  Currently, a plethora of saliency models based on deep neural networks have
led great breakthroughs in many complex high-level vision tasks (e.g. scene
description, object detection). The robustness of these models, however, has
not yet been studied. In this paper, we propose a sparse feature-space
adversarial attack method against deep saliency models for the first time. The
proposed attack only requires a part of the model information, and is able to
generate a sparser and more insidious adversarial perturbation, compared to
traditional image-space attacks. These adversarial perturbations are so subtle
that a human observer cannot notice their presences, but the model outputs will
be revolutionized. This phenomenon raises security threats to deep saliency
models in practical applications. We also explore some intriguing properties of
the feature-space attack, e.g. 1) the hidden layers with bigger receptive
fields generate sparser perturbations, 2) the deeper hidden layers achieve
higher attack success rates, and 3) different loss functions and different
attacked layers will result in diverse perturbations. Experiments indicate that
the proposed method is able to successfully attack different model
architectures across various image scenes.


http://arxiv.org/abs/1904.01160
Curls & Whey: Boosting Black-Box Adversarial Attacks.
Yucheng Shi; Siyu Wang; Yahong Han
  Image classifiers based on deep neural networks suffer from harassment caused
by adversarial examples. Two defects exist in black-box iterative attacks that
generate adversarial examples by incrementally adjusting the noise-adding
direction for each step. On the one hand, existing iterative attacks add noises
monotonically along the direction of gradient ascent, resulting in a lack of
diversity and adaptability of the generated iterative trajectories. On the
other hand, it is trivial to perform adversarial attack by adding excessive
noises, but currently there is no refinement mechanism to squeeze redundant
noises. In this work, we propose Curls & Whey black-box attack to fix the above
two defects. During Curls iteration, by combining gradient ascent and descent,
we `curl' up iterative trajectories to integrate more diversity and
transferability into adversarial examples. Curls iteration also alleviates the
diminishing marginal effect in existing iterative attacks. The Whey
optimization further squeezes the `whey' of noises by exploiting the robustness
of adversarial perturbation. Extensive experiments on Imagenet and
Tiny-Imagenet demonstrate that our approach achieves impressive decrease on
noise magnitude in l2 norm. Curls & Whey attack also shows promising
transferability against ensemble models as well as adversarially trained
models. In addition, we extend our attack to the targeted misclassification,
effectively reducing the difficulty of targeted attacks under black-box
condition.


http://arxiv.org/abs/1904.00979
Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses.
Yingwei Li; Song Bai; Cihang Xie; Zhenyu Liao; Xiaohui Shen; Alan L. Yuille
  This paper focuses on learning transferable adversarial examples specifically
against defense models (models to defense adversarial attacks). In particular,
we show that a simple universal perturbation can fool a series of
state-of-the-art defenses.
  Adversarial examples generated by existing attacks are generally hard to
transfer to defense models. We observe the property of regional homogeneity in
adversarial perturbations and suggest that the defenses are less robust to
regionally homogeneous perturbations. Therefore, we propose an effective
transforming paradigm and a customized gradient transformer module to transform
existing perturbations into regionally homogeneous ones. Without explicitly
forcing the perturbations to be universal, we observe that a well-trained
gradient transformer module tends to output input-independent gradients (hence
universal) benefiting from the under-fitting phenomenon. Thorough experiments
demonstrate that our work significantly outperforms the prior art attacking
algorithms (either image-dependent or universal ones) by an average improvement
of 14.0% when attacking 9 defenses in the black-box setting. In addition to the
cross-model transferability, we also verify that regionally homogeneous
perturbations can well transfer across different vision tasks (attacking with
the semantic segmentation task and testing on the object detection task).


http://arxiv.org/abs/1904.00923
Robustness of 3D Deep Learning in an Adversarial Setting.
Matthew Wicker; Marta Kwiatkowska
  Understanding the spatial arrangement and nature of real-world objects is of
paramount importance to many complex engineering tasks, including autonomous
navigation. Deep learning has revolutionized state-of-the-art performance for
tasks in 3D environments; however, relatively little is known about the
robustness of these approaches in an adversarial setting. The lack of
comprehensive analysis makes it difficult to justify deployment of 3D deep
learning models in real-world, safety-critical applications. In this work, we
develop an algorithm for analysis of pointwise robustness of neural networks
that operate on 3D data. We show that current approaches presented for
understanding the resilience of state-of-the-art models vastly overestimate
their robustness. We then use our algorithm to evaluate an array of
state-of-the-art models in order to demonstrate their vulnerability to
occlusion attacks. We show that, in the worst case, these networks can be
reduced to 0% classification accuracy after the occlusion of at most 6.5% of
the occupied input space.


http://arxiv.org/abs/1904.00887
Adversarial Defense by Restricting the Hidden Space of Deep Neural Networks.
Aamir Mustafa; Salman Khan; Munawar Hayat; Roland Goecke; Jianbing Shen; Ling Shao
  Deep neural networks are vulnerable to adversarial attacks, which can fool
them by adding minuscule perturbations to the input images. The robustness of
existing defenses suffers greatly under white-box attack settings, where an
adversary has full knowledge about the network and can iterate several times to
find strong perturbations. We observe that the main reason for the existence of
such perturbations is the close proximity of different class samples in the
learned feature space. This allows model decisions to be totally changed by
adding an imperceptible perturbation in the inputs. To counter this, we propose
to class-wise disentangle the intermediate feature representations of deep
networks. Specifically, we force the features for each class to lie inside a
convex polytope that is maximally separated from the polytopes of other
classes. In this manner, the network is forced to learn distinct and distant
decision regions for each class. We observe that this simple constraint on the
features greatly enhances the robustness of learned models, even against the
strongest white-box attacks, without degrading the classification performance
on clean images. We report extensive evaluations in both black-box and
white-box attack scenarios and show significant gains in comparison to
state-of-the art defenses.


http://arxiv.org/abs/1904.00689
Defending against adversarial attacks by randomized diversification.
Olga Taran; Shideh Rezaeifar; Taras Holotyak; Slava Voloshynovskiy
  The vulnerability of machine learning systems to adversarial attacks
questions their usage in many applications. In this paper, we propose a
randomized diversification as a defense strategy. We introduce a multi-channel
architecture in a gray-box scenario, which assumes that the architecture of the
classifier and the training data set are known to the attacker. The attacker
does not only have access to a secret key and to the internal states of the
system at the test time. The defender processes an input in multiple channels.
Each channel introduces its own randomization in a special transform domain
based on a secret key shared between the training and testing stages. Such a
transform based randomization with a shared key preserves the gradients in
key-defined sub-spaces for the defender but it prevents gradient back
propagation and the creation of various bypass systems for the attacker. An
additional benefit of multi-channel randomization is the aggregation that fuses
soft-outputs from all channels, thus increasing the reliability of the final
score. The sharing of a secret key creates an information advantage to the
defender. Experimental evaluation demonstrates an increased robustness of the
proposed method to a number of known state-of-the-art attacks.


http://arxiv.org/abs/1904.01002
On the Vulnerability of CNN Classifiers in EEG-Based BCIs.
Xiao Zhang; Dongrui Wu
  Deep learning has been successfully used in numerous applications because of
its outstanding performance and the ability to avoid manual feature
engineering. One such application is electroencephalogram (EEG) based
brain-computer interface (BCI), where multiple convolutional neural network
(CNN) models have been proposed for EEG classification. However, it has been
found that deep learning models can be easily fooled with adversarial examples,
which are normal examples with small deliberate perturbations. This paper
proposes an unsupervised fast gradient sign method (UFGSM) to attack three
popular CNN classifiers in BCIs, and demonstrates its effectiveness. We also
verify the transferability of adversarial examples in BCIs, which means we can
perform attacks even without knowing the architecture and parameters of the
target models, or the datasets they were trained on. To our knowledge, this is
the first study on the vulnerability of CNN classifiers in EEG-based BCIs, and
hopefully will trigger more attention on the security of BCI systems.


http://arxiv.org/abs/1903.12561
Second Rethinking of Network Pruning in the Adversarial Setting.
Shaokai Ye; Kaidi Xu; Sijia Liu; Hao Cheng; Jan-Henrik Lambrechts; Huan Zhang; Aojun Zhou; Kaisheng Ma; Yanzhi Wang; Xue Lin
  It is well known that deep neural networks (DNNs) are vulnerable to
adversarial attacks, which are implemented by adding crafted perturbations onto
benign examples. Min-max robust optimization based adversarial training can
provide a notion of security against adversarial attacks. However, adversarial
robustness requires a significantly larger capacity of the network than that
for the natural training with only benign examples. This paper proposes a
framework of concurrent adversarial training and weight pruning that enables
model compression while still preserving the adversarial robustness and
essentially tackles the dilemma of adversarial training. Furthermore, this work
studies two hypotheses about weight pruning in the conventional network pruning
setting and finds that weight pruning is essential for reducing the network
model size in the adversarial setting, i.e., training a small model from
scratch even with inherited initialization from the large model cannot achieve
both adversarial robustness and model compression.


http://arxiv.org/abs/1903.12261
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.
Dan Hendrycks; Thomas Dietterich
  In this paper we establish rigorous benchmarks for image classifier
robustness. Our first benchmark, ImageNet-C, standardizes and expands the
corruption robustness topic, while showing which classifiers are preferable in
safety-critical applications. Then we propose a new dataset called ImageNet-P
which enables researchers to benchmark a classifier's robustness to common
perturbations. Unlike recent robustness research, this benchmark evaluates
performance on common corruptions and perturbations not worst-case adversarial
perturbations. We find that there are negligible changes in relative corruption
robustness from AlexNet classifiers to ResNet classifiers. Afterward we
discover ways to enhance corruption and perturbation robustness. We even find
that a bypassed adversarial defense provides substantial common perturbation
robustness. Together our benchmarks may aid future work toward networks that
robustly generalize.


http://arxiv.org/abs/1903.11862
Smooth Adversarial Examples.
Hanwei Zhang; Yannis Avrithis; Teddy Furon; Laurent Amsaleg
  This paper investigates the visual quality of the adversarial examples.
Recent papers propose to smooth the perturbations to get rid of high frequency
artefacts. In this work, smoothing has a different meaning as it perceptually
shapes the perturbation according to the visual content of the image to be
attacked. The perturbation becomes locally smooth on the flat areas of the
input image, but it may be noisy on its textured areas and sharp across its
edges.
  This operation relies on Laplacian smoothing, well-known in graph signal
processing, which we integrate in the attack pipeline. We benchmark several
attacks with and without smoothing under a white-box scenario and evaluate
their transferability. Despite the additional constraint of smoothness, our
attack has the same probability of success at lower distortion.


http://arxiv.org/abs/1903.11688
Rallying Adversarial Techniques against Deep Learning for Network Security.
Joseph Clements; Yuzhe Yang; Ankur Sharma; Hongxin Hu; Yingjie Lao
  Recent advances in artificial intelligence and the increasing need for
powerful defensive measures in the domain of network security, have led to the
adoption of deep learning approaches for use in network intrusion detection
systems. These methods have achieved superior performance against conventional
network attacks, which enable the deployment of practical security systems to
unique and dynamic sectors. Adversarial machine learning, unfortunately, has
recently shown that deep learning models are inherently vulnerable to
adversarial modifications on their input data. Because of this susceptibility,
the deep learning models deployed to power a network defense could in fact be
the weakest entry point for compromising a network system. In this paper, we
show that by modifying on average as little as 1.38 of the input features, an
adversary can generate malicious inputs which effectively fool a deep learning
based NIDS. Therefore, when designing such systems, it is crucial to consider
the performance from not only the conventional network security perspective but
also the adversarial machine learning domain.


http://arxiv.org/abs/1903.11626
Bridging Adversarial Robustness and Gradient Interpretability.
Beomsu Kim; Junghoon Seo; Taegyun Jeon
  Adversarial training is a training scheme designed to counter adversarial
attacks by augmenting the training dataset with adversarial examples.
Surprisingly, several studies have observed that loss gradients from
adversarially trained DNNs are visually more interpretable than those from
standard DNNs. Although this phenomenon is interesting, there are only few
works that have offered an explanation. In this paper, we attempted to bridge
this gap between adversarial robustness and gradient interpretability. To this
end, we identified that loss gradients from adversarially trained DNNs align
better with human perception because adversarial training restricts gradients
closer to the image manifold. We then demonstrated that adversarial training
causes loss gradients to be quantitatively meaningful. Finally, we showed that
under the adversarial training framework, there exists an empirical trade-off
between test accuracy and loss gradient interpretability and proposed two
potential approaches to resolving this trade-off.


http://arxiv.org/abs/1903.11508
Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems.
Steffen Eger; Gözde Gül Şahin; Andreas Rücklé; Ji-Ung Lee; Claudia Schulz; Mohsen Mesgar; Krishnkant Swarnkar; Edwin Simpson; Iryna Gurevych
  Visual modifications to text are often used to obfuscate offensive comments
in social media (e.g., "!d10t") or as a writing style ("1337" in "leet speak"),
among other scenarios. We consider this as a new type of adversarial attack in
NLP, a setting to which humans are very robust, as our experiments with both
simple and more difficult visual input perturbations demonstrate. We then
investigate the impact of visual adversarial attacks on current NLP systems on
character-, word-, and sentence-level tasks, showing that both neural and
non-neural models are, in contrast to humans, extremely sensitive to such
attacks, suffering performance decreases of up to 82\%. We then explore three
shielding methods---visual character embeddings, adversarial training, and
rule-based recovery---which substantially improve the robustness of the models.
However, the shielding methods still fall behind performances achieved in
non-attack scenarios, which demonstrates the difficulty of dealing with visual
attacks.


http://arxiv.org/abs/1903.11359
Scaling up the randomized gradient-free adversarial attack reveals overestimation of robustness using established attacks.
Francesco Croce; Jonas Rauber; Matthias Hein
  Modern neural networks are highly non-robust against adversarial
manipulation. A significant amount of work has been invested in techniques to
compute lower bounds on robustness through formal guarantees and to build
provably robust model. However it is still difficult to apply them to larger
networks or in order to get robustness against larger perturbations. Thus
attack strategies are needed to provide tight upper bounds on the actual
robustness. We significantly improve the randomized gradient-free attack for
ReLU networks [9], in particular by scaling it up to large networks. We show
that our attack achieves similar or significantly smaller robust accuracy than
state-of-the-art attacks like PGD or the one of Carlini and Wagner, thus
revealing an overestimation of the robustness by these state-of-the-art
methods. Our attack is not based on a gradient descent scheme and in this sense
gradient-free, which makes it less sensitive to the choice of hyperparameters
as no careful selection of the stepsize is required.


http://arxiv.org/abs/1903.11220
On the Adversarial Robustness of Multivariate Robust Estimation.
Erhan Bayraktar; Lifeng Lai
  In this paper, we investigate the adversarial robustness of multivariate
$M$-Estimators. In the considered model, after observing the whole dataset, an
adversary can modify all data points with the goal of maximizing inference
errors. We use adversarial influence function (AIF) to measure the asymptotic
rate at which the adversary can change the inference result. We first
characterize the adversary's optimal modification strategy and its
corresponding AIF. From the defender's perspective, we would like to design an
estimator that has a small AIF. For the case of joint location and scale
estimation problem, we characterize the optimal $M$-estimator that has the
smallest AIF. We further identify a tradeoff between robustness against
adversarial modifications and robustness against outliers, and derive the
optimal $M$-estimator that achieves the best tradeoff.


http://arxiv.org/abs/1903.10826
A geometry-inspired decision-based attack.
Yujia Liu; Seyed-Mohsen Moosavi-Dezfooli; Pascal Frossard
  Deep neural networks have recently achieved tremendous success in image
classification. Recent studies have however shown that they are easily misled
into incorrect classification decisions by adversarial examples. Adversaries
can even craft attacks by querying the model in black-box settings, where no
information about the model is released except its final decision. Such
decision-based attacks usually require lots of queries, while real-world image
recognition systems might actually restrict the number of queries. In this
paper, we propose qFool, a novel decision-based attack algorithm that can
generate adversarial examples using a small number of queries. The qFool method
can drastically reduce the number of queries compared to previous
decision-based attacks while reaching the same quality of adversarial examples.
We also enhance our method by constraining adversarial perturbations in
low-frequency subspace, which can make qFool even more computationally
efficient. Altogether, we manage to fool commercial image recognition systems
with a small number of queries, which demonstrates the actual effectiveness of
our new algorithm in practice.


http://arxiv.org/abs/1903.10586
Defending against Whitebox Adversarial Attacks via Randomized Discretization.
Yuchen Zhang; Percy Liang
  Adversarial perturbations dramatically decrease the accuracy of
state-of-the-art image classifiers. In this paper, we propose and analyze a
simple and computationally efficient defense strategy: inject random Gaussian
noise, discretize each pixel, and then feed the result into any pre-trained
classifier. Theoretically, we show that our randomized discretization strategy
reduces the KL divergence between original and adversarial inputs, leading to a
lower bound on the classification accuracy of any classifier against any
(potentially whitebox) $\ell_\infty$-bounded adversarial attack. Empirically,
we evaluate our defense on adversarial examples generated by a strong iterative
PGD attack. On ImageNet, our defense is more robust than adversarially-trained
networks and the winning defenses of the NIPS 2017 Adversarial Attacks &
Defenses competition.


http://arxiv.org/abs/1903.10484
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness.
Jörn-Henrik Jacobsen; Jens Behrmannn; Nicholas Carlini; Florian Tramèr; Nicolas Papernot
  Adversarial examples are malicious inputs crafted to cause a model to
misclassify them. Their most common instantiation, "perturbation-based"
adversarial examples introduce changes to the input that leave its true label
unchanged, yet result in a different model prediction. Conversely,
"invariance-based" adversarial examples insert changes to the input that leave
the model's prediction unaffected despite the underlying input's label having
changed.
  In this paper, we demonstrate that robustness to perturbation-based
adversarial examples is not only insufficient for general robustness, but
worse, it can also increase vulnerability of the model to invariance-based
adversarial examples. In addition to analytical constructions, we empirically
study vision classifiers with state-of-the-art robustness to perturbation-based
adversaries constrained by an $\ell_p$ norm. We mount attacks that exploit
excessive model invariance in directions relevant to the task, which are able
to find adversarial examples within the $\ell_p$ ball. In fact, we find that
classifiers trained to be $\ell_p$-norm robust are more vulnerable to
invariance-based adversarial examples than their undefended counterparts.
  Excessive invariance is not limited to models trained to be robust to
perturbation-based $\ell_p$-norm adversaries. In fact, we argue that the term
adversarial example is used to capture a series of model limitations, some of
which may not have been discovered yet. Accordingly, we call for a set of
precise definitions that taxonomize and address each of these shortcomings in
learning.


http://arxiv.org/abs/1903.10396
The LogBarrier adversarial attack: making effective use of decision boundary information.
Chris Finlay; Aram-Alexandre Pooladian; Adam M. Oberman
  Adversarial attacks for image classification are small perturbations to
images that are designed to cause misclassification by a model. Adversarial
attacks formally correspond to an optimization problem: find a minimum norm
image perturbation, constrained to cause misclassification. A number of
effective attacks have been developed. However, to date, no gradient-based
attacks have used best practices from the optimization literature to solve this
constrained minimization problem. We design a new untargeted attack, based on
these best practices, using the established logarithmic barrier method. On
average, our attack distance is similar or better than all state-of-the-art
attacks on benchmark datasets (MNIST, CIFAR10, ImageNet-1K). In addition, our
method performs significantly better on the most challenging images, those
which normally require larger perturbations for misclassification. We employ
the LogBarrier attack on several adversarially defended models, and show that
it adversarially perturbs all images more efficiently than other attacks: the
distance needed to perturb all images is significantly smaller with the
LogBarrier attack than with other state-of-the-art attacks.


http://arxiv.org/abs/1903.10219
Robust Neural Networks using Randomized Adversarial Training.
Alexandre Araujo; Rafael Pinot; Benjamin Negrevergne; Laurent Meunier; Yann Chevaleyre; Florian Yger; Jamal Atif
  Since the discovery of adversarial examples in machine learning, researchers
have designed several techniques to train neural networks that are robust
against different types of attacks (most notably $\ell_\infty$ and $\ell_2$
based attacks). However, it has been observed that the defense mechanisms
designed to protect against one type of attack often offer poor performance
against the other. In this paper, we introduce Randomized Adversarial Training
(RAT), a technique that is efficient both against $\ell_2$ and $\ell_\infty$
attacks. To obtain this result, we build upon adversarial training, a technique
that is efficient against $\ell_\infty$ attacks, and demonstrate that adding
random noise at training and inference time further improves performance
against \ltwo attacks. We then show that RAT is as efficient as adversarial
training against $\ell_\infty$ attacks while being robust against strong
$\ell_2$ attacks. Our final comparative experiments demonstrate that RAT
outperforms all state-of-the-art approaches against $\ell_2$ and $\ell_\infty$
attacks.


http://arxiv.org/abs/1903.10033
A Formalization of Robustness for Deep Neural Networks.
Tommaso Dreossi; Shromona Ghosh; Alberto Sangiovanni-Vincentelli; Sanjit A. Seshia
  Deep neural networks have been shown to lack robustness to small input
perturbations. The process of generating the perturbations that expose the lack
of robustness of neural networks is known as adversarial input generation. This
process depends on the goals and capabilities of the adversary, In this paper,
we propose a unifying formalization of the adversarial input generation process
from a formal methods perspective. We provide a definition of robustness that
is general enough to capture different formulations. The expressiveness of our
formalization is shown by modeling and comparing a variety of adversarial
attack techniques.


http://arxiv.org/abs/1903.09940
Variational Inference with Latent Space Quantization for Adversarial Resilience.
Vinay Kyatham; Prathosh A. P.; Deepak Mishra; Tarun Kumar Yadav; Dheeraj Mundhra
  Despite their tremendous success in modelling high-dimensional data
manifolds, deep neural networks suffer from the threat of adversarial attacks -
Existence of perceptually valid input-like samples obtained through careful
perturbations that leads to degradation in the performance of underlying model.
Major concerns with existing defense mechanisms include non-generalizability
across different attacks, models and large inference time. In this paper, we
propose a generalized defense mechanism capitalizing on the expressive power of
regularized latent space based generative models. We design an adversarial
filter, devoid of access to classifier and adversaries, which makes it usable
in tandem with any classifier. The basic idea is to learn a Lipschitz
constrained mapping from the data manifold, incorporating adversarial
perturbations, to a quantized latent space and re-map it to the true data
manifold. Specifically, we simultaneously auto-encode the data manifold and its
perturbations implicitly through the perturbations of the regularized and
quantized generative latent space, realized using variational inference. We
demonstrate the efficacy of the proposed formulation in providing the
resilience against multiple attack types (Black and white box) and methods,
while being almost real-time. Our experiments show that the proposed method
surpasses the state-of-the-art techniques in several cases.


http://arxiv.org/abs/1903.09799
Improving Adversarial Robustness via Guided Complement Entropy.
Hao-Yun Chen; Jhao-Hong Liang; Shih-Chieh Chang; Jia-Yu Pan; Yu-Ting Chen; Wei Wei; Da-Cheng Juan
  Model robustness has been an important issue, since adding small adversarial
perturbations to images is sufficient to drive the model accuracy down to
nearly zero. In this paper, we propose a new training objective "Guided
Complement Entropy" (GCE) that has dual desirable effects: (a) neutralizing the
predicted probabilities of incorrect classes, and (b) maximizing the predicted
probability of the ground-truth class, particularly when (a) is achieved.
Training with GCE encourages models to learn latent representations where
samples of different classes form distinct clusters, which we argue, improves
the model robustness against adversarial perturbations. Furthermore, compared
with the state-of-the-arts trained with cross-entropy, same models trained with
GCE achieve significant improvements on the robustness against white-box
adversarial attacks, both with and without adversarial training. When no attack
is present, training with GCE also outperforms cross-entropy in terms of model
accuracy.


http://arxiv.org/abs/1903.10346
Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition.
Yao Qin; Nicholas Carlini; Ian Goodfellow; Garrison Cottrell; Colin Raffel
  Adversarial examples are inputs to machine learning models designed by an
adversary to cause an incorrect output. So far, adversarial examples have been
studied most extensively in the image domain. In this domain, adversarial
examples can be constructed by imperceptibly modifying images to cause
misclassification, and are practical in the physical world. In contrast,
current targeted adversarial examples applied to speech recognition systems
have neither of these properties: humans can easily identify the adversarial
perturbations, and they are not effective when played over-the-air. This paper
makes advances on both of these fronts. First, we develop effectively
imperceptible audio adversarial examples (verified through a human study) by
leveraging the psychoacoustic principle of auditory masking, while retaining
100% targeted success rate on arbitrary full-sentence targets. Next, we make
progress towards physical-world over-the-air audio adversarial examples by
constructing perturbations which remain effective even after applying realistic
simulated environmental distortions.


http://arxiv.org/abs/1904.00759
Adversarial camera stickers: A physical camera-based attack on deep learning systems.
Juncheng Li; Frank R. Schmidt; J. Zico Kolter
  Recent work has documented the susceptibility of deep learning systems to
adversarial examples, but most such attacks directly manipulate the digital
input to a classifier. Although a smaller line of work considers physical
adversarial attacks, in all cases these involve manipulating the object of
interest, e.g., putting a physical sticker on an object to misclassify it, or
manufacturing an object specifically intended to be misclassified. In this
work, we consider an alternative question: is it possible to fool deep
classifiers, over all perceived objects of a certain type, by physically
manipulating the camera itself? We show that by placing a carefully crafted and
mainly-translucent sticker over the lens of a camera, one can create universal
perturbations of the observed images that are inconspicuous, yet misclassify
target objects as a different (targeted) class. To accomplish this, we propose
an iterative procedure for both updating the attack perturbation (to make it
adversarial for a given classifier), and the threat model itself (to ensure it
is physically realizable). For example, we show that we can achieve
physically-realizable attacks that fool ImageNet classifiers in a targeted
fashion 49.6% of the time. This presents a new class of physically-realizable
threat models to consider in the context of adversarially robust machine
learning. Our demo video can be viewed at: https://youtu.be/wUVmL33Fx54


http://arxiv.org/abs/1903.08778
Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes.
Matt Jordan; Justin Lewis; Alexandros G. Dimakis
  We propose a novel method for computing exact pointwise robustness of deep
neural networks for all convex $\ell_p$ norms. Our algorithm, GeoCert, finds
the largest $\ell_p$ ball centered at an input point $x_0$, within which the
output class of a given neural network with ReLU nonlinearities remains
unchanged. We relate the problem of computing pointwise robustness of these
networks to that of computing the maximum norm ball with a fixed center that
can be contained in a non-convex polytope. This is a challenging problem in
general, however we show that there exists an efficient algorithm to compute
this for polyhedral complices. Further we show that piecewise linear neural
networks partition the input space into a polyhedral complex. Our algorithm has
the ability to almost immediately output a nontrivial lower bound to the
pointwise robustness which is iteratively improved until it ultimately becomes
tight. We empirically show that our approach generates distance lower bounds
that are tighter compared to prior work, under moderate time constraints.


http://arxiv.org/abs/1903.08333
On the Robustness of Deep K-Nearest Neighbors.
Chawin Sitawarin; David Wagner
  Despite a large amount of attention on adversarial examples, very few works
have demonstrated an effective defense against this threat. We examine Deep
k-Nearest Neighbor (DkNN), a proposed defense that combines k-Nearest Neighbor
(kNN) and deep learning to improve the model's robustness to adversarial
examples. It is challenging to evaluate the robustness of this scheme due to a
lack of efficient algorithm for attacking kNN classifiers with large k and
high-dimensional data. We propose a heuristic attack that allows us to use
gradient descent to find adversarial examples for kNN classifiers, and then
apply it to attack the DkNN defense as well. Results suggest that our attack is
moderately stronger than any naive attack on kNN and significantly outperforms
other attacks on DkNN.


http://arxiv.org/abs/1903.07282
Generating Adversarial Examples With Conditional Generative Adversarial Net.
Ping Yu; Kaitao Song; Jianfeng Lu
  Recently, deep neural networks have significant progress and successful
application in various fields, but they are found vulnerable to attack
instances, e.g., adversarial examples. State-of-art attack methods can generate
attack images by adding small perturbation to the source image. These attack
images can fool the classifier but have little impact to human. Therefore, such
attack instances are difficult to generate by searching the feature space. How
to design an effective and robust generating method has become a spotlight.
Inspired by adversarial examples, we propose two novel generative models to
produce adaptive attack instances directly, in which conditional generative
adversarial network is adopted and distinctive strategy is designed for
training. Compared with the common method, such as Fast Gradient Sign Method,
our models can reduce the generating cost and improve robustness and has about
one fifth running time for producing attack instance.


http://arxiv.org/abs/1904.05734
Practical Hidden Voice Attacks against Speech and Speaker Recognition Systems.
Hadi Abdullah; Washington Garcia; Christian Peeters; Patrick Traynor; Kevin R. B. Butler; Joseph Wilson
  Voice Processing Systems (VPSes), now widely deployed, have been made
significantly more accurate through the application of recent advances in
machine learning. However, adversarial machine learning has similarly advanced
and has been used to demonstrate that VPSes are vulnerable to the injection of
hidden commands - audio obscured by noise that is correctly recognized by a VPS
but not by human beings. Such attacks, though, are often highly dependent on
white-box knowledge of a specific machine learning model and limited to
specific microphones and speakers, making their use across different acoustic
hardware platforms (and thus their practicality) limited. In this paper, we
break these dependencies and make hidden command attacks more practical through
model-agnostic (blackbox) attacks, which exploit knowledge of the signal
processing algorithms commonly used by VPSes to generate the data fed into
machine learning systems. Specifically, we exploit the fact that multiple
source audio samples have similar feature vectors when transformed by acoustic
feature extraction algorithms (e.g., FFTs). We develop four classes of
perturbations that create unintelligible audio and test them against 12 machine
learning models, including 7 proprietary models (e.g., Google Speech API, Bing
Speech API, IBM Speech API, Azure Speaker API, etc), and demonstrate successful
attacks against all targets. Moreover, we successfully use our maliciously
generated audio samples in multiple hardware configurations, demonstrating
effectiveness across both models and real systems. In so doing, we demonstrate
that domain-specific knowledge of audio signal processing represents a
practical means of generating successful hidden voice command attacks.


http://arxiv.org/abs/1903.07054
Adversarial Attacks on Deep Neural Networks for Time Series Classification.
Hassan Ismail Fawaz; Germain Forestier; Jonathan Weber; Lhassane Idoumghar; Pierre-Alain Muller
  Time Series Classification (TSC) problems are encountered in many real life
data mining tasks ranging from medicine and security to human activity
recognition and food safety. With the recent success of deep neural networks in
various domains such as computer vision and natural language processing,
researchers started adopting these techniques for solving time series data
mining problems. However, to the best of our knowledge, no previous work has
considered the vulnerability of deep learning models to adversarial time series
examples, which could potentially make them unreliable in situations where the
decision taken by the classifier is crucial such as in medicine and security.
For computer vision problems, such attacks have been shown to be very easy to
perform by altering the image and adding an imperceptible amount of noise to
trick the network into wrongly classifying the input image. Following this line
of work, we propose to leverage existing adversarial attack mechanisms to add a
special noise to the input time series in order to decrease the network's
confidence when classifying instances at test time. Our results reveal that
current state-of-the-art deep learning time series classifiers are vulnerable
to adversarial attacks which can have major consequences in multiple domains
such as food safety and quality assurance.


http://arxiv.org/abs/1903.06620
On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models.
Paul Michel; Xian Li; Graham Neubig; Juan Miguel Pino
  Adversarial examples --- perturbations to the input of a model that elicit
large changes in the output --- have been shown to be an effective way of
assessing the robustness of sequence-to-sequence (seq2seq) models. However,
these perturbations only indicate weaknesses in the model if they do not change
the input so significantly that it legitimately results in changes in the
expected output. This fact has largely been ignored in the evaluations of the
growing body of related literature. Using the example of untargeted attacks on
machine translation (MT), we propose a new evaluation framework for adversarial
attacks on seq2seq models that takes the semantic equivalence of the pre- and
post-perturbation input into account. Using this framework, we demonstrate that
existing methods may not preserve meaning in general, breaking the
aforementioned assumption that source side perturbations should not result in
changes in the expected output. We further use this framework to demonstrate
that adding additional constraints on attacks allows for adversarial
perturbations that are more meaning-preserving, but nonetheless largely change
the output sequence. Finally, we show that performing untargeted adversarial
training with meaning-preserving attacks is beneficial to the model in terms of
adversarial robustness, without hurting test performance. A toolkit
implementing our evaluation framework is released at
https://github.com/pmichel31415/teapot-nlp.


http://arxiv.org/abs/1903.06603
On Certifying Non-uniform Bound against Adversarial Attacks.
Chen Liu; Ryota Tomioka; Volkan Cevher
  This work studies the robustness certification problem of neural network
models, which aims to find certified adversary-free regions as large as
possible around data points. In contrast to the existing approaches that seek
regions bounded uniformly along all input features, we consider non-uniform
bounds and use it to study the decision boundary of neural network models. We
formulate our target as an optimization problem with nonlinear constraints.
Then, a framework applicable for general feedforward neural networks is
proposed to bound the output logits so that the relaxed problem can be solved
by the augmented Lagrangian method. Our experiments show the non-uniform bounds
have larger volumes than uniform ones and the geometric similarity of the
non-uniform bounds gives a quantitative, data-agnostic metric of input
features' robustness. Further, compared with normal models, the robust models
have even larger non-uniform bounds and better interpretability.


http://arxiv.org/abs/1903.06293
A Research Agenda: Dynamic Models to Defend Against Correlated Attacks.
Ian Goodfellow
  In this article I describe a research agenda for securing machine learning
models against adversarial inputs at test time. This article does not present
results but instead shares some of my thoughts about where I think that the
field needs to go. Modern machine learning works very well on I.I.D. data: data
for which each example is drawn {\em independently} and for which the
distribution generating each example is {\em identical}. When these assumptions
are relaxed, modern machine learning can perform very poorly. When machine
learning is used in contexts where security is a concern, it is desirable to
design models that perform well even when the input is designed by a malicious
adversary. So far most research in this direction has focused on an adversary
who violates the {\em identical} assumption, and imposes some kind of
restricted worst-case distribution shift. I argue that machine learning
security researchers should also address the problem of relaxing the {\em
independence} assumption and that current strategies designed for robustness to
distribution shift will not do so. I recommend {\em dynamic models} that change
each time they are run as a potential solution path to this problem, and show
an example of a simple attack using correlated data that can be mitigated by a
simple dynamic defense. This is not intended as a real-world security measure,
but as a recommendation to explore this research direction and develop more
realistic defenses.


http://arxiv.org/abs/1903.05821
Attribution-driven Causal Analysis for Detection of Adversarial Examples.
Susmit Jha; Sunny Raj; Steven Lawrence Fernandes; Sumit Kumar Jha; Somesh Jha; Gunjan Verma; Brian Jalaian; Ananthram Swami
  Attribution methods have been developed to explain the decision of a machine
learning model on a given input. We use the Integrated Gradient method for
finding attributions to define the causal neighborhood of an input by
incrementally masking high attribution features. We study the robustness of
machine learning models on benign and adversarial inputs in this neighborhood.
Our study indicates that benign inputs are robust to the masking of high
attribution features but adversarial inputs generated by the state-of-the-art
adversarial attack methods such as DeepFool, FGSM, CW and PGD, are not robust
to such masking. Further, our study demonstrates that this concentration of
high-attribution features responsible for the incorrect decision is more
pronounced in physically realizable adversarial examples. This difference in
attribution of benign and adversarial inputs can be used to detect adversarial
examples. Such a defense approach is independent of training data and attack
method, and we demonstrate its effectiveness on digital and physically
realizable perturbations.


http://arxiv.org/abs/1903.05543
Adversarial attacks against Fact Extraction and VERification.
James Thorne; Andreas Vlachos
  This paper describes a baseline for the second iteration of the Fact
Extraction and VERification shared task (FEVER2.0) which explores the
resilience of systems through adversarial evaluation. We present a collection
of simple adversarial attacks against systems that participated in the first
FEVER shared task. FEVER modeled the assessment of truthfulness of written
claims as a joint information retrieval and natural language inference task
using evidence from Wikipedia. A large number of participants made use of deep
neural networks in their submissions to the shared task. The extent as to
whether such models understand language has been the subject of a number of
recent investigations and discussion in literature. In this paper, we present a
simple method of generating entailment-preserving and entailment-altering
perturbations of instances by common patterns within the training data. We find
that a number of systems are greatly affected with absolute losses in
classification accuracy of up to $29\%$ on the newly perturbed instances. Using
these newly generated instances, we construct a sample submission for the
FEVER2.0 shared task. Addressing these types of attacks will aid in building
more robust fact-checking models, as well as suggest directions to expand the
datasets.


http://arxiv.org/abs/1903.05157
Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models.
Adith Boloor; Xin He; Christopher Gill; Yevgeniy Vorobeychik; Xuan Zhang
  Recent advances in machine learning, especially techniques such as deep
neural networks, are promoting a range of high-stakes applications, including
autonomous driving, which often relies on deep learning for perception. While
deep learning for perception has been shown to be vulnerable to a host of
subtle adversarial manipulations of images, end-to-end demonstrations of
successful attacks, which manipulate the physical environment and result in
physical consequences, are scarce. Moreover, attacks typically involve
carefully constructed adversarial examples at the level of pixels. We
demonstrate the first end-to-end attacks on autonomous driving in simulation,
using simple physically realizable attacks: the painting of black lines on the
road. These attacks target deep neural network models for end-to-end autonomous
driving control. A systematic investigation shows that such attacks are
surprisingly easy to engineer, and we describe scenarios (e.g., right turns) in
which they are highly effective, and others that are less vulnerable (e.g.,
driving straight). Further, we use network deconvolution to demonstrate that
the attacks succeed by inducing activation patterns similar to entirely
different scenarios used in training.


http://arxiv.org/abs/1903.05994
Can Adversarial Network Attack be Defended?.
Jinyin Chen; Yangyang Wu; Xiang Lin; Qi Xuan
  Machine learning has been successfully applied to complex network analysis in
various areas, and graph neural networks (GNNs) based methods outperform
others. Recently, adversarial attack on networks has attracted special
attention since carefully crafted adversarial networks with slight
perturbations on clean network may invalid lots of network applications, such
as node classification, link prediction, and community detection etc. Such
attacks are easily constructed with serious security threat to various analyze
methods, including traditional methods and deep models. To the best of our
knowledge, it is the first time that defense method against network adversarial
attack is discussed. In this paper, we are interested in the possibility of
defense against adversarial attack on network, and propose defense strategies
for GNNs against attacks. First, we propose novel adversarial training
strategies to improve GNNs' defensibility against attacks. Then, we
analytically investigate the robustness properties for GNNs granted by the use
of smooth defense, and propose two special smooth defense strategies: smoothing
distillation and smoothing cross-entropy loss function. Both of them are
capable of smoothing gradient of GNNs, and consequently reduce the amplitude of
adversarial gradients, which benefits gradient masking from attackers. The
comprehensive experiments show that our proposed strategies have great
defensibility against different adversarial attacks on four real-world networks
in different network analyze tasks.


http://arxiv.org/abs/1903.03905
Manifold Preserving Adversarial Learning.
Ousmane Amadou Dia; Elnaz Barshan; Reza Babanezhad
  How to generate semantically meaningful and structurally sound adversarial
examples? We propose to answer this question by restricting the search for
adversaries in the true data manifold. To this end, we introduce a stochastic
variational inference method to learn the data manifold, in the presence of
continuous latent variables with intractable posterior distributions, without
requiring an a priori form for the data underlying distribution. We then
propose a manifold perturbation strategy that ensures the cases we perturb
remain in the manifold of the original examples and thereby generate the
adversaries. We evaluate our approach on a number of image and text datasets.
Our results show the effectiveness of our approach in producing coherent, and
realistic-looking adversaries that can evade strong defenses known to be
resilient to traditional adversarial attacks


http://arxiv.org/abs/1903.03029
Attack Type Agnostic Perceptual Enhancement of Adversarial Images.
Bilgin Aksoy; Alptekin Temizel
  Adversarial images are samples that are intentionally modified to deceive
machine learning systems. They are widely used in applications such as CAPTHAs
to help distinguish legitimate human users from bots. However, the noise
introduced during the adversarial image generation process degrades the
perceptual quality and introduces artificial colours; making it also difficult
for humans to classify images and recognise objects. In this letter, we propose
a method to enhance the perceptual quality of these adversarial images. The
proposed method is attack type agnostic and could be used in association with
the existing attacks in the literature. Our experiments show that the generated
adversarial images have lower Euclidean distance values while maintaining the
same adversarial attack performance. Distances are reduced by 5.88% to 41.27%
with an average reduction of 22% over the different attack and network types.


http://arxiv.org/abs/1903.02926
Out-domain examples for generative models.
Dario Pasquini; Marco Mingione; Massimo Bernaschi
  Deep generative models are being increasingly used in a wide variety of
applications. However, the generative process is not fully predictable and at
times, it produces an unexpected output. We will refer to those outputs as
out-domain examples. In the present paper we show that an attacker can force a
pre-trained generator to reproduce an arbitrary out-domain example if fed by a
suitable adversarial input. The main assumption is that those outputs lie in an
unexplored region of the generator's codomain and hence they have a very low
probability of being naturally generated. Moreover, we show that this
adversarial input can be shaped so as to be statistically indistinguishable
from the set of genuine inputs. The goal is to look for an efficient way of
finding these inputs in the generator's latent space.


http://arxiv.org/abs/1903.02585
GanDef: A GAN based Adversarial Training Defense for Neural Network Classifier.
Guanxiong Liu; Issa Khalil; Abdallah Khreishah
  Machine learning models, especially neural network (NN) classifiers, are
widely used in many applications including natural language processing,
computer vision and cybersecurity. They provide high accuracy under the
assumption of attack-free scenarios. However, this assumption has been defied
by the introduction of adversarial examples -- carefully perturbed samples of
input that are usually misclassified. Many researchers have tried to develop a
defense against adversarial examples; however, we are still far from achieving
that goal. In this paper, we design a Generative Adversarial Net (GAN) based
adversarial training defense, dubbed GanDef, which utilizes a competition game
to regulate the feature selection during the training. We analytically show
that GanDef can train a classifier so it can defend against adversarial
examples. Through extensive evaluation on different white-box adversarial
examples, the classifier trained by GanDef shows the same level of test
accuracy as those trained by state-of-the-art adversarial training defenses.
More importantly, GanDef-Comb, a variant of GanDef, could utilize the
discriminator to achieve a dynamic trade-off between correctly classifying
original and adversarial examples. As a result, it achieves the highest overall
test accuracy when the ratio of adversarial examples exceeds 41.7%.


http://arxiv.org/abs/1903.01980
Statistical Guarantees for the Robustness of Bayesian Neural Networks.
Luca Cardelli; Marta Kwiatkowska; Luca Laurenti; Nicola Paoletti; Andrea Patane; Matthew Wicker
  We introduce a probabilistic robustness measure for Bayesian Neural Networks
(BNNs), defined as the probability that, given a test point, there exists a
point within a bounded set such that the BNN prediction differs between the
two. Such a measure can be used, for instance, to quantify the probability of
the existence of adversarial examples. Building on statistical verification
techniques for probabilistic models, we develop a framework that allows us to
estimate probabilistic robustness for a BNN with statistical guarantees, i.e.,
with a priori error and confidence bounds. We provide experimental comparison
for several approximate BNN inference techniques on image classification tasks
associated to MNIST and a two-class subset of the GTSRB dataset. Our results
enable quantification of uncertainty of BNN predictions in adversarial
settings.


http://arxiv.org/abs/1903.01715
L 1-norm double backpropagation adversarial defense.
Ismaïla LIMOS, LITIS Seck; Gaëlle LIMOS Loosli; Stephane LITIS Canu
  Adversarial examples are a challenging open problem for deep neural networks.
We propose in this paper to add a penalization term that forces the decision
function to be at in some regions of the input space, such that it becomes, at
least locally, less sensitive to attacks. Our proposition is theoretically
motivated and shows on a first set of carefully conducted experiments that it
behaves as expected when used alone, and seems promising when coupled with
adversarial training.


http://arxiv.org/abs/1903.01612
Defense Against Adversarial Images using Web-Scale Nearest-Neighbor Search.
Abhimanyu Dubey; der Maaten Laurens van; Zeki Yalniz; Yixuan Li; Dhruv Mahajan
  A plethora of recent work has shown that convolutional networks are not
robust to adversarial images: images that are created by perturbing a sample
from the data distribution as to maximize the loss on the perturbed example. In
this work, we hypothesize that adversarial perturbations move the image away
from the image manifold in the sense that there exists no physical process that
could have produced the adversarial image. This hypothesis suggests that a
successful defense mechanism against adversarial images should aim to project
the images back onto the image manifold. We study such defense mechanisms,
which approximate the projection onto the unknown image manifold by a
nearest-neighbor search against a web-scale image database containing tens of
billions of images. Empirical evaluations of this defense strategy on ImageNet
suggest that it is very effective in attack settings in which the adversary
does not have access to the image database. We also propose two novel attack
methods to break nearest-neighbor defenses, and demonstrate conditions under
which nearest-neighbor defense fails. We perform a series of ablation
experiments, which suggest that there is a trade-off between robustness and
accuracy in our defenses, that a large image database (with hundreds of
millions of images) is crucial to get good performance, and that careful
construction the image database is important to be robust against attacks
tailored to circumvent our defenses.


http://arxiv.org/abs/1903.01610
The Vulnerabilities of Graph Convolutional Networks: Stronger Attacks and Defensive Techniques.
Huijun Wu; Chen Wang; Yuriy Tyshetskiy; Andrew Dotcherty; Kai Lu; Liming Zhu
  Graph deep learning models, such as graph convolutional networks (GCN)
achieve remarkable performance for tasks on graph data. Similar to other types
of deep models, graph deep learning models often suffer from adversarial
attacks. However, compared with non-graph data, the discrete features, graph
connections and different definitions of imperceptible perturbations bring
unique challenges and opportunities for the adversarial attacks and defences
for graph data. In this paper, we propose both attack and defence techniques.
For attack, we show that the discrete feature problem could easily be resolved
by introducing integrated gradients which could accurately reflect the effect
of perturbing certain features or edges while still benefiting from the
parallel computations. For defence, we propose to partially learn the adjacency
matrix to integrate the information of distant nodes so that the prediction of
a certain target is supported by more global graph information rather than just
few neighbour nodes. This, therefore, makes the attacks harder since one need
to perturb more features/edges to make the attacks succeed. Our experiments on
a number of datasets show the effectiveness of the proposed methods.


http://arxiv.org/abs/1903.01287
Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming.
Mahyar Fazlyab; Manfred Morari; George J. Pappas
  Analyzing the robustness of neural networks against norm-bounded
uncertainties and adversarial attacks has found many applications ranging from
safety verification to robust training. In this paper, we propose a
semidefinite programming (SDP) framework for safety verification and robustness
analysis of neural networks with general activation functions. Our main idea is
to abstract various properties of activation functions (e.g., monotonicity,
bounded slope, bounded values, and repetition across layers) with the formalism
of quadratic constraints. We then analyze the safety properties of the
abstracted network via the S-procedure and semidefinite programming. Compared
to other semidefinite relaxations proposed in the literature, our method is
less conservative, especially for deep networks, with an order of magnitude
reduction in computational complexity. Furthermore, our approach is applicable
to any activation functions.


http://arxiv.org/abs/1903.01182
Complement Objective Training.
Hao-Yun Chen; Pei-Hsin Wang; Chun-Hao Liu; Shih-Chieh Chang; Jia-Yu Pan; Yu-Ting Chen; Wei Wei; Da-Cheng Juan
  Learning with a primary objective, such as softmax cross entropy for
classification and sequence generation, has been the norm for training deep
neural networks for years. Although being a widely-adopted approach, using
cross entropy as the primary objective exploits mostly the information from the
ground-truth class for maximizing data likelihood, and largely ignores
information from the complement (incorrect) classes. We argue that, in addition
to the primary objective, training also using a complement objective that
leverages information from the complement classes can be effective in improving
model performance. This motivates us to study a new training paradigm that
maximizes the likelihood of the groundtruth class while neutralizing the
probabilities of the complement classes. We conduct extensive experiments on
multiple tasks ranging from computer vision to natural language understanding.
The experimental results confirm that, compared to the conventional training
with just one primary objective, training also with the complement objective
further improves the performance of the state-of-the-art models across all
tasks. In addition to the accuracy improvement, we also show that models
trained with both primary and complement objectives are more robust to
single-step adversarial attacks.


http://arxiv.org/abs/1903.01015
A Kernelized Manifold Mapping to Diminish the Effect of Adversarial Perturbations.
Saeid Asgari Taghanaki; Kumar Abhishek; Shekoofeh Azizi; Ghassan Hamarneh
  The linear and non-flexible nature of deep convolutional models makes them
vulnerable to carefully crafted adversarial perturbations. To tackle this
problem, we propose a non-linear radial basis convolutional feature mapping by
learning a Mahalanobis-like distance function. Our method then maps the
convolutional features onto a linearly well-separated manifold, which prevents
small adversarial perturbations from forcing a sample to cross the decision
boundary. We test the proposed method on three publicly available image
classification and segmentation datasets namely, MNIST, ISBI ISIC 2017 skin
lesion segmentation, and NIH Chest X-Ray-14. We evaluate the robustness of our
method to different gradient (targeted and untargeted) and non-gradient based
attacks and compare it to several non-gradient masking defense strategies. Our
results demonstrate that the proposed method can increase the resilience of
deep convolutional neural networks to adversarial perturbations without
accuracy drop on clean data.


http://arxiv.org/abs/1903.01563
Evaluating Adversarial Evasion Attacks in the Context of Wireless Communications.
Bryse Flowers; R. Michael Buehrer; William C. Headley
  Recent advancements in radio frequency machine learning (RFML) have
demonstrated the use of raw in-phase and quadrature (IQ) samples for multiple
spectrum sensing tasks. Yet, deep learning techniques have been shown, in other
applications, to be vulnerable to adversarial machine learning (ML) techniques,
which seek to craft small perturbations that are added to the input to cause a
misclassification. The current work differentiates the threats that adversarial
ML poses to RFML systems based on where the attack is executed from: direct
access to classifier input, synchronously transmitted over the air (OTA), or
asynchronously transmitted from a separate device. Additionally, the current
work develops a methodology for evaluating adversarial success in the context
of wireless communications, where the primary metric of interest is bit error
rate and not human perception, as is the case in image recognition. The
methodology is demonstrated using the well known Fast Gradient Sign Method to
evaluate the vulnerabilities of raw IQ based Automatic Modulation
Classification and concludes RFML is vulnerable to adversarial examples, even
in OTA attacks. However, RFML domain specific receiver effects, which would be
encountered in an OTA attack, can present significant impairments to
adversarial evasion.


http://arxiv.org/abs/1903.00585
PuVAE: A Variational Autoencoder to Purify Adversarial Examples.
Uiwon Hwang; Jaewoo Park; Hyemi Jang; Sungroh Yoon; Nam Ik Cho
  Deep neural networks are widely used and exhibit excellent performance in
many areas. However, they are vulnerable to adversarial attacks that compromise
the network at the inference time by applying elaborately designed perturbation
to input data. Although several defense methods have been proposed to address
specific attacks, other attack methods can circumvent these defense mechanisms.
Therefore, we propose Purifying Variational Autoencoder (PuVAE), a method to
purify adversarial examples. The proposed method eliminates an adversarial
perturbation by projecting an adversarial example on the manifold of each
class, and determines the closest projection as a purified sample. We
experimentally illustrate the robustness of PuVAE against various attack
methods without any prior knowledge. In our experiments, the proposed method
exhibits performances competitive with state-of-the-art defense methods, and
the inference time is approximately 130 times faster than that of Defense-GAN
that is the state-of-the art purifier model.


http://arxiv.org/abs/1903.00553
Attacking Graph-based Classification via Manipulating the Graph Structure.
Binghui Wang; Neil Zhenqiang Gong
  Graph-based classification methods are widely used for security and privacy
analytics. Roughly speaking, graph-based classification methods include
collective classification and graph neural network. Evading a graph-based
classification method enables an attacker to evade detection in security
analytics and can be used as a privacy defense against inference attacks.
Existing adversarial machine learning studies mainly focused on machine
learning for non-graph data. Only a few recent studies touched adversarial
graph-based classification methods. However, they focused on graph neural
network methods, leaving adversarial collective classification largely
unexplored. We aim to bridge this gap in this work. We first propose a threat
model to characterize the attack surface of a collective classification method.
Specifically, we characterize an attacker's background knowledge along three
dimensions: parameters of the method, training dataset, and the complete graph;
an attacker's goal is to evade detection via manipulating the graph structure.
We formulate our attack as a graph-based optimization problem, solving which
produces the edges that an attacker needs to manipulate to achieve its attack
goal. Moreover, we propose several approximation techniques to solve the
optimization problem. We evaluate our attacks and compare them with a recent
attack designed for graph neural networks. Results show that our attacks 1) can
effectively evade graph-based classification methods; 2) do not require access
to the true parameters, true training dataset, and/or complete graph; and 3)
outperform the existing attack for evading collective classification methods
and some graph neural network methods. We also apply our attacks to evade Sybil
detection using a large-scale Twitter dataset and apply our attacks as a
defense against attribute inference attacks using a large-scale Google+
dataset.


http://arxiv.org/abs/1903.00073
On the Effectiveness of Low Frequency Perturbations.
Yash Sharma; Gavin Weiguang Ding; Marcus Brubaker
  Carefully crafted, often imperceptible, adversarial perturbations have been
shown to cause state-of-the-art models to yield extremely inaccurate outputs,
rendering them unsuitable for safety-critical application domains. In addition,
recent work has shown that constraining the attack space to a low frequency
regime is particularly effective. Yet, it remains unclear whether this is due
to generally constraining the attack search space or specifically removing high
frequency components from consideration. By systematically controlling the
frequency components of the perturbation, evaluating against the top-placing
defense submissions in the NeurIPS 2017 competition, we empirically show that
performance improvements in both optimization and generalization are yielded
only when low frequency components are preserved. In fact, the defended models
based on (ensemble) adversarial training are roughly as vulnerable to low
frequency perturbations as undefended models, suggesting that the purported
robustness of proposed defenses is reliant upon adversarial perturbations being
high frequency in nature. We do find that under $\ell_\infty$
$\epsilon=16/255$, a commonly used distortion bound, low frequency
perturbations are indeed perceptible. This questions the use of the
$\ell_\infty$-norm, in particular, as a distortion metric, and suggests that
explicitly considering the frequency space is promising for learning robust
models which better align with human perception.


http://arxiv.org/abs/1902.11029
Enhancing the Robustness of Deep Neural Networks by Boundary Conditional GAN.
Ke Sun; Zhanxing Zhu; Zhouchen Lin
  Deep neural networks have been widely deployed in various machine learning
tasks. However, recent works have demonstrated that they are vulnerable to
adversarial examples: carefully crafted small perturbations to cause
misclassification by the network. In this work, we propose a novel defense
mechanism called Boundary Conditional GAN to enhance the robustness of deep
neural networks against adversarial examples. Boundary Conditional GAN, a
modified version of Conditional GAN, can generate boundary samples with true
labels near the decision boundary of a pre-trained classifier. These boundary
samples are fed to the pre-trained classifier as data augmentation to make the
decision boundary more robust. We empirically show that the model improved by
our approach consistently defenses against various types of adversarial attacks
successfully. Further quantitative investigations about the improvement of
robustness and visualization of decision boundaries are also provided to
justify the effectiveness of our strategy. This new defense mechanism that uses
boundary samples to enhance the robustness of networks opens up a new way to
defense adversarial attacks consistently.


http://arxiv.org/abs/1902.11019
Towards Understanding Adversarial Examples Systematically: Exploring Data Size, Task and Model Factors.
Ke Sun; Zhanxing Zhu; Zhouchen Lin
  Most previous works usually explained adversarial examples from several
specific perspectives, lacking relatively integral comprehension about this
problem. In this paper, we present a systematic study on adversarial examples
from three aspects: the amount of training data, task-dependent and
model-specific factors. Particularly, we show that adversarial generalization
(i.e. test accuracy on adversarial examples) for standard training requires
more data than standard generalization (i.e. test accuracy on clean examples);
and uncover the global relationship between generalization and robustness with
respect to the data size especially when data is augmented by generative
models. This reveals the trade-off correlation between standard generalization
and robustness in limited training data regime and their consistency when data
size is large enough. Furthermore, we explore how different task-dependent and
model-specific factors influence the vulnerability of deep neural networks by
extensive empirical analysis. Relevant recommendations on defense against
adversarial attacks are provided as well. Our results outline a potential path
towards the luminous and systematic understanding of adversarial examples.


http://arxiv.org/abs/1902.10899
Adversarial Attack and Defense on Point Sets.
Jiancheng Yang; Qiang Zhang; Rongyao Fang; Bingbing Ni; Jinxian Liu; Qi Tian
  Emergence of the utility of 3D point cloud data in critical vision tasks
(e.g., ADAS) urges researchers to pay more attention to the robustness of 3D
representations and deep networks. To this end, we develop an attack and
defense scheme, dedicated to 3D point cloud data, for preventing 3D point
clouds from manipulated as well as pursuing noise-tolerable 3D representation.
A set of novel 3D point cloud attack operations are proposed via pointwise
gradient perturbation and adversarial point attachment / detachment. We then
develop a flexible perturbation-measurement scheme for 3D point cloud data to
detect potential attack data or noisy sensing data. Extensive experimental
results on common point cloud benchmarks demonstrate the validity of the
proposed 3D attack and defense framework.


http://arxiv.org/abs/1902.10758
Stochastically Rank-Regularized Tensor Regression Networks.
Arinbjörn Kolbeinsson; Jean Kossaifi; Yannis Panagakis; Anima Anandkumar; Ioanna Tzoulaki; Paul Matthews
  Over-parametrization of deep neural networks has recently been shown to be
key to their successful training. However, it also renders them prone to
overfitting and makes them expensive to store and train. Tensor regression
networks significantly reduce the number of effective parameters in deep neural
networks while retaining accuracy and the ease of training. They replace the
flattening and fully-connected layers with a tensor regression layer, where the
regression weights are expressed through the factors of a low-rank tensor
decomposition. In this paper, to further improve tensor regression networks, we
propose a novel stochastic rank-regularization. It consists of a novel
randomized tensor sketching method to approximate the weights of tensor
regression layers. We theoretically and empirically establish the link between
our proposed stochastic rank-regularization and the dropout on low-rank tensor
regression. Extensive experimental results with both synthetic data and real
world datasets (i.e., CIFAR-100 and the UK Biobank brain MRI dataset) support
that the proposed approach i) improves performance in both classification and
regression tasks, ii) decreases overfitting, iii) leads to more stable training
and iv) improves robustness to adversarial attacks and random noise.


http://arxiv.org/abs/1902.10755
Adversarial Attacks on Time Series.
Fazle Karim; Somshubra Majumdar; Houshang Darabi
  Time series classification models have been garnering significant importance
in the research community. However, not much research has been done on
generating adversarial samples for these models. These adversarial samples can
become a security concern. In this paper, we propose utilizing an adversarial
transformation network (ATN) on a distilled model to attack various time series
classification models. The proposed attack on the classification model utilizes
a distilled model as a surrogate that mimics the behavior of the attacked
classical time series classification models. Our proposed methodology is
applied onto 1-Nearest Neighbor Dynamic Time Warping (1-NN ) DTW, a Fully
Connected Network and a Fully Convolutional Network (FCN), all of which are
trained on 42 University of California Riverside (UCR) datasets. In this paper,
we show both models were susceptible to attacks on all 42 datasets. To the best
of our knowledge, such an attack on time series classification models has never
been done before. Finally, we recommend future researchers that develop time
series classification models to incorporating adversarial data samples into
their training data sets to improve resilience on adversarial samples and to
consider model robustness as an evaluative metric.


http://arxiv.org/abs/1902.10674
Communication without Interception: Defense against Deep-Learning-based Modulation Detection.
Muhammad Zaid Hameed; Andras Gyorgy; Deniz Gunduz
  We consider a communication scenario, in which an intruder, employing a deep
neural network (DNN), tries to determine the modulation scheme of the
intercepted signal. Our aim is to minimize the accuracy of the intruder, while
guaranteeing that the intended receiver can still recover the underlying
message with the highest reliability. This is achieved by constellation
perturbation at the encoder, similarly to adversarial attacks against DNN-based
classifiers. In the latter perturbation is limited to be imperceptible to a
human observer, while in our case perturbation is constrained so that the
message can still be reliably decoded by the legitimate receiver which is
oblivious to the perturbation. Simulation results demonstrate the viability of
our approach to make wireless communication secure against DNN-based intruders
with minimal sacrifice in the communication performance.


http://arxiv.org/abs/1902.10660
Robust Decision Trees Against Adversarial Examples.
Hongge Chen; Huan Zhang; Duane Boning; Cho-Jui Hsieh
  Although adversarial examples and model robustness have been extensively
studied in the context of linear models and neural networks, research on this
issue in tree-based models and how to make tree-based models robust against
adversarial examples is still limited. In this paper, we show that tree based
models are also vulnerable to adversarial examples and develop a novel
algorithm to learn robust trees. At its core, our method aims to optimize the
performance under the worst-case perturbation of input features, which leads to
a max-min saddle point problem. Incorporating this saddle point objective into
the decision tree building procedure is non-trivial due to the discrete nature
of trees --- a naive approach to finding the best split according to this
saddle point objective will take exponential time. To make our approach
practical and scalable, we propose efficient tree building algorithms by
approximating the inner minimizer in this saddle point problem, and present
efficient implementations for classical information gain based trees as well as
state-of-the-art tree boosting models such as XGBoost. Experimental results on
real world datasets demonstrate that the proposed algorithms can substantially
improve the robustness of tree-based models against adversarial examples.


http://arxiv.org/abs/1902.11134
Disentangled Deep Autoencoding Regularization for Robust Image Classification.
Zhenyu Duan; Martin Renqiang Min; Li Erran Li; Mingbo Cai; Yi Xu; Bingbing Ni
  In spite of achieving revolutionary successes in machine learning, deep
convolutional neural networks have been recently found to be vulnerable to
adversarial attacks and difficult to generalize to novel test images with
reasonably large geometric transformations. Inspired by a recent neuroscience
discovery revealing that primate brain employs disentangled shape and
appearance representations for object recognition, we propose a general
disentangled deep autoencoding regularization framework that can be easily
applied to any deep embedding based classification model for improving the
robustness of deep neural networks. Our framework effectively learns
disentangled appearance code and geometric code for robust image
classification, which is the first disentangling based method defending against
adversarial attacks and complementary to standard defense methods. Extensive
experiments on several benchmark datasets show that, our proposed
regularization framework leveraging disentangled embedding significantly
outperforms traditional unregularized convolutional neural networks for image
classification on robustness against adversarial attacks and generalization to
novel test data.


http://arxiv.org/abs/1902.09866
Analyzing Deep Neural Networks with Symbolic Propagation: Towards Higher Precision and Faster Verification.
Pengfei Yang; Jiangchao Liu; Jianlin Li; Liqian Chen; Xiaowei Huang
  Deep neural networks (DNNs) have been shown lack of robustness for the
vulnerability of their classification to small perturbations on the inputs.
This has led to safety concerns of applying DNNs to safety-critical domains.
Several verification approaches have been developed to automatically prove or
disprove safety properties of DNNs. However, these approaches suffer from
either the scalability problem, i.e., only small DNNs can be handled, or the
precision problem, i.e., the obtained bounds are loose. This paper improves on
a recent proposal of analyzing DNNs through the classic abstract interpretation
technique, by a novel symbolic propagation technique. More specifically, the
values of neurons are represented symbolically and propagated forwardly from
the input layer to the output layer, on top of abstract domains. We show that
our approach can achieve significantly higher precision and thus can prove more
properties than using only abstract domains. Moreover, we show that the bounds
derived from our approach on the hidden neurons, when applied to a
state-of-the-art SMT based verification tool, can improve its performance. We
implement our approach into a software tool and validate it over a few DNNs
trained on benchmark datasets such as MNIST, etc.


http://arxiv.org/abs/1902.09592
Verification of Non-Linear Specifications for Neural Networks.
Chongli Dj Qin; Dj Krishnamurthy; Dvijotham; Brendan O'Donoghue; Rudy Bunel; Robert Stanforth; Sven Gowal; Jonathan Uesato; Grzegorz Swirszcz; Pushmeet Kohli
  Prior work on neural network verification has focused on specifications that
are linear functions of the output of the network, e.g., invariance of the
classifier output under adversarial perturbations of the input. In this paper,
we extend verification algorithms to be able to certify richer properties of
neural networks. To do this we introduce the class of convex-relaxable
specifications, which constitute nonlinear specifications that can be verified
using a convex relaxation. We show that a number of important properties of
interest can be modeled within this class, including conservation of energy in
a learned dynamics model of a physical system; semantic consistency of a
classifier's output labels under adversarial perturbations and bounding errors
in a system that predicts the summation of handwritten digits. Our experimental
evaluation shows that our method is able to effectively verify these
specifications. Moreover, our evaluation exposes the failure modes in models
which cannot be verified to satisfy these specifications. Thus, emphasizing the
importance of training models not just to fit training data but also to be
consistent with specifications.


http://arxiv.org/abs/1902.09286
Adversarial attacks hidden in plain sight.
Jan Philip Göpfert; Heiko Wersing; Barbara Hammer
  Convolutional neural networks have been used to achieve a string of successes
during recent years, but their lack of interpretability remains a serious
issue. Adversarial examples are designed to deliberately fool neural networks
into making any desired incorrect classification, potentially with very high
certainty. We underline the severity of the issue by presenting a technique
that allows to hide such adversarial attacks in regions of high complexity,
such that they are imperceptible even to an astute observer.


http://arxiv.org/abs/1902.08909
MaskDGA: A Black-box Evasion Technique Against DGA Classifiers and Adversarial Defenses.
Lior Sidi; Asaf Nadler; Asaf Shabtai
  Domain generation algorithms (DGAs) are commonly used by botnets to generate
domain names through which bots can establish a resilient communication channel
with their command and control servers. Recent publications presented deep
learning, character-level classifiers that are able to detect algorithmically
generated domain (AGD) names with high accuracy, and correspondingly,
significantly reduce the effectiveness of DGAs for botnet communication. In
this paper we present MaskDGA, a practical adversarial learning technique that
adds perturbation to the character-level representation of algorithmically
generated domain names in order to evade DGA classifiers, without the attacker
having any knowledge about the DGA classifier's architecture and parameters.
MaskDGA was evaluated using the DMD-2018 dataset of AGD names and four recently
published DGA classifiers, in which the average F1-score of the classifiers
degrades from 0.977 to 0.495 when applying the evasion technique. An additional
evaluation was conducted using the same classifiers but with adversarial
defenses implemented: adversarial re-training and distillation. The results of
this evaluation show that MaskDGA can be used for improving the robustness of
the character-level DGA classifiers against adversarial attacks, but that
ideally DGA classifiers should incorporate additional features alongside
character-level features that are demonstrated in this study to be vulnerable
to adversarial attacks.


http://arxiv.org/abs/1902.09062
Adversarial Reinforcement Learning under Partial Observability in Software-Defined Networking.
Yi Han; David Hubczenko; Paul Montague; Vel Olivier De; Tamas Abraham; Benjamin I. P. Rubinstein; Christopher Leckie; Tansu Alpcan; Sarah Erfani
  Recent studies have demonstrated that reinforcement learning (RL) agents are
susceptible to adversarial manipulation, similar to vulnerabilities previously
demonstrated in the supervised setting. Accordingly focus has remained with
computer vision, and full observability. This paper focuses on reinforcement
learning in the context of autonomous defence in Software-Defined Networking
(SDN). We demonstrate that causative attacks---attacks that target the training
process---can poison RL agents even if the attacker only has partial
observability of the environment. In addition, we propose an inversion defence
method that aims to apply the opposite perturbation to that which an attacker
might use to generate their adversarial samples. Our experimental results
illustrate that the countermeasure can effectively reduce the impact of the
causative attack, while not significantly affecting the training process in
non-attack scenarios.


http://arxiv.org/abs/1902.08832
Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses.
Ananya B. Sai; Mithun Das Gupta; Mitesh M. Khapra; Mukundhan Srinivasan
  Automatically evaluating the quality of dialogue responses for unstructured
domains is a challenging problem. ADEM(Lowe et al. 2017) formulated the
automatic evaluation of dialogue systems as a learning problem and showed that
such a model was able to predict responses which correlate significantly with
human judgements, both at utterance and system level. Their system was shown to
have beaten word-overlap metrics such as BLEU with large margins. We start with
the question of whether an adversary can game the ADEM model. We design a
battery of targeted attacks at the neural network based ADEM evaluation system
and show that automatic evaluation of dialogue systems still has a long way to
go. ADEM can get confused with a variation as simple as reversing the word
order in the text! We report experiments on several such adversarial scenarios
that draw out counterintuitive scores on the dialogue responses. We take a
systematic look at the scoring function proposed by ADEM and connect it to
linear system theory to predict the shortcomings evident in the system. We also
devise an attack that can fool such a system to rate a response generation
system as favorable. Finally, we allude to future research directions of using
the adversarial attacks to design a truly automated dialogue evaluation system.


http://arxiv.org/abs/1902.08785
A Deep, Information-theoretic Framework for Robust Biometric Recognition.
Renjie Xie; Yanzhi Chen; Yan Wo; Qiao Wang
  Deep neural networks (DNN) have been a de facto standard for nowadays
biometric recognition solutions. A serious, but still overlooked problem in
these DNN-based recognition systems is their vulnerability against adversarial
attacks. Adversarial attacks can easily cause the output of a DNN system to
greatly distort with only tiny changes in its input. Such distortions can
potentially lead to an unexpected match between a valid biometric and a
synthetic one constructed by a strategic attacker, raising security issue. In
this work, we show how this issue can be resolved by learning robust biometric
features through a deep, information-theoretic framework, which builds upon the
recent deep variational information bottleneck method but is carefully adapted
to biometric recognition tasks. Empirical evaluation demonstrates that our
method not only offers stronger robustness against adversarial attacks but also
provides better recognition performance over state-of-the-art approaches.


http://arxiv.org/abs/1902.08412
Adversarial Attacks on Graph Neural Networks via Meta Learning.
Daniel Zügner; Stephan Günnemann
  Deep learning models for graphs have advanced the state of the art on many
tasks. Despite their recent success, little is known about their robustness. We
investigate training time attacks on graph neural networks for node
classification that perturb the discrete graph structure. Our core principle is
to use meta-gradients to solve the bilevel problem underlying training-time
attacks, essentially treating the graph as a hyperparameter to optimize. Our
experiments show that small graph perturbations consistently lead to a strong
decrease in performance for graph convolutional networks, and even transfer to
unsupervised embeddings. Remarkably, the perturbations created by our algorithm
can misguide the graph neural networks such that they perform worse than a
simple baseline that ignores all relational information. Our attacks do not
assume any knowledge about or access to the target classifiers.


http://arxiv.org/abs/1902.08391
Physical Adversarial Attacks Against End-to-End Autoencoder Communication Systems.
Meysam Sadeghi; Erik G. Larsson
  We show that end-to-end learning of communication systems through deep neural
network (DNN) autoencoders can be extremely vulnerable to physical adversarial
attacks. Specifically, we elaborate how an attacker can craft effective
physical black-box adversarial attacks. Due to the openness (broadcast nature)
of the wireless channel, an adversary transmitter can increase the
block-error-rate of a communication system by orders of magnitude by
transmitting a well-designed perturbation signal over the channel. We reveal
that the adversarial attacks are more destructive than jamming attacks. We also
show that classical coding schemes are more robust than autoencoders against
both adversarial and jamming attacks. The codes are available at [1].


http://arxiv.org/abs/1902.08722
A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks.
Hadi Salman; Greg Yang; Huan Zhang; Cho-Jui Hsieh; Pengchuan Zhang
  Verification of neural networks enables us to gauge their robustness against
adversarial attacks. Verification algorithms fall into two categories: exact
verifiers that run in exponential time and relaxed verifiers that are efficient
but incomplete. In this paper, we unify all existing LP-relaxed verifiers, to
the best of our knowledge, under a general convex relaxation framework. This
framework works for neural networks with diverse architectures and
nonlinearities and covers both primal and dual views of robustness
verification. We further prove strong duality between the primal and dual
problems under very mild conditions. Next, we perform large-scale experiments,
amounting to more than 22 CPU-years, to obtain exact solution to the
convex-relaxed problem that is optimal within our framework for ReLU networks.
We find the exact solution does not significantly improve upon the gap between
PGD and existing relaxed verifiers for various networks trained normally or
robustly on MNIST and CIFAR datasets. Our results suggest there is an inherent
barrier to tight verification for the large class of methods captured by our
framework. We discuss possible causes of this barrier and potential future
directions for bypassing it.


http://arxiv.org/abs/1902.08336
On the Sensitivity of Adversarial Robustness to Input Data Distributions.
Gavin Weiguang Ding; Kry Yik Chau Lui; Xiaomeng Jin; Luyu Wang; Ruitong Huang
  Neural networks are vulnerable to small adversarial perturbations. Existing
literature largely focused on understanding and mitigating the vulnerability of
learned models. In this paper, we demonstrate an intriguing phenomenon about
the most popular robust training method in the literature, adversarial
training: Adversarial robustness, unlike clean accuracy, is sensitive to the
input data distribution. Even a semantics-preserving transformations on the
input data distribution can cause a significantly different robustness for the
adversarial trained model that is both trained and evaluated on the new
distribution. Our discovery of such sensitivity on data distribution is based
on a study which disentangles the behaviors of clean accuracy and robust
accuracy of the Bayes classifier. Empirical investigations further confirm our
finding. We construct semantically-identical variants for MNIST and CIFAR10
respectively, and show that standardly trained models achieve comparable clean
accuracies on them, but adversarially trained models achieve significantly
different robustness accuracies. This counter-intuitive phenomenon indicates
that input data distribution alone can affect the adversarial robustness of
trained neural networks, not necessarily the tasks themselves. Lastly, we
discuss the practical implications on evaluating adversarial robustness, and
make initial attempts to understand this complex phenomenon.


http://arxiv.org/abs/1902.08265
Quantifying Perceptual Distortion of Adversarial Examples.
Matt Jordan; Naren Manoj; Surbhi Goel; Alexandros G. Dimakis
  Recent work has shown that additive threat models, which only permit the
addition of bounded noise to the pixels of an image, are insufficient for fully
capturing the space of imperceivable adversarial examples. For example, small
rotations and spatial transformations can fool classifiers, remain
imperceivable to humans, but have large additive distance from the original
images. In this work, we leverage quantitative perceptual metrics like LPIPS
and SSIM to define a novel threat model for adversarial attacks.
  To demonstrate the value of quantifying the perceptual distortion of
adversarial examples, we present and employ a unifying framework fusing
different attack styles. We first prove that our framework results in images
that are unattainable by attack styles in isolation. We then perform
adversarial training using attacks generated by our framework to demonstrate
that networks are only robust to classes of adversarial perturbations they have
been trained against, and combination attacks are stronger than any of their
individual components. Finally, we experimentally demonstrate that our combined
attacks retain the same perceptual distortion but induce far higher
misclassification rates when compared against individual attacks.


http://arxiv.org/abs/1902.07906
Wasserstein Adversarial Examples via Projected Sinkhorn Iterations.
Eric Wong; Frank R. Schmidt; J. Zico Kolter
  A rapidly growing area of work has studied the existence of adversarial
examples, datapoints which have been perturbed to fool a classifier, but the
vast majority of these works have focused primarily on threat models defined by
$\ell_p$ norm-bounded perturbations. In this paper, we propose a new threat
model for adversarial attacks based on the Wasserstein distance. In the image
classification setting, such distances measure the cost of moving pixel mass,
which naturally cover "standard" image manipulations such as scaling, rotation,
translation, and distortion (and can potentially be applied to other settings
as well). To generate Wasserstein adversarial examples, we develop a procedure
for projecting onto the Wasserstein ball, based upon a modified version of the
Sinkhorn iteration. The resulting algorithm can successfully attack image
classification models, bringing traditional CIFAR10 models down to 3% accuracy
within a Wasserstein ball with radius 0.1 (i.e., moving 10% of the image mass 1
pixel), and we demonstrate that PGD-based adversarial training can improve this
adversarial accuracy to 76%. In total, this work opens up a new direction of
study in adversarial robustness, more formally considering convex metrics that
accurately capture the invariances that we typically believe should exist in
classifiers. Code for all experiments in the paper is available at
https://github.com/locuslab/projected_sinkhorn.


http://arxiv.org/abs/1902.08226
Graph Adversarial Training: Dynamically Regularizing Based on Graph Structure.
Fuli Feng; Xiangnan He; Jie Tang; Tat-Seng Chua
  Recent efforts show that neural networks are vulnerable to small but
intentional perturbations on input features in visual classification tasks. Due
to the additional consideration of connections between examples (e.g., articles
with citation link tend to be in the same class), graph neural networks could
be more sensitive to the perturbations, since the perturbations from connected
examples exacerbate the impact on a target example. Adversarial Training (AT),
a dynamic regularization technique, can resist the worst-case perturbations on
input features and is a promising choice to improve model robustness and
generalization. However, existing AT methods focus on standard classification,
being less effective when training models on graph since it does not model the
impact from connected examples.
  In this work, we explore adversarial training on graph, aiming to improve the
robustness and generalization of models learned on graph. We propose Graph
Adversarial Training (GAT), which takes the impact from connected examples into
account when learning to construct and resist perturbations. We give a general
formulation of GAT, which can be seen as a dynamic regularization scheme based
on the graph structure. To demonstrate the utility of GAT, we employ it on a
state-of-the-art graph neural network model --- Graph Convolutional Network
(GCN). We conduct experiments on two citation graphs (Citeseer and Cora) and a
knowledge graph (NELL), verifying the effectiveness of GAT which outperforms
normal training on GCN by 4.51% in node classification accuracy. Codes will be
released upon acceptance.


http://arxiv.org/abs/1902.07776
Perceptual Quality-preserving Black-Box Attack against Deep Learning Image Classifiers.
Diego Gragnaniello; Francesco Marra; Giovanni Poggi; Luisa Verdoliva
  Deep neural networks provide unprecedented performance in all image
classification problems, leveraging the availability of huge amounts of data
for training. Recent studies, however, have shown their vulnerability to
adversarial attacks, spawning an intense research effort in this field. With
the aim of building better systems, new countermeasures and stronger attacks
are proposed by the day. On the attacker's side, there is growing interest for
the realistic black-box scenario, in which the user has no access to the neural
network parameters. The problem is to design limited-complexity attacks which
mislead the neural network without impairing image quality too much, not to
raise the attention of human observers. In this work, we put special emphasis
on this latter requirement and propose a powerful and low-complexity black-box
attack which preserves perceptual image quality. Numerical experiments prove
the effectiveness of the proposed techniques both for tasks commonly considered
in this context, and for other applications in biometrics (face recognition)
and forensics (camera model identification).


http://arxiv.org/abs/1902.07623
advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch.
Gavin Weiguang Ding; Luyu Wang; Xiaomeng Jin
  advertorch is a toolbox for adversarial robustness research. It contains
various implementations for attacks, defenses and robust training methods.
advertorch is built on PyTorch (Paszke et al., 2017), and leverages the
advantages of the dynamic computational graph to provide concise and efficient
reference implementations. The code is licensed under the LGPL license and is
open sourced at https://github.com/BorealisAI/advertorch .


http://arxiv.org/abs/1902.06894
There are No Bit Parts for Sign Bits in Black-Box Attacks.
Abdullah Al-Dujaili; Una-May O'Reilly
  We present a black-box adversarial attack algorithm which sets new
state-of-the-art model evasion rates for query efficiency in the $\ell_\infty$
and $\ell_2$ metrics, where only loss-oracle access to the model is available.
On two public black-box attack challenges, the algorithm achieves the highest
evasion rate, surpassing all of the submitted attacks. Similar performance is
observed on a model that is secure against substitute-model attacks. For
standard models trained on the MNIST, CIFAR10, and IMAGENET datasets, averaged
over the datasets and metrics, the algorithm is 3.8x less failure-prone, and
spends in total 2.5x fewer queries than the current state-of-the-art attacks
combined given a budget of 10, 000 queries per attack attempt. Notably, it
requires no hyperparameter tuning or any data/time-dependent prior. The
algorithm exploits a new approach, namely sign-based rather than
magnitude-based gradient estimation. This shifts the estimation from continuous
to binary black-box optimization. With three properties of the directional
derivative, we examine three approaches to adversarial attacks. This yields a
superior algorithm breaking a standard MNIST model using just 12 queries on
average!


http://arxiv.org/abs/1902.06705
On Evaluating Adversarial Robustness.
Nicholas Carlini; Anish Athalye; Nicolas Papernot; Wieland Brendel; Jonas Rauber; Dimitris Tsipras; Ian Goodfellow; Aleksander Madry; Alexey Kurakin
  Correctly evaluating defenses against adversarial examples has proven to be
extremely difficult. Despite the significant amount of recent work attempting
to design defenses that withstand adaptive attacks, few have succeeded; most
papers that propose defenses are quickly shown to be incorrect.
  We believe a large contributing factor is the difficulty of performing
security evaluations. In this paper, we discuss the methodological foundations,
review commonly accepted best practices, and suggest new methods for evaluating
defenses to adversarial examples. We hope that both researchers developing
defenses as well as readers and reviewers who wish to understand the
completeness of an evaluation consider our advice in order to avoid common
pitfalls.


http://arxiv.org/abs/1902.06415
AuxBlocks: Defense Adversarial Example via Auxiliary Blocks.
Yueyao Yu; Pengfei Yu; Wenye Li
  Deep learning models are vulnerable to adversarial examples, which poses an
indisputable threat to their applications. However, recent studies observe
gradient-masking defenses are self-deceiving methods if an attacker can realize
this defense. In this paper, we propose a new defense method based on appending
information. We introduce the Aux Block model to produce extra outputs as a
self-ensemble algorithm and analytically investigate the robustness mechanism
of Aux Block. We have empirically studied the efficiency of our method against
adversarial examples in two types of white-box attacks, and found that even in
the full white-box attack where an adversary can craft malicious examples from
defense models, our method has a more robust performance of about 54.6%
precision on Cifar10 dataset and 38.7% precision on Mini-Imagenet dataset.
Another advantage of our method is that it is able to maintain the prediction
accuracy of the classification model on clean images, and thereby exhibits its
high potential in practical applications


http://arxiv.org/abs/1902.06626
Mockingbird: Defending Against Deep-Learning-Based Website Fingerprinting Attacks with Adversarial Traces.
Mohsen Imani; Mohammad Saidur Rahman; Nate Mathews; Matthew Wright
  Website Fingerprinting (WF) is a type of traffic analysis attack that enables
a local passive eavesdropper to infer the victim's activity even when the
traffic is protected by encryption, a VPN, or some other anonymity system like
Tor. Leveraging a deep-learning classifier, a WF attacker can gain up to 98%
accuracy against Tor. Existing WF defenses are either too expensive in terms of
bandwidth and latency overheads (e.g. two-to-three times as large or slow) or
ineffective against the latest attacks. In this paper, we explore a novel
defense, Mockingbird, based on the idea of adversarial examples that have been
shown to undermine machine learning classifiers in other domains. To make the
technique more robust than existing algorithms for finding adversarial
examples, our approach aims to make the source trace more similar to a randomly
selected target and use the neural network only to determine when the modified
trace will be misclassified with high confidence. The technique drops the
accuracy of the state-of-the-art attack hardened with adversarial training from
95% to between 29-57%, depending on the scenario, while incurring a reasonable
56% bandwidth overhead. The attack accuracy is generally lower than
state-of-the-art defenses, and much lower when considering Top-2 accuracy,
while incurring lower overheads in most settings. In addition, the information
leakage of Mockingbird is at most 1.9 bits, whereas it is at most 2.2 bits for
W-T and 2.6 bits for WTF-PAD.


http://arxiv.org/abs/1902.08034
Mitigation of Adversarial Examples in RF Deep Classifiers Utilizing AutoEncoder Pre-training.
Silvija Kokalj-Filipovic; Rob Miller; Nicholas Chang; Chi Leung Lau
  Adversarial examples in machine learning for images are widely publicized and
explored. Illustrations of misclassifications caused by slightly perturbed
inputs are abundant and commonly known (e.g., a picture of panda imperceptibly
perturbed to fool the classifier into incorrectly labeling it as a gibbon).
Similar attacks on deep learning (DL) for radio frequency (RF) signals and
their mitigation strategies are scarcely addressed in the published work. Yet,
RF adversarial examples (AdExs) with minimal waveform perturbations can cause
drastic, targeted misclassification results, particularly against spectrum
sensing/survey applications (e.g. BPSK is mistaken for 8-PSK). Our research on
deep learning AdExs and proposed defense mechanisms are RF-centric, and
incorporate physical world, over-the-air (OTA) effects. We herein present
defense mechanisms based on pre-training the target classifier using an
autoencoder. Our results validate this approach as a viable mitigation method
to subvert adversarial attacks against deep learning-based communications and
radar sensing systems.


http://arxiv.org/abs/1902.06044
Adversarial Examples in RF Deep Learning: Detection of the Attack and its Physical Robustness.
Silvija Kokalj-Filipovic; Rob Miller
  While research on adversarial examples in machine learning for images has
been prolific, similar attacks on deep learning (DL) for radio frequency (RF)
signals and their mitigation strategies are scarcely addressed in the published
work, with only one recent publication in the RF domain [1]. RF adversarial
examples (AdExs) can cause drastic, targeted misclassification results mostly
in spectrum sensing/ survey applications (e.g. BPSK mistaken for 8-PSK) with
minimal waveform perturbation. It is not clear if the RF AdExs maintain their
effects in the physical world, i.e., when AdExs are delivered over-the-air
(OTA). Our research on deep learning AdExs and proposed defense mechanisms are
RF-centric, and incorporate physical world, OTA effects. We here present
defense mechanisms based on statistical tests. One test to detect AdExs
utilizes Peak-to- Average-Power-Ratio (PAPR) of the DL data points delivered
OTA, while another statistical test uses the Softmax outputs of the DL
classifier, which corresponds to the probabilities the classifier assigns to
each of the trained classes. The former test leverages the RF nature of the
data, and the latter is universally applicable to AdExs regardless of their
origin. Both solutions are shown as viable mitigation methods to subvert
adversarial attacks against communications and radar sensing systems.


http://arxiv.org/abs/1902.05974
DeepFault: Fault Localization for Deep Neural Networks.
Hasan Ferit Eniser; Simos Gerasimou; Alper Sen
  Deep Neural Networks (DNNs) are increasingly deployed in safety-critical
applications including autonomous vehicles and medical diagnostics. To reduce
the residual risk for unexpected DNN behaviour and provide evidence for their
trustworthy operation, DNNs should be thoroughly tested. The DeepFault whitebox
DNN testing approach presented in our paper addresses this challenge by
employing suspiciousness measures inspired by fault localization to establish
the hit spectrum of neurons and identify suspicious neurons whose weights have
not been calibrated correctly and thus are considered responsible for
inadequate DNN performance. DeepFault also uses a suspiciousness-guided
algorithm to synthesize new inputs, from correctly classified inputs, that
increase the activation values of suspicious neurons. Our empirical evaluation
on several DNN instances trained on MNIST and CIFAR-10 datasets shows that
DeepFault is effective in identifying suspicious neurons. Also, the inputs
synthesized by DeepFault closely resemble the original inputs, exercise the
identified suspicious neurons and are highly adversarial.


http://arxiv.org/abs/1902.05586
Can Intelligent Hyperparameter Selection Improve Resistance to Adversarial Examples?.
Cody Burkard; Brent Lagesse
  Convolutional Neural Networks and Deep Learning classification systems in
general have been shown to be vulnerable to attack by specially crafted data
samples that appear to belong to one class but are instead classified as
another, commonly known as adversarial examples. A variety of attack strategies
have been proposed to craft these samples; however, there is no standard model
that is used to compare the success of each type of attack. Furthermore, there
is no literature currently available that evaluates how common hyperparameters
and optimization strategies may impact a model's ability to resist these
samples. This research bridges that lack of awareness and provides a means for
the selection of training and model parameters in future research on evasion
attacks against convolutional neural networks. The findings of this work
indicate that the selection of model hyperparameters does impact the ability of
a model to resist attack, although they alone cannot prevent the existence of
adversarial examples.


http://arxiv.org/abs/1902.04818
The Odds are Odd: A Statistical Test for Detecting Adversarial Examples.
Kevin Roth; Yannic Kilcher; Thomas Hofmann
  We investigate conditions under which test statistics exist that can reliably
detect examples, which have been adversarially manipulated in a white-box
attack. These statistics can be easily computed and calibrated by randomly
corrupting inputs. They exploit certain anomalies that adversarial attacks
introduce, in particular if they follow the paradigm of choosing perturbations
optimally under p-norm constraints. Access to the log-odds is the only
requirement to defend models. We justify our approach empirically, but also
provide conditions under which detectability via the suggested test statistics
is guaranteed to be effective. In our experiments, we show that it is even
possible to correct test time predictions for adversarial attacks with high
accuracy.


http://arxiv.org/abs/1902.04416
Examining Adversarial Learning against Graph-based IoT Malware Detection Systems.
Ahmed Abusnaina; Aminollah Khormali; Hisham Alasmary; Jeman Park; Afsah Anwar; Ulku Meteriz; Aziz Mohaisen
  The main goal of this study is to investigate the robustness of graph-based
Deep Learning (DL) models used for Internet of Things (IoT) malware
classification against Adversarial Learning (AL). We designed two approaches to
craft adversarial IoT software, including Off-the-Shelf Adversarial Attack
(OSAA) methods, using six different AL attack approaches, and Graph Embedding
and Augmentation (GEA). The GEA approach aims to preserve the functionality and
practicality of the generated adversarial sample through a careful embedding of
a benign sample to a malicious one. Our evaluations demonstrate that OSAAs are
able to achieve a misclassification rate (MR) of 100%. Moreover, we observed
that the GEA approach is able to misclassify all IoT malware samples as benign.


http://arxiv.org/abs/1902.07285
Towards a Robust Deep Neural Network in Text Domain A Survey.
Wenqi Wang; Lina Wang; Benxiao Tang; Run Wang; Aoshuang Ye
  Deep neural networks (DNNs) have shown an inherent vulnerability to
adversarial examples which are maliciously crafted on real examples by
attackers, aiming at making target DNNs misbehave. The threats of adversarial
examples are widely existed in image, voice, speech, and text recognition and
classification. Inspired by the previous work, researches on adversarial
attacks and defenses in text domain develop rapidly. In order to make people
have a general understanding about the field, this article presents a
comprehensive review on adversarial examples in text. We analyze the advantages
and shortcomings of recent adversarial examples generation methods and
elaborate the efficiency and limitations on countermeasures. Finally, we
discuss the challenges in adversarial texts and provide a research direction of
this aspect.


http://arxiv.org/abs/1902.04238
Adversarial Samples on Android Malware Detection Systems for IoT Systems.
Xiaolei Liu; Xiaojiang Du; Xiaosong Zhang; Qingxin Zhu; Mohsen Guizani
  Many IoT(Internet of Things) systems run Android systems or Android-like
systems. With the continuous development of machine learning algorithms, the
learning-based Android malware detection system for IoT devices has gradually
increased. However, these learning-based detection models are often vulnerable
to adversarial samples. An automated testing framework is needed to help these
learning-based malware detection systems for IoT devices perform security
analysis. The current methods of generating adversarial samples mostly require
training parameters of models and most of the methods are aimed at image data.
To solve this problem, we propose a \textbf{t}esting framework for
\textbf{l}earning-based \textbf{A}ndroid \textbf{m}alware \textbf{d}etection
systems(TLAMD) for IoT Devices. The key challenge is how to construct a
suitable fitness function to generate an effective adversarial sample without
affecting the features of the application. By introducing genetic algorithms
and some technical improvements, our test framework can generate adversarial
samples for the IoT Android Application with a success rate of nearly 100\% and
can perform black-box testing on the system.


http://arxiv.org/abs/1902.03538
Adversarially Trained Model Compression: When Robustness Meets Efficiency.
Shupeng University of Rochester Gui; Haotao Texas A&M University Wang; Chen University of Rochester Yu; Haichuan University of Rochester Yang; Zhangyang Texas A&M University Wang; Ji University of Rochester Liu
  The robustness of deep models to adversarial attacks has gained significant
attention in recent years, so has the model compactness and efficiency: yet the
two have been mostly studied separately, with few relationships drawn between
each other. This paper is concerned with: how can we combine the best of both
worlds, obtaining a robust and compact network? The answer is not as
straightforward as it may seem, since the two goals of model robustness and
compactness may contradict from time to time. We formally study this new
question, by proposing a novel Adversarially Trained Model Compression (ATMC)
framework. A unified constrained optimization formulation is designed, with an
efficient algorithm developed. An extensive group of experiments are then
carefully designed and presented, demonstrating that ATMC obtains remarkably
more favorable trade-off among model size, accuracy and robustness, over
currently available alternatives in various settings.


http://arxiv.org/abs/1902.03380
When Causal Intervention Meets Adversarial Examples and Image Masking for Deep Neural Networks.
Chao-Han Huck Yang; Yi-Chieh Liu; Pin-Yu Chen; Xiaoli Ma; Yi-Chang James Tsai
  Discovering and exploiting the causality in deep neural networks (DNNs) are
crucial challenges for understanding and reasoning causal effects (CE) on an
explainable visual model. "Intervention" has been widely used for recognizing a
causal relation ontologically. In this paper, we propose a causal inference
framework for visual reasoning via do-calculus. To study the intervention
effects on pixel-level features for causal reasoning, we introduce pixel-wise
masking and adversarial perturbation. In our framework, CE is calculated using
features in a latent space and perturbed prediction from a DNN-based model. We
further provide the first look into the characteristics of discovered CE of
adversarially perturbed images generated by gradient-based methods
\footnote{~~https://github.com/jjaacckkyy63/Causal-Intervention-AE-wAdvImg}.
Experimental results show that CE is a competitive and robust index for
understanding DNNs when compared with conventional methods such as
class-activation mappings (CAMs) on the Chest X-Ray-14 dataset for
human-interpretable feature(s) (e.g., symptom) reasoning. Moreover, CE holds
promises for detecting adversarial examples as it possesses distinct
characteristics in the presence of adversarial perturbations.


http://arxiv.org/abs/1902.03227
Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Images.
Sanjana Srivastava; Guy Ben-Yosef; Xavier Boix
  The human ability to recognize objects is impaired when the object is not
shown in full. "Minimal images" are the smallest regions of an image that
remain recognizable for humans. Ullman et al. 2016 show that a slight
modification of the location and size of the visible region of the minimal
image produces a sharp drop in human recognition accuracy. In this paper, we
demonstrate that such drops in accuracy due to changes of the visible region
are a common phenomenon between humans and existing state-of-the-art deep
neural networks (DNNs), and are much more prominent in DNNs. We found many
cases where DNNs classified one region correctly and the other incorrectly,
though they only differed by one row or column of pixels, and were often bigger
than the average human minimal image size. We show that this phenomenon is
independent from previous works that have reported lack of invariance to minor
modifications in object location in DNNs. Our results thus reveal a new failure
mode of DNNs that also affects humans to a much lesser degree. They expose how
fragile DNN recognition ability is for natural images even without adversarial
patterns being introduced. Bringing the robustness of DNNs in natural images to
the human level remains an open challenge for the community.


http://arxiv.org/abs/1902.02947
Understanding the One-Pixel Attack: Propagation Maps and Locality Analysis.
Danilo Vasconcellos Vargas; Jiawei Su
  Deep neural networks were shown to be vulnerable to single pixel
modifications. However, the reason behind such phenomena has never been
elucidated. Here, we propose Propagation Maps which show the influence of the
perturbation in each layer of the network. Propagation Maps reveal that even in
extremely deep networks such as Resnet, modification in one pixel easily
propagates until the last layer. In fact, this initial local perturbation is
also shown to spread becoming a global one and reaching absolute difference
values that are close to the maximum value of the original feature maps in a
given layer. Moreover, we do a locality analysis in which we demonstrate that
nearby pixels of the perturbed one in the one-pixel attack tend to share the
same vulnerability, revealing that the main vulnerability lies in neither
neurons nor pixels but receptive fields. Hopefully, the analysis conducted in
this work together with a new technique called propagation maps shall shed
light into the inner workings of other adversarial samples and be the basis of
new defense systems to come.


http://arxiv.org/abs/1902.03151
Discretization based Solutions for Secure Machine Learning against Adversarial Attacks.
Priyadarshini Panda; Indranil Chakraborty; Kaushik Roy
  Adversarial examples are perturbed inputs that are designed (from a deep
learning network's (DLN) parameter gradients) to mislead the DLN during test
time. Intuitively, constraining the dimensionality of inputs or parameters of a
network reduces the 'space' in which adversarial examples exist. Guided by this
intuition, we demonstrate that discretization greatly improves the robustness
of DLNs against adversarial attacks. Specifically, discretizing the input space
(or allowed pixel levels from 256 values or 8-bit to 4 values or 2-bit)
extensively improves the adversarial robustness of DLNs for a substantial range
of perturbations for minimal loss in test accuracy. Furthermore, we find that
Binary Neural Networks (BNNs) and related variants are intrinsically more
robust than their full precision counterparts in adversarial scenarios.
Combining input discretization with BNNs furthers the robustness even waiving
the need for adversarial training for certain magnitude of perturbation values.
We evaluate the effect of discretization on MNIST, CIFAR10, CIFAR100 and
Imagenet datasets. Across all datasets, we observe maximal adversarial
resistance with 2-bit input discretization that incurs an adversarial accuracy
loss of just ~1-2% as compared to clean test accuracy.


http://arxiv.org/abs/1902.02826
Robustness Of Saak Transform Against Adversarial Attacks.
Thiyagarajan Ramanathan; Abinaya Manimaran; Suya You; C-C Jay Kuo
  Image classification is vulnerable to adversarial attacks. This work
investigates the robustness of Saak transform against adversarial attacks
towards high performance image classification. We develop a complete image
classification system based on multi-stage Saak transform. In the Saak
transform domain, clean and adversarial images demonstrate different
distributions at different spectral dimensions. Selection of the spectral
dimensions at every stage can be viewed as an automatic denoising process.
Motivated by this observation, we carefully design strategies of feature
extraction, representation and classification that increase adversarial
robustness. The performances with well-known datasets and attacks are
demonstrated by extensive experimental evaluations.


http://arxiv.org/abs/1902.02918
Certified Adversarial Robustness via Randomized Smoothing.
Jeremy M Cohen; Elan Rosenfeld; J. Zico Kolter
  We show how to turn any classifier that classifies well under Gaussian noise
into a new classifier that is certifiably robust to adversarial perturbations
under the $\ell_2$ norm. This "randomized smoothing" technique has been
proposed recently in the literature, but existing guarantees are loose. We
prove a tight robustness guarantee in $\ell_2$ norm for smoothing with Gaussian
noise. We use randomized smoothing to obtain an ImageNet classifier with e.g. a
certified top-1 accuracy of 49% under adversarial perturbations with $\ell_2$
norm less than 0.5 (=127/255). No certified defense has been shown feasible on
ImageNet except for smoothing. On smaller-scale datasets where competing
approaches to certified $\ell_2$ robustness are viable, smoothing delivers
higher certified accuracies. Our strong empirical results suggest that
randomized smoothing is a promising direction for future research into
adversarially robust classification. Code and models are available at
http://github.com/locuslab/smoothing.


http://arxiv.org/abs/1902.02067
Daedalus: Breaking Non-Maximum Suppression in Object Detection via Adversarial Examples.
Derui Wang; Chaoran Li; Sheng Wen; Surya Nepal; Yang Xiang
  We demonstrated that Non-Maximum Suppression (NMS), which is commonly used in
object detection tasks to filter redundant detection results, is no longer
secure. NMS has always been an integral part of object detection algorithms.
Currently, Fully Convolutional Network (FCN) is widely used as the backbone
architecture of object detection models. Given an input instance, since FCN
generates end-to-end detection results in a single stage, it outputs a large
number of raw detection boxes. These bounding boxes are then filtered by NMS to
make the final detection results.
  In this paper, we propose an adversarial example attack which triggers
malfunctioning of NMS in the end-to-end object detection models. Our attack,
namely Daedalus, manipulates the detection box regression values to compress
the dimensions of detection boxes. Henceforth, NMS will no longer be able to
filter redundant detection boxes correctly. And as a result, the final
detection output contains extremely dense false positives. This can be fatal
for many object detection applications such as autonomous vehicle and smart
manufacturing industry. Our attack can be applied to different end-to-end
object detection models. Furthermore, we suggest crafting robust adversarial
examples by using an ensemble of popular detection models as the substitutes.
Considering that model reusing is commonly seen in real-world object detection
scenarios, Daedalus examples crafted based on an ensemble of substitutes can
launch attacks without knowing the details of the victim models. Our
experiments demonstrate that our attack effectively stops NMS from filtering
redundant bounding boxes. As the evaluation results suggest, Daedalus increases
the false positive rate in detection results to 99.9% and reduces the mean
average precision scores to 0, while maintaining a low cost of distortion on
the original inputs.


http://arxiv.org/abs/1902.02041
Fooling Neural Network Interpretations via Adversarial Model Manipulation.
Juyeon Heo; Sunghwan Joo; Taesup Moon
  We ask whether the neural network interpretation methods can be fooled via
adversarial model manipulation, which is defined as a model fine-tuning step
that aims to radically alter the explanations without hurting the accuracy of
the original model. By incorporating the interpretation results directly in the
regularization term of the objective function for fine-tuning, we show that the
state-of-the-art interpreters, e.g., LRP and Grad-CAM, can be easily fooled
with our model manipulation. We propose two types of fooling, passive and
active, and demonstrate such foolings generalize well to the entire validation
set as well as transfer to other interpretation methods. Our results are
validated by both visually showing the fooled explanations and reporting
quantitative metrics that measure the deviations from the original
explanations. We claim that the stability of neural network interpretation
method with respect to our adversarial model manipulation is an important
criterion to check for developing robust and reliable neural network
interpretation method.


http://arxiv.org/abs/1902.01686
Fatal Brain Damage.
El Mahdi El Mhamdi; Rachid Guerraoui; Sergei Volodin
  The loss of a few neurons in a brain often does not result in a visible loss
of function. We propose to advance the understanding of neural networks through
their remarkable ability to sustain individual neuron failures, i.e. their
fault tolerance. Before the last AI winter, fault tolerance in NNs was a
popular topic as NNs were expected to be implemented in neuromorphic hardware,
which for a while did not happen. Moreover, since the number of possible crash
subsets grows exponentially with the network size, additional assumptions are
required to practically study this phenomenon for modern architectures. We
prove a series of bounds on error propagation using justified assumptions,
applicable to deep networks, show their location on the complexity versus
tightness trade-off scale and test them empirically. We demonstrate how fault
tolerance is connected to generalization and show that the data jacobian of a
network determines its fault tolerance properties. We investigate this quantity
and show how it is interlinked with other mathematical properties of the
network such as Lipschitzness, singular values, weight matrices norms, and the
loss gradients. Known results give a connection between the data jacobian and
robustness to adversarial examples, providing another piece of the puzzle.
Combining that with our results, we call for a unifying research endeavor
encompassing fault tolerance, generalization capacity, and robustness to
adversarial inputs together as we demonstrate a strong connection between these
areas. Moreover, we argue that fault tolerance is an important overlooked AI
safety problem since neuromorphic hardware is becoming popular again.


http://arxiv.org/abs/1902.01147
SNN under Attack: are Spiking Deep Belief Networks vulnerable to Adversarial Examples?.
Alberto Marchisio; Giorgio Nanfa; Faiq Khalid; Muhammad Abdullah Hanif; Maurizio Martina; Muhammad Shafique
  Recently, many adversarial examples have emerged for Deep Neural Networks
(DNNs) causing misclassifications. However, in-depth work still needs to be
performed to demonstrate such attacks and security vulnerabilities for spiking
neural networks (SNNs), i.e. the 3rd generation NNs. This paper aims at
addressing the fundamental questions:"Are SNNs vulnerable to the adversarial
attacks as well?" and "if yes, to what extent?" Using a Spiking Deep Belief
Network (SDBN) for the MNIST database classification, we show that the SNN
accuracy decreases accordingly to the noise magnitude in data poisoning random
attacks applied to the test images. Moreover, SDBNs generalization capabilities
increase by applying noise to the training images. We develop a novel black box
attack methodology to automatically generate imperceptible and robust
adversarial examples through a greedy algorithm, which is first of its kind for
SNNs.


http://arxiv.org/abs/1902.01148
Theoretical evidence for adversarial robustness through randomization.
Rafael Pinot; Laurent Meunier; Alexandre Araujo; Hisashi Kashima; Florian Yger; Cédric Gouy-Pailler; Jamal Atif
  This paper investigates the theory of robustness against adversarial attacks.
It focuses on the family of randomization techniques that consist in injecting
noise in the network at inference time. These techniques have proven effective
in many contexts, but lack theoretical arguments. We close this gap by
presenting a theoretical analysis of these approaches, hence explaining why
they perform well in practice. More precisely, we make two new contributions.
The first one relates the randomization rate to robustness to adversarial
attacks. This result applies for the general family of exponential
distributions, and thus extends and unifies the previous approaches. The second
contribution consists in devising a new upper bound on the adversarial
generalization gap of randomized neural networks. We support our theoretical
claims with a set of experiments.


http://arxiv.org/abs/1902.01235
Robustness Certificates Against Adversarial Examples for ReLU Networks.
Sahil Singla; Soheil Feizi
  While neural networks have achieved high performance in different learning
tasks, their accuracy drops significantly in the presence of small adversarial
perturbations to inputs. Defenses based on regularization and adversarial
training are often followed by new attacks to defeat them. In this paper, we
propose attack-agnostic robustness certificates for a multi-label
classification problem using a deep ReLU network. Although computing the exact
distance of a given input sample to the classification decision boundary
requires solving a non-convex optimization, we characterize two lower bounds
for such distances, namely the simplex certificate and the decision boundary
certificate. These robustness certificates leverage the piece-wise linear
structure of ReLU networks and use the fact that in a polyhedron around a given
sample, the prediction function is linear. In particular, the proposed simplex
certificate has a closed-form, is differentiable and is an order of magnitude
faster to compute than the existing methods even for deep networks. In addition
to theoretical bounds, we provide numerical results for our certificates over
MNIST and compare them with some existing upper bounds.


http://arxiv.org/abs/1902.00541
The Efficacy of SHIELD under Different Threat Models.
Cory Cornelius
  We study the efficacy of SHIELD in the face of alternative threat models. We
find that SHIELD's robustness decreases by 65% (accuracy drops from 63% to 22%)
against an adaptive adversary (one who knows JPEG compression is being used as
a pre-processing step but not necessarily the compression level) in the
gray-box threat model (adversary is aware of the model architecture but not
necessarily the weights of that model). However, these adversarial examples
are, so far, unable to force a targeted prediction. We also find that the
robustness of the JPEG-trained models used in SHIELD decreases by 67% (accuracy
drops from 57% to 19% on average) against an adaptive adversary in the gray-box
threat model. The addition of SLQ pre-processing to these JPEG-trained models
is also not a robust defense (accuracy drops to 0.1%) against an adaptive
adversary in the gray-box threat model, and an adversary can create adversarial
perturbations that force a chosen prediction. We find that neither JPEG-trained
models with SLQ pre-processing nor SHIELD are robust against an adaptive
adversary in the white-box threat model (accuracy is 0.1%) and the adversary
can control the predicted output of their adversarial images. Finally,
ensemble-based attacks transfer better (29.8% targeted accuracy) than
non-ensemble based attacks (1.4%) against the JPEG-trained models in SHIELD.


http://arxiv.org/abs/1902.00236
Natural and Adversarial Error Detection using Invariance to Image Transformations.
Yuval Bahat; Michal Irani; Gregory Shakhnarovich
  We propose an approach to distinguish between correct and incorrect image
classifications. Our approach can detect misclassifications which either occur
$\it{unintentionally}$ ("natural errors"), or due to
$\it{intentional~adversarial~attacks}$ ("adversarial errors"), both in a single
$\it{unified~framework}$. Our approach is based on the observation that
correctly classified images tend to exhibit robust and consistent
classifications under certain image transformations (e.g., horizontal flip,
small image translation, etc.). In contrast, incorrectly classified images
(whether due to adversarial errors or natural errors) tend to exhibit large
variations in classification results under such transformations. Our approach
does not require any modifications or retraining of the classifier, hence can
be applied to any pre-trained classifier. We further use state of the art
targeted adversarial attacks to demonstrate that even when the adversary has
full knowledge of our method, the adversarial distortion needed for bypassing
our detector is $\it{no~longer~imperceptible~to~the~human~eye}$. Our approach
obtains state-of-the-art results compared to previous adversarial detection
methods, surpassing them by a large margin.


http://arxiv.org/abs/1902.01220
Adaptive Gradient for Adversarial Perturbations Generation.
Yatie Xiao; Chi-Man Pun
  Deep Neural Networks have achieved remarkable success in computer vision,
natural language processing, and audio tasks.


http://arxiv.org/abs/1902.00577
Robustness of Generalized Learning Vector Quantization Models against Adversarial Attacks.
Sascha Saralajew; Lars Holdijk; Maike Rees; Thomas Villmann
  Adversarial attacks and the development of (deep) neural networks robust
against them are currently two widely researched topics. The robustness of
Learning Vector Quantization (LVQ) models against adversarial attacks has
however not yet been studied to the same extent. We therefore present an
extensive evaluation of three LVQ models: Generalized LVQ, Generalized Matrix
LVQ and Generalized Tangent LVQ. The evaluation suggests that both Generalized
LVQ and Generalized Tangent LVQ have a high base robustness, on par with the
current state-of-the-art in robust neural network methods. In contrast to this,
Generalized Matrix LVQ shows a high susceptibility to adversarial attacks,
scoring consistently behind all other models. Additionally, our numerical
evaluation indicates that increasing the number of prototypes per class
improves the robustness of the models.


http://arxiv.org/abs/1902.01208
A New Family of Neural Networks Provably Resistant to Adversarial Attacks.
Rakshit Agrawal; Alfaro Luca de; David Helmbold
  Adversarial attacks add perturbations to the input features with the intent
of changing the classification produced by a machine learning system. Small
perturbations can yield adversarial examples which are misclassified despite
being virtually indistinguishable from the unperturbed input. Classifiers
trained with standard neural network techniques are highly susceptible to
adversarial examples, allowing an adversary to create misclassifications of
their choice.
  We introduce a new type of network unit, called MWD (max of weighed distance)
units that have a built-in resistant to adversarial attacks. These units are
highly non-linear, and we develop the techniques needed to effectively train
them. We show that simple interval techniques for propagating perturbation
effects through the network enables the efficient computation of robustness
(i.e., accuracy guarantees) for MWD networks under any perturbations, including
adversarial attacks.
  MWD networks are significantly more robust to input perturbations than ReLU
networks. On permutation invariant MNIST, when test examples can be perturbed
by 20% of the input range, MWD networks provably retain accuracy above 83%,
while the accuracy of ReLU networks drops below 5%. The provable accuracy of
MWD networks is superior even to the observed accuracy of ReLU networks trained
with the help of adversarial examples. In the absence of adversarial attacks,
MWD networks match the performance of sigmoid networks, and have accuracy only
slightly below that of ReLU networks.


http://arxiv.org/abs/1902.00358
Training Artificial Neural Networks by Generalized Likelihood Ratio Method: Exploring Brain-like Learning to Improve Robustness.
Li Xiao; Yijie Peng; Jeff Hong; Zewu Ke; Shuhuai Yang
  In this work, we propose a generalized likelihood ratio method capable of
training the artificial neural networks with some biological brain-like
mechanisms,.e.g., (a) learning by the loss value, (b) learning via neurons with
discontinuous activation and loss functions. The traditional back propagation
method cannot train the artificial neural networks with aforementioned
brain-like learning mechanisms. Numerical results show that the robustness of
various artificial neural networks trained by the new method is significantly
improved when the input data is affected by both the natural noises and
adversarial attacks. Code is available:
\url{https://github.com/LX-doctorAI/GLR_ADV} .


http://arxiv.org/abs/1901.10861
A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance.
Adi Shamir; Itay Safran; Eyal Ronen; Orr Dunkelman
  The existence of adversarial examples in which an imperceptible change in the
input can fool well trained neural networks was experimentally discovered by
Szegedy et al in 2013, who called them "Intriguing properties of neural
networks". Since then, this topic had become one of the hottest research areas
within machine learning, but the ease with which we can switch between any two
decisions in targeted attacks is still far from being understood, and in
particular it is not clear which parameters determine the number of input
coordinates we have to change in order to mislead the network. In this paper we
develop a simple mathematical framework which enables us to think about this
baffling phenomenon from a fresh perspective, turning it into a natural
consequence of the geometry of $\mathbb{R}^n$ with the $L_0$ (Hamming) metric,
which can be quantitatively analyzed. In particular, we explain why we should
expect to find targeted adversarial examples with Hamming distance of roughly
$m$ in arbitrarily deep neural networks which are designed to distinguish
between $m$ input classes.


http://arxiv.org/abs/1901.11188
Augmenting Model Robustness with Transformation-Invariant Attacks.
Houpu Yao; Zhe Wang; Guangyu Nie; Yassine Mazboudi; Yezhou Yang; Yi Ren
  The vulnerability of neural networks under adversarial attacks has raised
serious concerns and motivated extensive research. It has been shown that both
neural networks and adversarial attacks against them can be sensitive to input
transformations such as linear translation and rotation, and that human vision,
which is robust against adversarial attacks, is invariant to natural input
transformations. Based on these, this paper tests the hypothesis that model
robustness can be further improved when it is adversarially trained against
transformed attacks and transformation-invariant attacks. Experiments on MNIST,
CIFAR-10, and restricted ImageNet show that while transformations of attacks
alone do not affect robustness, transformation-invariant attacks can improve
model robustness by 2.5\% on MNIST, 3.7\% on CIFAR-10, and 1.1\% on restricted
ImageNet. We discuss the intuition behind this phenomenon.


http://arxiv.org/abs/1901.10650
Metric Attack and Defense for Person Re-identification.
Song Bai; Yingwei Li; Yuyin Zhou; Qizhu Li; Philip H. S. Torr
  Person re-identification (re-ID) has attracted much attention recently due to
its great importance in video surveillance. In general, distance metrics used
to identify two person images are expected to be robust under various
appearance changes. However, our work observes the extreme vulnerability of
existing distance metrics to adversarial examples, generated by simply adding
human-imperceptible perturbations to person images. Hence, the security danger
is dramatically increased when deploying commercial re-ID systems in video
surveillance.
  Although adversarial examples have been extensively applied for
classification analysis, it is rarely studied in metric analysis like person
re-identification. The most likely reason is the natural gap between the
training and testing of re-ID networks, that is, the predictions of a re-ID
network cannot be directly used during testing without an effective metric. In
this work, we bridge the gap by proposing Adversarial Metric Attack, a parallel
methodology to adversarial classification attacks. Comprehensive experiments
clearly reveal the adversarial effects in re-ID systems. Meanwhile, we also
present an early attempt of training a metric-preserving network, thereby
defending the metric against adversarial attacks. At last, by benchmarking
various adversarial settings, we expect that our work can facilitate the
development of adversarial attack and defense in metric-based applications.


http://arxiv.org/abs/1901.10513
Adversarial Examples Are a Natural Consequence of Test Error in Noise.
Nic Ford; Justin Gilmer; Nicolas Carlini; Dogus Cubuk
  Over the last few years, the phenomenon of adversarial examples ---
maliciously constructed inputs that fool trained machine learning models ---
has captured the attention of the research community, especially when the
adversary is restricted to small modifications of a correctly handled input.
Less surprisingly, image classifiers also lack human-level performance on
randomly corrupted images, such as images with additive Gaussian noise. In this
paper we provide both empirical and theoretical evidence that these are two
manifestations of the same underlying phenomenon, establishing close
connections between the adversarial robustness and corruption robustness
research programs. This suggests that improving adversarial robustness should
go hand in hand with improving performance in the presence of more general and
realistic image corruptions. Based on our results we recommend that future
adversarial defenses consider evaluating the robustness of their methods to
distributional shift with benchmarks such as Imagenet-C.


http://arxiv.org/abs/1901.10371
On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks.
Peter Langeberg; Emilio Rafael Balda; Arash Behboodi; Rudolf Mathar
  Recently, there has been an abundance of works on designing Deep Neural
Networks (DNNs) that are robust to adversarial examples. In particular, a
central question is which features of DNNs influence adversarial robustness
and, therefore, can be to used to design robust DNNs. In this work, this
problem is studied through the lens of compression which is captured by the
low-rank structure of weight matrices. It is first shown that adversarial
training tends to promote simultaneously low-rank and sparse structure in the
weight matrices of neural networks. This is measured through the notions of
effective rank and effective sparsity. In the reverse direction, when the low
rank structure is promoted by nuclear norm regularization and combined with
sparsity inducing regularizations, neural networks show significantly improved
adversarial robustness. The effect of nuclear norm regularization on
adversarial robustness is paramount when it is applied to convolutional neural
networks. Although still not competing with adversarial training, this result
contributes to understanding the key properties of robust classifiers.


http://arxiv.org/abs/1901.10258
RED-Attack: Resource Efficient Decision based Attack for Machine Learning.
Faiq Khalid; Hassan Ali; Muhammad Abdullah Hanif; Semeen Rehman; Rehan Ahmed; Muhammad Shafique
  Due to data dependency and model leakage properties, Deep Neural Networks
(DNNs) exhibit several security vulnerabilities. Several security attacks
exploited them but most of them require the output probability vector. These
attacks can be mitigated by concealing the output probability vector. To
address this limitation, decision-based attacks have been proposed which can
estimate the model but they require several thousand queries to generate a
single untargeted attack image. However, in real-time attacks, resources and
attack time are very crucial parameters. Therefore, in resource-constrained
systems, e.g., autonomous vehicles where an untargeted attack can have a
catastrophic effect, these attacks may not work efficiently. To address this
limitation, we propose a resource efficient decision-based methodology which
generates the imperceptible attack, i.e., the RED-Attack, for a given black-box
model. The proposed methodology follows two main steps to generate the
imperceptible attack, i.e., classification boundary estimation and adversarial
noise optimization. Firstly, we propose a half-interval search-based algorithm
for estimating a sample on the classification boundary using a target image and
a randomly selected image from another class. Secondly, we propose an
optimization algorithm which first, introduces a small perturbation in some
randomly selected pixels of the estimated sample. Then to ensure
imperceptibility, it optimizes the distance between the perturbed and target
samples. For illustration, we evaluate it for CFAR-10 and German Traffic Sign
Recognition (GTSR) using state-of-the-art networks.


http://arxiv.org/abs/1901.10622
Reliable Smart Road Signs.
Muhammed O. Sayin; Chung-Wei Lin; Eunsuk Kang; Shinichi Shiraishi; Tamer Basar
  In this paper, we propose a game theoretical adversarial intervention
detection mechanism for reliable smart road signs. A future trend in
intelligent transportation systems is ``smart road signs" that incorporate
smart codes (e.g., visible at infrared) on their surface to provide more
detailed information to smart vehicles. Such smart codes make road sign
classification problem aligned with communication settings more than
conventional classification. This enables us to integrate well-established
results in communication theory, e.g., error-correction methods, into road sign
classification problem. Recently, vision-based road sign classification
algorithms have been shown to be vulnerable against (even) small scale
adversarial interventions that are imperceptible for humans. On the other hand,
smart codes constructed via error-correction methods can lead to robustness
against small scale intelligent or random perturbations on them. In the
recognition of smart road signs, however, humans are out of the loop since they
cannot see or interpret them. Therefore, there is no equivalent concept of
imperceptible perturbations in order to achieve a comparable performance with
humans. Robustness against small scale perturbations would not be sufficient
since the attacker can attack more aggressively without such a constraint.
Under a game theoretical solution concept, we seek to ensure certain measure of
guarantees against even the worst case (intelligent) attackers that can perturb
the signal even at large scale. We provide a randomized detection strategy
based on the distance between the decoder output and the received input, i.e.,
error rate. Finally, we examine the performance of the proposed scheme over
various scenarios.


http://arxiv.org/abs/1901.09981
Improving Adversarial Robustness of Ensembles with Diversity Training.
Sanjay Kariyappa; Moinuddin K. Qureshi
  Deep Neural Networks are vulnerable to adversarial attacks even in settings
where the attacker has no direct access to the model being attacked. Such
attacks usually rely on the principle of transferability, whereby an attack
crafted on a surrogate model tends to transfer to the target model. We show
that an ensemble of models with misaligned loss gradients can provide an
effective defense against transfer-based attacks. Our key insight is that an
adversarial example is less likely to fool multiple models in the ensemble if
their loss functions do not increase in a correlated fashion. To this end, we
propose Diversity Training, a novel method to train an ensemble of models with
uncorrelated loss functions. We show that our method significantly improves the
adversarial robustness of ensembles and can also be combined with existing
methods to create a stronger defense.


http://arxiv.org/abs/1901.09878
CapsAttacks: Robust and Imperceptible Adversarial Attacks on Capsule Networks.
Alberto Marchisio; Giorgio Nanfa; Faiq Khalid; Muhammad Abdullah Hanif; Maurizio Martina; Muhammad Shafique
  Capsule Networks preserve the hierarchical spatial relationships between
objects, and thereby bears a potential to surpass the performance of
traditional Convolutional Neural Networks (CNNs) in performing tasks like image
classification. A large body of work has explored adversarial examples for
CNNs, but their effectiveness on Capsule Networks has not yet been well
studied. In our work, we perform an analysis to study the vulnerabilities in
Capsule Networks to adversarial attacks. These perturbations, added to the test
inputs, are small and imperceptible to humans, but can fool the network to
mispredict. We propose a greedy algorithm to automatically generate targeted
imperceptible adversarial examples in a black-box attack scenario. We show that
this kind of attacks, when applied to the German Traffic Sign Recognition
Benchmark (GTSRB), mislead Capsule Networks. Moreover, we apply the same kind
of adversarial attacks to a 5-layer CNN and a 9-layer CNN, and analyze the
outcome, compared to the Capsule Networks to study differences in their
behavior.


http://arxiv.org/abs/1901.09963
Defense Methods Against Adversarial Examples for Recurrent Neural Networks.
Ishai Rosenberg; Asaf Shabtai; Yuval Elovici; Lior Rokach
  Adversarial examples are known to mislead deep learning models to incorrectly
classify them, even in domains where such models achieve state-of-the-art
performance. Until recently, research on both attack and defense methods
focused on image recognition, primarily using convolutional neural networks
(CNNs). In recent years, adversarial example generation methods for recurrent
neural networks (RNNs) have been published, demonstrating that RNN classifiers
are also vulnerable. In this paper, we present a novel defense method, termed
sequence squeezing, to make RNN classifiers more robust against such attacks.
Our method differs from previous defense methods which were designed only for
non-sequence based models. We also implement four additional RNN defense
methods inspired by current CNN defense methods. We evaluate our methods
against state-of-the-art attacks in the cyber security domain where real
adversaries (malware developers) exist, but our methods can be applied against
any sequence based adversarial attack, e.g., in the NLP domain. Using our
methods we were able to decrease the effectiveness of such attack from 99.9% to
15%.


http://arxiv.org/abs/1901.09960
Using Pre-Training Can Improve Model Robustness and Uncertainty.
Dan Hendrycks; Kimin Lee; Mantas Mazeika
  He et al. (2018) have called into question the utility of pre-training by
showing that training from scratch can often yield similar performance to
pre-training. We show that although pre-training may not improve performance on
traditional classification metrics, it improves model robustness and
uncertainty estimates. Through extensive experiments on adversarial examples,
label corruption, class imbalance, out-of-distribution detection, and
confidence calibration, we demonstrate large gains from pre-training and
complementary effects with task-specific methods. We introduce adversarial
pre-training and show approximately a 10% absolute improvement over the
previous state-of-the-art in adversarial robustness. In some cases, using
pre-training without task-specific methods also surpasses the state-of-the-art,
highlighting the need for pre-training when evaluating future methods on
robustness and uncertainty tasks.


http://arxiv.org/abs/1901.09413
An Information-Theoretic Explanation for the Adversarial Fragility of AI Classifiers.
Hui Xie; Jirong Yi; Weiyu Xu; Raghu Mudumbai
  We present a simple hypothesis about a compression property of artificial
intelligence (AI) classifiers and present theoretical arguments to show that
this hypothesis successfully accounts for the observed fragility of AI
classifiers to small adversarial perturbations. We also propose a new method
for detecting when small input perturbations cause classifier errors, and show
theoretical guarantees for the performance of this detection method. We present
experimental results with a voice recognition system to demonstrate this
method. The ideas in this paper are motivated by a simple analogy between AI
classifiers and the standard Shannon model of a communication system.


http://arxiv.org/abs/1901.09496
Characterizing the Shape of Activation Space in Deep Neural Networks.
Thomas Gebhart; Paul Schrater; Alan Hylton
  The representations learned by deep neural networks are difficult to
interpret in part due to their large parameter space and the complexities
introduced by their multi-layer structure. We introduce a method for computing
persistent homology over the graphical activation structure of neural networks,
which provides access to the task-relevant substructures activated throughout
the network for a given input. This topological perspective provides unique
insights into the distributed representations encoded by neural networks in
terms of the shape of their activation structures. We demonstrate the value of
this approach by showing an alternative explanation for the existence of
adversarial examples. By studying the topology of network activations across
multiple architectures and datasets, we find that adversarial perturbations do
not add activations that target the semantic structure of the adversarial class
as previously hypothesized. Rather, adversarial examples are explainable as
alterations to the dominant activation structures induced by the original
image, suggesting the class representations learned by deep networks are
problematically sparse on the input space.


http://arxiv.org/abs/1901.09493
Strong Black-box Adversarial Attacks on Unsupervised Machine Learning Models.
Anshuman Chhabra; Abhishek Roy; Prasant Mohapatra
  Machine Learning (ML) and Deep Learning (DL) models have achieved
state-of-the-art performance on multiple learning tasks, from vision to natural
language modelling. With the growing adoption of ML and DL to many areas of
computer science, recent research has also started focusing on the security
properties of these models. There has been a lot of work undertaken to
understand if (deep) neural network architectures are resilient to black-box
adversarial attacks which craft perturbed input samples that fool the
classifier without knowing the architecture used. Recent work has also focused
on the transferability of adversarial attacks and found that adversarial
attacks are generally easily transferable between models, datasets, and
techniques. However, such attacks and their analysis have not been covered from
the perspective of unsupervised machine learning algorithms. In this paper, we
seek to bridge this gap through multiple contributions. We first provide a
strong (iterative) black-box adversarial attack that can craft adversarial
samples which will be incorrectly clustered irrespective of the choice of
clustering algorithm. We choose 4 prominent clustering algorithms, and a
real-world dataset to show the working of the proposed adversarial algorithm.
Using these clustering algorithms we also carry out a simple study of
cross-technique adversarial attack transferability.


http://arxiv.org/abs/1901.09892
A Black-box Attack on Neural Networks Based on Swarm Evolutionary Algorithm.
Xiaolei Liu; Yuheng Luo; Xiaosong Zhang; Qingxin Zhu
  Neural networks play an increasingly important role in the field of machine
learning and are included in many applications in society. Unfortunately,
neural networks suffer from adversarial samples generated to attack them.
However, most of the generation approaches either assume that the attacker has
full knowledge of the neural network model or are limited by the type of
attacked model. In this paper, we propose a new approach that generates a
black-box attack to neural networks based on the swarm evolutionary algorithm.
Benefiting from the improvements in the technology and theoretical
characteristics of evolutionary algorithms, our approach has the advantages of
effectiveness, black-box attack, generality, and randomness. Our experimental
results show that both the MNIST images and the CIFAR-10 images can be
perturbed to successful generate a black-box attack with 100\% probability on
average. In addition, the proposed attack, which is successful on distilled
neural networks with almost 100\% probability, is resistant to defensive
distillation. The experimental results also indicate that the robustness of the
artificial intelligence algorithm is related to the complexity of the model and
the data set. In addition, we find that the adversarial samples to some extent
reproduce the characteristics of the sample data learned by the neural network
model.


http://arxiv.org/abs/1901.10300
Towards Weighted-Sampling Audio Adversarial Example Attack.
Xiaolei Liu; Xiaosong Zhang; Kun Wan; Qingxin Zhu; Yufei Ding
  Recent studies have highlighted audio adversarial examples as a ubiquitous
threat to state-of-the-art automatic speech recognition systems. Thorough
studies on how to effectively generate adversarial examples are essential to
prevent potential attacks. Despite many research on this, the efficiency and
the robustness of existing works are not yet satisfactory. In this paper, we
propose~\textit{weighted-sampling audio adversarial examples}, focusing on the
numbers and the weights of distortion to reinforce the attack. Further, we
apply a denoising method in the loss function to make the adversarial attack
more imperceptible. Experiments show that our method is the first in the field
to generate audio adversarial examples with low noise and high audio robustness
at the minute time-consuming level.


http://arxiv.org/abs/1901.09113
Generative Adversarial Networks for Black-Box API Attacks with Limited Training Data.
Yi Shi; Yalin E. Sagduyu; Kemal Davaslioglu; Jason H. Li
  As online systems based on machine learning are offered to public or paid
subscribers via application programming interfaces (APIs), they become
vulnerable to frequent exploits and attacks. This paper studies adversarial
machine learning in the practical case when there are rate limitations on API
calls. The adversary launches an exploratory (inference) attack by querying the
API of an online machine learning system (in particular, a classifier) with
input data samples, collecting returned labels to build up the training data,
and training an adversarial classifier that is functionally equivalent and
statistically close to the target classifier. The exploratory attack with
limited training data is shown to fail to reliably infer the target classifier
of a real text classifier API that is available online to the public. In
return, a generative adversarial network (GAN) based on deep learning is built
to generate synthetic training data from a limited number of real training data
samples, thereby extending the training data and improving the performance of
the inferred classifier. The exploratory attack provides the basis to launch
the causative attack (that aims to poison the training process) and evasion
attack (that aims to fool the classifier into making wrong decisions) by
selecting training and test data samples, respectively, based on the confidence
scores obtained from the inferred classifier. These stealth attacks with small
footprint (using a small number of API calls) make adversarial machine learning
practical under the realistic case with limited training data available to the
adversary.


http://arxiv.org/abs/1901.08846
Improving Adversarial Robustness via Promoting Ensemble Diversity.
Tianyu Pang; Kun Xu; Chao Du; Ning Chen; Jun Zhu
  Though deep neural networks have achieved significant progress on various
tasks, often enhanced by model ensemble, existing high-performance models can
be vulnerable to adversarial attacks. Many efforts have been devoted to
enhancing the robustness of individual networks and then constructing a
straightforward ensemble, e.g., by directly averaging the outputs, which
ignores the interaction among networks. This paper presents a new method that
explores the interaction among individual networks to improve robustness for
ensemble models. Technically, we define a new notion of ensemble diversity in
the adversarial setting as the diversity among non-maximal predictions of
individual members, and present an adaptive diversity promoting (ADP)
regularizer to encourage the diversity, which leads to globally better
robustness for the ensemble by making adversarial examples difficult to
transfer among individual members. Our method is computationally efficient and
compatible with the defense methods acting on individual networks. Empirical
results on various datasets verify that our method can improve adversarial
robustness while maintaining state-of-the-art accuracy on normal examples.


http://arxiv.org/abs/1901.09035
Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples.
Yinpeng Dong; Fan Bao; Hang Su; Jun Zhu
  Sometimes it is not enough for a DNN to produce an outcome. For example, in
applications such as healthcare, users need to understand the rationale of the
decisions. Therefore, it is imperative to develop algorithms to learn models
with good interpretability (Doshi-Velez 2017). An important factor that leads
to the lack of interpretability of DNNs is the ambiguity of neurons, where a
neuron may fire for various unrelated concepts. This work aims to increase the
interpretability of DNNs on the whole image space by reducing the ambiguity of
neurons. In this paper, we make the following contributions:
  1) We propose a metric to evaluate the consistency level of neurons in a
network quantitatively.
  2) We find that the learned features of neurons are ambiguous by leveraging
adversarial examples.
  3) We propose to improve the consistency of neurons on adversarial example
subset by an adversarial training algorithm with a consistent loss.


http://arxiv.org/abs/1901.08360
Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples.
Kamil Nar; Orhan Ocal; S. Shankar Sastry; Kannan Ramchandran
  State-of-the-art neural networks are vulnerable to adversarial examples; they
can easily misclassify inputs that are imperceptibly different than their
training and test data. In this work, we establish that the use of
cross-entropy loss function and the low-rank features of the training data have
responsibility for the existence of these inputs. Based on this observation, we
suggest that addressing adversarial examples requires rethinking the use of
cross-entropy loss function and looking for an alternative that is more suited
for minimization with low-rank features. In this direction, we present a
training scheme called differential training, which uses a loss function
defined on the differences between the features of points from opposite
classes. We show that differential training can ensure a large margin between
the decision boundary of the neural network and the points in the training
dataset. This larger margin increases the amount of perturbation needed to flip
the prediction of the classifier and makes it harder to find an adversarial
example with small perturbations. We test differential training on a binary
classification task with CIFAR-10 dataset and demonstrate that it radically
reduces the ratio of images for which an adversarial example could be found --
not only in the training dataset, but in the test dataset as well.


http://arxiv.org/abs/1901.08573
Theoretically Principled Trade-off between Robustness and Accuracy.
Hongyang Zhang; Yaodong Yu; Jiantao Jiao; Eric P. Xing; Laurent El Ghaoui; Michael I. Jordan
  We identify a trade-off between robustness and accuracy that serves as a
guiding principle in the design of defenses against adversarial examples.
Although this problem has been widely studied empirically, much remains unknown
concerning the theory underlying this trade-off. In this work, we decompose the
prediction error for adversarial examples (robust error) as the sum of the
natural (classification) error and boundary error, and provide a differentiable
upper bound using the theory of classification-calibrated loss, which is shown
to be the tightest possible upper bound uniform over all probability
distributions and measurable predictors. Inspired by our theoretical analysis,
we also design a new defense method, TRADES, to trade adversarial robustness
off against accuracy. Our proposed algorithm performs well experimentally in
real-world datasets. The methodology is the foundation of our entry to the
NeurIPS 2018 Adversarial Vision Challenge in which we won the 1st place out of
~2,000 submissions, surpassing the runner-up approach by $11.41\%$ in terms of
mean $\ell_2$ perturbation distance.


http://arxiv.org/abs/1901.08121
Sitatapatra: Blocking the Transfer of Adversarial Samples.
Ilia Shumailov; Xitong Gao; Yiren Zhao; Robert Mullins; Ross Anderson; Cheng-Zhong Xu
  Convolutional Neural Networks (CNNs) are widely used to solve classification
tasks in computer vision. However, they can be tricked into misclassifying
specially crafted `adversarial' samples -- and samples built to trick one model
often work alarmingly well against other models trained on the same task. In
this paper we introduce Sitatapatra, a system designed to block the transfer of
adversarial samples. It diversifies neural networks using a key, as in
cryptography, and provides a mechanism for detecting attacks. What's more, when
adversarial samples are detected they can typically be traced back to the
individual device that was used to develop them. The run-time overheads are
minimal permitting the use of Sitatapatra on constrained systems.


http://arxiv.org/abs/1901.07846
SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systems.
Tianyu Du; Shouling Ji; Jinfeng Li; Qinchen Gu; Ting Wang; Raheem Beyah
  Despite their immense popularity, deep learning-based acoustic systems are
inherently vulnerable to adversarial attacks, wherein maliciously crafted
audios trigger target systems to misbehave. In this paper, we present
SirenAttack, a new class of attacks to generate adversarial audios. Compared
with existing attacks, SirenAttack highlights with a set of significant
features: (i) versatile -- it is able to deceive a range of end-to-end acoustic
systems under both white-box and black-box settings; (ii) effective -- it is
able to generate adversarial audios that can be recognized as specific phrases
by target acoustic systems; and (iii) stealthy -- it is able to generate
adversarial audios indistinguishable from their benign counterparts to human
perception. We empirically evaluate SirenAttack on a set of state-of-the-art
deep learning-based acoustic systems (including speech command recognition,
speaker recognition and sound event classification), with results showing the
versatility, effectiveness, and stealthiness of SirenAttack. For instance, it
achieves 99.45% attack success rate on the IEMOCAP dataset against the ResNet18
model, while the generated adversarial audios are also misinterpreted by
multiple popular ASR platforms, including Google Cloud Speech, Microsoft Bing
Voice, and IBM Speech-to-Text. We further evaluate three potential defense
methods to mitigate such attacks, including adversarial training, audio
downsampling, and moving average filtering, which leads to promising directions
for further research.


http://arxiv.org/abs/1901.07152
Sensitivity Analysis of Deep Neural Networks.
Hai Shu; Hongtu Zhu
  Deep neural networks (DNNs) have achieved superior performance in various
prediction tasks, but can be very vulnerable to adversarial examples or
perturbations. Therefore, it is crucial to measure the sensitivity of DNNs to
various forms of perturbations in real applications. We introduce a novel
perturbation manifold and its associated influence measure to quantify the
effects of various perturbations on DNN classifiers. Such perturbations include
various external and internal perturbations to input samples and network
parameters. The proposed measure is motivated by information geometry and
provides desirable invariance properties. We demonstrate that our influence
measure is useful for four model building tasks: detecting potential
'outliers', analyzing the sensitivity of model architectures, comparing network
sensitivity between training and test sets, and locating vulnerable areas.
Experiments show reasonably good performance of the proposed measure for the
popular DNN models ResNet50 and DenseNet121 on CIFAR10 and MNIST datasets.


http://arxiv.org/abs/1901.07132
Universal Rules for Fooling Deep Neural Networks based Text Classification.
Di Li; Danilo Vasconcellos Vargas; Sakurai Kouichi
  Recently, deep learning based natural language processing techniques are
being extensively used to deal with spam mail, censorship evaluation in social
networks, among others. However, there is only a couple of works evaluating the
vulnerabilities of such deep neural networks. Here, we go beyond attacks to
investigate, for the first time, universal rules, i.e., rules that are sample
agnostic and therefore could turn any text sample in an adversarial one. In
fact, the universal rules do not use any information from the method itself (no
information from the method, gradient information or training dataset
information is used), making them black-box universal attacks. In other words,
the universal rules are sample and method agnostic. By proposing a
coevolutionary optimization algorithm we show that it is possible to create
universal rules that can automatically craft imperceptible adversarial samples
(only less than five perturbations which are close to misspelling are inserted
in the text sample). A comparison with a random search algorithm further
justifies the strength of the method. Thus, universal rules for fooling
networks are here shown to exist. Hopefully, the results from this work will
impact the development of yet more sample and model agnostic attacks as well as
their defenses, culminating in perhaps a new age for artificial intelligence.


http://arxiv.org/abs/1901.06834
Perception-in-the-Loop Adversarial Examples.
Mahmoud Salamati; Sadegh Soudjani; Rupak Majumdar
  We present a scalable, black box, perception-in-the-loop technique to find
adversarial examples for deep neural network classifiers. Black box means that
our procedure only has input-output access to the classifier, and not to the
internal structure, parameters, or intermediate confidence values.
Perception-in-the-loop means that the notion of proximity between inputs can be
directly queried from human participants rather than an arbitrarily chosen
metric. Our technique is based on covariance matrix adaptation evolution
strategy (CMA-ES), a black box optimization approach. CMA-ES explores the
search space iteratively in a black box manner, by generating populations of
candidates according to a distribution, choosing the best candidates according
to a cost function, and updating the posterior distribution to favor the best
candidates. We run CMA-ES using human participants to provide the fitness
function, using the insight that the choice of best candidates in CMA-ES can be
naturally modeled as a perception task: pick the top $k$ inputs perceptually
closest to a fixed input. We empirically demonstrate that finding adversarial
examples is feasible using small populations and few iterations. We compare the
performance of CMA-ES on the MNIST benchmark with other black-box approaches
using $L_p$ norms as a cost function, and show that it performs favorably both
in terms of success in finding adversarial examples and in minimizing the
distance between the original and the adversarial input. In experiments on the
MNIST, CIFAR10, and GTSRB benchmarks, we demonstrate that CMA-ES can find
perceptually similar adversarial inputs with a small number of iterations and
small population sizes when using perception-in-the-loop. Finally, we show that
networks trained specifically to be robust against $L_\infty$ norm can still be
susceptible to perceptually similar adversarial examples.


http://arxiv.org/abs/1901.06796
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey.
Wei Emma Zhang; Quan Z. Sheng; Ahoud Alhazmi; Chenliang Li
  With the development of high computational devices, deep neural networks
(DNNs), in recent years, have gained significant popularity in many Artificial
Intelligence (AI) applications. However, previous efforts have shown that DNNs
were vulnerable to strategically modified samples, named adversarial examples.
These samples are generated with some imperceptible perturbations but can fool
the DNNs to give false predictions. Inspired by the popularity of generating
adversarial examples for image DNNs, research efforts on attacking DNNs for
textual applications emerges in recent years. However, existing perturbation
methods for images cannotbe directly applied to texts as text data is discrete.
In this article, we review research works that address this difference and
generatetextual adversarial examples on DNNs. We collect, select, summarize,
discuss and analyze these works in a comprehensive way andcover all the related
information to make the article self-contained. Finally, drawing on the
reviewed literature, we provide further discussions and suggestions on this
topic.


http://arxiv.org/abs/1901.05674
Easy to Fool? Testing the Anti-evasion Capabilities of PDF Malware Scanners.
Saeed TU Darmstadt Ehteshamifar; Antonio xorlab Barresi; Thomas R. ETH Zurich Gross; Michael TU Darmstadt Pradel
  Malware scanners try to protect users from opening malicious documents by
statically or dynamically analyzing documents. However, malware developers may
apply evasions that conceal the maliciousness of a document. Given the variety
of existing evasions, systematically assessing the impact of evasions on
malware scanners remains an open challenge. This paper presents a novel
methodology for testing the capability of malware scanners to cope with
evasions. We apply the methodology to malicious Portable Document Format (PDF)
documents and present an in-depth study of how current PDF evasions affect 41
state-of-the-art malware scanners. The study is based on a framework for
creating malicious PDF documents that use one or more evasions. Based on such
documents, we measure how effective different evasions are at concealing the
maliciousness of a document. We find that many static and dynamic scanners can
be easily fooled by relatively simple evasions and that the effectiveness of
different evasions varies drastically. Our work not only is a call to arms for
improving current malware scanners, but by providing a large-scale corpus of
malicious PDF documents with evasions, we directly support the development of
improved tools to detect document-based malware. Moreover, our methodology
paves the way for a quantitative evaluation of evasions in other kinds of
malware.


http://arxiv.org/abs/1901.04684
The Limitations of Adversarial Training and the Blind-Spot Attack.
Huan Zhang; Hongge Chen; Zhao Song; Duane Boning; Inderjit S. Dhillon; Cho-Jui Hsieh
  The adversarial training procedure proposed by Madry et al. (2018) is one of
the most effective methods to defend against adversarial examples in deep
neural networks (DNNs). In our paper, we shed some lights on the practicality
and the hardness of adversarial training by showing that the effectiveness
(robustness on test set) of adversarial training has a strong correlation with
the distance between a test point and the manifold of training data embedded by
the network. Test examples that are relatively far away from this manifold are
more likely to be vulnerable to adversarial attacks. Consequentially, an
adversarial training based defense is susceptible to a new class of attacks,
the "blind-spot attack", where the input images reside in "blind-spots" (low
density regions) of the empirical distribution of training data but is still on
the ground-truth data manifold. For MNIST, we found that these blind-spots can
be easily found by simply scaling and shifting image pixel values. Most
importantly, for large datasets with high dimensional and complex data manifold
(CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training
makes defending on any valid test examples difficult due to the curse of
dimensionality and the scarcity of training data. Additionally, we find that
blind-spots also exist on provable defenses including (Wong & Kolter, 2018) and
(Sinha et al., 2018) because these trainable robustness certificates can only
be practically optimized on a limited set of training data.


http://arxiv.org/abs/1901.03706
Generating Adversarial Perturbation with Root Mean Square Gradient.
Yatie Xiao; Chi-Man Pun; Jizhe Zhou
  Deep Neural Models are vulnerable to adversarial perturbations in
classification. Many attack methods generate adversarial examples with large
pixel modification and low cosine similarity with original images. In this
paper, we propose an adversarial method generating perturbations based on root
mean square gradient which formulates adversarial perturbation size in root
mean square level and update gradient in direction, due to updating gradients
with adaptive and root mean square stride, our method map origin, and
corresponding adversarial image directly which shows good transferability in
adversarial examples generation. We evaluate several traditional perturbations
creating ways in image classification with our methods. Experimental results
show that our approach works well and outperform recent techniques in the
change of misclassifying image classification with slight pixel modification,
and excellent efficiency in fooling deep network models.


http://arxiv.org/abs/1901.03808
ECGadv: Generating Adversarial Electrocardiogram to Misguide Arrhythmia Classification System.
Huangxun Chen; Chenyu Huang; Qianyi Huang; Qian Zhang; Wei Wang
  Deep neural networks (DNNs)-powered Electrocardiogram (ECG) diagnosis systems
emerge recently, and are expected to take over tedious examinations by
cardiologists. However, their vulnerability to adversarial attacks still lack
of comprehensive investigation. ECG recordings differ from images in the
visualization, dynamic property and accessibility, thus, the existing
image-targeted attack may not directly applicable. To fill this gap, this paper
proposes ECGadv to explore the feasibility of adversarial attacks on arrhythmia
classification system. We identify the main issues under two different
deployment models(i.e., cloud-based and local-based) and propose effective
attack schemes respectively. Our results demonstrate the blind spots of
DNN-powered diagnosis system under adversarial attacks, which facilitates
future researches on countermeasures.


http://arxiv.org/abs/1901.03583
Explaining Vulnerabilities of Deep Learning to Adversarial Malware Binaries.
Luca Demetrio; Battista Biggio; Giovanni Lagorio; Fabio Roli; Alessandro Armando
  Recent work has shown that deep-learning algorithms for malware detection are
also susceptible to adversarial examples, i.e., carefully-crafted perturbations
to input malware that enable misleading classification. Although this has
questioned their suitability for this task, it is not yet clear why such
algorithms are easily fooled also in this particular application domain. In
this work, we take a first step to tackle this issue by leveraging explainable
machine-learning algorithms developed to interpret the black-box decisions of
deep neural networks. In particular, we use an explainable technique known as
feature attribution to identify the most influential input features
contributing to each decision, and adapt it to provide meaningful explanations
to the classification of malware binaries. In this case, we find that a
recently-proposed convolutional neural network does not learn any meaningful
characteristic for malware detection from the data and text sections of
executable files, but rather tends to learn to discriminate between benign and
malware samples based on the characteristics found in the file header. Based on
this finding, we propose a novel attack algorithm that generates adversarial
malware binaries by only changing few tens of bytes in the file header. With
respect to the other state-of-the-art attack algorithms, our attack does not
require injecting any padding bytes at the end of the file, and it is much more
efficient, as it requires manipulating much fewer bytes.


http://arxiv.org/abs/1901.03398
Characterizing and evaluating adversarial examples for Offline Handwritten Signature Verification.
Luiz G. Hafemann; Robert Sabourin; Luiz S. Oliveira
  The phenomenon of Adversarial Examples is attracting increasing interest from
the Machine Learning community, due to its significant impact to the security
of Machine Learning systems. Adversarial examples are similar (from a
perceptual notion of similarity) to samples from the data distribution, that
"fool" a machine learning classifier. For computer vision applications, these
are images with carefully crafted but almost imperceptible changes, that are
misclassified. In this work, we characterize this phenomenon under an existing
taxonomy of threats to biometric systems, in particular identifying new attacks
for Offline Handwritten Signature Verification systems. We conducted an
extensive set of experiments on four widely used datasets: MCYT-75, CEDAR,
GPDS-160 and the Brazilian PUC-PR, considering both a CNN-based system and a
system using a handcrafted feature extractor (CLBP). We found that attacks that
aim to get a genuine signature rejected are easy to generate, even in a limited
knowledge scenario, where the attacker does not have access to the trained
classifier nor the signatures used for training. Attacks that get a forgery to
be accepted are harder to produce, and often require a higher level of noise -
in most cases, no longer "imperceptible" as previous findings in object
recognition. We also evaluated the impact of two countermeasures on the success
rate of the attacks and the amount of noise required for generating successful
attacks.


http://arxiv.org/abs/1901.03037
Image Transformation can make Neural Networks more robust against Adversarial Examples.
Dang Duy Thang; Toshihiro Matsui
  Neural networks are being applied in many tasks related to IoT with
encouraging results. For example, neural networks can precisely detect human,
objects and animal via surveillance camera for security purpose. However,
neural networks have been recently found vulnerable to well-designed input
samples that called adversarial examples. Such issue causes neural networks to
misclassify adversarial examples that are imperceptible to humans. We found
giving a rotation to an adversarial example image can defeat the effect of
adversarial examples. Using MNIST number images as the original images, we
first generated adversarial examples to neural network recognizer, which was
completely fooled by the forged examples. Then we rotated the adversarial image
and gave them to the recognizer to find the recognizer to regain the correct
recognition. Thus, we empirically confirmed rotation to images can protect
pattern recognizer based on neural networks from adversarial example attacks.


http://arxiv.org/abs/1901.03006
Extending Adversarial Attacks and Defenses to Deep 3D Point Cloud Classifiers.
Daniel Liu; Ronald Yu; Hao Su
  3D object classification and segmentation using deep neural networks has been
extremely successful. As the problem of identifying 3D objects has many
safety-critical applications, the neural networks have to be robust against
adversarial changes to the input data set. There is a growing body of research
on generating human-imperceptible adversarial attacks and defenses against them
in the 2D image classification domain. However, 3D objects have various
differences with 2D images, and this specific domain has not been rigorously
studied so far.
  We present a preliminary evaluation of adversarial attacks on deep 3D point
cloud classifiers, namely PointNet and PointNet++, by evaluating both white-box
and black-box adversarial attacks that were proposed for 2D images and
extending those attacks to reduce the perceptibility of the perturbations in 3D
space. We also show the high effectiveness of simple defenses against those
attacks by proposing new defenses that exploit the unique structure of 3D point
clouds. Finally, we attempt to explain the effectiveness of the defenses
through the intrinsic structures of both the point clouds and the neural
network architectures. Overall, we find that networks that process 3D point
cloud data are weak to adversarial attacks, but they are also more easily
defensible compared to 2D image classifiers. Our investigation will provide the
groundwork for future studies on improving the robustness of deep neural
networks that handle 3D data.


http://arxiv.org/abs/1901.02229
Interpretable BoW Networks for Adversarial Example Detection.
Krishna Kanth Nakka; Mathieu Salzmann
  The standard approach to providing interpretability to deep convolutional
neural networks (CNNs) consists of visualizing either their feature maps, or
the image regions that contribute the most to the prediction. In this paper, we
introduce an alternative strategy to interpret the results of a CNN. To this
end, we leverage a Bag of visual Word representation within the network and
associate a visual and semantic meaning to the corresponding codebook elements
via the use of a generative adversarial network. The reason behind the
prediction for a new sample can then be interpreted by looking at the visual
representation of the most highly activated codeword. We then propose to
exploit our interpretable BoW networks for adversarial example detection. To
this end, we build upon the intuition that, while adversarial samples look very
similar to real images, to produce incorrect predictions, they should activate
codewords with a significantly different visual representation. We therefore
cast the adversarial example detection problem as that of comparing the input
image with the most highly activated visual codeword. As evidenced by our
experiments, this allows us to outperform the state-of-the-art adversarial
example detection methods on standard benchmarks, independently of the attack
strategy.


http://arxiv.org/abs/1901.01677
Image Super-Resolution as a Defense Against Adversarial Attacks.
Aamir Mustafa; Salman H. Khan; Munawar Hayat; Jianbing Shen; Ling Shao
  Convolutional Neural Networks have achieved significant success across
multiple computer vision tasks. However, they are vulnerable to carefully
crafted, human imperceptible adversarial noise patterns which constrain their
deployment in critical security-sensitive systems. This paper proposes a
computationally efficient image enhancement approach that provides a strong
defense mechanism to effectively mitigate the effect of such adversarial
perturbations. We show that the deep image restoration networks learn mapping
functions that can bring \textit{off-the-manifold} adversarial samples onto the
natural image manifold, thus restoring classifier beliefs towards correct
classes. A distinguishing feature of our approach is that, in addition to
providing robustness against attacks, it simultaneously enhances image quality
and retains models performance on clean images. Furthermore, the proposed
method does not modify the classifier or requires a separate mechanism to
detect adversarial images. The effectiveness of the scheme has been
demonstrated through extensive experiments, where it has proven a strong
defense in both white-box and black-box attack settings. The proposed scheme is
simple and has the following advantages: (1) it does not require any model
training or parameter optimization, (2) it complements other existing defense
mechanisms, (3) it is agnostic to the attacked model and attack type and (4) it
provides superior performance across all popular attack algorithms. Our codes
are publicly available at
https://github.com/aamir-mustafa/super-resolution-adversarial-defense.


http://arxiv.org/abs/1901.09657
Fake News Detection via NLP is Vulnerable to Adversarial Attacks.
Zhixuan Zhou; Huankang Guan; Meghana Moorthy Bhat; Justin Hsu
  News plays a significant role in shaping people's beliefs and opinions. Fake
news has always been a problem, which wasn't exposed to the mass public until
the past election cycle for the 45th President of the United States. While
quite a few detection methods have been proposed to combat fake news since
2015, they focus mainly on linguistic aspects of an article without any fact
checking. In this paper, we argue that these models have the potential to
misclassify fact-tampering fake news as well as under-written real news.
Through experiments on Fakebox, a state-of-the-art fake news detector, we show
that fact tampering attacks can be effective. To address these weaknesses, we
argue that fact checking should be adopted in conjunction with linguistic
characteristics analysis, so as to truly separate fake news from real news. A
crowdsourced knowledge graph is proposed as a straw man solution to collecting
timely facts about news events.


http://arxiv.org/abs/1901.01223
Adversarial Examples Versus Cloud-based Detectors: A Black-box Empirical Study.
Xurong Li; Shouling Ji; Meng Han; Juntao Ji; Zhenyu Ren; Yushan Liu; Chunming Wu
  Deep learning has been broadly leveraged by major cloud providers such as
Google, AWS, Baidu, to offer various computer vision related services including
image auto-classification, object identification, and illegal image detection,
etc. While the recent works demonstrated that the deep learning classification
models are vulnerable to adversarial examples, the more complicated cloud-based
image detections models also have similar security concern but not get enough
attention in literature yet. In this paper, we mainly focus on the security
issues of real-world cloud-based image detectors. Specifically, (1) based on
effective semantic segmentation, we propose four different attacks to generate
semantics-aware adversarial examples via only interacting with black-box APIs;
and (2) we make the first attempt to conduct an extensive empirical study of
black-box attacks against real-world cloud-based image detectors. Through the
comprehensive evaluations on five major cloud platforms on AWS, Azure, Google
Cloud, Baidu Cloud, and Alibaba Cloud, we demonstrate that our Image Processing
attacks have a success rate of approximately 100%, and the semantic
segmentation based attacks have a success rate over 90% among different
detection services, such as violence, politician, and pornography detection,
etc. We also proposed several possible defense strategies for these security
challenges in the real-life situation.


http://arxiv.org/abs/1901.00546
Multi-Label Adversarial Perturbations.
Qingquan Song; Haifeng Jin; Xiao Huang; Xia Hu
  Adversarial examples are delicately perturbed inputs, which aim to mislead
machine learning models towards incorrect outputs. While most of the existing
work focuses on generating adversarial perturbations in multi-class
classification problems, many real-world applications fall into the multi-label
setting in which one instance could be associated with more than one label. For
example, a spammer may generate adversarial spams with malicious advertising
while maintaining the other labels such as topic labels unchanged. To analyze
the vulnerability and robustness of multi-label learning models, we investigate
the generation of multi-label adversarial perturbations. This is a challenging
task due to the uncertain number of positive labels associated with one
instance, as well as the fact that multiple labels are usually not mutually
exclusive with each other. To bridge this gap, in this paper, we propose a
general attacking framework targeting on multi-label classification problem and
conduct a premier analysis on the perturbations for deep neural networks.
Leveraging the ranking relationships among labels, we further design a
ranking-based framework to attack multi-label ranking algorithms. We specify
the connection between the two proposed frameworks and separately design two
specific methods grounded on each of them to generate targeted multi-label
perturbations. Experiments on real-world multi-label image classification and
ranking problems demonstrate the effectiveness of our proposed frameworks and
provide insights of the vulnerability of multi-label deep learning models under
diverse targeted attacking strategies. Several interesting findings including
an unpolished defensive strategy, which could potentially enhance the
interpretability and robustness of multi-label deep learning models, are
further presented and discussed at the end.


http://arxiv.org/abs/1901.00532
Adversarial Robustness May Be at Odds With Simplicity.
Preetum Nakkiran
  Current techniques in machine learning are so far are unable to learn
classifiers that are robust to adversarial perturbations. However, they are
able to learn non-robust classifiers with very high accuracy, even in the
presence of random perturbations. Towards explaining this gap, we highlight the
hypothesis that $\textit{robust classification may require more complex
classifiers (i.e. more capacity) than standard classification.}$
  In this note, we show that this hypothesis is indeed possible, by giving
several theoretical examples of classification tasks and sets of "simple"
classifiers for which: (1) There exists a simple classifier with high standard
accuracy, and also high accuracy under random $\ell_\infty$ noise. (2) Any
simple classifier is not robust: it must have high adversarial loss with
$\ell_\infty$ perturbations. (3) Robust classification is possible, but only
with more complex classifiers (exponentially more complex, in some examples).
  Moreover, $\textit{there is a quantitative trade-off between robustness and
standard accuracy among simple classifiers.}$ This suggests an alternate
explanation of this phenomenon, which appears in practice: the tradeoff may
occur not because the classification task inherently requires such a tradeoff
(as in [Tsipras-Santurkar-Engstrom-Turner-Madry `18]), but because the
structure of our current classifiers imposes such a tradeoff.


http://arxiv.org/abs/1901.00054
A Noise-Sensitivity-Analysis-Based Test Prioritization Technique for Deep Neural Networks.
Long Zhang; Xuechao Sun; Yong Li; Zhenyu Zhang
  Deep neural networks (DNNs) have been widely used in the fields such as
natural language processing, computer vision and image recognition. But several
studies have been shown that deep neural networks can be easily fooled by
artificial examples with some perturbations, which are widely known as
adversarial examples. Adversarial examples can be used to attack deep neural
networks or to improve the robustness of deep neural networks. A common way of
generating adversarial examples is to first generate some noises and then add
them into original examples. In practice, different examples have different
noise-sensitive. To generate an effective adversarial example, it may be
necessary to add a lot of noise to low noise-sensitive example, which may make
the adversarial example meaningless. In this paper, we propose a
noise-sensitivity-analysis-based test prioritization technique to pick out
examples by their noise sensitivity. We construct an experiment to validate our
approach on four image sets and two DNN models, which shows that examples are
sensitive to noise and our method can effectively pick out examples by their
noise sensitivity.


http://arxiv.org/abs/1812.10812
DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems.
Husheng Zhou; Wei Li; Yuankun Zhu; Yuqun Zhang; Bei Yu; Lingming Zhang; Cong Liu
  Deep Neural Networks (DNNs) have been widely applied in many autonomous
systems such as autonomous driving. Recently, DNN testing has been intensively
studied to automatically generate adversarial examples, which inject
small-magnitude perturbations into inputs to test DNNs under extreme
situations. While existing testing techniques prove to be effective, they
mostly focus on generating digital adversarial perturbations (particularly for
autonomous driving), e.g., changing image pixels, which may never happen in
physical world. There is a critical missing piece in the literature on
autonomous driving testing: understanding and exploiting both digital and
physical adversarial perturbation generation for impacting steering decisions.
In this paper, we present DeepBillboard, a systematic physical-world testing
approach targeting at a common and practical driving scenario: drive-by
billboards. DeepBillboard is capable of generating a robust and resilient
printable adversarial billboard, which works under dynamic changing driving
conditions including viewing angle, distance, and lighting. The objective is to
maximize the possibility, degree, and duration of the steering-angle errors of
an autonomous vehicle driving by the generated adversarial billboard. We have
extensively evaluated the efficacy and robustness of DeepBillboard through
conducting both digital and physical-world experiments. Results show that
DeepBillboard is effective for various steering models and scenes. Furthermore,
DeepBillboard is sufficiently robust and resilient for generating
physical-world adversarial billboard tests for real-world driving under various
weather conditions. To the best of our knowledge, this is the first study
demonstrating the possibility of generating realistic and continuous
physical-world tests for practical autonomous driving systems.


http://arxiv.org/abs/1812.10528
Adversarial Attack and Defense on Graph Data: A Survey.
Lichao Sun; Ji Wang; Philip S. Yu; Bo Li
  Deep neural networks (DNNs) have been widely applied in various applications
involving image, text, audio, and graph data. However, recent studies have
shown that DNNs are vulnerable to adversarial attack. Though there are several
works studying adversarial attack and defense on domains such as images and
text processing, it is difficult to directly transfer the learned knowledge to
graph data due to its representation challenge. Given the importance of graph
analysis, increasing number of works start to analyze the robustness of machine
learning models on graph. Nevertheless, current studies considering adversarial
behaviors on graph data usually focus on specific types of attacks with certain
assumptions. In addition, each work proposes its own mathematical formulation
which makes the comparison among different methods difficult. Therefore, in
this paper, we aim to survey existing adversarial attack strategies on graph
data and provide an unified problem formulation which can cover all current
adversarial learning studies on graph. We also compare different attacks on
graph data and discuss their corresponding contributions and limitations.
Finally, we discuss several future research directions in this area.


http://arxiv.org/abs/1812.10217
Seeing isn't Believing: Practical Adversarial Attack Against Object Detectors.
Yue Zhao; Hong Zhu; Ruigang Liang; Qintao Shen; Shengzhi Zhang; Kai Chen
  In this paper, we design the first practical adversarial attacks against
object detectors in realistic situations: the printed adversarial examples
(AEs) placed with different angles and distances, could hide or create an
object to deceive object detectors. To improve the robustness of AEs, we
propose the novel nested AEs and introduce the image transformation techniques
to simulate the variance factors such as distances, angles, illuminations, in
the physical world. To let the AEs converge with so many constraints, we also
design batch-variation momentum training. Evaluation results show that our
adversarial examples work in real environments, capable of attacking
state-of-the-art real-time object detectors (e.g., YOLO V3 and faster-RCNN)
with the success rate up to 92.4% (with the distance ranging from 1m to 25m and
wide angles from -60{\deg}to 60{\deg}). Our AEs are also shown to be highly
transferable. Four state-of-the-art black-box models could be attacked by our
AEs with high success rates.


http://arxiv.org/abs/1812.10199
A Multiversion Programming Inspired Approach to Detecting Audio Adversarial Examples.
Qiang Zeng; Jianhai Su; Chenglong Fu; Golam Kayas; Lannan Luo
  Adversarial examples (AEs) are crafted by adding human-imperceptible
perturbations to inputs such that a machine-learning based classifier
incorrectly labels them. They have become a severe threat to the
trustworthiness of machine learning. While AEs in the image domain have been
well studied, audio AEs are less investigated. Recently, multiple techniques
are proposed to generate audio AEs, which makes countermeasures against them an
urgent task. Our experiments show that, given an AE, the transcription results
by different Automatic Speech Recognition (ASR) systems differ significantly,
as they use different architectures, parameters, and training datasets.
Inspired by Multiversion Programming, we propose a novel audio AE detection
approach, which utilizes multiple off-the-shelf ASR systems to determine
whether an audio input is an AE. The evaluation shows that the detection
achieves accuracies over 98.6%.


http://arxiv.org/abs/1812.10085
A Data-driven Adversarial Examples Recognition Framework via Adversarial Feature Genome.
Li Chen; Hailun Ding; Qi Li; Jiawei Zhu; Jian Peng; Haifeng Li
  Convolutional neural networks (CNNs) are easily spoofed by adversarial
examples which lead to wrong classification results. Most of the defense
methods focus only on how to improve the robustness of CNNs or to detect
adversarial examples. They are incapable of detecting and correctly classifying
adversarial examples simultaneously. We find that adversarial examples and
original images have diverse representations in the feature space, and this
difference grows as layers go deeper, which we call Adversarial Feature
Separability (AFS). Inspired by AFS, we propose a defense framework based on
Adversarial Feature Genome (AFG), which can detect and correctly classify
adversarial examples into original classes simultaneously. AFG is an innovative
encoding for both image and adversarial example. It consists of group features
and a mixed label. With group features which are visual representations of
adversarial and original images via group visualization method, one can detect
adversarial examples because of ASF of group features. With a mixed label, one
can trace back to the original label of an adversarial example. Then, the
classification of adversarial example is modeled as a multi-label
classification trained on the AFG dataset, which can get the original class of
adversarial example. Experiments show that the proposed framework not only
effectively detects adversarial examples from different attack algorithms, but
also correctly classifies adversarial examples. Our framework potentially gives
a new perspective, i.e., a data-driven way, to improve the robustness of a CNN
model.


http://arxiv.org/abs/1812.10061
Noise Flooding for Detecting Audio Adversarial Examples Against Automatic Speech Recognition.
Krishan Rajaratnam; Jugal Kalita
  Neural models enjoy widespread use across a variety of tasks and have grown
to become crucial components of many industrial systems. Despite their
effectiveness and extensive popularity, they are not without their exploitable
flaws. Initially applied to computer vision systems, the generation of
adversarial examples is a process in which seemingly imperceptible
perturbations are made to an image, with the purpose of inducing a deep
learning based classifier to misclassify the image. Due to recent trends in
speech processing, this has become a noticeable issue in speech recognition
models. In late 2017, an attack was shown to be quite effective against the
Speech Commands classification model. Limited-vocabulary speech classifiers,
such as the Speech Commands model, are used quite frequently in a variety of
applications, particularly in managing automated attendants in telephony
contexts. As such, adversarial examples produced by this attack could have
real-world consequences. While previous work in defending against these
adversarial examples has investigated using audio preprocessing to reduce or
distort adversarial noise, this work explores the idea of flooding particular
frequency bands of an audio signal with random noise in order to detect
adversarial examples. This technique of flooding, which does not require
retraining or modifying the model, is inspired by work done in computer vision
and builds on the idea that speech classifiers are relatively robust to natural
noise. A combined defense incorporating 5 different frequency bands for
flooding the signal with noise outperformed other existing defenses in the
audio space, detecting adversarial examples with 91.8% precision and 93.5%
recall.


http://arxiv.org/abs/1812.10049
PPD: Permutation Phase Defense Against Adversarial Examples in Deep Learning.
Mehdi Jafarnia-Jahromi; Tasmin Chowdhury; Hsin-Tai Wu; Sayandev Mukherjee
  Deep neural networks have demonstrated cutting edge performance on various
tasks including classification. However, it is well known that adversarially
designed imperceptible perturbation of the input can mislead advanced
classifiers. In this paper, Permutation Phase Defense (PPD), is proposed as a
novel method to resist adversarial attacks. PPD combines random permutation of
the image with phase component of its Fourier transform. The basic idea behind
this approach is to turn adversarial defense problems analogously into
symmetric cryptography, which relies solely on safekeeping of the keys for
security. In PPD, safe keeping of the selected permutation ensures
effectiveness against adversarial attacks. Testing PPD on MNIST and CIFAR-10
datasets yielded state-of-the-art robustness against the most powerful
adversarial attacks currently available.


http://arxiv.org/abs/1812.11017
Deflecting 3D Adversarial Point Clouds Through Outlier-Guided Removal.
Hang Zhou; Kejiang Chen; Weiming Zhang; Han Fang; Wenbo Zhou; Nenghai Yu
  Neural networks are vulnerable to adversarial examples, which poses a threat
to their application in security sensitive systems. We propose simple random
sampling (SRS) and statistical outlier removal (SOR) as defenses for 3D point
cloud classification, where both methods remove points by estimating
probability of points serving as adversarial points. Compared with ensemble
adversarial training which is the state-of-the-art defending method, SOR has
several advantages: better defense performance, randomization makes the network
more robust to adversarial point clouds, no additional training or fine-tuning
required, and few computations are needed by adding the points-removal layer.
In particular, our experiments on ModelNet40 show that SOR is very effective as
defense in practice. The strength of those defenses lies in their
non-differentiable nature and inherent randomness, which makes it difficult for
an adversary to circumvent the defenses. Our best defense eliminates 81.4% of
strong white-box attacks by C&W and l2 loss based attack methods.


http://arxiv.org/abs/1812.09803
Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks.
Thomas Brunner; Frederik Diehl; Michael Truong Le; Alois Knoll
  We consider adversarial examples for image classification in the black-box
decision-based setting. Here, an attacker cannot access confidence scores, but
only the final label. Most attacks for this scenario are either unreliable or
inefficient. Focusing on the latter, we show that a specific class of attacks,
Boundary Attacks, can be reinterpreted as a biased sampling framework that
gains efficiency from domain knowledge. We identify three such biases, image
frequency, regional masks and surrogate gradients, and evaluate their
performance against an ImageNet classifier. We show that the combination of
these biases outperforms the state of the art by a wide margin. We also
showcase an efficient way to attack the Google Cloud Vision API, where we craft
convincing perturbations with just a few hundred queries. Finally, the methods
we propose have also been found to work very well against strong defenses: Our
targeted attack won second place in the NeurIPS 2018 Adversarial Vision
Challenge.


http://arxiv.org/abs/1812.09660
Markov Game Modeling of Moving Target Defense for Strategic Detection of Threats in Cloud Networks.
Ankur Chowdhary; Sailik Sengupta; Dijiang Huang; Subbarao Kambhampati
  The processing and storage of critical data in large-scale cloud networks
necessitate the need for scalable security solutions. It has been shown that
deploying all possible security measures incurs a cost on performance by using
up valuable computing and networking resources which are the primary selling
points for cloud service providers. Thus, there has been a recent interest in
developing Moving Target Defense (MTD) mechanisms that helps one optimize the
joint objective of maximizing security while ensuring that the impact on
performance is minimized. Often, these techniques model the problem of
multi-stage attacks by stealthy adversaries as a single-step attack detection
game using graph connectivity measures as a heuristic to measure performance,
thereby (1) losing out on valuable information that is inherently present in
graph-theoretic models designed for large cloud networks, and (2) coming up
with certain strategies that have asymmetric impacts on performance. In this
work, we leverage knowledge in attack graphs of a cloud network in formulating
a zero-sum Markov Game and use the Common Vulnerability Scoring System (CVSS)
to come up with meaningful utility values for this game. Then, we show that the
optimal strategy of placing detecting mechanisms against an adversary is
equivalent to computing the mixed Min-max Equilibrium of the Markov Game. We
compare the gains obtained by using our method to other techniques presently
used in cloud network security, thereby showing its effectiveness. Finally, we
highlight how the method was used for a small real-world cloud system.


http://arxiv.org/abs/1812.09638
AEPecker: L0 Adversarial Examples are not Strong Enough.
Fei Zuo; Bokai Yang; Xiaopeng Li; Lannan Luo; Qiang Zeng
  Despite the great achievements made by neural networks on tasks such as image
classification, they are brittle and vulnerable to adversarial examples (AEs).
By adding adversarial noise to input images, adversarial examples can be
crafted to mislead neural network based image classifiers. One type of AE
attack in particular, known as an L0 AE, has been used in several notable
real-world incidents. Our observation is that, while L0 corruptions modify as
few pixels as possible, they tend to cause large-amplitude perturbations to the
modified pixels. We consider this to be an inherent limitation of L0 AEs which
can be exploited. To show the weakness of L0 AEs, we thwart samples of these
attacks by both detecting and rectifying them. The main novelty of the proposed
detector is that we convert the AE detection problem into an image comparison
problem by exploiting the inherent characteristics of L0 AEs. More concretely,
given an image I, it is pre-processed to obtain another image I'. We use a
Siamese network which is known to be effective in comparison, to take I and I'
as the input pair. A well trained Siamese network can automatically capture the
discrepancy between I and I' to detect L0 noises. In addition, the
straightforward pre-processor based on heuristics can be deployed as an
effective defense, having a high probability of removing the adversarial
influence of L0 perturbations. The proposed technique shows not only a high
accuracy but also a resilience to the adaptive adversary, which outperforms
other state-of-the-art methods. We accordingly argue that L0 attacks are not
strong enough.


http://arxiv.org/abs/1812.09431
Dissociable neural representations of adversarially perturbed images in deep neural networks and the human brain.
Chi Zhang; Xiaohan Duan; Linyuan Wang; Yongli Li; Bin Yan; Guoen Hu; Ruyuan Zhang; Li Tong
  Despite the remarkable similarities between deep neural networks (DNN) and
the human brain as shown in previous studies, the fact that DNNs still fall
behind humans in many visual tasks suggests that considerable differences still
exist between the two systems. To probe their dissimilarities, we leverage
adversarial noise (AN) and adversarial interference (AI) images that yield
distinct recognition performance in a prototypical DNN (AlexNet) and human
vision. The evoked activity by regular (RE) and adversarial images in both
systems is thoroughly compared. We find that representational similarity
between RE and adversarial images in the human brain resembles their perceptual
similarity. However, such representation-perception association is disrupted in
the DNN. Especially, the representational similarity between RE and AN images
idiosyncratically increases from low- to high-level layers. Furthermore,
forward encoding modeling reveals that the DNN-brain hierarchical
correspondence proposed in previous studies only holds when the two systems
process RE and AI images but not AN images. These results might be due to the
deterministic modeling approach of current DNNs. Taken together, our results
provide a complementary perspective on the comparison between DNNs and the
human brain, and highlight the need to characterize their differences to
further bridge artificial and human intelligence research.


http://arxiv.org/abs/1812.08108
Enhancing Robustness of Deep Neural Networks Against Adversarial Malware Samples: Principles, Framework, and AICS'2019 Challenge.
Deqiang Li; Qianmu Li; Yanfang Ye; Shouhuai Xu
  Malware continues to be a major cyber threat, despite the tremendous effort
that has been made to combat them. The number of malware in the wild steadily
increases over time, meaning that we must resort to automated defense
techniques. This naturally calls for machine learning based malware detection.
However, machine learning is known to be vulnerable to adversarial evasion
attacks that manipulate a small number of features to make classifiers wrongly
recognize a malware sample as a benign one. The state-of-the-art is that there
are no effective countermeasures against these attacks. Inspired by the
AICS'2019 Challenge, we systematize a number of principles for enhancing the
robustness of neural networks against adversarial malware evasion attacks. Some
of these principles have been scattered in the literature, but others are
proposed in this paper for the first time. Under the guidance of these
principles, we propose a framework and an accompanying training algorithm,
which are then applied to the AICS'2019 challenge. Our experimental results
have been submitted to the challenge organizer for evaluation.


http://arxiv.org/abs/1812.08329
PROVEN: Certifying Robustness of Neural Networks with a Probabilistic Approach.
Tsui-Wei Weng; Pin-Yu Chen; Lam M. Nguyen; Mark S. Squillante; Ivan Oseledets; Luca Daniel
  With deep neural networks providing state-of-the-art machine learning models
for numerous machine learning tasks, quantifying the robustness of these models
has become an important area of research. However, most of the research
literature merely focuses on the \textit{worst-case} setting where the input of
the neural network is perturbed with noises that are constrained within an
$\ell_p$ ball; and several algorithms have been proposed to compute certified
lower bounds of minimum adversarial distortion based on such worst-case
analysis. In this paper, we address these limitations and extend the approach
to a \textit{probabilistic} setting where the additive noises can follow a
given distributional characterization. We propose a novel probabilistic
framework PROVEN to PRObabilistically VErify Neural networks with statistical
guarantees -- i.e., PROVEN certifies the probability that the classifier's
top-1 prediction cannot be altered under any constrained $\ell_p$ norm
perturbation to a given input. Importantly, we show that it is possible to
derive closed-form probabilistic certificates based on current state-of-the-art
neural network robustness verification frameworks. Hence, the probabilistic
certificates provided by PROVEN come naturally and with almost no overhead when
obtaining the worst-case certified lower bounds from existing methods such as
Fast-Lin, CROWN and CNN-Cert. Experiments on small and large MNIST and CIFAR
neural network models demonstrate our probabilistic approach can achieve up to
around $75\%$ improvement in the robustness certification with at least a
$99.99\%$ confidence compared with the worst-case robustness certificate
delivered by CROWN.


http://arxiv.org/abs/1812.08342
Safety and Trustworthiness of Deep Neural Networks: A Survey.
Xiaowei Huang; Daniel Kroening; Marta Kwiatkowska; Wenjie Ruan; Youcheng Sun; Emese Thamo; Min Wu; Xinping Yi
  In the past few years, significant progress has been made on deep neural
networks (DNNs) in achieving human-level intelligence on several long-standing
tasks. With broader deployment of DNNs on various applications, the concerns on
its safety and trustworthiness have been raised, particularly after the fatal
incidents of self-driving cars. Research to address these concerns is very
active, with many papers released in the past few years. This survey paper is
to conduct a review of the current research efforts on making DNNs safe and
trustworthy, by focusing on four aspects, i.e., verification, testing,
adversarial attack and defence, and interpretability. In total, we surveyed 178
papers, most of which were published in the most recent two years, i.e., 2017
and 2018.


http://arxiv.org/abs/1812.06815
Spartan Networks: Self-Feature-Squeezing Neural Networks for increased robustness in adversarial settings.
François Menet; Paul Berthier; José M. Fernandez; Michel Gagnon
  Deep learning models are vulnerable to adversarial examples which are input
samples modified in order to maximize the error on the system. We introduce
Spartan Networks, resistant deep neural networks that do not require input
preprocessing nor adversarial training. These networks have an adversarial
layer designed to discard some information of the network, thus forcing the
system to focus on relevant input. This is done using a new activation function
to discard data. The added layer trains the neural network to filter-out
usually-irrelevant parts of its input. Our performance evaluation shows that
Spartan Networks have a slightly lower precision but report a higher robustness
under attack when compared to unprotected models. Results of this study of
Adversarial AI as a new attack vector are based on tests conducted on the MNIST
dataset.


http://arxiv.org/abs/1812.06626
Designing Adversarially Resilient Classifiers using Resilient Feature Engineering.
Kevin Eykholt; Atul Prakash
  We provide a methodology, resilient feature engineering, for creating
adversarially resilient classifiers. According to existing work, adversarial
attacks identify weakly correlated or non-predictive features learned by the
classifier during training and design the adversarial noise to utilize these
features. Therefore, highly predictive features should be used first during
classification in order to determine the set of possible output labels. Our
methodology focuses the problem of designing resilient classifiers into a
problem of designing resilient feature extractors for these highly predictive
features. We provide two theorems, which support our methodology. The Serial
Composition Resilience and Parallel Composition Resilience theorems show that
the output of adversarially resilient feature extractors can be combined to
create an equally resilient classifier. Based on our theoretical results, we
outline the design of an adversarially resilient classifier.


http://arxiv.org/abs/1812.06570
Defense-VAE: A Fast and Accurate Defense against Adversarial Attacks.
Xiang Li; Shihao Ji
  Deep neural networks (DNNs) have been enormously successful across a variety
of prediction tasks. However, recent research shows that DNNs are particularly
vulnerable to adversarial attacks, which poses a serous threat to their
applications in security-sensitive systems. In this paper, we propose a simple
yet effective defense algorithm Defense-VAE that uses variational autoencoder
(VAE) to purge adversarial perturbations from contaminated images. The proposed
method is generic and can defend white-box and black-box attacks without the
need of retraining the original CNN classifiers, and can further strengthen the
defense by retraining CNN or end-to-end finetuning the whole pipeline. In
addition, the proposed method is very efficient compared to the
optimization-based alternatives, such as Defense-GAN, since no iterative
optimization is needed for online prediction. Extensive experiments on MNIST,
Fashion-MNIST, CelebA and CIFAR-10 demonstrate the superior defense accuracy of
Defense-VAE compared to Defense-GAN, while being 50x faster than the latter.
This makes Defense-VAE widely deployable in real-time security-sensitive
systems. We plan to open source our implementation to facilitate the research
in this area.


http://arxiv.org/abs/1812.07385
Perturbation Analysis of Learning Algorithms: A Unifying Perspective on Generation of Adversarial Examples.
Emilio Rafael Balda; Arash Behboodi; Rudolf Mathar
  Despite the tremendous success of deep neural networks in various learning
problems, it has been observed that adding an intentionally designed
adversarial perturbation to inputs of these architectures leads to erroneous
classification with high confidence in the prediction. In this work, we propose
a general framework based on the perturbation analysis of learning algorithms
which consists of convex programming and is able to recover many current
adversarial attacks as special cases. The framework can be used to propose
novel attacks against learning algorithms for classification and regression
tasks under various new constraints with closed form solutions in many
instances. In particular we derive new attacks against classification
algorithms which are shown to achieve comparable performances to notable
existing attacks. The framework is then used to generate adversarial
perturbations for regression tasks which include single pixel and single subset
attacks. By applying this method to autoencoding and image colorization tasks,
it is shown that adversarial perturbations can effectively perturb the output
of regression tasks as well.


http://arxiv.org/abs/1812.06371
Trust Region Based Adversarial Attack on Neural Networks.
Zhewei Yao; Amir Gholami; Peng Xu; Kurt Keutzer; Michael Mahoney
  Deep Neural Networks are quite vulnerable to adversarial perturbations.
Current state-of-the-art adversarial attack methods typically require very time
consuming hyper-parameter tuning, or require many iterations to solve an
optimization based adversarial attack. To address this problem, we present a
new family of trust region based adversarial attacks, with the goal of
computing adversarial perturbations efficiently. We propose several attacks
based on variants of the trust region optimization method. We test the proposed
methods on Cifar-10 and ImageNet datasets using several different models
including AlexNet, ResNet-50, VGG-16, and DenseNet-121 models. Our methods
achieve comparable results with the Carlini-Wagner (CW) attack, but with
significant speed up of up to $37\times$, for the VGG-16 model on a Titan Xp
GPU. For the case of ResNet-50 on ImageNet, we can bring down its
classification accuracy to less than 0.1\% with at most $1.5\%$ relative
$L_\infty$ (or $L_2$) perturbation requiring only $1.02$ seconds as compared to
$27.04$ seconds for the CW attack. We have open sourced our method which can be
accessed at [1].


http://arxiv.org/abs/1812.05793
Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing.
Jingyi Wang; Guoliang Dong; Jun Sun; Xinyu Wang; Peixin Zhang
  Deep neural networks (DNN) have been shown to be useful in a wide range of
applications. However, they are also known to be vulnerable to adversarial
samples. By transforming a normal sample with some carefully crafted human
imperceptible perturbations, even highly accurate DNN make wrong decisions.
Multiple defense mechanisms have been proposed which aim to hinder the
generation of such adversarial samples. However, a recent work show that most
of them are ineffective. In this work, we propose an alternative approach to
detect adversarial samples at runtime. Our main observation is that adversarial
samples are much more sensitive than normal samples if we impose random
mutations on the DNN. We thus first propose a measure of `sensitivity' and show
empirically that normal samples and adversarial samples have distinguishable
sensitivity. We then integrate statistical hypothesis testing and model
mutation testing to check whether an input sample is likely to be normal or
adversarial at runtime by measuring its sensitivity. We evaluated our approach
on the MNIST and CIFAR10 datasets. The results show that our approach detects
adversarial samples generated by state-of-the-art attacking methods efficiently
and accurately.


http://arxiv.org/abs/1812.05271
TextBugger: Generating Adversarial Text Against Real-world Applications.
Jinfeng Li; Shouling Ji; Tianyu Du; Bo Li; Ting Wang
  Deep Learning-based Text Understanding (DLTU) is the backbone technique
behind various applications, including question answering, machine translation,
and text classification. Despite its tremendous popularity, the security
vulnerabilities of DLTU are still largely unknown, which is highly concerning
given its increasing use in security-sensitive applications such as sentiment
analysis and toxic content detection. In this paper, we show that DLTU is
inherently vulnerable to adversarial text attacks, in which maliciously crafted
texts trigger target DLTU systems and services to misbehave. Specifically, we
present TextBugger, a general attack framework for generating adversarial
texts. In contrast to prior works, TextBugger differs in significant ways: (i)
effective -- it outperforms state-of-the-art attacks in terms of attack success
rate; (ii) evasive -- it preserves the utility of benign text, with 94.9\% of
the adversarial text correctly recognized by human readers; and (iii) efficient
-- it generates adversarial text with computational complexity sub-linear to
the text length. We empirically evaluate TextBugger on a set of real-world DLTU
systems and services used for sentiment analysis and toxic content detection,
demonstrating its effectiveness, evasiveness, and efficiency. For instance,
TextBugger achieves 100\% success rate on the IMDB dataset based on Amazon AWS
Comprehend within 4.61 seconds and preserves 97\% semantic similarity. We
further discuss possible defense mechanisms to mitigate such attack and the
adversary's potential countermeasures, which leads to promising directions for
further research.


http://arxiv.org/abs/1812.05720
Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem.
Matthias Hein; Maksym Andriushchenko; Julian Bitterwolf
  Classifiers used in the wild, in particular for safety-critical systems,
should not only have good generalization properties but also should know when
they don't know, in particular make low confidence predictions far away from
the training data. We show that ReLU type neural networks which yield a
piecewise linear classifier function fail in this regard as they produce almost
always high confidence predictions far away from the training data. For bounded
domains like images we propose a new robust optimization technique similar to
adversarial training which enforces low confidence predictions far away from
the training data. We show that this technique is surprisingly effective in
reducing the confidence of predictions far away from the training data while
maintaining high confidence predictions and test error on the original
classification task compared to standard training.


http://arxiv.org/abs/1812.05013
Thwarting Adversarial Examples: An $L_0$-RobustSparse Fourier Transform.
Mitali Bafna; Jack Murtagh; Nikhil Vyas
  We give a new algorithm for approximating the Discrete Fourier transform of
an approximately sparse signal that has been corrupted by worst-case $L_0$
noise, namely a bounded number of coordinates of the signal have been corrupted
arbitrarily. Our techniques generalize to a wide range of linear
transformations that are used in data analysis such as the Discrete Cosine and
Sine transforms, the Hadamard transform, and their high-dimensional analogs. We
use our algorithm to successfully defend against well known $L_0$ adversaries
in the setting of image classification. We give experimental results on the
Jacobian-based Saliency Map Attack (JSMA) and the Carlini Wagner (CW) $L_0$
attack on the MNIST and Fashion-MNIST datasets as well as the Adversarial Patch
on the ImageNet dataset.


http://arxiv.org/abs/1812.04599
Adversarial Framing for Image and Video Classification.
Michał Zając; Konrad Żołna; Negar Rostamzadeh; Pedro O. Pinheiro
  Neural networks are prone to adversarial attacks. In general, such attacks
deteriorate the quality of the input by either slightly modifying most of its
pixels, or by occluding it with a patch. In this paper, we propose a method
that keeps the image unchanged and only adds an adversarial framing on the
border of the image. We show empirically that our method is able to
successfully attack state-of-the-art methods on both image and video
classification problems. Notably, the proposed method results in a universal
attack which is very fast at test time. Source code can be found at
https://github.com/zajaczajac/adv_framing .


http://arxiv.org/abs/1812.04293
Mix'n'Squeeze: Thwarting Adaptive Adversarial Samples Using Randomized Squeezing.
Kumar Sharad; Giorgia Azzurra Marson; Hien Thi Thu Truong; Ghassan Karame
  Deep Learning (DL) has been shown to be particularly vulnerable to
adversarial samples. To combat adversarial strategies, numerous defenses have
been proposed in the literature. Among these, feature squeezing emerges as an
effective defense by reducing unnecessary features without changing the DL
model. However, feature squeezing is a static defense and does not resist
adaptive attacks. Namely, feature squeezing is a deterministic process: as soon
as an adversarial sample is found, this sample will always succeed against the
classifier.
  In this work, we address this problem and introduce Mix'n'Squeeze, the first
randomized feature squeezing defense that leverages key-based randomness and is
secure against adaptive whitebox adversaries. Our defense consists of
pre-processing the classifier inputs by embedding carefully selected randomness
within each feature, before applying feature squeezing, so that an adaptive
whitebox attacker can no longer predict the effect of their own perturbations
on the resulting sample. We thoroughly implement and evaluate Mix'n'Squeeze in
the context of image classification in light of the various reported strategies
to generate adversarial samples. We also analyze the resilience of
Mix'n'Squeeze with respect to state of the art adaptive strategies and we show
that---in contrast to common belief---Mix'n'Squeeze does not hamper the
classifier's accuracy while significantly decreasing the success probability of
an adaptive whitebox adversary.


http://arxiv.org/abs/1812.03705
Defending against Universal Perturbations with Shared Adversarial Training.
Chaithanya Kumar Mummadi; Thomas Brox; Jan Hendrik Metzen
  Classifiers such as deep neural networks have been shown to be vulnerable
against adversarial perturbations on problems with high-dimensional input
space. While adversarial training improves the robustness of image classifiers
against such adversarial perturbations, it leaves them sensitive to
perturbations on a non-negligible fraction of the inputs. In this work, we show
that adversarial training is more effective in preventing universal
perturbations, where the same perturbation needs to fool a classifier on many
inputs. Moreover, we investigate the trade-off between robustness against
universal perturbations and performance on unperturbed data and propose an
extension of adversarial training that handles this trade-off more gracefully.
We present results for image classification and semantic segmentation to
showcase that universal perturbations that fool a model hardened with
adversarial training become clearly perceptible and show patterns of the target
scene.


http://arxiv.org/abs/1812.03413
Learning Transferable Adversarial Examples via Ghost Networks.
Yingwei Li; Song Bai; Yuyin Zhou; Cihang Xie; Zhishuai Zhang; Alan Yuille
  Recent development of adversarial attacks has proven that ensemble-based
methods outperform traditional, non-ensemble ones in black-box attack. However,
these methods generally require a family of diverse models, and ensembling them
together afterward, both of which are computationally expensive. In this paper,
we propose Ghost Networks to generate transferable adversarial examples
efficiently. The critical principle of ghost networks is to apply feature-level
perturbations to an existing model to potentially create a huge set of diverse
models. After that, models are subsequently fused by longitudinal ensemble.
Compared to traditional ensemble methods, both steps require almost no extra
time and space consumption. Extensive experimental results suggest that the
number of networks is essential for improving the transferability of
adversarial examples, but it is less necessary to independently train different
networks and ensemble them in an intensive aggregation way. Instead, our work
can be used as a computationally cheap and easily applied plug-in to improve
adversarial approaches both in single-model and multi-model attack, compatible
with residual and non-residual networks. By reproducing the NeurIPS 2017
adversarial competition, our method outperforms the No.1 attack submission by a
large margin, demonstrating its effectiveness and efficiency.


http://arxiv.org/abs/1812.03411
Feature Denoising for Improving Adversarial Robustness.
Cihang Xie; Yuxin Wu; der Maaten Laurens van; Alan Yuille; Kaiming He
  Adversarial attacks to image classification systems present challenges to
convolutional networks and opportunities for understanding them. This study
suggests that adversarial perturbations on images lead to noise in the features
constructed by these networks. Motivated by this observation, we develop new
network architectures that increase adversarial robustness by performing
feature denoising. Specifically, our networks contain blocks that denoise the
features using non-local means or other filters; the entire networks are
trained end-to-end. When combined with adversarial training, our feature
denoising networks substantially improve the state-of-the-art in adversarial
robustness in both white-box and black-box attack settings. On ImageNet, under
10-iteration PGD white-box attacks where prior art has 27.9% accuracy, our
method achieves 55.7%; even under extreme 2000-iteration PGD white-box attacks,
our method secures 42.6% accuracy. Our method was ranked first in Competition
on Adversarial Attacks and Defenses (CAAD) 2018 --- it achieved 50.6%
classification accuracy on a secret, ImageNet-like test dataset against 48
unknown attackers, surpassing the runner-up approach by ~10%. Code is available
at https://github.com/facebookresearch/ImageNet-Adversarial-Training.


http://arxiv.org/abs/1812.03405
AutoGAN: Robust Classifier Against Adversarial Attacks.
Blerta Lindqvist; Shridatt Sugrim; Rauf Izmailov
  Classifiers fail to classify correctly input images that have been
purposefully and imperceptibly perturbed to cause misclassification. This
susceptability has been shown to be consistent across classifiers, regardless
of their type, architecture or parameters. Common defenses against adversarial
attacks modify the classifer boundary by training on additional adversarial
examples created in various ways. In this paper, we introduce AutoGAN, which
counters adversarial attacks by enhancing the lower-dimensional manifold
defined by the training data and by projecting perturbed data points onto it.
AutoGAN mitigates the need for knowing the attack type and magnitude as well as
the need for having adversarial samples of the attack. Our approach uses a
Generative Adversarial Network (GAN) with an autoencoder generator and a
discriminator that also serves as a classifier. We test AutoGAN against
adversarial samples generated with state-of-the-art Fast Gradient Sign Method
(FGSM) as well as samples generated with random Gaussian noise, both using the
MNIST dataset. For different magnitudes of perturbation in training and
testing, AutoGAN can surpass the accuracy of FGSM method by up to 25\% points
on samples perturbed using FGSM. Without an augmented training dataset, AutoGAN
achieves an accuracy of 89\% compared to 1\% achieved by FGSM method on FGSM
testing adversarial samples.


http://arxiv.org/abs/1812.03303
Detecting Adversarial Examples in Convolutional Neural Networks.
Stefanos Pertigkiozoglou; Petros Maragos
  The great success of convolutional neural networks has caused a massive
spread of the use of such models in a large variety of Computer Vision
applications. However, these models are vulnerable to certain inputs, the
adversarial examples, which although are not easily perceived by humans, they
can lead a neural network to produce faulty results. This paper focuses on the
detection of adversarial examples, which are created for convolutional neural
networks that perform image classification. We propose three methods for
detecting possible adversarial examples and after we analyze and compare their
performance, we combine their best aspects to develop an even more robust
approach. The first proposed method is based on the regularization of the
feature vector that the neural network produces as output. The second method
detects adversarial examples by using histograms, which are created from the
outputs of the hidden layers of the neural network. These histograms create a
feature vector which is used as the input of an SVM classifier, which
classifies the original input either as an adversarial or as a real input.
Finally, for the third method we introduce the concept of the residual image,
which contains information about the parts of the input pattern that are
ignored by the neural network. This method aims at the detection of possible
adversarial examples, by using the residual image and reinforcing the parts of
the input pattern that are ignored by the neural network. Each one of these
methods has some novelties and by combining them we can further improve the
detection results. For the proposed methods and their combination, we present
the results of detecting adversarial examples on the MNIST dataset. The
combination of the proposed methods offers some improvements over similar state
of the art approaches.


http://arxiv.org/abs/1812.03190
Deep-RBF Networks Revisited: Robust Classification with Rejection.
Pourya Habib Zadeh; Reshad Hosseini; Suvrit Sra
  One of the main drawbacks of deep neural networks, like many other
classifiers, is their vulnerability to adversarial attacks. An important reason
for their vulnerability is assigning high confidence to regions with few or
even no feature points. By feature points, we mean a nonlinear transformation
of the input space extracting a meaningful representation of the input data. On
the other hand, deep-RBF networks assign high confidence only to the regions
containing enough feature points, but they have been discounted due to the
widely-held belief that they have the vanishing gradient problem. In this
paper, we revisit the deep-RBF networks by first giving a general formulation
for them, and then proposing a family of cost functions thereof inspired by
metric learning. In the proposed deep-RBF learning algorithm, the vanishing
gradient problem does not occur. We make these networks robust to adversarial
attack by adding the reject option to their output layer. Through several
experiments on the MNIST dataset, we demonstrate that our proposed method not
only achieves significant classification accuracy but is also very resistant to
various adversarial attacks.


http://arxiv.org/abs/1812.03087
Combatting Adversarial Attacks through Denoising and Dimensionality Reduction: A Cascaded Autoencoder Approach.
Rajeev Sahay; Rehana Mahfuz; Aly El Gamal
  Machine Learning models are vulnerable to adversarial attacks that rely on
perturbing the input data. This work proposes a novel strategy using
Autoencoder Deep Neural Networks to defend a machine learning model against two
gradient-based attacks: The Fast Gradient Sign attack and Fast Gradient attack.
First we use an autoencoder to denoise the test data, which is trained with
both clean and corrupted data. Then, we reduce the dimension of the denoised
data using the hidden layer representation of another autoencoder. We perform
this experiment for multiple values of the bound of adversarial perturbations,
and consider different numbers of reduced dimensions. When the test data is
preprocessed using this cascaded pipeline, the tested deep neural network
classifier yields a much higher accuracy, thus mitigating the effect of the
adversarial perturbation.


http://arxiv.org/abs/1812.02891
Adversarial Defense of Image Classification Using a Variational Auto-Encoder.
Yi Luo; Henry Pfister
  Deep neural networks are known to be vulnerable to adversarial attacks. This
exposes them to potential exploits in security-sensitive applications and
highlights their lack of robustness. This paper uses a variational auto-encoder
(VAE) to defend against adversarial attacks for image classification tasks.
This VAE defense has a few nice properties: (1) it is quite flexible and its
use of randomness makes it harder to attack; (2) it can learn disentangled
representations that prevent blurry reconstruction; and (3) a patch-wise VAE
defense strategy is used that does not require retraining for different size
images. For moderate to severe attacks, this system outperforms or closely
matches the performance of JPEG compression, with the best quality parameter.
It also has more flexibility and potential for improvement via training.


http://arxiv.org/abs/1812.02885
Adversarial Attacks, Regression, and Numerical Stability Regularization.
Andre T. Nguyen; Edward Raff
  Adversarial attacks against neural networks in a regression setting are a
critical yet understudied problem. In this work, we advance the state of the
art by investigating adversarial attacks against regression networks and by
formulating a more effective defense against these attacks. In particular, we
take the perspective that adversarial attacks are likely caused by numerical
instability in learned functions. We introduce a stability inducing,
regularization based defense against adversarial attacks in the regression
setting. Our new and easy to implement defense is shown to outperform prior
approaches and to improve the numerical stability of learned functions.


http://arxiv.org/abs/1812.02843
Towards Hiding Adversarial Examples from Network Interpretation.
Akshayvarun Subramanya; Vipin Pillai; Hamed Pirsiavash
  Deep networks have been shown to be fooled rather easily using adversarial
attack algorithms. Practical methods such as adversarial patches have been
shown to be extremely effective in causing misclassification. However, these
patches can be highlighted using standard network interpretation algorithms,
thus revealing the identity of the adversary. We show that it is possible to
create adversarial patches which not only fool the prediction, but also change
what we interpret regarding the cause of prediction. We show that our
algorithms can empower adversarial patches, by hiding them from network
interpretation tools. We believe our algorithms can facilitate developing more
robust network interpretation tools that truly explain the network's underlying
decision making process.


http://arxiv.org/abs/1812.02606
The Limitations of Model Uncertainty in Adversarial Settings.
Kathrin Grosse; David Pfaff; Michael T. Smith; Michael Backes
  Machine learning models are vulnerable to adversarial examples: minor
perturbations to input samples intended to deliberately cause
misclassification. Many defenses have led to an arms race-we thus study a
promising, recent trend in this setting, Bayesian uncertainty measures. These
measures allow a classifier to provide principled confidence and uncertainty
for an input, where the latter refers to how usual the input is. We focus on
Gaussian processes (GP), a classifier providing such principled uncertainty and
confidence measures. Using correctly classified benign data as comparison, GP's
intrinsic uncertainty and confidence deviate for misclassified benign samples
and misclassified adversarial examples. We therefore introduce
high-confidence-low-uncertainty adversarial examples: adversarial examples
crafted maximizing GP confidence and minimizing GP uncertainty. Visual
inspection shows HCLU adversarial examples are malicious, and resemble the
original rather than the target class. HCLU adversarial examples also transfer
to other classifiers. We focus on transferability to other algorithms providing
uncertainty measures, and find that a Bayesian neural network confidently
misclassifies HCLU adversarial examples. We conclude that uncertainty and
confidence, even in the Bayesian sense, can be circumvented by both white-box
and black-box attackers.


http://arxiv.org/abs/1812.02575
Prior Networks for Detection of Adversarial Attacks.
Andrey Malinin; Mark Gales
  Adversarial examples are considered a serious issue for safety critical
applications of AI, such as finance, autonomous vehicle control and medicinal
applications. Though significant work has resulted in increased robustness of
systems to these attacks, systems are still vulnerable to well-crafted attacks.
To address this problem, several adversarial attack detection methods have been
proposed. However, a system can still be vulnerable to adversarial samples that
are designed to specifically evade these detection methods. One recent
detection scheme that has shown good performance is based on uncertainty
estimates derived from Monte-Carlo dropout ensembles. Prior Networks, a new
method of estimating predictive uncertainty, has been shown to outperform
Monte-Carlo dropout on a range of tasks. One of the advantages of this approach
is that the behaviour of a Prior Network can be explicitly tuned to, for
example, predict high uncertainty in regions where there are no training data
samples. In this work, Prior Networks are applied to adversarial attack
detection using measures of uncertainty in a similar fashion to Monte-Carlo
Dropout. Detection based on measures of uncertainty derived from DNNs and
Monte-Carlo dropout ensembles are used as a baseline. Prior Networks are shown
to significantly out-perform these baseline approaches over a range of
adversarial attacks in both detection of whitebox and blackbox configurations.
Even when the adversarial attacks are constructed with full knowledge of the
detection mechanism, it is shown to be highly challenging to successfully
generate an adversarial sample.


http://arxiv.org/abs/1812.02524
Towards Leveraging the Information of Gradients in Optimization-based Adversarial Attack.
Jingyang Zhang; Hsin-Pai Cheng; Chunpeng Wu; Hai Li; Yiran Chen
  In recent years, deep neural networks demonstrated state-of-the-art
performance in a large variety of tasks and therefore have been adopted in many
applications. On the other hand, the latest studies revealed that neural
networks are vulnerable to adversarial examples obtained by carefully adding
small perturbation to legitimate samples. Based upon the observation, many
attack methods were proposed. Among them, the optimization-based CW attack is
the most powerful as the produced adversarial samples present much less
distortion compared to other methods. The better attacking effect, however,
comes at the cost of running more iterations and thus longer computation time
to reach desirable results. In this work, we propose to leverage the
information of gradients as a guidance during the search of adversaries. More
specifically, directly incorporating the gradients into the perturbation can be
regarded as a constraint added to the optimization process. We intuitively and
empirically prove the rationality of our method in reducing the search space.
Our experiments show that compared to the original CW attack, the proposed
method requires fewer iterations towards adversarial samples, obtaining a
higher success rate and resulting in smaller $\ell_2$ distortion.


http://arxiv.org/abs/1812.02637
Max-Margin Adversarial (MMA) Training: Direct Input Space Margin Maximization through Adversarial Training.
Gavin Weiguang Ding; Yash Sharma; Kry Yik Chau Lui; Ruitong Huang
  We study adversarial robustness of neural networks from a margin maximization
perspective, where margins are defined as the distances from inputs to a
classifier's decision boundary. In theory, we show that maximizing margins can
be achieved by minimizing the adversarial loss on the decision boundary at the
"shortest successful perturbation". This max-margin perspective also provides
an alternative interpretation on adversarial training with a fixed perturbation
magnitude $\epsilon$: adversarial training is maximizing either a lower bound
or an upper bound of the margin. Motivated by our theoretical analysis, we
propose Max-Margin Adversarial (MMA) training to directly maximize the margins.
Instead of adversarial training with a fixed $\epsilon$, MMA offers an
improvement by selecting the margin as the "correct" $\epsilon$ individually
for each point. We demonstrate MMA training's efficacy and analyze its
properties on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$
robustness.


http://arxiv.org/abs/1812.02737
On Configurable Defense against Adversarial Example Attacks.
Bo Luo; Min Li; Yu Li; Qiang Xu
  Machine learning systems based on deep neural networks (DNNs) have gained
mainstream adoption in many applications. Recently, however, DNNs are shown to
be vulnerable to adversarial example attacks with slight perturbations on the
inputs. Existing defense mechanisms against such attacks try to improve the
overall robustness of the system, but they do not differentiate different
targeted attacks even though the corresponding impacts may vary significantly.
To tackle this problem, we propose a novel configurable defense mechanism in
this work, wherein we are able to flexibly tune the robustness of the system
against different targeted attacks to satisfy application requirements. This is
achieved by refining the DNN loss function with an attack sensitive matrix to
represent the impacts of different targeted attacks. Experimental results on
CIFAR-10 and GTSRB data sets demonstrate the efficacy of the proposed solution.


http://arxiv.org/abs/1812.02132
SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications.
Abdullah Hamdi; Matthias Müller; Bernard Ghanem
  One major factor impeding more widespread adoption of deep neural networks
(DNNs) is their issues with robustness, which is essential for safety critical
applications such as autonomous driving. This has motivated much recent work on
adversarial attacks for DNNs, which mostly focus on pixel-level perturbations
void of semantic meaning. In contrast, we present a general framework for
adversarial black-box attacks on trained agents, which covers semantic
perturbations to the environment of the agent performing the task as well as
pixel-level attacks. To do this, we re-frame the adversarial attack problem as
learning a distribution of parameters that always fool the agent. In the
semantic case, our proposed adversary (denoted as BBGAN) is trained to sample
parameters that describe the environment with which the black-box agent
interacts, such that the agent performs its dedicated task poorly in this
environment. We apply BBGAN on three different tasks (primarily targeting
aspects of autonomous navigation): object detection, self-driving, and
autonomous UAV racing. On these tasks, BBGAN can generate failure cases that
consistently fool an agent. We also demonstrate the usefulness of our framework
as an analysis tool by visualizing systemic failure cases and uncovering
semantic insights about the agents themselves.


http://arxiv.org/abs/1812.01821
Regularized Ensembles and Transferability in Adversarial Learning.
Yifan Chen; Yevgeniy Vorobeychik
  Despite the considerable success of convolutional neural networks in a broad
array of domains, recent research has shown these to be vulnerable to small
adversarial perturbations, commonly known as adversarial examples. Moreover,
such examples have shown to be remarkably portable, or transferable, from one
model to another, enabling highly successful black-box attacks. We explore this
issue of transferability and robustness from two dimensions: first, considering
the impact of conventional $l_p$ regularization as well as replacing the top
layer with a linear support vector machine (SVM), and second, the value of
combining regularized models into an ensemble. We show that models trained with
different regularizers present barriers to transferability, as does partial
information about the models comprising the ensemble.


http://arxiv.org/abs/1812.01804
Random Spiking and Systematic Evaluation of Defenses Against Adversarial Examples.
Huangyi Ge; Sze Yiu Chau; Bruno Ribeiro; Ninghui Li
  Image classifiers often suffer from adversarial examples, which are generated
by strategically adding a small amount of noise to input images to trick
classifiers into misclassification. Over the years, many defense mechanisms
have been proposed, and different researchers have made seemingly contradictory
claims on their effectiveness. We present an analysis of possible adversarial
models, and propose an evaluation framework for comparing different defense
mechanisms. As part of the framework, we introduced a more powerful and
realistic adversary strategy. We propose a new defense mechanism called Random
Spiking (RS), which generalizes dropout and introduces random noises in the
training process in a controlled manner. Evaluations under our proposed
framework suggest RS delivers better protection against adversarial examples
than many existing schemes.


http://arxiv.org/abs/1812.01647
Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures.
Jonathan Dj Uesato; Ananya Dj Kumar; Csaba Dj Szepesvari; Tom Dj Erez; Avraham Dj Ruderman; Keith Dj Anderson; Dj Krishmamurthy; Dvijotham; Nicolas Heess; Pushmeet Kohli
  This paper addresses the problem of evaluating learning systems in safety
critical domains such as autonomous driving, where failures can have
catastrophic consequences. We focus on two problems: searching for scenarios
when learned agents fail and assessing their probability of failure. The
standard method for agent evaluation in reinforcement learning, Vanilla Monte
Carlo, can miss failures entirely, leading to the deployment of unsafe agents.
We demonstrate this is an issue for current agents, where even matching the
compute used for training is sometimes insufficient for evaluation. To address
this shortcoming, we draw upon the rare event probability estimation literature
and propose an adversarial evaluation approach. Our approach focuses evaluation
on adversarially chosen situations, while still providing unbiased estimates of
failure probabilities. The key difficulty is in identifying these adversarial
situations -- since failures are rare there is little signal to drive
optimization. To solve this we propose a continuation approach that learns
failure modes in related but less robust agents. Our approach also allows reuse
of data already collected for training the agent. We demonstrate the efficacy
of adversarial evaluation on two standard domains: humanoid control and
simulated driving. Experimental results show that our methods can find
catastrophic failures and estimate failures rates of agents multiple orders of
magnitude faster than standard evaluation schemes, in minutes to hours rather
than days.


http://arxiv.org/abs/1812.00740
Disentangling Adversarial Robustness and Generalization.
David Stutz; Matthias Hein; Bernt Schiele
  Obtaining deep networks that are robust against adversarial examples and
generalize well is an open problem. A recent hypothesis even states that both
robust and accurate models are impossible, i.e., adversarial robustness and
generalization are conflicting goals. In an effort to clarify the relationship
between robustness and generalization, we assume an underlying, low-dimensional
data manifold and show that: 1. regular adversarial examples leave the
manifold; 2. adversarial examples constrained to the manifold, i.e.,
on-manifold adversarial examples, exist; 3. on-manifold adversarial examples
are generalization errors, and on-manifold adversarial training boosts
generalization; 4. regular robustness and generalization are not necessarily
contradicting goals. These assumptions imply that both robust and accurate
models are possible. However, different models (architectures, training
strategies etc.) can exhibit different robustness and generalization
characteristics. To confirm our claims, we present extensive experiments on
synthetic data (with known manifold) as well as on EMNIST, Fashion-MNIST and
CelebA.


http://arxiv.org/abs/1812.00891
Interpretable Deep Learning under Fire.
Xinyang Zhang; Ningfei Wang; Hua Shen; Shouling Ji; Xiapu Luo; Ting Wang
  Providing explanations for deep neural network (DNN) models is crucial for
their usability in security-sensitive domains. A plethora of interpretation
models have been proposed to help end users understand the inner workings of
DNNs: how does a DNN arrive at a specific decision for a given input? The
improved interpretability is believed to offer a sense of security by involving
human in the decision-making process. However, due to its data-driven nature,
this interpretability itself is potentially susceptible to malicious
manipulations, about which little is known thus far.
  In this paper, we bridge this gap by conducting the first systematic study on
the security of interpretable deep learning systems (IDLSes). We demonstrate
that existing IDLSes are highly vulnerable to adversarial manipulations.
Specifically, we present ADV^2, a general class of attacks that generate
adversarial inputs not only misleading target DNNs but also deceiving their
coupled interpretation models. Through empirical evaluation against four major
types of IDLSes on benchmark datasets and in security-critical applications
(e.g., skin cancer diagnosis), we show that with ADV^2 the adversary is able to
arbitrarily designate an input's prediction and interpretation simultaneously.
Moreover, with both analytical and empirical evidence, we identify the
prediction-interpretation gap as one possible cause of this vulnerability -- a
DNN and its interpretation model are often only partially aligned, resulting in
the possibility to exploit both models simultaneously. Finally, we explore
potential countermeasures against ADV^2, including leveraging its low
transferability and incorporating it in an adversarial training framework. Our
findings shed light on designing and operating IDLSes in a more secure and
informative fashion, leading to several promising research directions.


http://arxiv.org/abs/1812.01198
Adversarial Example Decomposition.
Horace He; Aaron Lou; Qingxuan Jiang; Isay Katsman; Serge Belongie; Ser-Nam Lim
  Research has shown that widely used deep neural networks are vulnerable to
carefully crafted adversarial perturbations. Moreover, these adversarial
perturbations often transfer across models. We hypothesize that adversarial
weakness is composed of three sources of bias: architecture, dataset, and
random initialization. We show that one can decompose adversarial examples into
an architecture-dependent component, data-dependent component, and
noise-dependent component and that these components behave intuitively. For
example, noise-dependent components transfer poorly to all other models, while
architecture-dependent components transfer better to retrained models with the
same architecture. In addition, we demonstrate that these components can be
recombined to improve transferability without sacrificing efficacy on the
original model.


http://arxiv.org/abs/1812.00552
Universal Perturbation Attack Against Image Retrieval.
Jie Li; Rongrong Ji; Hong Liu; Xiaopeng Hong; Yue Gao; Qi Tian
  Despite the remarkable success, deep learning models have shown to be
vulnerable to the universal adversarial perturbation (UAP). The existing
endeavors on UAP methods mainly focus on attacking the image classification
models. Nevertheless, little attention has been paid to attacking image
retrieval systems. In this paper, we make the first attempt for UAP attacking
to deep feature based image retrieval. Concretely, attacking image retrieval is
to make the retrieval system return more irrelevant images to the query at the
top ranking list, whose key design is to corrupt the relationships among
features. To this end, we propose a unified method to generate retrieval-based
UAP to break the relationships between image features from point-wise,
label-wise, and list-wise aspects. We further analyze the impact of the
resizing operation in generating UAP, and thus provide a solution to attack
high-performance retrieval systems with query resizing. We evaluate the
proposed methods on four widely-used image retrieval datasets, i.e., Oxford5k
and Paris6k with their revised versions, which lead to a significant
performance drop in terms of different metrics, such as mAP, and mP@10.
Finally, we test our attacking methods on the real-world visual search engine,
i.e., Google Images, which demonstrates the potential of our methods.


http://arxiv.org/abs/1812.00483
Model-Reuse Attacks on Deep Learning Systems.
Yujie Ji; Xinyang Zhang; Shouling Ji; Xiapu Luo; Ting Wang
  Many of today's machine learning (ML) systems are built by reusing an array
of, often pre-trained, primitive models, each fulfilling distinct functionality
(e.g., feature extraction). The increasing use of primitive models
significantly simplifies and expedites the development cycles of ML systems.
Yet, because most of such models are contributed and maintained by untrusted
sources, their lack of standardization or regulation entails profound security
implications, about which little is known thus far.
  In this paper, we demonstrate that malicious primitive models pose immense
threats to the security of ML systems. We present a broad class of {\em
model-reuse} attacks wherein maliciously crafted models trigger host ML systems
to misbehave on targeted inputs in a highly predictable manner. By empirically
studying four deep learning systems (including both individual and ensemble
systems) used in skin cancer screening, speech recognition, face verification,
and autonomous steering, we show that such attacks are (i) effective - the host
systems misbehave on the targeted inputs as desired by the adversary with high
probability, (ii) evasive - the malicious models function indistinguishably
from their benign counterparts on non-targeted inputs, (iii) elastic - the
malicious models remain effective regardless of various system design choices
and tuning strategies, and (iv) easy - the adversary needs little prior
knowledge about the data used for system tuning or inference. We provide
analytical justification for the effectiveness of model-reuse attacks, which
points to the unprecedented complexity of today's primitive models. This issue
thus seems fundamental to many ML systems. We further discuss potential
countermeasures and their challenges, which lead to several promising research
directions.


http://arxiv.org/abs/1812.01713
FineFool: Fine Object Contour Attack via Attention.
Jinyin Chen; Haibin Zheng; Hui Xiong; Mengmeng Su
  Machine learning models have been shown vulnerable to adversarial attacks
launched by adversarial examples which are carefully crafted by attacker to
defeat classifiers. Deep learning models cannot escape the attack either. Most
of adversarial attack methods are focused on success rate or perturbations
size, while we are more interested in the relationship between adversarial
perturbation and the image itself. In this paper, we put forward a novel
adversarial attack based on contour, named FineFool. Finefool not only has
better attack performance compared with other state-of-art white-box attacks in
aspect of higher attack success rate and smaller perturbation, but also capable
of visualization the optimal adversarial perturbation via attention on object
contour. To the best of our knowledge, Finefool is for the first time combines
the critical feature of the original clean image with the optimal perturbations
in a visible manner. Inspired by the correlations between adversarial
perturbations and object contour, slighter perturbations is produced via
focusing on object contour features, which is more imperceptible and difficult
to be defended, especially network add-on defense methods with the trade-off
between perturbations filtering and contour feature loss. Compared with
existing state-of-art attacks, extensive experiments are conducted to show that
Finefool is capable of efficient attack against defensive deep models.


http://arxiv.org/abs/1812.00292
SentiNet: Detecting Physical Attacks Against Deep Learning Systems.
Edward Chou; Florian Tramèr; Giancarlo Pellegrino; Dan Boneh
  SentiNet is a novel detection framework for physical attacks on neural
networks, a class of attacks that constrains an adversarial region to a visible
portion of an image. Physical attacks have been shown to be robust and flexible
techniques suited for deployment in real-world scenarios. Unlike most other
adversarial detection works, SentiNet does not require training a model or
preknowledge of an attack prior to detection. This attack-agnostic approach is
appealing due to the large number of possible mechanisms and vectors of attack
an attack-specific defense would have to consider. By leveraging the neural
network's susceptibility to attacks and by using techniques from model
interpretability and object detection as detection mechanisms, SentiNet turns a
weakness of a model into a strength. We demonstrate the effectiveness of
SentiNet on three different attacks - i.e., adversarial examples, data
poisoning attacks, and trojaned networks - that have large variations in
deployment mechanisms, and show that our defense is able to achieve very
competitive performance metrics for all three threats, even against strong
adaptive adversaries with full knowledge of SentiNet.


http://arxiv.org/abs/1812.00239
Building robust classifiers through generation of confident out of distribution examples.
Kumar Sricharan; Ashok Srivastava
  Deep learning models are known to be overconfident in their predictions on
out of distribution inputs. There have been several pieces of work to address
this issue, including a number of approaches for building Bayesian neural
networks, as well as closely related work on detection of out of distribution
samples. Recently, there has been work on building classifiers that are robust
to out of distribution samples by adding a regularization term that maximizes
the entropy of the classifier output on out of distribution data. To
approximate out of distribution samples (which are not known apriori), a GAN
was used for generation of samples at the edges of the training distribution.
In this paper, we introduce an alternative GAN based approach for building a
robust classifier, where the idea is to use the GAN to explicitly generate out
of distribution samples that the classifier is confident on (low entropy), and
have the classifier maximize the entropy for these samples. We showcase the
effectiveness of our approach relative to state-of-the-art on hand-written
characters as well as on a variety of natural image datasets.


http://arxiv.org/abs/1812.00181
Effects of Loss Functions And Target Representations on Adversarial Robustness.
Sean Saito; Sujoy Roy
  Understanding and evaluating the robustness of neural networks under
adversarial settings is a subject of growing interest. Attacks proposed in the
literature usually work with models trained to minimize cross-entropy loss and
output softmax probabilities. In this work, we present interesting experimental
results that suggest the importance of considering other loss functions and
target representations, specifically, (1) training on mean-squared error and
(2) representing targets as codewords generated from a random codebook. We
evaluate the robustness of neural networks that implement these proposed
modifications using existing attacks, showing an increase in accuracy against
untargeted attacks of up to 98.7\% and a decrease of targeted attack success
rates of up to 99.8\%. Our model demonstrates more robustness compared to its
conventional counterpart even against attacks that are tailored to our
modifications. Furthermore, we find that the parameters of our modified model
have significantly smaller Lipschitz bounds, an important measure correlated
with a model's sensitivity to adversarial perturbations.


http://arxiv.org/abs/1812.00151
Discrete Adversarial Attacks and Submodular Optimization with Applications to Text Classification.
Qi Lei; Lingfei Wu; Pin-Yu Chen; Alexandros G. Dimakis; Inderjit S. Dhillon; Michael Witbrock
  Adversarial examples are carefully constructed modifications to an input that
completely change the output of a classifier but are imperceptible to humans.
Despite these successful attacks for continuous data (such as image and audio
samples), generating adversarial examples for discrete structures such as text
has proven significantly more challenging. In this paper we formulate the
attacks with discrete input on a set function as an optimization task. We prove
that this set function is submodular for some popular neural network text
classifiers under simplifying assumption. This finding guarantees a $1-1/e$
approximation factor for attacks that use the greedy algorithm. Meanwhile, we
show how to use the gradient of the attacked classifier to guide the greedy
search. Empirical studies with our proposed optimization scheme show
significantly improved attack ability and efficiency, on three different text
classification tasks over various baselines. We also use a joint sentence and
word paraphrasing technique to maintain the original semantics and syntax of
the text. This is validated by a human subject evaluation in subjective metrics
on the quality and semantic coherence of our generated adversarial text.


http://arxiv.org/abs/1811.12641
Transferable Adversarial Attacks for Image and Video Object Detection.
Xingxing Wei; Siyuan Liang; Xiaochun Cao; Jun Zhu
  Adversarial examples have been demonstrated to threaten many computer vision
tasks including object detection. However, the existing attacking methods for
object detection have two limitations: poor transferability, which denotes that
the generated adversarial examples have low success rate to attack other kinds
of detection methods, and high computation cost, which means that they need
more time to generate an adversarial image, and therefore are difficult to deal
with the video data. To address these issues, we utilize a generative mechanism
to obtain the adversarial image and video. In this way, the processing time is
reduced. To enhance the transferability, we destroy the feature maps extracted
from the feature network, which usually constitutes the basis of object
detectors. The proposed method is based on the Generative Adversarial Network
(GAN) framework, where we combine the high-level class loss and low-level
feature loss to jointly train the adversarial example generator. A series of
experiments conducted on PASCAL VOC and ImageNet VID datasets show that our
method can efficiently generate image and video adversarial examples, and more
importantly, these adversarial examples have better transferability, and thus,
are able to simultaneously attack two kinds of representative object detection
models: proposal based models like Faster-RCNN, and regression based models
like SSD.


http://arxiv.org/abs/1811.12673
ComDefend: An Efficient Image Compression Model to Defend Adversarial Examples.
Xiaojun Jia; Xingxing Wei; Xiaochun Cao; Hassan Foroosh
  Deep neural networks (DNNs) have been demonstrated to be vulnerable to
adversarial examples. Specifically, adding imperceptible perturbations to clean
images can fool the well trained deep neural networks. In this paper, we
propose an end-to-end image compression model to defend adversarial examples:
\textbf{ComDefend}. The proposed model consists of a compression convolutional
neural network (ComCNN) and a reconstruction convolutional neural network
(ResCNN). The ComCNN is used to maintain the structure information of the
original image and purify adversarial perturbations. And the ResCNN is used to
reconstruct the original image with high quality. In other words, ComDefend can
transform the adversarial image to its clean version, which is then fed to the
trained classifier. Our method is a pre-processing module, and does not modify
the classifier's structure during the whole process. Therefore, it can be
combined with other model-specific defense models to jointly improve the
classifier's robustness. A series of experiments conducted on MNIST, CIFAR10
and ImageNet show that the proposed method outperforms the state-of-the-art
defense methods, and is consistently effective to protect classifiers against
adversarial attacks.


http://arxiv.org/abs/1812.00037
Adversarial Defense by Stratified Convolutional Sparse Coding.
Bo Sun; Nian-hsuan Tsai; Fangchen Liu; Ronald Yu; Hao Su
  We propose an adversarial defense method that achieves state-of-the-art
performance among attack-agnostic adversarial defense methods while also
maintaining robustness to input resolution, scale of adversarial perturbation,
and scale of dataset size. Based on convolutional sparse coding, we construct a
stratified low-dimensional quasi-natural image space that faithfully
approximates the natural image space while also removing adversarial
perturbations. We introduce a novel Sparse Transformation Layer (STL) in
between the input image and the first layer of the neural network to
efficiently project images into our quasi-natural image space. Our experiments
show state-of-the-art performance of our method compared to other
attack-agnostic adversarial defense methods in various adversarial settings.


http://arxiv.org/abs/1811.12395
CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networks.
Akhilan Boopathy; Tsui-Wei Weng; Pin-Yu Chen; Sijia Liu; Luca Daniel
  Verifying robustness of neural network classifiers has attracted great
interests and attention due to the success of deep neural networks and their
unexpected vulnerability to adversarial perturbations. Although finding minimum
adversarial distortion of neural networks (with ReLU activations) has been
shown to be an NP-complete problem, obtaining a non-trivial lower bound of
minimum distortion as a provable robustness guarantee is possible. However,
most previous works only focused on simple fully-connected layers (multilayer
perceptrons) and were limited to ReLU activations. This motivates us to propose
a general and efficient framework, CNN-Cert, that is capable of certifying
robustness on general convolutional neural networks. Our framework is general
-- we can handle various architectures including convolutional layers,
max-pooling layers, batch normalization layer, residual blocks, as well as
general activation functions; our approach is efficient -- by exploiting the
special structure of convolutional layers, we achieve up to 17 and 11 times of
speed-up compared to the state-of-the-art certification algorithms (e.g.
Fast-Lin, CROWN) and 366 times of speed-up compared to the dual-LP approach
while our algorithm obtains similar or even better verification bounds. In
addition, CNN-Cert generalizes state-of-the-art algorithms e.g. Fast-Lin and
CROWN. We demonstrate by extensive experiments that our method outperforms
state-of-the-art lower-bound-based certification algorithms in terms of both
bound quality and speed.


http://arxiv.org/abs/1811.12335
Bayesian Adversarial Spheres: Bayesian Inference and Adversarial Examples in a Noiseless Setting.
Artur Bekasov; Iain Murray
  Modern deep neural network models suffer from adversarial examples, i.e.
confidently misclassified points in the input space. It has been shown that
Bayesian neural networks are a promising approach for detecting adversarial
points, but careful analysis is problematic due to the complexity of these
models. Recently Gilmer et al. (2018) introduced adversarial spheres, a toy
set-up that simplifies both practical and theoretical analysis of the problem.
In this work, we use the adversarial sphere set-up to understand the properties
of approximate Bayesian inference methods for a linear model in a noiseless
setting. We compare predictions of Bayesian and non-Bayesian methods,
showcasing the advantages of the former, although revealing open challenges for
deep learning applications.


http://arxiv.org/abs/1811.12601
Adversarial Examples as an Input-Fault Tolerance Problem.
Angus Galloway; Anna Golubeva; Graham W. Taylor
  We analyze the adversarial examples problem in terms of a model's fault
tolerance with respect to its input. Whereas previous work focuses on
arbitrarily strict threat models, i.e., $\epsilon$-perturbations, we consider
arbitrary valid inputs and propose an information-based characteristic for
evaluating tolerance to diverse input faults.


http://arxiv.org/abs/1811.12470
Analyzing Federated Learning through an Adversarial Lens.
Arjun Nitin Bhagoji; Supriyo Chakraborty; Prateek Mittal; Seraphin Calo
  Federated learning distributes model training among a multitude of agents,
who, guided by privacy concerns, perform training using their local data but
share only model parameter updates, for iterative aggregation at the server. In
this work, we explore the threat of model poisoning attacks on federated
learning initiated by a single, non-colluding malicious agent where the
adversarial objective is to cause the model to misclassify a set of chosen
inputs with high confidence. We explore a number of strategies to carry out
this attack, starting with simple boosting of the malicious agent's update to
overcome the effects of other agents' updates. To increase attack stealth, we
propose an alternating minimization strategy, which alternately optimizes for
the training loss and the adversarial objective. We follow up by using
parameter estimation for the benign agents' updates to improve on attack
success. Finally, we use a suite of interpretability techniques to generate
visual explanations of model decisions for both benign and malicious models and
show that the explanations are nearly visually indistinguishable. Our results
indicate that even a highly constrained adversary can carry out model poisoning
attacks while simultaneously maintaining stealth, thus highlighting the
vulnerability of the federated learning setting and the need to develop
effective defense strategies.


http://arxiv.org/abs/1811.11875
Adversarial Attacks for Optical Flow-Based Action Recognition Classifiers.
Nathan Inkawhich; Matthew Inkawhich; Yiran Chen; Hai Li
  The success of deep learning research has catapulted deep models into
production systems that our society is becoming increasingly dependent on,
especially in the image and video domains. However, recent work has shown that
these largely uninterpretable models exhibit glaring security vulnerabilities
in the presence of an adversary. In this work, we develop a powerful untargeted
adversarial attack for action recognition systems in both white-box and
black-box settings. Action recognition models differ from image-classification
models in that their inputs contain a temporal dimension, which we explicitly
target in the attack. Drawing inspiration from image classifier attacks, we
create new attacks which achieve state-of-the-art success rates on a two-stream
classifier trained on the UCF-101 dataset. We find that our attacks can
significantly degrade a model's performance with sparsely and imperceptibly
perturbed examples. We also demonstrate the transferability of our attacks to
black-box action recognition systems.


http://arxiv.org/abs/1811.11553
Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects.
Michael A. Alcorn; Qi Li; Zhitao Gong; Chengfei Wang; Long Mai; Wei-Shinn Ku; Anh Nguyen
  Despite excellent performance on stationary test sets, deep neural networks
(DNNs) can fail to generalize to out-of-distribution (OoD) inputs, including
natural, non-adversarial ones, which are common in real-world settings. In this
paper, we present a framework for discovering DNN failures that harnesses 3D
renderers and 3D models. That is, we estimate the parameters of a 3D renderer
that cause a target DNN to misbehave in response to the rendered image. Using
our framework and a self-assembled dataset of 3D objects, we investigate the
vulnerability of DNNs to OoD poses of well-known objects in ImageNet. For
objects that are readily recognized by DNNs in their canonical poses, DNNs
incorrectly classify 97% of their pose space. In addition, DNNs are highly
sensitive to slight pose perturbations. Importantly, adversarial poses transfer
across models and datasets. We find that 99.9% and 99.4% of the poses
misclassified by Inception-v3 also transfer to the AlexNet and ResNet-50 image
classifiers trained on the same ImageNet dataset, respectively, and 75.5%
transfer to the YOLOv3 object detector trained on MS COCO.


http://arxiv.org/abs/1811.11493
A randomized gradient-free attack on ReLU networks.
Francesco Croce; Matthias Hein
  It has recently been shown that neural networks but also other classifiers
are vulnerable to so called adversarial attacks e.g. in object recognition an
almost non-perceivable change of the image changes the decision of the
classifier. Relatively fast heuristics have been proposed to produce these
adversarial inputs but the problem of finding the optimal adversarial input,
that is with the minimal change of the input, is NP-hard. While methods based
on mixed-integer optimization which find the optimal adversarial input have
been developed, they do not scale to large networks. Currently, the attack
scheme proposed by Carlini and Wagner is considered to produce the best
adversarial inputs. In this paper we propose a new attack scheme for the class
of ReLU networks based on a direct optimization on the resulting linear
regions. In our experimental validation we improve in all except one experiment
out of 18 over the Carlini-Wagner attack with a relative improvement of up to
9\%. As our approach is based on the geometrical structure of ReLU networks, it
is less susceptible to defences targeting their functional properties.


http://arxiv.org/abs/1811.11402
Adversarial Machine Learning And Speech Emotion Recognition: Utilizing Generative Adversarial Networks For Robustness.
Siddique Latif; Rajib Rana; Junaid Qadir
  Deep learning has undoubtedly offered tremendous improvements in the
performance of state-of-the-art speech emotion recognition (SER) systems.
However, recent research on adversarial examples poses enormous challenges on
the robustness of SER systems by showing the susceptibility of deep neural
networks to adversarial examples as they rely only on small and imperceptible
perturbations. In this study, we evaluate how adversarial examples can be used
to attack SER systems and propose the first black-box adversarial attack on SER
systems. We also explore potential defenses including adversarial training and
generative adversarial network (GAN) to enhance robustness. Experimental
evaluations suggest various interesting aspects of the effective utilization of
adversarial examples useful for achieving robustness for SER systems opening up
opportunities for researchers to further innovate in this space.


http://arxiv.org/abs/1811.11304
Universal Adversarial Training.
Ali Shafahi; Mahyar Najibi; Zheng Xu; John Dickerson; Larry S. Davis; Tom Goldstein
  Standard adversarial attacks change the predicted class label of an image by
adding specially tailored small perturbations to its pixels. In contrast, a
universal perturbation is an update that can be added to any image in a broad
class of images, while still changing the predicted class label. We study the
efficient generation of universal adversarial perturbations, and also efficient
methods for hardening networks to these attacks. We propose a simple
optimization-based universal attack that reduces the top-1 accuracy of various
network architectures on ImageNet to less than 20%, while learning the
universal perturbation 13X faster than the standard method. To defend against
these perturbations, we propose universal adversarial training, which models
the problem of robust classifier generation as a two-player min-max game. This
method is much faster and more scalable than conventional adversarial training
with a strong adversary (PGD), and yet yields models that are extremely
resistant to universal attacks, and comparably resistant to standard
(per-instance) black box attacks. We also discover a rather fascinating
side-effect of universal adversarial training: attacks built for universally
robust models transfer better to other (black box) models than those built with
conventional adversarial training.


http://arxiv.org/abs/1811.11079
Robust Classification of Financial Risk.
Suproteem K. Sarkar; Kojin Oshiba; Daniel Giebisch; Yaron Singer
  Algorithms are increasingly common components of high-impact decision-making,
and a growing body of literature on adversarial examples in laboratory settings
indicates that standard machine learning models are not robust. This suggests
that real-world systems are also susceptible to manipulation or
misclassification, which especially poses a challenge to machine learning
models used in financial services. We use the loan grade classification problem
to explore how machine learning models are sensitive to small changes in
user-reported data, using adversarial attacks documented in the literature and
an original, domain-specific attack. Our work shows that a robust optimization
algorithm can build models for financial services that are resistant to
misclassification on perturbations. To the best of our knowledge, this is the
first study of adversarial attacks and defenses for deep learning in financial
services.


http://arxiv.org/abs/1811.10828
A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks.
Jinghui Chen; Jinfeng Yi; Quanquan Gu
  Depending on how much information an adversary can access to, adversarial
attacks can be classified as white-box attack and black-box attack. In both
cases, optimization-based attack algorithms can achieve relatively low
distortions and high attack success rates. However, they usually suffer from
poor time and query complexities, thereby limiting their practical usefulness.
In this work, we focus on the problem of developing efficient and effective
optimization-based adversarial attack algorithms. In particular, we propose a
novel adversarial attack framework for both white-box and black-box settings
based on the non-convex Frank-Wolfe algorithm. We show in theory that the
proposed attack algorithms are efficient with an $O(1/\sqrt{T})$ convergence
rate. The empirical results of attacking Inception V3 model and ResNet V2 model
on the ImageNet dataset also verify the efficiency and effectiveness of the
proposed algorithms. More specific, our proposed algorithms attain the highest
attack success rate in both white-box and black-box attacks among all
baselines, and are more time and query efficient than the state-of-the-art.


http://arxiv.org/abs/1811.11310
Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry.
Kevin McCloskey; Ankur Taly; Federico Monti; Michael P. Brenner; Lucy Colwell
  Deep neural networks have achieved state of the art accuracy at classifying
molecules with respect to whether they bind to specific protein targets. A key
breakthrough would occur if these models could reveal the fragment
pharmacophores that are causally involved in binding. Extracting chemical
details of binding from the networks could potentially lead to scientific
discoveries about the mechanisms of drug actions. But doing so requires shining
light into the black box that is the trained neural network model, a task that
has proved difficult across many domains. Here we show how the binding
mechanism learned by deep neural network models can be interrogated, using a
recently described attribution method. We first work with carefully constructed
synthetic datasets, in which the 'fragment logic' of binding is fully known. We
find that networks that achieve perfect accuracy on held out test datasets
still learn spurious correlations due to biases in the datasets, and we are
able to exploit this non-robustness to construct adversarial examples that fool
the model. The dataset bias makes these models unreliable for accurately
revealing information about the mechanisms of protein-ligand binding. In light
of our findings, we prescribe a test that checks for dataset bias given a
hypothesis. If the test fails, it indicates that either the model must be
simplified or regularized and/or that the training dataset requires
augmentation.


http://arxiv.org/abs/1811.10716
Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks.
Jianyu Wang
  In this paper, we study fast training of adversarially robust models. From
the analyses on the state-of-the-art defense method, i.e., the multi-step
adversarial training~\cite{madry2017towards}, we hypothesize that the gradient
magnitude links to the model robustness. Motivated by this, we propose to
perturb both the image and the label during training, which we call Bilateral
Adversarial Training (BAT). To generate the adversarial label, we derive an
closed-form heuristic solution. To generate the adversarial image, we use
one-step targeted attack with the target label being the most confusing class.
In the experiment, we first show that random start and the most confusing
target attack effectively prevent the label leaking and gradient masking
problem. Then coupled with the adversarial label part, our model significantly
improves the state-of-the-art results. For example, against PGD100 attack with
cross-entropy loss, on CIFAR10, we achieve 63.7\% versus 47.2\%; on SVHN, we
achieve 59.1\% versus 42.1\%; on CIFAR100, we achieve 25.3\% versus 23.4\%.
Note that these results are obtained by the fast one-step adversarial training.


http://arxiv.org/abs/1811.10745
ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies.
Bao Wang; Binjie Yuan; Zuoqiang Shi; Stanley J. Osher
  Empirical adversarial risk minimization (EARM) is a widely used mathematical
framework to robustly train deep neural nets (DNNs) that are resistant to
adversarial attacks. However, both natural and robust accuracies, in
classifying clean and adversarial images, respectively, of the trained robust
models are far from satisfactory. In this work, we unify the theory of optimal
control of transport equations with the practice of training and testing of
ResNets. Based on this unified viewpoint, we propose a simple yet effective
ResNets ensemble algorithm to boost the accuracy of the robustly trained model
on both clean and adversarial images. The proposed algorithm consists of two
components: First, we modify the base ResNets by injecting a variance specified
Gaussian noise to the output of each residual mapping. Second, we average over
the production of multiple jointly trained modified ResNets to get the final
prediction. These two steps give an approximation to the Feynman-Kac formula
for representing the solution of a transport equation with viscosity, or a
convection-diffusion equation. For the CIFAR10 benchmark, this simple algorithm
leads to a robust model with a natural accuracy of {\bf 85.62}\% on clean
images and a robust accuracy of ${\bf 57.94 \%}$ under the 20 iterations of the
IFGSM attack, which outperforms the current state-of-the-art in defending
against IFGSM attack on the CIFAR10. Both natural and robust accuracies of the
proposed ResNets ensemble can be improved dynamically as the building block
ResNet advances. The code is available at:
\url{https://github.com/BaoWangMath/EnResNet}.


http://arxiv.org/abs/1811.09982
Is Data Clustering in Adversarial Settings Secure?.
Battista Biggio; Ignazio Pillai; Samuel Rota Bulò; Davide Ariu; Marcello Pelillo; Fabio Roli
  Clustering algorithms have been increasingly adopted in security applications
to spot dangerous or illicit activities. However, they have not been originally
devised to deal with deliberate attack attempts that may aim to subvert the
clustering process itself. Whether clustering can be safely adopted in such
settings remains thus questionable. In this work we propose a general framework
that allows one to identify potential attacks against clustering algorithms,
and to evaluate their impact, by making specific assumptions on the adversary's
goal, knowledge of the attacked system, and capabilities of manipulating the
input data. We show that an attacker may significantly poison the whole
clustering process by adding a relatively small percentage of attack samples to
the input data, and that some attack samples may be obfuscated to be hidden
within some existing clusters. We present a case study on single-linkage
hierarchical clustering, and report experiments on clustering of malware
samples and handwritten digits.


http://arxiv.org/abs/1811.09831
Attention, Please! Adversarial Defense via Attention Rectification and Preservation.
Shangxi Wu; Jitao Sang; Kaiyuan Xu; Jiaming Zhang; Yanfeng Sun; Liping Jing; Jian Yu
  This study provides a new understanding of the adversarial attack problem by
examining the correlation between adversarial attack and visual attention
change. In particular, we observed that: (1) images with incomplete attention
regions are more vulnerable to adversarial attacks; and (2) successful
adversarial attacks lead to deviated and scattered attention map. Accordingly,
an attention-based adversarial defense framework is designed to simultaneously
rectify the attention map for prediction and preserve the attention area
between adversarial and original images. The problem of adding iteratively
attacked samples is also discussed in the context of visual attention change.
We hope the attention-related data analysis and defense solution in this study
will shed some light on the mechanism behind the adversarial attack and also
facilitate future adversarial defense/attack model design.


http://arxiv.org/abs/1811.09716
Robustness via curvature regularization, and vice versa.
Seyed-Mohsen Moosavi-Dezfooli; Alhussein Fawzi; Jonathan Uesato; Pascal Frossard
  State-of-the-art classifiers have been shown to be largely vulnerable to
adversarial perturbations. One of the most effective strategies to improve
robustness is adversarial training. In this paper, we investigate the effect of
adversarial training on the geometry of the classification landscape and
decision boundaries. We show in particular that adversarial training leads to a
significant decrease in the curvature of the loss surface with respect to
inputs, leading to a drastically more "linear" behaviour of the network. Using
a locally quadratic approximation, we provide theoretical evidence on the
existence of a strong relation between large robustness and small curvature. To
further show the importance of reduced curvature for improving the robustness,
we propose a new regularizer that directly minimizes curvature of the loss
surface, and leads to adversarial robustness that is on par with adversarial
training. Besides being a more efficient and principled alternative to
adversarial training, the proposed regularizer confirms our claims on the
importance of exhibiting quasi-linear behavior in the vicinity of data points
in order to achieve robustness.


http://arxiv.org/abs/1811.09600
Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses.
Jérôme Rony; Luiz G. Hafemann; Luiz S. Oliveira; Ismail Ben Ayed; Robert Sabourin; Eric Granger
  Research on adversarial examples in computer vision tasks has shown that
small, often imperceptible changes to an image can induce misclassification,
which has security implications for a wide range of image processing systems.
Considering $L_2$ norm distortions, the Carlini and Wagner attack is presently
the most effective white-box attack in the literature. However, this method is
slow since it performs a line-search for one of the optimization terms, and
often requires thousands of iterations. In this paper, an efficient approach is
proposed to generate gradient-based attacks that induce misclassifications with
low $L_2$ norm, by decoupling the direction and the norm of the adversarial
perturbation that is added to the image. Experiments conducted on the MNIST,
CIFAR-10 and ImageNet datasets indicate that our attack achieves comparable
results to the state-of-the-art (in terms of $L_2$ norm) with considerably
fewer iterations (as few as 100 iterations), which opens the possibility of
using these attacks for adversarial training. Models trained with our attack
achieve state-of-the-art robustness against white-box gradient-based $L_2$
attacks on the MNIST and CIFAR-10 datasets, outperforming the Madry defense
when the attacks are limited to a maximum norm.


http://arxiv.org/abs/1811.09310
Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack.
Adnan Siraj Rakin; Zhezhi He; Deliang Fan
  Recent development in the field of Deep Learning have exposed the underlying
vulnerability of Deep Neural Network (DNN) against adversarial examples. In
image classification, an adversarial example is a carefully modified image that
is visually imperceptible to the original image but can cause DNN model to
misclassify it. Training the network with Gaussian noise is an effective
technique to perform model regularization, thus improving model robustness
against input variation. Inspired by this classical method, we explore to
utilize the regularization characteristic of noise injection to improve DNN's
robustness against adversarial attack. In this work, we propose
Parametric-Noise-Injection (PNI) which involves trainable Gaussian noise
injection at each layer on either activation or weights through solving the
min-max optimization problem, embedded with adversarial training. These
parameters are trained explicitly to achieve improved robustness. To the best
of our knowledge, this is the first work that uses trainable noise injection to
improve network robustness against adversarial attacks, rather than manually
configuring the injected noise level through cross-validation. The extensive
results show that our proposed PNI technique effectively improves the
robustness against a variety of powerful white-box and black-box attacks such
as PGD, C & W, FGSM, transferable attack and ZOO attack. Last but not the
least, PNI method improves both clean- and perturbed-data accuracy in
comparison to the state-of-the-art defense methods, which outperforms current
unbroken PGD defense by 1.1 % and 6.8 % on clean test data and perturbed test
data respectively using Resnet-20 architecture.


http://arxiv.org/abs/1811.09300
Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles.
Edward Grefenstette; Robert Stanforth; Brendan O'Donoghue; Jonathan Uesato; Grzegorz Swirszcz; Pushmeet Kohli
  While deep learning has led to remarkable results on a number of challenging
problems, researchers have discovered a vulnerability of neural networks in
adversarial settings, where small but carefully chosen perturbations to the
input can make the models produce extremely inaccurate outputs. This makes
these models particularly unsuitable for safety-critical application domains
(e.g. self-driving cars) where robustness is extremely important. Recent work
has shown that augmenting training with adversarially generated data provides
some degree of robustness against test-time attacks. In this paper we
investigate how this approach scales as we increase the computational budget
given to the defender. We show that increasing the number of parameters in
adversarially-trained models increases their robustness, and in particular that
ensembling smaller models while adversarially training the entire ensemble as a
single model is a more efficient way of spending said budget than simply using
a larger single model. Crucially, we show that it is the adversarial training
of the ensemble, rather than the ensembling of adversarially trained models,
which provides robustness.


http://arxiv.org/abs/1811.09043
Detecting Adversarial Perturbations Through Spatial Behavior in Activation Spaces.
Ziv Katzir; Yuval Elovici
  Neural network based classifiers are still prone to manipulation through
adversarial perturbations. State of the art attacks can overcome most of the
defense or detection mechanisms suggested so far, and adversaries have the
upper hand in this arms race. Adversarial examples are designed to resemble the
normal input from which they were constructed, while triggering an incorrect
classification. This basic design goal leads to a characteristic spatial
behavior within the context of Activation Spaces, a term coined by the authors
to refer to the hyperspaces formed by the activation values of the network's
layers. Within the output of the first layers of the network, an adversarial
example is likely to resemble normal instances of the source class, while in
the final layers such examples will diverge towards the adversary's target
class. The steps below enable us to leverage this inherent shift from one class
to another in order to form a novel adversarial example detector. We construct
Euclidian spaces out of the activation values of each of the deep neural
network layers. Then, we induce a set of k-nearest neighbor classifiers (k-NN),
one per activation space of each neural network layer, using the
non-adversarial examples. We leverage those classifiers to produce a sequence
of class labels for each nonperturbed input sample and estimate the a priori
probability for a class label change between one activation space and another.
During the detection phase we compute a sequence of classification labels for
each input using the trained classifiers. We then estimate the likelihood of
those classification sequences and show that adversarial sequences are far less
likely than normal ones. We evaluated our detection method against the state of
the art C&W attack method, using two image classification datasets (MNIST,
CIFAR-10) reaching an AUC 0f 0.95 for the CIFAR-10 dataset.


http://arxiv.org/abs/1811.09020
Task-generalizable Adversarial Attack based on Perceptual Metric.
Muzammal Naseer; Salman H. Khan; Shafin Rahman; Fatih Porikli
  Deep neural networks (DNNs) can be easily fooled by adding human
imperceptible perturbations to the images. These perturbed images are known as
`adversarial examples' and pose a serious threat to security and safety
critical systems. A litmus test for the strength of adversarial examples is
their transferability across different DNN models in a black box setting (i.e.
when the target model's architecture and parameters are not known to attacker).
Current attack algorithms that seek to enhance adversarial transferability work
on the decision level i.e. generate perturbations that alter the network
decisions. This leads to two key limitations: (a) An attack is dependent on the
task-specific loss function (e.g. softmax cross-entropy for object recognition)
and therefore does not generalize beyond its original task. (b) The adversarial
examples are specific to the network architecture and demonstrate poor
transferability to other network architectures. We propose a novel approach to
create adversarial examples that can broadly fool different networks on
multiple tasks. Our approach is based on the following intuition: "Perpetual
metrics based on neural network features are highly generalizable and show
excellent performance in measuring and stabilizing input distortions. Therefore
an ideal attack that creates maximum distortions in the network feature space
should realize highly transferable examples". We report extensive experiments
to show how adversarial examples generalize across multiple networks for
classification, object detection and segmentation tasks.


http://arxiv.org/abs/1811.09008
Towards Robust Neural Networks with Lipschitz Continuity.
Muhammad Usama; Dong Eui Chang
  Deep neural networks have shown remarkable performance across a wide range of
vision-based tasks, particularly due to the availability of large-scale
datasets for training and better architectures. However, data seen in the real
world are often affected by distortions that not accounted for by the training
datasets. In this paper, we address the challenge of robustness and stability
of neural networks and propose a general training method that can be used to
make the existing neural network architectures more robust and stable to input
visual perturbations while using only available datasets for training. Proposed
training method is convenient to use as it does not require data augmentation
or changes in the network architecture. We provide theoretical proof as well as
empirical evidence for the efficiency of the proposed training method by
performing experiments with existing neural network architectures and
demonstrate that same architecture when trained with the proposed training
method perform better than when trained with conventional training approach in
the presence of noisy datasets.


http://arxiv.org/abs/1811.08577
How the Softmax Output is Misleading for Evaluating the Strength of Adversarial Examples.
Utku Ozbulak; Neve Wesley De; Messem Arnout Van
  Even before deep learning architectures became the de facto models for
complex computer vision tasks, the softmax function was, given its elegant
properties, already used to analyze the predictions of feedforward neural
networks. Nowadays, the output of the softmax function is also commonly used to
assess the strength of adversarial examples: malicious data points designed to
fail machine learning models during the testing phase. However, in this paper,
we show that it is possible to generate adversarial examples that take
advantage of some properties of the softmax function, leading to undesired
outcomes when interpreting the strength of the adversarial examples at hand.
Specifically, we argue that the output of the softmax function is a poor
indicator when the strength of an adversarial example is analyzed and that this
indicator can be easily tricked by already existing methods for adversarial
example generation.


http://arxiv.org/abs/1811.08484
MimicGAN: Corruption-Mimicking for Blind Image Recovery & Adversarial Defense.
Rushil Anirudh; Jayaraman J. Thiagarajan; Bhavya Kailkhura; Timo Bremer
  Solving inverse problems continues to be a central challenge in computer
vision. Existing techniques either explicitly construct an inverse mapping
using prior knowledge about the corruption, or learn the inverse directly using
a large collection of examples. However, in practice, the nature of corruption
may be unknown, and thus it is challenging to regularize the problem of
inferring a plausible solution. On the other hand, collecting task-specific
training data is tedious for known corruptions and impossible for unknown ones.
We present MimicGAN, an unsupervised technique to solve general inverse
problems based on image priors in the form of generative adversarial networks
(GANs). Using a GAN prior, we show that one can reliably recover solutions to
underdetermined inverse problems through a surrogate network that learns to
mimic the corruption at test time. Our system successively estimates the
corruption and the clean image without the need for supervisory training, while
outperforming existing baselines in blind image recovery. We also demonstrate
that MimicGAN improves upon recent GAN-based defenses against adversarial
attacks and represents one of the strongest test-time defenses available today.


http://arxiv.org/abs/1811.08458
Intermediate Level Adversarial Attack for Enhanced Transferability.
Qian Huang; Zeqi Gu; Isay Katsman; Horace He; Pian Pawakapan; Zhiqiu Lin; Serge Belongie; Ser-Nam Lim
  Neural networks are vulnerable to adversarial examples, malicious inputs
crafted to fool trained models. Adversarial examples often exhibit black-box
transfer, meaning that adversarial examples for one model can fool another
model. However, adversarial examples may be overfit to exploit the particular
architecture and feature representation of a source model, resulting in
sub-optimal black-box transfer attacks to other target models. This leads us to
introduce the Intermediate Level Attack (ILA), which attempts to fine-tune an
existing adversarial example for greater black-box transferability by
increasing its perturbation on a pre-specified layer of the source model. We
show that our method can effectively achieve this goal and that we can decide a
nearly-optimal layer of the source model to perturb without any knowledge of
the target models.


http://arxiv.org/abs/1811.08080
Lightweight Lipschitz Margin Training for Certified Defense against Adversarial Examples.
Hajime Ono; Tsubasa Takahashi; Kazuya Kakizaki
  How can we make machine learning provably robust against adversarial examples
in a scalable way? Since certified defense methods, which ensure
$\epsilon$-robust, consume huge resources, they can only achieve small degree
of robustness in practice. Lipschitz margin training (LMT) is a scalable
certified defense, but it can also only achieve small robustness due to
over-regularization. How can we make certified defense more efficiently? We
present LC-LMT, a light weight Lipschitz margin training which solves the above
problem. Our method has the following properties; (a) efficient: it can achieve
$\epsilon$-robustness at early epoch, and (b) robust: it has a potential to get
higher robustness than LMT. In the evaluation, we demonstrate the benefits of
the proposed method. LC-LMT can achieve required robustness more than 30 epoch
earlier than LMT in MNIST, and shows more than 90 $\%$ accuracy against both
legitimate and adversarial inputs.


http://arxiv.org/abs/1812.02622
Convolutional Neural Networks with Transformed Input based on Robust Tensor Network Decomposition.
Jenn-Bing Ong; Wee-Keong Ng; C. -C. Jay Kuo
  Tensor network decomposition, originated from quantum physics to model
entangled many-particle quantum systems, turns out to be a promising
mathematical technique to efficiently represent and process big data in
parsimonious manner. In this study, we show that tensor networks can
systematically partition structured data, e.g. color images, for distributed
storage and communication in privacy-preserving manner. Leveraging the sea of
big data and metadata privacy, empirical results show that neighbouring
subtensors with implicit information stored in tensor network formats cannot be
identified for data reconstruction. This technique complements the existing
encryption and randomization techniques which store explicit data
representation at one place and highly susceptible to adversarial attacks such
as side-channel attacks and de-anonymization. Furthermore, we propose a theory
for adversarial examples that mislead convolutional neural networks to
misclassification using subspace analysis based on singular value decomposition
(SVD). The theory is extended to analyze higher-order tensors using
tensor-train SVD (TT-SVD); it helps to explain the level of susceptibility of
different datasets to adversarial attacks, the structural similarity of
different adversarial attacks including global and localized attacks, and the
efficacy of different adversarial defenses based on input transformation. An
efficient and adaptive algorithm based on robust TT-SVD is then developed to
detect strong and static adversarial attacks.


http://arxiv.org/abs/1811.07950
Optimal Transport Classifier: Defending Against Adversarial Attacks by Regularized Deep Embedding.
Yao Li; Martin Renqiang Min; Wenchao Yu; Cho-Jui Hsieh; Thomas C. M. Lee; Erik Kruus
  Recent studies have demonstrated the vulnerability of deep convolutional
neural networks against adversarial examples. Inspired by the observation that
the intrinsic dimension of image data is much smaller than its pixel space
dimension and the vulnerability of neural networks grows with the input
dimension, we propose to embed high-dimensional input images into a
low-dimensional space to perform classification. However, arbitrarily
projecting the input images to a low-dimensional space without regularization
will not improve the robustness of deep neural networks. Leveraging optimal
transport theory, we propose a new framework, Optimal Transport Classifier
(OT-Classifier), and derive an objective that minimizes the discrepancy between
the distribution of the true label and the distribution of the OT-Classifier
output. Experimental results on several benchmark datasets show that, our
proposed framework achieves state-of-the-art performance against strong
adversarial attack methods.


http://arxiv.org/abs/1811.07457
Generalizable Adversarial Training via Spectral Normalization.
Farzan Farnia; Jesse M. Zhang; David Tse
  Deep neural networks (DNNs) have set benchmarks on a wide array of supervised
learning tasks. Trained DNNs, however, often lack robustness to minor
adversarial perturbations to the input, which undermines their true
practicality. Recent works have increased the robustness of DNNs by fitting
networks using adversarially-perturbed training samples, but the improved
performance can still be far below the performance seen in non-adversarial
settings. A significant portion of this gap can be attributed to the decrease
in generalization performance due to adversarial training. In this work, we
extend the notion of margin loss to adversarial settings and bound the
generalization error for DNNs trained under several well-known gradient-based
attack schemes, motivating an effective regularization scheme based on spectral
normalization of the DNN's weight matrices. We also provide a
computationally-efficient method for normalizing the spectral norm of
convolutional layers with arbitrary stride and padding schemes in deep
convolutional networks. We evaluate the power of spectral normalization
extensively on combinations of datasets, network architectures, and adversarial
training schemes. The code is available at
https://github.com/jessemzhang/dl_spectral_normalization.


http://arxiv.org/abs/1811.07375
The Taboo Trap: Behavioural Detection of Adversarial Samples.
Ilia Shumailov; Yiren Zhao; Robert Mullins; Ross Anderson
  Deep Neural Networks (DNNs) have become a powerful tool for a wide range of
problems. Yet recent work has shown an increasing variety of adversarial
samples that can fool them. Most existing detection mechanisms impose
significant costs, either by using additional classifiers to spot adversarial
samples, or by requiring the DNN to be restructured. In this paper, we
introduce a novel defence. We train our DNN so that, as long as it is working
as intended on the kind of inputs we expect, its behavior is constrained, in
that a set of behaviors are taboo. If it is exposed to adversarial samples,
they will often cause a taboo behavior, which we can detect. As an analogy, we
can imagine that we are teaching our robot good manners; if it's ever rude, we
know it's come under some bad influence. This defence mechanism is very simple
and, although it involves a modest increase in training, has almost zero
computation overhead at runtime -- making it particularly suitable for use in
embedded systems. Taboos can be both subtle and diverse. Just as humans' choice
of language can convey a lot of information about location, affiliation, class
and much else that can be opaque to outsiders but that enables members of the
same group to recognise each other, so also taboo choice can encode and hide
information. We can use this to make adversarial attacks much harder. It is a
well-established design principle that the security of a system should not
depend on the obscurity of its design, but of some variable (the key) which can
differ between implementations and be changed as necessary. We explain how
taboos can be used to equip a classifier with just such a key, and to tune the
keying mechanism to adversaries of various capabilities. We evaluate the
performance of a prototype against a wide range of attacks and show how our
simple defense can work well in practice.


http://arxiv.org/abs/1811.07311
Regularized adversarial examples for model interpretability.
Yoel Shoshan; Vadim Ratner
  As machine learning algorithms continue to improve, there is an increasing
need for explaining why a model produces a certain prediction for a certain
input. In recent years, several methods for model interpretability have been
developed, aiming to provide explanation of which subset regions of the model
input is the main reason for the model prediction. In parallel, a significant
research community effort is occurring in recent years for developing
adversarial example generation methods for fooling models, while not altering
the true label of the input,as it would have been classified by a human
annotator. In this paper, we bridge the gap between adversarial example
generation and model interpretability, and introduce a modification to the
adversarial example generation process which encourages better
interpretability. We analyze the proposed method on a public medical imaging
dataset, both quantitatively and qualitatively, and show that it significantly
outperforms the leading known alternative method. Our suggested method is
simple to implement, and can be easily plugged into most common adversarial
example generation frameworks. Additionally, we propose an explanation quality
metric - $APE$ - "Adversarial Perturbative Explanation", which measures how
well an explanation describes model decisions.


http://arxiv.org/abs/1811.07266
DeepConsensus: using the consensus of features from multiple layers to attain robust image classification.
Yuchen Li; Safwan Hossain; Kiarash Jamali; Frank Rudzicz
  We consider a classifier whose test set is exposed to various perturbations
that are not present in the training set. These test samples still contain
enough features to map them to the same class as their unperturbed counterpart.
Current architectures exhibit rapid degradation of accuracy when trained on
standard datasets but then used to classify perturbed samples of that data. To
address this, we present a novel architecture named DeepConsensus that
significantly improves generalization to these test-time perturbations. Our key
insight is that deep neural networks should directly consider summaries of low
and high level features when making classifications. Existing convolutional
neural networks can be augmented with DeepConsensus, leading to improved
resistance against large and small perturbations on MNIST, EMNIST,
FashionMNIST, CIFAR10 and SVHN datasets.


http://arxiv.org/abs/1811.07211
Classifiers Based on Deep Sparse Coding Architectures are Robust to Deep Learning Transferable Examples.
Jacob M. Springer; Charles S. Strauss; Austin M. Thresher; Edward Kim; Garrett T. Kenyon
  Although deep learning has shown great success in recent years, researchers
have discovered a critical flaw where small, imperceptible changes in the input
to the system can drastically change the output classification. These attacks
are exploitable in nearly all of the existing deep learning classification
frameworks. However, the susceptibility of deep sparse coding models to
adversarial examples has not been examined. Here, we show that classifiers
based on a deep sparse coding model whose classification accuracy is
competitive with a variety of deep neural network models are robust to
adversarial examples that effectively fool those same deep learning models. We
demonstrate both quantitatively and qualitatively that the robustness of deep
sparse coding models to adversarial examples arises from two key properties.
First, because deep sparse coding models learn general features corresponding
to generators of the dataset as a whole, rather than highly discriminative
features for distinguishing specific classes, the resulting classifiers are
less dependent on idiosyncratic features that might be more easily exploited.
Second, because deep sparse coding models utilize fixed point attractor
dynamics with top-down feedback, it is more difficult to find small changes to
the input that drive the resulting representations out of the correct attractor
basin.


http://arxiv.org/abs/1811.07108
Boosting the Robustness Verification of DNN by Identifying the Achilles's Heel.
Chengdong Feng; Zhenbang Chen; Weijiang Hong; Hengbiao Yu; Wei Dong; Ji Wang
  Deep Neural Network (DNN) is a widely used deep learning technique. How to
ensure the safety of DNN-based system is a critical problem for the research
and application of DNN. Robustness is an important safety property of DNN.
However, existing work of verifying DNN's robustness is time-consuming and hard
to scale to large-scale DNNs. In this paper, we propose a boosting method for
DNN robustness verification, aiming to find counter-examples earlier. Our
observation is DNN's different inputs have different possibilities of existing
counter-examples around them, and the input with a small difference between the
largest output value and the second largest output value tends to be the
achilles's heel of the DNN. We have implemented our method and applied it on
Reluplex, a state-of-the-art DNN verification tool, and four DNN attacking
methods. The results of the extensive experiments on two benchmarks indicate
the effectiveness of our boosting method.


http://arxiv.org/abs/1811.07018
Protecting Voice Controlled Systems Using Sound Source Identification Based on Acoustic Cues.
Yuan Gong; Christian Poellabauer
  Over the last few years, a rapidly increasing number of Internet-of-Things
(IoT) systems that adopt voice as the primary user input have emerged. These
systems have been shown to be vulnerable to various types of voice spoofing
attacks. Existing defense techniques can usually only protect from a specific
type of attack or require an additional authentication step that involves
another device. Such defense strategies are either not strong enough or lower
the usability of the system. Based on the fact that legitimate voice commands
should only come from humans rather than a playback device, we propose a novel
defense strategy that is able to detect the sound source of a voice command
based on its acoustic features. The proposed defense strategy does not require
any information other than the voice command itself and can protect a system
from multiple types of spoofing attacks. Our proof-of-concept experiments
verify the feasibility and effectiveness of this defense strategy.


http://arxiv.org/abs/1811.06969
DARCCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules.
Nicholas Frosst; Sara Sabour; Geoffrey Hinton
  We present a simple technique that allows capsule models to detect
adversarial images. In addition to being trained to classify images, the
capsule model is trained to reconstruct the images from the pose parameters and
identity of the correct top-level capsule. Adversarial images do not look like
a typical member of the predicted class and they have much larger
reconstruction errors when the reconstruction is produced from the top-level
capsule for that class. We show that setting a threshold on the $l2$ distance
between the input image and its reconstruction from the winning capsule is very
effective at detecting adversarial images for three different datasets. The
same technique works quite well for CNNs that have been trained to reconstruct
the image from all or part of the last hidden layer before the softmax. We then
explore a stronger, white-box attack that takes the reconstruction error into
account. This attack is able to fool our detection technique but in order to
make the model change its prediction to another class, the attack must
typically make the "adversarial" image resemble images of the other class.


http://arxiv.org/abs/1811.06609
A Spectral View of Adversarially Robust Features.
Shivam Garg; Vatsal Sharan; Brian Hu Zhang; Gregory Valiant
  Given the apparent difficulty of learning models that are robust to
adversarial perturbations, we propose tackling the simpler problem of
developing adversarially robust features. Specifically, given a dataset and
metric of interest, the goal is to return a function (or multiple functions)
that 1) is robust to adversarial perturbations, and 2) has significant
variation across the datapoints. We establish strong connections between
adversarially robust features and a natural spectral property of the geometry
of the dataset and metric of interest. This connection can be leveraged to
provide both robust features, and a lower bound on the robustness of any
function that has significant variance across the dataset. Finally, we provide
empirical evidence that the adversarially robust features given by this
spectral approach can be fruitfully leveraged to learn a robust (and accurate)
model.


http://arxiv.org/abs/1811.06539
A note on hyperparameters in black-box adversarial examples.
Jamie Hayes
  Since Biggio et al. (2013) and Szegedy et al. (2013) first drew attention to
adversarial examples, there has been a flood of research into defending and
attacking machine learning models. However, almost all proposed attacks assume
white-box access to a model. In other words, the attacker is assumed to have
perfect knowledge of the models weights and architecture. With this insider
knowledge, a white-box attack can leverage gradient information to craft
adversarial examples. Black-box attacks assume no knowledge of the model
weights or architecture. These attacks craft adversarial examples using
information only contained in the logits or hard classification label. Here, we
assume the attacker can use the logits in order to find an adversarial example.
Empirically, we show that 2-sided stochastic gradient estimation techniques are
not sensitive to scaling parameters, and can be used to mount powerful
black-box attacks requiring relatively few model queries.


http://arxiv.org/abs/1811.06492
Mathematical Analysis of Adversarial Attacks.
Zehao Dou; Stanley J. Osher; Bao Wang
  In this paper, we analyze efficacy of the fast gradient sign method (FGSM)
and the Carlini-Wagner's L2 (CW-L2) attack. We prove that, within a certain
regime, the untargeted FGSM can fool any convolutional neural nets (CNNs) with
ReLU activation; the targeted FGSM can mislead any CNNs with ReLU activation to
classify any given image into any prescribed class. For a special two-layer
neural network: a linear layer followed by the softmax output activation, we
show that the CW-L2 attack increases the ratio of the classification
probability between the target and ground truth classes. Moreover, we provide
numerical results to verify all our theoretical results.


http://arxiv.org/abs/1811.06418
Adversarial Examples from Cryptographic Pseudo-Random Generators.
Sébastien Bubeck; Yin Tat Lee; Eric Price; Ilya Razenshteyn
  In our recent work (Bubeck, Price, Razenshteyn, arXiv:1805.10204) we argued
that adversarial examples in machine learning might be due to an inherent
computational hardness of the problem. More precisely, we constructed a binary
classification task for which (i) a robust classifier exists; yet no
non-trivial accuracy can be obtained with an efficient algorithm in (ii) the
statistical query model. In the present paper we significantly strengthen both
(i) and (ii): we now construct a task which admits (i') a maximally robust
classifier (that is it can tolerate perturbations of size comparable to the
size of the examples themselves); and moreover we prove computational hardness
of learning this task under (ii') a standard cryptographic assumption.


http://arxiv.org/abs/1811.06029
Verification of Recurrent Neural Networks Through Rule Extraction.
Qinglong Wang; Kaixuan Zhang; Xue Liu; C. Lee Giles
  The verification problem for neural networks is verifying whether a neural
network will suffer from adversarial samples, or approximating the maximal
allowed scale of adversarial perturbation that can be endured. While most prior
work contributes to verifying feed-forward networks, little has been explored
for verifying recurrent networks. This is due to the existence of a more
rigorous constraint on the perturbation space for sequential data, and the lack
of a proper metric for measuring the perturbation. In this work, we address
these challenges by proposing a metric which measures the distance between
strings, and use deterministic finite automata (DFA) to represent a rigorous
oracle which examines if the generated adversarial samples violate certain
constraints on a perturbation. More specifically, we empirically show that
certain recurrent networks allow relatively stable DFA extraction. As such,
DFAs extracted from these recurrent networks can serve as a surrogate oracle
for when the ground truth DFA is unknown. We apply our verification mechanism
to several widely used recurrent networks on a set of the Tomita grammars. The
results demonstrate that only a few models remain robust against adversarial
samples. In addition, we show that for grammars with different levels of
complexity, there is also a difference in the difficulty of robust learning of
these grammars.


http://arxiv.org/abs/1811.05808
Robustness of spectral methods for community detection.
Ludovic Stephan; Laurent Massoulié
  The present work is concerned with community detection. Specifically, we
consider a random graph drawn according to the stochastic block model~: its
vertex set is partitioned into blocks, or communities, and edges are placed
randomly and independently of each other with probability depending only on the
communities of their two endpoints. In this context, our aim is to recover the
community labels better than by random guess, based only on the observation of
the graph.
  In the sparse case, where edge probabilities are in $O(1/n)$, we introduce a
new spectral method based on the distance matrix $D^{(l)}$, where $D^{(l)}_{ij}
= 1$ iff the graph distance between $i$ and $j$, noted $d(i, j)$ is equal to
$\ell$. We show that when $\ell \sim c\log(n)$ for carefully chosen $c$, the
eigenvectors associated to the largest eigenvalues of $D^{(l)}$ provide enough
information to perform non-trivial community recovery with high probability,
provided we are above the so-called Kesten-Stigum threshold. This yields an
efficient algorithm for community detection, since computation of the matrix
$D^{(l)}$ can be done in $O(n^{1+\kappa})$ operations for a small constant
$\kappa$.
  We then study the sensitivity of the eigendecomposition of $D^{(l)}$ when we
allow an adversarial perturbation of the edges of $G$. We show that when the
considered perturbation does not affect more than $O(n^\varepsilon)$ vertices
for some small $\varepsilon > 0$, the highest eigenvalues and their
corresponding eigenvectors incur negligible perturbations, which allows us to
still perform efficient recovery.


http://arxiv.org/abs/1811.05521
Deep Q learning for fooling neural networks.
Mandar Kulkarni
  Deep learning models are vulnerable to external attacks. In this paper, we
propose a Reinforcement Learning (RL) based approach to generate adversarial
examples for the pre-trained (target) models. We assume a semi black-box
setting where the only access an adversary has to the target model is the class
probabilities obtained for the input queries. We train a Deep Q Network (DQN)
agent which, with experience, learns to attack only a small portion of image
pixels to generate non-targeted adversarial images. Initially, an agent
explores an environment by sequentially modifying random sets of image pixels
and observes its effect on the class probabilities. At the end of an episode,
it receives a positive (negative) reward if it succeeds (fails) to alter the
label of the image. Experimental results with MNIST, CIFAR-10 and Imagenet
datasets demonstrate that our RL framework is able to learn an effective attack
policy.


http://arxiv.org/abs/1811.03733
Universal Decision-Based Black-Box Perturbations: Breaking Security-Through-Obscurity Defenses.
Thomas A. Hogan; Bhavya Kailkhura
  We study the problem of finding a universal (image-agnostic) perturbation to
fool machine learning (ML) classifiers (e.g., neural nets, decision tress) in
the hard-label black-box setting. Recent work in adversarial ML in the
white-box setting (model parameters are known) has shown that many
state-of-the-art image classifiers are vulnerable to universal adversarial
perturbations: a fixed human-imperceptible perturbation that, when added to any
image, causes it to be misclassified with high probability Kurakin et al.
[2016], Szegedy et al. [2013], Chen et al. [2017a], Carlini and Wagner [2017].
This paper considers a more practical and challenging problem of finding such
universal perturbations in an obscure (or black-box) setting. More
specifically, we use zeroth order optimization algorithms to find such a
universal adversarial perturbation when no model information is revealed-except
that the attacker can make queries to probe the classifier. We further relax
the assumption that the output of a query is continuous valued confidence
scores for all the classes and consider the case where the output is a
hard-label decision. Surprisingly, we found that even in these extremely
obscure regimes, state-of-the-art ML classifiers can be fooled with a very high
probability just by adding a single human-imperceptible image perturbation to
any natural image. The surprising existence of universal perturbations in a
hard-label black-box setting raises serious security concerns with the
existence of a universal noise vector that adversaries can possibly exploit to
break a classifier on most natural images.


http://arxiv.org/abs/1811.03685
New CleverHans Feature: Better Adversarial Robustness Evaluations with Attack Bundling.
Ian Goodfellow
  This technical report describes a new feature of the CleverHans library
called "attack bundling". Many papers about adversarial examples present lists
of error rates corresponding to different attack algorithms. A common approach
is to take the maximum across this list and compare defenses against that error
rate. We argue that a better approach is to use attack bundling: the max should
be taken across many examples at the level of individual examples, then the
error rate should be calculated by averaging after this maximization operation.
Reporting the bundled attacker error rate provides a lower bound on the true
worst-case error rate. The traditional approach of reporting the maximum error
rate across attacks can underestimate the true worst-case error rate by an
amount approaching 100\% as the number of attacks approaches infinity. Attack
bundling can be used with different prioritization schemes to optimize
quantities such as error rate on adversarial examples, perturbation size needed
to cause misclassification, or failure rate when using a specific confidence
threshold.


http://arxiv.org/abs/1811.03531
A Geometric Perspective on the Transferability of Adversarial Directions.
Zachary Charles; Harrison Rosenberg; Dimitris Papailiopoulos
  State-of-the-art machine learning models frequently misclassify inputs that
have been perturbed in an adversarial manner. Adversarial perturbations
generated for a given input and a specific classifier often seem to be
effective on other inputs and even different classifiers. In other words,
adversarial perturbations seem to transfer between different inputs, models,
and even different neural network architectures. In this work, we show that in
the context of linear classifiers and two-layer ReLU networks, there provably
exist directions that give rise to adversarial perturbations for many
classifiers and data points simultaneously. We show that these "transferable
adversarial directions" are guaranteed to exist for linear separators of a
given set, and will exist with high probability for linear classifiers trained
on independent sets drawn from the same distribution. We extend our results to
large classes of two-layer ReLU networks. We further show that adversarial
directions for ReLU networks transfer to linear classifiers while the reverse
need not hold, suggesting that adversarial perturbations for more complex
models are more likely to transfer to other classifiers. We validate our
findings empirically, even for deeper ReLU networks.


http://arxiv.org/abs/1811.03456
CAAD 2018: Iterative Ensemble Adversarial Attack.
Jiayang Liu; Weiming Zhang; Nenghai Yu
  Deep Neural Networks (DNNs) have recently led to significant improvements in
many fields. However, DNNs are vulnerable to adversarial examples which are
samples with imperceptible perturbations while dramatically misleading the
DNNs. Adversarial attacks can be used to evaluate the robustness of deep
learning models before they are deployed. Unfortunately, most of existing
adversarial attacks can only fool a black-box model with a low success rate. To
improve the success rates for black-box adversarial attacks, we proposed an
iterated adversarial attack against an ensemble of image classifiers. With this
method, we won the 5th place in CAAD 2018 Targeted Adversarial Attack
competition.


http://arxiv.org/abs/1811.03194
Ad-versarial: Perceptual Ad-Blocking meets Adversarial Machine Learning.
Florian Tramèr; Pascal Dupré; Gili Rusak; Giancarlo Pellegrino; Dan Boneh
  Perceptual ad-blocking is a novel approach that detects online advertisements
using visual cues. Compared to traditional filter lists, perceptual ad-blocking
is believed to be less prone to an arms race with web publishers and ad
networks. In this work, we use techniques from adversarial machine learning to
demonstrate that this may not be the case. We show that perceptual ad-blocking
engenders a new arms race that likely disfavors ad-blockers. Unexpectedly,
perceptual ad-blocking can also introduce new vulnerabilities that let an
attacker bypass web security boundaries and mount DDoS attacks. We first
analyze the design space of perceptual ad-blockers and present a unified
architecture that incorporates prior academic and commercial work. We then
explore a variety of attacks on the ad-blocker's visual detection pipeline,
that enable publishers or ad networks to evade or detect ad-blocking, and at
times even abuse its high privilege level to bypass web security boundaries.
Our attacks exploit the unreasonably strong threat model that perceptual
ad-blockers must resist. Finally, we present a concrete set of attacks on an
ad-blocker's internal ad-classifier by constructing adversarial examples in a
real Web page context. For six ad-detection techniques, we create perturbed
ads, perturbed ad-disclosure logos, and native web content that misleads
perceptual ad-blocking with 100% success rates. In one of our attacks, we
demonstrate how a malicious user can upload adversarial content, such as a
perturbed image in a Facebook post, that fools the ad-blocker into removing
another users' non-ad content. Moving beyond the Web and visual domain, we also
build adversarial examples for AdblockRadio, an open source radio client that
uses machine learning to detects ads in raw audio.


http://arxiv.org/abs/1811.02625
MixTrain: Scalable Training of Verifiably Robust Neural Networks.
Shiqi Wang; Yizheng Chen; Ahmed Abdou; Suman Jana
  Making neural networks robust against adversarial inputs has resulted in an
arms race between new defenses and attacks. The most promising defenses,
adversarially robust training and verifiably robust training, have limitations
that restrict their practical applications. The adversarially robust training
only makes the networks robust against a subclass of attackers and we reveal
such weaknesses by developing a new attack based on interval gradients. By
contrast, verifiably robust training provides protection against any L-p
norm-bounded attacker but incurs orders of magnitude more computational and
memory overhead than adversarially robust training.
  We propose two novel techniques, stochastic robust approximation and dynamic
mixed training, to drastically improve the efficiency of verifiably robust
training without sacrificing verified robustness. We leverage two critical
insights: (1) instead of over the entire training set, sound
over-approximations over randomly subsampled training data points are
sufficient for efficiently guiding the robust training process; and (2) We
observe that the test accuracy and verifiable robustness often conflict after
certain training epochs. Therefore, we use a dynamic loss function to
adaptively balance them for each epoch.
  We designed and implemented our techniques as part of MixTrain and evaluated
it on six networks trained on three popular datasets including MNIST, CIFAR,
and ImageNet-200. Our evaluations show that MixTrain can achieve up to $95.2\%$
verified robust accuracy against $L_\infty$ norm-bounded attackers while taking
$15$ and $3$ times less training time than state-of-the-art verifiably robust
training and adversarially robust training schemes, respectively. Furthermore,
MixTrain easily scales to larger networks like the one trained on ImageNet-200,
significantly outperforming the existing verifiably robust training methods.


http://arxiv.org/abs/1811.02248
SparseFool: a few pixels make a big difference.
Apostolos Modas; Seyed-Mohsen Moosavi-Dezfooli; Pascal Frossard
  Deep Neural Networks have achieved extraordinary results on image
classification tasks, but have been shown to be vulnerable to attacks with
carefully crafted perturbations of the input data. Although most attacks
usually change values of many image's pixels, it has been shown that deep
networks are also vulnerable to sparse alterations of the input. However, no
computationally efficient method has been proposed to compute sparse
perturbations. In this paper, we exploit the low mean curvature of the decision
boundary, and propose SparseFool, a geometry inspired sparse attack that
controls the sparsity of the perturbations. Extensive evaluations show that our
approach computes sparse perturbations very fast, and scales efficiently to
high dimensional data. We further analyze the transferability and the visual
effects of the perturbations, and show the existence of shared semantic
information across the images and the networks. Finally, we show that
adversarial training can only slightly improve the robustness against sparse
additive perturbations computed with SparseFool.


http://arxiv.org/abs/1811.01811
Active Deep Learning Attacks under Strict Rate Limitations for Online API Calls.
Yi Shi; Yalin E. Sagduyu; Kemal Davaslioglu; Jason H. Li
  Machine learning has been applied to a broad range of applications and some
of them are available online as application programming interfaces (APIs) with
either free (trial) or paid subscriptions. In this paper, we study adversarial
machine learning in the form of back-box attacks on online classifier APIs. We
start with a deep learning based exploratory (inference) attack, which aims to
build a classifier that can provide similar classification results (labels) as
the target classifier. To minimize the difference between the labels returned
by the inferred classifier and the target classifier, we show that the deep
learning based exploratory attack requires a large number of labeled training
data samples. These labels can be collected by calling the online API, but
usually there is some strict rate limitation on the number of allowed API
calls. To mitigate the impact of limited training data, we develop an active
learning approach that first builds a classifier based on a small number of API
calls and uses this classifier to select samples to further collect their
labels. Then, a new classifier is built using more training data samples. This
updating process can be repeated multiple times. We show that this active
learning approach can build an adversarial classifier with a small statistical
difference from the target classifier using only a limited number of training
data samples. We further consider evasion and causative (poisoning) attacks
based on the inferred classifier that is built by the exploratory attack.
Evasion attack determines samples that the target classifier is likely to
misclassify, whereas causative attack provides erroneous training data samples
to reduce the reliability of the re-trained classifier. The success of these
attacks show that adversarial machine learning emerges as a feasible threat in
the realistic case with limited training data.


http://arxiv.org/abs/1811.01749
FUNN: Flexible Unsupervised Neural Network.
David Vigouroux; Sylvain Picard
  Deep neural networks have demonstrated high accuracy in image classification
tasks. However, they were shown to be weak against adversarial examples: a
small perturbation in the image which changes the classification output
dramatically. In recent years, several defenses have been proposed to solve
this issue in supervised classification tasks. We propose a method to obtain
robust features in unsupervised learning tasks against adversarial attacks. Our
method differs from existing solutions by directly learning the robust features
without the need to project the adversarial examples in the original examples
distribution space. A first auto-encoder A1 is in charge of perturbing the
input image to fool another auto-encoder A2 which is in charge of regenerating
the original image. A1 tries to find the less perturbed image under the
constraint that the error in the output of A2 should be at least equal to a
threshold. Thanks to this training, the encoder of A2 will be robust against
adversarial attacks and could be used in different tasks like classification.
Using state-of-art network architectures, we demonstrate the robustness of the
features obtained thanks to this method in classification tasks.


http://arxiv.org/abs/1811.01629
On the Transferability of Adversarial Examples Against CNN-Based Image Forensics.
Mauro Barni; Kassem Kallas; Ehsan Nowroozi; Benedetta Tondi
  Recent studies have shown that Convolutional Neural Networks (CNN) are
relatively easy to attack through the generation of so-called adversarial
examples. Such vulnerability also affects CNN-based image forensic tools.
Research in deep learning has shown that adversarial examples exhibit a certain
degree of transferability, i.e., they maintain part of their effectiveness even
against CNN models other than the one targeted by the attack. This is a very
strong property undermining the usability of CNN's in security-oriented
applications. In this paper, we investigate if attack transferability also
holds in image forensics applications. With specific reference to the case of
manipulation detection, we analyse the results of several experiments
considering different sources of mismatch between the CNN used to build the
adversarial examples and the one adopted by the forensic analyst. The analysis
ranges from cases in which the mismatch involves only the training dataset, to
cases in which the attacker and the forensic analyst adopt different
architectures. The results of our experiments show that, in the majority of the
cases, the attacks are not transferable, thus easing the design of proper
countermeasures at least when the attacker does not have a perfect knowledge of
the target detector.


http://arxiv.org/abs/1811.01444
FAdeML: Understanding the Impact of Pre-Processing Noise Filtering on Adversarial Machine Learning.
Faiq Khalid; Muhammmad Abdullah Hanif; Semeen Rehman; Junaid Qadir; Muhammad Shafique
  Deep neural networks (DNN)-based machine learning (ML) algorithms have
recently emerged as the leading ML paradigm particularly for the task of
classification due to their superior capability of learning efficiently from
large datasets. The discovery of a number of well-known attacks such as dataset
poisoning, adversarial examples, and network manipulation (through the addition
of malicious nodes) has, however, put the spotlight squarely on the lack of
security in DNN-based ML systems. In particular, malicious actors can use these
well-known attacks to cause random/targeted misclassification, or cause a
change in the prediction confidence, by only slightly but systematically
manipulating the environmental parameters, inference data, or the data
acquisition block. Most of the prior adversarial attacks have, however, not
accounted for the pre-processing noise filters commonly integrated with the
ML-inference module. Our contribution in this work is to show that this is a
major omission since these noise filters can render ineffective the majority of
the existing attacks, which rely essentially on introducing adversarial noise.
Apart from this, we also extend the state of the art by proposing a novel
pre-processing noise Filter-aware Adversarial ML attack called FAdeML. To
demonstrate the effectiveness of the proposed methodology, we generate an
adversarial attack image by exploiting the "VGGNet" DNN trained for the "German
Traffic Sign Recognition Benchmarks (GTSRB" dataset, which despite having no
visual noise, can cause a classifier to misclassify even in the presence of
pre-processing noise filters.


http://arxiv.org/abs/1811.01443
SSCNets: A Selective Sobel Convolution-based Technique to Enhance the Robustness of Deep Neural Networks against Security Attacks.
Hammad Tariq; Hassan Ali; Muhammad Abdullah Hanif; Faiq Khalid; Semeen Rehman; Rehan Ahmed; Muhammad Shafique
  Recent studies have shown that slight perturbations in the input data can
significantly affect the robustness of Deep Neural Networks (DNNs), leading to
misclassification and confidence reduction. In this paper, we introduce a novel
technique based on the Selective Sobel Convolution (SSC) operation in the
training loop, that increases the robustness of a given DNN by allowing it to
learn important edges in the input in a controlled fashion. This is achieved by
introducing a trainable parameter, which acts as a threshold for eliminating
the weaker edges. We validate our technique against the attacks of Cleverhans
library on Convolutional DNNs against adversarial attacks. Our experimental
results on the MNIST and CIFAR10 datasets illustrate that this controlled
learning considerably increases the accuracy of the DNNs by 1.53% even when
subjected to adversarial attacks.


http://arxiv.org/abs/1811.01437
QuSecNets: Quantization-based Defense Mechanism for Securing Deep Neural Network against Adversarial Attacks.
Hassan Ali; Hammad Tariq; Muhammad Abdullah Hanif; Faiq Khalid; Semeen Rehman; Rehan Ahmed; Muhammad Shafique
  Deep Neural Networks (DNNs) have recently been shown vulnerable to
adversarial attacks in which the input examples are perturbed to fool these
DNNs towards confidence reduction and (targeted or random) misclassification.
In this paper, we demonstrate that how an efficient quantization technique can
be leveraged to increase the robustness of a given DNN against adversarial
attacks. We present two quantization-based defense mechanisms, namely Constant
Quantization (CQ) and Variable Quantization (VQ), applied at the input to
increase the robustness of DNNs. In CQ, the intensity of the input pixel is
quantized according to the number of quantization levels. While in VQ, the
quantization levels are recursively updated during the training phase, thereby
providing a stronger defense mechanism. We apply our techniques on the
Convolutional Neural Networks (CNNs, a particular type of DNN which is heavily
used in vision-based applications) against adversarial attacks from the
open-source Cleverhans library. Our experimental results show 1%-5% increase in
the adversarial accuracy for MNIST and 0%-2.4% increase in the adversarial
accuracy for CIFAR10.


http://arxiv.org/abs/1811.01302
Adversarial Gain.
Peter Henderson; Koustuv Sinha; Rosemary Nan Ke; Joelle Pineau
  Adversarial examples can be defined as inputs to a model which induce a
mistake - where the model output is different than that of an oracle, perhaps
in surprising or malicious ways. Original models of adversarial attacks are
primarily studied in the context of classification and computer vision tasks.
While several attacks have been proposed in natural language processing (NLP)
settings, they often vary in defining the parameters of an attack and what a
successful attack would look like. The goal of this work is to propose a
unifying model of adversarial examples suitable for NLP tasks in both
generative and classification settings. We define the notion of adversarial
gain: based in control theory, it is a measure of the change in the output of a
system relative to the perturbation of the input (caused by the so-called
adversary) presented to the learner. This definition, as we show, can be used
under different feature spaces and distance conditions to determine attack or
defense effectiveness across different intuitive manifolds. This notion of
adversarial gain not only provides a useful way for evaluating adversaries and
defenses, but can act as a building block for future work in robustness under
adversaries due to its rooted nature in stability and manifold theory.


http://arxiv.org/abs/1811.01225
CAAD 2018: Powerful None-Access Black-Box Attack Based on Adversarial Transformation Network.
Xiaoyi Dong; Weiming Zhang; Nenghai Yu
  In this paper, we propose an improvement of Adversarial Transformation
Networks(ATN) to generate adversarial examples, which can fool white-box models
and black-box models with a state of the art performance and won the 2rd place
in the non-target task in CAAD 2018.


http://arxiv.org/abs/1811.01213
Learning to Defense by Learning to Attack.
Zhehui Chen; Haoming Jiang; Bo Dai; Tuo Zhao
  Adversarial training provides a principled approach for training robust
neural networks. From an optimization perspective, the adversarial training is
essentially solving a minmax robust optimization problem. The outer
minimization is trying to learn a robust classifier, while the inner
maximization is trying to generate adversarial samples. Unfortunately, such a
minmax problem is very difficult to solve due to the lack of convex-concave
structure. This work proposes a new adversarial training method based on a
general learning-to-learn framework. Specifically, instead of applying the
existing hand-design algorithms for the inner problem, we learn an optimizer,
which is parametrized as a convolutional neural network. At the same time, a
robust classifier is learned to defense the adversarial attack generated by the
learned optimizer. Our experiments demonstrate that our proposed method
significantly outperforms existing adversarial training methods on CIFAR-10 and
CIFAR-100 datasets.


http://arxiv.org/abs/1811.01312
Adversarial Black-Box Attacks on Automatic Speech Recognition Systems using Multi-Objective Evolutionary Optimization.
Shreya Khare; Rahul Aralikatte; Senthil Mani
  Fooling deep neural networks with adversarial input have exposed a
significant vulnerability in the current state-of-the-art systems in multiple
domains. Both black-box and white-box approaches have been used to either
replicate the model itself or to craft examples which cause the model to fail.
In this work, we propose a framework which uses multi-objective evolutionary
optimization to perform both targeted and un-targeted black-box attacks on
Automatic Speech Recognition (ASR) systems. We apply this framework on two ASR
systems: Deepspeech and Kaldi-ASR, which increases the Word Error Rates (WER)
of these systems by upto 980%, indicating the potency of our approach. During
both un-targeted and targeted attacks, the adversarial samples maintain a high
acoustic similarity of 0.98 and 0.97 with the original audio.


http://arxiv.org/abs/1811.01134
A Marauder's Map of Security and Privacy in Machine Learning.
Nicolas Papernot
  There is growing recognition that machine learning (ML) exposes new security
and privacy vulnerabilities in software systems, yet the technical community's
understanding of the nature and extent of these vulnerabilities remains limited
but expanding. In this talk, we explore the threat model space of ML algorithms
through the lens of Saltzer and Schroeder's principles for the design of secure
computer systems. This characterization of the threat space prompts an
investigation of current and future research directions. We structure our
discussion around three of these directions, which we believe are likely to
lead to significant progress. The first encompasses a spectrum of approaches to
verification and admission control, which is a prerequisite to enable fail-safe
defaults in machine learning systems. The second seeks to design mechanisms for
assembling reliable records of compromise that would help understand the degree
to which vulnerabilities are exploited by adversaries, as well as favor
psychological acceptability of machine learning applications. The third pursues
formal frameworks for security and privacy in machine learning, which we argue
should strive to align machine learning goals such as generalization with
security and privacy desiderata like robustness or privacy. Key insights
resulting from these three directions pursued both in the ML and security
communities are identified and the effectiveness of approaches are related to
structural elements of ML algorithms and the data used to train them. We
conclude by systematizing best practices in our community.


http://arxiv.org/abs/1811.01057
Semidefinite relaxations for certifying robustness to adversarial examples.
Aditi Raghunathan; Jacob Steinhardt; Percy Liang
  Despite their impressive performance on diverse tasks, neural networks fail
catastrophically in the presence of adversarial inputs---imperceptibly but
adversarially perturbed versions of natural inputs. We have witnessed an arms
race between defenders who attempt to train robust networks and attackers who
try to construct adversarial examples. One promise of ending the arms race is
developing certified defenses, ones which are provably robust against all
attackers in some family. These certified defenses are based on convex
relaxations which construct an upper bound on the worst case loss over all
attackers in the family. Previous relaxations are loose on networks that are
not trained against the respective relaxation. In this paper, we propose a new
semidefinite relaxation for certifying robustness that applies to arbitrary
ReLU networks. We show that our proposed relaxation is tighter than previous
relaxations and produces meaningful robustness guarantees on three different
"foreign networks" whose training objectives are agnostic to our proposed
relaxation.


http://arxiv.org/abs/1811.01031
TrISec: Training Data-Unaware Imperceptible Security Attacks on Deep Neural Networks.
Faiq Khalid; Muhammad Abdullah Hanif; Semeen Rehman; Rehan Ahmed; Muhammad Shafique
  Most of the data manipulation attacks on deep neural networks (DNNs) during
the training stage introduce a perceptible noise that can be catered by
preprocessing during inference. Therefore, data poisoning attacks during
inference (e.g., adversarial attacks) are becoming more popular. However, they
do not consider the imperceptibility factor in their optimization algorithms,
and can be detected by correlation and structural testing. Therefore, in this
paper, we propose a novel methodology which automatically generates
imperceptible attack images by using the back-propagation algorithm on
pre-trained DNNs. We present a case study on traffic sign detection using the
VGGNet and the German Traffic Sign Recognition Benchmarks dataset in an
autonomous driving use case. Our results demonstrate that the generated attack
images successfully perform misclassification while remaining imperceptible in
both `subjective' and `objective' quality tests


http://arxiv.org/abs/1811.00866
Efficient Neural Network Robustness Certification with General Activation Functions.
Huan Zhang; Tsui-Wei Weng; Pin-Yu Chen; Cho-Jui Hsieh; Luca Daniel
  Finding minimum distortion of adversarial examples and thus certifying
robustness in neural network classifiers for given data points is known to be a
challenging problem. Nevertheless, recently it has been shown to be possible to
give a non-trivial certified lower bound of minimum adversarial distortion, and
some recent progress has been made towards this direction by exploiting the
piece-wise linear nature of ReLU activations. However, a generic robustness
certification for general activation functions still remains largely
unexplored. To address this issue, in this paper we introduce CROWN, a general
framework to certify robustness of neural networks with general activation
functions for given input data points. The novelty in our algorithm consists of
bounding a given activation function with linear and quadratic functions, hence
allowing it to tackle general activation functions including but not limited to
four popular choices: ReLU, tanh, sigmoid and arctan. In addition, we
facilitate the search for a tighter certified lower bound by adaptively
selecting appropriate surrogates for each neuron activation. Experimental
results show that CROWN on ReLU networks can notably improve the certified
lower bounds compared to the current state-of-the-art algorithm Fast-Lin, while
having comparable computational efficiency. Furthermore, CROWN also
demonstrates its effectiveness and flexibility on networks with general
activation functions, including tanh, sigmoid and arctan.


http://arxiv.org/abs/1811.00830
Towards Adversarial Malware Detection: Lessons Learned from PDF-based Attacks.
Davide Maiorca; Battista Biggio; Giorgio Giacinto
  Malware still constitutes a major threat in the cybersecurity landscape, also
due to the widespread use of infection vectors such as documents. These
infection vectors hide embedded malicious code to the victim users,
facilitating the use of social engineering techniques to infect their machines.
Research showed that machine-learning algorithms provide effective detection
mechanisms against such threats, but the existence of an arms race in
adversarial settings has recently challenged such systems. In this work, we
focus on malware embedded in PDF files as a representative case of such an arms
race. We start by providing a comprehensive taxonomy of the different
approaches used to generate PDF malware, and of the corresponding
learning-based detection systems. We then categorize threats specifically
targeted against learning-based PDF malware detectors, using a well-established
framework in the field of adversarial machine learning. This framework allows
us to categorize known vulnerabilities of learning-based PDF malware detectors
and to identify novel attacks that may threaten such systems, along with the
potential defense mechanisms that can mitigate the impact of such threats. We
conclude the paper by discussing how such findings highlight promising research
directions towards tackling the more general challenge of designing robust
malware detectors in adversarial settings.


http://arxiv.org/abs/1811.00621
Improving Adversarial Robustness by Encouraging Discriminative Features.
Chirag Agarwal; Anh Nguyen; Dan Schonfeld
  Deep neural networks (DNNs) have achieved state-of-the-art results in various
pattern recognition tasks. However, they perform poorly on out-of-distribution
adversarial examples i.e. inputs that are specifically crafted by an adversary
to cause DNNs to misbehave, questioning the security and reliability of
applications. In this paper, we encourage DNN classifiers to learn more
discriminative features by imposing a center loss in addition to the regular
softmax cross-entropy loss. Intuitively, the center loss encourages DNNs to
simultaneously learns a center for the deep features of each class, and
minimize the distances between the intra-class deep features and their
corresponding class centers. We hypothesize that minimizing distances between
intra-class features and maximizing the distances between inter-class features
at the same time would improve a classifier's robustness to adversarial
examples. Our results on state-of-the-art architectures on MNIST, CIFAR-10, and
CIFAR-100 confirmed that intuition and highlight the importance of
discriminative features.


http://arxiv.org/abs/1811.00525
On the Geometry of Adversarial Examples.
Marc Khoury; Dylan Hadfield-Menell
  Adversarial examples are a pervasive phenomenon of machine learning models
where seemingly imperceptible perturbations to the input lead to
misclassifications for otherwise statistically accurate models. We propose a
geometric framework, drawing on tools from the manifold reconstruction
literature, to analyze the high-dimensional geometry of adversarial examples.
In particular, we highlight the importance of codimension: for low-dimensional
data manifolds embedded in high-dimensional space there are many directions off
the manifold in which to construct adversarial examples. Adversarial examples
are a natural consequence of learning a decision boundary that classifies the
low-dimensional data manifold well, but classifies points near the manifold
incorrectly. Using our geometric framework we prove (1) a tradeoff between
robustness under different norms, (2) that adversarial training in balls around
the data is sample inefficient, and (3) sufficient sampling conditions under
which nearest neighbor classifiers and ball-based adversarial training are
robust.


http://arxiv.org/abs/1811.00401
Excessive Invariance Causes Adversarial Vulnerability.
Jörn-Henrik Jacobsen; Jens Behrmann; Richard Zemel; Matthias Bethge
  Despite their impressive performance, deep neural networks exhibit striking
failures on out-of-distribution inputs. One core idea of adversarial example
research is to reveal neural network errors under such distribution shifts. We
decompose these errors into two complementary sources: sensitivity and
invariance. We show deep networks are not only too sensitive to task-irrelevant
changes of their input, as is well-known from epsilon-adversarial examples, but
are also too invariant to a wide range of task-relevant changes, thus making
vast regions in input space vulnerable to adversarial attacks. We show such
excessive invariance occurs across various tasks and architecture types. On
MNIST and ImageNet one can manipulate the class-specific content of almost any
image without changing the hidden activations. We identify an insufficiency of
the standard cross-entropy loss as a reason for these failures. Further, we
extend this objective based on an information-theoretic analysis so it
encourages the model to consider all task-dependent features in its decision.
This provides the first approach tailored explicitly to overcome excessive
invariance and resulting vulnerabilities.


http://arxiv.org/abs/1811.02658
When Not to Classify: Detection of Reverse Engineering Attacks on DNN Image Classifiers.
Yujia Wang; David J. Miller; George Kesidis
  This paper addresses detection of a reverse engineering (RE) attack targeting
a deep neural network (DNN) image classifier; by querying, RE's aim is to
discover the classifier's decision rule. RE can enable test-time evasion
attacks, which require knowledge of the classifier. Recently, we proposed a
quite effective approach (ADA) to detect test-time evasion attacks. In this
paper, we extend ADA to detect RE attacks (ADA-RE). We demonstrate our method
is successful in detecting "stealthy" RE attacks before they learn enough to
launch effective test-time evasion attacks.


http://arxiv.org/abs/1811.00189
Reversible Adversarial Examples.
Jiayang Liu; Dongdong Hou; Weiming Zhang; Nenghai Yu
  Deep Neural Networks have recently led to significant improvement in many
fields such as image classification and speech recognition. However, these
machine learning models are vulnerable to adversarial examples which can
mislead machine learning classifiers to give incorrect classifications. In this
paper, we take advantage of reversible data hiding to construct reversible
adversarial examples which are still misclassified by Deep Neural Networks.
Furthermore, the proposed method can recover original images from reversible
adversarial examples with no distortion.


http://arxiv.org/abs/1810.12715
On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models.
Sven Gowal; Krishnamurthy Dvijotham; Robert Stanforth; Rudy Bunel; Chongli Qin; Jonathan Uesato; Relja Arandjelovic; Timothy Mann; Pushmeet Kohli
  Recent work has shown that it is possible to train deep neural networks that
are verifiably robust to norm-bounded adversarial perturbations. Most of these
methods are based on minimizing an upper bound on the worst-case loss over all
possible adversarial perturbations. While these techniques show promise, they
remain hard to scale to larger networks. Through a comprehensive analysis, we
show how a careful implementation of a simple bounding technique, interval
bound propagation (IBP), can be exploited to train verifiably robust neural
networks that beat the state-of-the-art in verified accuracy. While the upper
bound computed by IBP can be quite weak for general networks, we demonstrate
that an appropriate loss and choice of hyper-parameters allows the network to
adapt such that the IBP bound is tight. This results in a fast and stable
learning algorithm that outperforms more sophisticated methods and achieves
state-of-the-art results on MNIST, CIFAR-10 and SVHN. It also allows us to
obtain the first verifiably robust model on a downscaled version of ImageNet.


http://arxiv.org/abs/1810.12576
Improved Network Robustness with Adversary Critic.
Alexander Matyasko; Lap-Pui Chau
  Ideally, what confuses neural network should be confusing to humans. However,
recent experiments have shown that small, imperceptible perturbations can
change the network prediction. To address this gap in perception, we propose a
novel approach for learning robust classifier. Our main idea is: adversarial
examples for the robust classifier should be indistinguishable from the regular
data of the adversarial target. We formulate a problem of learning robust
classifier in the framework of Generative Adversarial Networks (GAN), where the
adversarial attack on classifier acts as a generator, and the critic network
learns to distinguish between regular and adversarial images. The classifier
cost is augmented with the objective that its adversarial examples should
confuse the adversary critic. To improve the stability of the adversarial
mapping, we introduce adversarial cycle-consistency constraint which ensures
that the adversarial mapping of the adversarial examples is close to the
original. In the experiments, we show the effectiveness of our defense. Our
method surpasses in terms of robustness networks trained with adversarial
training. Additionally, we verify in the experiments with human annotators on
MTurk that adversarial examples are indeed visually confusing. Codes for the
project are available at https://github.com/aam-at/adversary_critic.


http://arxiv.org/abs/1810.12272
Adversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution.
Dimitrios I. Diochnos; Saeed Mahloujifar; Mohammad Mahmoody
  We study adversarial perturbations when the instances are uniformly
distributed over $\{0,1\}^n$. We study both "inherent" bounds that apply to any
problem and any classifier for such a problem as well as bounds that apply to
specific problems and specific hypothesis classes.
  As the current literature contains multiple definitions of adversarial risk
and robustness, we start by giving a taxonomy for these definitions based on
their goals, we identify one of them as the one guaranteeing misclassification
by pushing the instances to the error region. We then study some classic
algorithms for learning monotone conjunctions and compare their adversarial
risk and robustness under different definitions by attacking the hypotheses
using instances drawn from the uniform distribution. We observe that sometimes
these definitions lead to significantly different bounds. Thus, this study
advocates for the use of the error-region definition, even though other
definitions, in other contexts, may coincide with the error-region definition.
  Using the error-region definition of adversarial perturbations, we then study
inherent bounds on risk and robustness of any classifier for any classification
problem whose instances are uniformly distributed over $\{0,1\}^n$. Using the
isoperimetric inequality for the Boolean hypercube, we show that for initial
error $0.01$, there always exists an adversarial perturbation that changes
$O(\sqrt{n})$ bits of the instances to increase the risk to $0.5$, making
classifier's decisions meaningless. Furthermore, by also using the central
limit theorem we show that when $n\to \infty$, at most $c \cdot \sqrt{n}$ bits
of perturbations, for a universal constant $c< 1.17$, suffice for increasing
the risk to $0.5$, and the same $c \cdot \sqrt{n} $ bits of perturbations on
average suffice to increase the risk to $1$, hence bounding the robustness by
$c \cdot \sqrt{n}$.


http://arxiv.org/abs/1810.12042
Logit Pairing Methods Can Fool Gradient-Based Attacks.
Marius Mosbach; Maksym Andriushchenko; Thomas Trost; Matthias Hein; Dietrich Klakow
  Recently, Kannan et al. [2018] proposed several logit regularization methods
to improve the adversarial robustness of classifiers. We show that the
computationally fast methods they propose - Clean Logit Pairing (CLP) and Logit
Squeezing (LSQ) - just make the gradient-based optimization problem of crafting
adversarial examples harder without providing actual robustness. We find that
Adversarial Logit Pairing (ALP) may indeed provide robustness against
adversarial examples, especially when combined with adversarial training, and
we examine it in a variety of settings. However, the increase in adversarial
accuracy is much smaller than previously claimed. Finally, our results suggest
that the evaluation against an iterative PGD attack relies heavily on the
parameters used and may result in false conclusions regarding robustness of a
model.


http://arxiv.org/abs/1810.11914
Rademacher Complexity for Adversarially Robust Generalization.
Dong Yin; Kannan Ramchandran; Peter Bartlett
  Many machine learning models are vulnerable to adversarial attacks; for
example, adding adversarial perturbations that are imperceptible to humans can
often make machine learning models produce wrong predictions with high
confidence. Moreover, although we may obtain robust models on the training
dataset via adversarial training, in some problems the learned models cannot
generalize well to the test data. In this paper, we focus on $\ell_\infty$
attacks, and study the adversarially robust generalization problem through the
lens of Rademacher complexity. For binary linear classifiers, we prove tight
bounds for the adversarial Rademacher complexity, and show that the adversarial
Rademacher complexity is never smaller than its natural counterpart, and it has
an unavoidable dimension dependence, unless the weight vector has bounded
$\ell_1$ norm. The results also extend to multi-class linear classifiers. For
(nonlinear) neural networks, we show that the dimension dependence in the
adversarial Rademacher complexity also exists. We further consider a surrogate
adversarial loss for one-hidden layer ReLU network and prove margin bounds for
this setting. Our results indicate that having $\ell_1$ norm constraints on the
weight matrices might be a potential way to improve generalization in the
adversarial setting. We demonstrate experimental results that validate our
theoretical findings.


http://arxiv.org/abs/1810.11793
Robust Audio Adversarial Example for a Physical Attack.
Hiromu Yakura; Jun Sakuma
  We propose a method to generate audio adversarial examples that can attack a
state-of-the-art speech recognition model in the physical world. Previous work
assumes that generated adversarial examples are directly fed to the recognition
model, and is not able to perform such a physical attack because of
reverberation and noise from playback environments. In contrast, our method
obtains robust adversarial examples by simulating transformations caused by
playback or recording in the physical world and incorporating the
transformations into the generation process. Evaluation and a listening
experiment demonstrated that our adversarial examples are able to attack
without being noticed by humans. This result suggests that audio adversarial
examples generated by the proposed method may become a real threat.


http://arxiv.org/abs/1810.11783
RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications.
Huan Zhang; Pengchuan Zhang; Cho-Jui Hsieh
  The Jacobian matrix (or the gradient for single-output networks) is directly
related to many important properties of neural networks, such as the function
landscape, stationary points, (local) Lipschitz constants and robustness to
adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac,
to compute both upper and lower bounds for each element in the Jacobian matrix
of a neural network with respect to network's input, and the network can
contain a wide range of activation functions. As a byproduct, we can
efficiently obtain a (local) Lipschitz constant, which plays a crucial role in
neural network robustness verification, as well as the training stability of
GANs. Experiments show that (local) Lipschitz constants produced by our method
is of better quality than previous approaches, thus providing better robustness
verification results. Our algorithm has polynomial time complexity, and its
computation time is reasonable even for relatively large networks.
Additionally, we use our bounds of Jacobian matrix to characterize the
landscape of the neural network, for example, to determine whether there exist
stationary points in a local neighborhood. Source code available at
\url{http://github.com/huanzhang12/RecurJac-Jacobian-bounds}.


http://arxiv.org/abs/1810.11726
Towards Robust Deep Neural Networks.
Timothy E. Wang; Yiming Gu; Dhagash Mehta; Xiaojun Zhao; Edgar A. Bernal
  We investigate the topics of sensitivity and robustness in feedforward and
convolutional neural networks. Combining energy landscape techniques developed
in computational chemistry with tools drawn from formal methods, we produce
empirical evidence indicating that networks corresponding to lower-lying minima
in the optimization landscape of the learning objective tend to be more robust.
The robustness estimate used is the inverse of a proposed sensitivity measure,
which we define as the volume of an over-approximation of the reachable set of
network outputs under all additive $l_{\infty}$-bounded perturbations on the
input data. We present a novel loss function which includes a sensitivity term
in addition to the traditional task-oriented and regularization terms. In our
experiments on standard machine learning and computer vision datasets, we show
that the proposed loss function leads to networks which reliably optimize the
robustness measure as well as other related metrics of adversarial robustness
without significant degradation in the classification error. Experimental
results indicate that the proposed method outperforms state-of-the-art
sensitivity-based learning approaches with regards to robustness to adversarial
attacks. We also show that although the introduced framework does not
explicitly enforce an adversarial loss, it achieves competitive overall
performance relative to methods that do.


http://arxiv.org/abs/1810.11711
Regularization Effect of Fast Gradient Sign Method and its Generalization.
Chandler Zuo
  Fast Gradient Sign Method (FGSM) is a popular method to generate adversarial
examples that make neural network models robust against perturbations. Despite
its empirical success, its theoretical property is not well understood. This
paper develops theory to explain the regularization effect of Generalized FGSM,
a class of methods to generate adversarial examples. Motivated from the
relationship between FGSM and LASSO penalty, the asymptotic properties of
Generalized FGSM are derived in the Generalized Linear Model setting, which is
essentially the 1-layer neural network setting with certain activation
functions. In such simple neural network models, I prove that Generalized FGSM
estimation is root n-consistent and weakly oracle under proper conditions. The
asymptotic results are also highly similar to penalized likelihood estimation.
Nevertheless, Generalized FGSM introduces additional bias when data sampling is
not sign neutral, a concept I introduce to describe the balance-ness of the
noise signs. Although the theory in this paper is developed under simple neural
network settings, I argue that it may give insights and justification for FGSM
in deep neural network settings as well.


http://arxiv.org/abs/1810.11580
Attacks Meet Interpretability: Attribute-steered Detection of Adversarial Samples.
Guanhong Tao; Shiqing Ma; Yingqi Liu; Xiangyu Zhang
  Adversarial sample attacks perturb benign inputs to induce DNN misbehaviors.
Recent research has demonstrated the widespread presence and the devastating
consequences of such attacks. Existing defense techniques either assume prior
knowledge of specific attacks or may not work well on complex models due to
their underlying assumptions. We argue that adversarial sample attacks are
deeply entangled with interpretability of DNN models: while classification
results on benign inputs can be reasoned based on the human perceptible
features/attributes, results on adversarial samples can hardly be explained.
Therefore, we propose a novel adversarial sample detection technique for face
recognition models, based on interpretability. It features a novel
bi-directional correspondence inference between attributes and internal neurons
to identify neurons critical for individual attributes. The activation values
of critical neurons are enhanced to amplify the reasoning part of the
computation and the values of other neurons are weakened to suppress the
uninterpretable part. The classification results after such transformation are
compared with those of the original model to detect adversaries. Results show
that our technique can achieve 94% detection accuracy for 7 different kinds of
attacks with 9.91% false positives on benign inputs. In contrast, a
state-of-the-art feature squeezing technique can only achieve 55% accuracy with
23.3% false positives.


http://arxiv.org/abs/1810.10751
Attack Graph Convolutional Networks by Adding Fake Nodes.
Xiaoyun Wang; Joe Eaton; Cho-Jui Hsieh; Felix Wu
  Graph convolutional networks (GCNs) have been widely used for classifying
graph nodes in the semi-supervised setting. Previous work have shown that GCNs
are vulnerable to the perturbation on adjacency and feature matrices of
existing nodes. However, it is unrealistic to change existing nodes in many
applications, such as existing users in social networks. In this paper, we
design algorithms to attack GCNs by adding fake nodes. A greedy algorithm is
proposed to generate adjacency and feature matrices of fake nodes, aiming to
minimize the classification accuracy on the existing nodes. In additional, we
introduce a discriminator to classify fake nodes from real nodes, and propose a
Greedy-GAN attack to simultaneously update the discriminator and the attacker,
to make fake nodes indistinguishable to the real ones. Our non-targeted attack
decreases the accuracy of GCN down to 0.10, and our targeted attack reaches a
success rate of 99% on the whole datasets, and 94% on average for attacking a
single target node.


http://arxiv.org/abs/1810.10731
Law and Adversarial Machine Learning.
Ram Shankar Siva Kumar; David R. O'Brien; Kendra Albert; Salome Vilojen
  When machine learning systems fail because of adversarial manipulation, how
should society expect the law to respond? Through scenarios grounded in
adversarial ML literature, we explore how some aspects of computer crime,
copyright, and tort law interface with perturbation, poisoning, model stealing
and model inversion attacks to show how some attacks are more likely to result
in liability than others. We end with a call for action to ML researchers to
invest in transparent benchmarks of attacks and defenses; architect ML systems
with forensics in mind and finally, think more about adversarial machine
learning in the context of civil liberties. The paper is targeted towards ML
researchers who have no legal background.


http://arxiv.org/abs/1810.10939
Evading classifiers in discrete domains with provable optimality guarantees.
Bogdan Kulynych; Jamie Hayes; Nikita Samarin; Carmela Troncoso
  Machine-learning models for security-critical applications such as bot,
malware, or spam detection, operate in constrained discrete domains. These
applications would benefit from having provable guarantees against adversarial
examples. The existing literature on provable adversarial robustness of models,
however, exclusively focuses on robustness to gradient-based attacks in domains
such as images. These attacks model the adversarial cost, e.g., amount of
distortion applied to an image, as a $p$-norm. We argue that this approach is
not well-suited to model adversarial costs in constrained domains where not all
examples are feasible.
  We introduce a graphical framework that (1) generalizes existing attacks in
discrete domains, (2) can accommodate complex cost functions beyond $p$-norms,
including financial cost incurred when attacking a classifier, and (3)
efficiently produces valid adversarial examples with guarantees of minimal
adversarial cost. These guarantees directly translate into a notion of
adversarial robustness that takes into account domain constraints and the
adversary's capabilities. We show how our framework can be used to evaluate
security by crafting adversarial examples that evade a Twitter-bot detection
classifier with provably minimal number of changes; and to build privacy
defenses by crafting adversarial examples that evade a privacy-invasive
website-fingerprinting classifier.


http://arxiv.org/abs/1810.10625
Robust Adversarial Learning via Sparsifying Front Ends.
Soorya Gopalakrishnan; Zhinus Marzi; Upamanyu Madhow; Ramtin Pedarsani
  It is by now well-known that small adversarial perturbations can induce
classification errors in deep neural networks. In this paper, we take a
bottom-up signal processing perspective to this problem and show that a
systematic exploitation of sparsity in natural data is a promising tool for
defense. For linear classifiers, we show that a sparsifying front end is
provably effective against $\ell_{\infty}$-bounded attacks, reducing output
distortion due to the attack by a factor of roughly $K/N$ where $N$ is the data
dimension and $K$ is the sparsity level. We then extend this concept to deep
networks, showing that a "locally linear" model can be used to develop a
theoretical foundation for crafting attacks and defenses. We also devise
attacks based on the locally linear model that outperform the well-known FGSM
attack. We supplement our theoretical results with experiments on the MNIST
handwritten digit database, showing the efficacy of the proposed sparsity-based
defense schemes.


http://arxiv.org/abs/1810.10031
Stochastic Substitute Training: A Gray-box Approach to Craft Adversarial Examples Against Gradient Obfuscation Defenses.
Mohammad Hashemi; Greg Cusack; Eric Keller
  It has been shown that adversaries can craft example inputs to neural
networks which are similar to legitimate inputs but have been created to
purposely cause the neural network to misclassify the input. These adversarial
examples are crafted, for example, by calculating gradients of a carefully
defined loss function with respect to the input. As a countermeasure, some
researchers have tried to design robust models by blocking or obfuscating
gradients, even in white-box settings. Another line of research proposes
introducing a separate detector to attempt to detect adversarial examples. This
approach also makes use of gradient obfuscation techniques, for example, to
prevent the adversary from trying to fool the detector. In this paper, we
introduce stochastic substitute training, a gray-box approach that can craft
adversarial examples for defenses which obfuscate gradients. For those defenses
that have tried to make models more robust, with our technique, an adversary
can craft adversarial examples with no knowledge of the defense. For defenses
that attempt to detect the adversarial examples, with our technique, an
adversary only needs very limited information about the defense to craft
adversarial examples. We demonstrate our technique by applying it against two
defenses which make models more robust and two defenses which detect
adversarial examples.


http://arxiv.org/abs/1810.09650
One Bit Matters: Understanding Adversarial Examples as the Abuse of Redundancy.
Jingkang Wang; Ruoxi Jia; Gerald Friedland; Bo Li; Costas Spanos
  Despite the great success achieved in machine learning (ML), adversarial
examples have caused concerns with regards to its trustworthiness: A small
perturbation of an input results in an arbitrary failure of an otherwise
seemingly well-trained ML model. While studies are being conducted to discover
the intrinsic properties of adversarial examples, such as their transferability
and universality, there is insufficient theoretic analysis to help understand
the phenomenon in a way that can influence the design process of ML
experiments. In this paper, we deduce an information-theoretic model which
explains adversarial attacks as the abuse of feature redundancies in ML
algorithms. We prove that feature redundancy is a necessary condition for the
existence of adversarial examples. Our model helps to explain some major
questions raised in many anecdotal studies on adversarial examples. Our theory
is backed up by empirical measurements of the information content of benign and
adversarial examples on both image and text datasets. Our measurements show
that typical adversarial examples introduce just enough redundancy to overflow
the decision making of an ML model trained on corresponding benign examples. We
conclude with actionable recommendations to improve the robustness of machine
learners against adversarial examples.


http://arxiv.org/abs/1810.09619
Sparse DNNs with Improved Adversarial Robustness.
Yiwen Guo; Chao Zhang; Changshui Zhang; Yurong Chen
  Deep neural networks (DNNs) are computationally/memory-intensive and
vulnerable to adversarial attacks, making them prohibitive in some real-world
applications. By converting dense models into sparse ones, pruning appears to
be a promising solution to reducing the computation/memory cost. This paper
studies classification models, especially DNN-based ones, to demonstrate that
there exists intrinsic relationships between their sparsity and adversarial
robustness. Our analyses reveal, both theoretically and empirically, that
nonlinear DNN-based classifiers behave differently under $l_2$ attacks from
some linear ones. We further demonstrate that an appropriately higher model
sparsity implies better robustness of nonlinear DNNs, whereas over-sparsified
models can be more difficult to resist adversarial examples.


http://arxiv.org/abs/1810.09519
Adversarial Risk Bounds via Function Transformation.
Justin Khim; Po-Ling Loh
  We derive bounds for a notion of adversarial risk, designed to characterize
the robustness of linear and neural network classifiers to adversarial
perturbations. Specifically, we introduce a new class of function
transformations with the property that the risk of the transformed functions
upper-bounds the adversarial risk of the original functions. This reduces the
problem of deriving bounds on the adversarial risk to the problem of deriving
risk bounds using standard learning-theoretic techniques. We then derive bounds
on the Rademacher complexities of the transformed function classes, obtaining
error rates on the same order as the generalization error of the original
function classes. We also discuss extensions of our theory to multiclass
classification and regression. Finally, we provide two algorithms for
optimizing the adversarial risk bounds in the linear case, and discuss
connections to regularization and distributional robustness.


http://arxiv.org/abs/1810.09225
Cost-Sensitive Robustness against Adversarial Examples.
Xiao Zhang; David Evans
  Several recent works have developed methods for training classifiers that are
certifiably robust against norm-bounded adversarial perturbations. These
methods assume that all the adversarial transformations are equally important,
which is seldom the case in real-world applications. We advocate for
cost-sensitive robustness as the criteria for measuring the classifier's
performance for tasks where some adversarial transformation are more important
than others. We encode the potential harm of each adversarial transformation in
a cost matrix, and propose a general objective function to adapt the robust
training method of Wong & Kolter (2018) to optimize for cost-sensitive
robustness. Our experiments on simple MNIST and CIFAR10 models with a variety
of cost matrices show that the proposed approach can produce models with
substantially reduced cost-sensitive robust error, while maintaining
classification accuracy.


http://arxiv.org/abs/1810.08640
On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm.
Tsui-Wei Weng; Huan Zhang; Pin-Yu Chen; Aurelie Lozano; Cho-Jui Hsieh; Luca Daniel
  CLEVER (Cross-Lipschitz Extreme Value for nEtwork Robustness) is an Extreme
Value Theory (EVT) based robustness score for large-scale deep neural networks
(DNNs). In this paper, we propose two extensions on this robustness score.
First, we provide a new formal robustness guarantee for classifier functions
that are twice differentiable. We apply extreme value theory on the new formal
robustness guarantee and the estimated robustness is called second-order CLEVER
score. Second, we discuss how to handle gradient masking, a common defensive
technique, using CLEVER with Backward Pass Differentiable Approximation (BPDA).
With BPDA applied, CLEVER can evaluate the intrinsic robustness of neural
networks of a broader class -- networks with non-differentiable input
transformations. We demonstrate the effectiveness of CLEVER with BPDA in
experiments on a 121-layer Densenet model trained on the ImageNet dataset.


http://arxiv.org/abs/1810.08280
Exploring Adversarial Examples in Malware Detection.
Octavian Suciu; Scott E. Coull; Jeffrey Johns
  The convolutional neural network (CNN) architecture is increasingly being
applied to new domains, such as malware detection, where it is able to learn
malicious behavior from raw bytes extracted from executables. These
architectures reach impressive performance with no feature engineering effort
involved, but their robustness against active attackers is yet to be
understood. Such malware detectors could face a new attack vector in the form
of adversarial interference with the classification model. Existing evasion
attacks intended to cause misclassification on test-time instances, which have
been extensively studied for image classifiers, are not applicable because of
the input semantics that prevents arbitrary changes to the binaries. This paper
explores the area of adversarial examples for malware detection. By training an
existing model on a production-scale dataset, we show that some previous
attacks are less effective than initially reported, while simultaneously
highlighting architectural weaknesses that facilitate new attack strategies for
malware classification. Finally, we explore how generalizable different attack
strategies are, the trade-offs when aiming to increase their effectiveness, and
the transferability of single-step attacks.


http://arxiv.org/abs/1810.08070
A Training-based Identification Approach to VIN Adversarial Examples.
Yingdi Wang; Wenjia Niu; Tong Chen; Yingxiao Xiang; Jingjing Liu; Gang Li; Jiqiang Liu
  With the rapid development of Artificial Intelligence (AI), the problem of AI
security has gradually emerged. Most existing machine learning algorithms may
be attacked by adversarial examples. An adversarial example is a slightly
modified input sample that can lead to a false result of machine learning
algorithms. The adversarial examples pose a potential security threat for many
AI application areas, especially in the domain of robot path planning. In this
field, the adversarial examples obstruct the algorithm by adding obstacles to
the normal maps, resulting in multiple effects on the predicted path. However,
there is no suitable approach to automatically identify them. To our knowledge,
all previous work uses manual observation method to estimate the attack results
of adversarial maps, which is time-consuming. Aiming at the existing problem,
this paper explores a method to automatically identify the adversarial examples
in Value Iteration Networks (VIN), which has a strong generalization ability.
We analyze the possible scenarios caused by the adversarial maps. We propose a
training-based identification approach to VIN adversarial examples by combing
the path feature comparison and path image classification. We evaluate our
method using the adversarial maps dataset, show that our method can achieve a
high-accuracy and faster identification than manual observation method.


http://arxiv.org/abs/1810.07481
Provable Robustness of ReLU networks via Maximization of Linear Regions.
Francesco University of Tübingen Croce; Maksym Saarland University Andriushchenko; Matthias University of Tübingen Hein
  It has been shown that neural network classifiers are not robust. This raises
concerns about their usage in safety-critical systems. We propose in this paper
a regularization scheme for ReLU networks which provably improves the
robustness of the classifier by maximizing the linear regions of the classifier
as well as the distance to the decision boundary. Our techniques allow even to
find the minimal adversarial perturbation for a fraction of test points for
large networks. In the experiments we show that our approach improves upon
adversarial training both in terms of lower and upper bounds on the robustness
and is comparable or better than the state-of-the-art in terms of test error
and robustness.


http://arxiv.org/abs/1810.10337
Projecting Trouble: Light Based Adversarial Attacks on Deep Learning Classifiers.
Nicole Nichols; Robert Jasper
  This work demonstrates a physical attack on a deep learning image
classification system using projected light onto a physical scene. Prior work
is dominated by techniques for creating adversarial examples which directly
manipulate the digital input of the classifier. Such an attack is limited to
scenarios where the adversary can directly update the inputs to the classifier.
This could happen by intercepting and modifying the inputs to an online API
such as Clarifai or Cloud Vision. Such limitations have led to a vein of
research around physical attacks where objects are constructed to be inherently
adversarial or adversarial modifications are added to cause misclassification.
Our work differs from other physical attacks in that we can cause
misclassification dynamically without altering physical objects in a permanent
way.
  We construct an experimental setup which includes a light projection source,
an object for classification, and a camera to capture the scene. Experiments
are conducted against 2D and 3D objects from CIFAR-10. Initial tests show
projected light patterns selected via differential evolution could degrade
classification from 98% to 22% and 89% to 43% probability for 2D and 3D targets
respectively. Subsequent experiments explore sensitivity to physical setup and
compare two additional baseline conditions for all 10 CIFAR classes. Some
physical targets are more susceptible to perturbation. Simple attacks show near
equivalent success, and 6 of the 10 classes were disrupted by light.


http://arxiv.org/abs/1810.07339
Security Matters: A Survey on Adversarial Machine Learning.
Guofu Li; Pengjia Zhu; Jin Li; Zhemin Yang; Ning Cao; Zhiyi Chen
  Adversarial machine learning is a fast growing research area, which considers
the scenarios when machine learning systems may face potential adversarial
attackers, who intentionally synthesize input data to make a well-trained model
to make mistake. It always involves a defending side, usually a classifier, and
an attacking side that aims to cause incorrect output. The earliest studies on
the adversarial examples for machine learning algorithms start from the
information security area, which considers a much wider varieties of attacking
methods. But recent research focus that popularized by the deep learning
community places strong emphasis on how the "imperceivable" perturbations on
the normal inputs may cause dramatic mistakes by the deep learning with
supposed super-human accuracy. This paper serves to give a comprehensive
introduction to a range of aspects of the adversarial deep learning topic,
including its foundations, typical attacking and defending strategies, and some
extended studies.


http://arxiv.org/abs/1810.06583
Adversarial Learning and Explainability in Structured Datasets.
Prasad Chalasani; Somesh Jha; Aravind Sadagopan; Xi Wu
  We theoretically and empirically explore the explainability benefits of
adversarial learning in logistic regression models on structured datasets. In
particular we focus on improved explainability due to significantly higher
$\textit{feature-concentration}$ in adversarially-learned models: Compared to
natural training, adversarial training tends to more efficiently shrink the
weights of non-predictive and weakly-predictive features, while model
performance on natural test data only degrades slightly (and even sometimes
improves), compared to that of a naturally trained model. We provide
theoretical insight into this phenomenon via an analysis of the expectation of
the logistic model weight updates by an SGD-based adversarial learning
algorithm, where examples are drawn from a random binary data-generation
process. We empirically demonstrate the feature-pruning effect on a synthetic
dataset, some datasets from the UCI repository, and real-world large-scale
advertising response-prediction data-sets from MediaMath. In several of the
MediaMath datasets there are 10s of millions of data points, and on the order
of 100,000 sparse categorical features, and adversarial learning often results
in model-size reduction by a factor of 20 or higher, and yet the model
performance on natural test data (measured by AUC) is comparable to (and
sometimes even better than) that of the naturally trained model. We also show
that traditional $\ell_1$ regularization does not even come close to achieving
this level of feature-concentration. We measure "feature concentration" using
the Integrated Gradients-based feature-attribution method of Sundararajan et.
al (2017), and derive a new closed-form expression for 1-layer networks, which
substantially speeds up computation of aggregate feature attributions across a
large dataset.


http://arxiv.org/abs/1810.05162
Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation.
Chaowei Xiao; Ruizhi Deng; Bo Li; Fisher Yu; Mingyan Liu; Dawn Song
  Deep Neural Networks (DNNs) have been widely applied in various recognition
tasks. However, recently DNNs have been shown to be vulnerable against
adversarial examples, which can mislead DNNs to make arbitrary incorrect
predictions. While adversarial examples are well studied in classification
tasks, other learning problems may have different properties. For instance,
semantic segmentation requires additional components such as dilated
convolutions and multiscale processing. In this paper, we aim to characterize
adversarial examples based on spatial context information in semantic
segmentation. We observe that spatial consistency information can be
potentially leveraged to detect adversarial examples robustly even when a
strong adaptive attacker has access to the model and detection strategies. We
also show that adversarial examples based on attacks considered within the
paper barely transfer among models, even though transferability is common in
classification. Our observations shed new light on developing adversarial
attacks and defenses to better understand the vulnerabilities of DNNs.


http://arxiv.org/abs/1810.05206
MeshAdv: Adversarial Meshes for Visual Recognition.
Chaowei Xiao; Dawei Yang; Bo Li; Jia Deng; Mingyan Liu
  Highly expressive models such as deep neural networks (DNNs) have been widely
applied to various applications. However, recent studies show that DNNs are
vulnerable to adversarial examples, which are carefully crafted inputs aiming
to mislead the predictions. Currently, the majority of these studies have
focused on perturbation added to image pixels, while such manipulation is not
physically realistic. Some works have tried to overcome this limitation by
attaching printable 2D patches or painting patterns onto surfaces, but can be
potentially defended because 3D shape features are intact. In this paper, we
propose meshAdv to generate "adversarial 3D meshes" from objects that have rich
shape features but minimal textural variation. To manipulate the shape or
texture of the objects, we make use of a differentiable renderer to compute
accurate shading on the shape and propagate the gradient. Extensive experiments
show that the generated 3D meshes are effective in attacking both classifiers
and object detectors. We evaluate the attack under different viewpoints. In
addition, we design a pipeline to perform black-box attack on a photorealistic
renderer with unknown rendering parameters.


http://arxiv.org/abs/1810.05665
Is PGD-Adversarial Training Necessary? Alternative Training via a Soft-Quantization Network with Noisy-Natural Samples Only.
Tianhang Zheng; Changyou Chen; Kui Ren
  Recent work on adversarial attack and defense suggests that PGD is a
universal $l_\infty$ first-order attack, and PGD adversarial training can
significantly improve network robustness against a wide range of first-order
$l_\infty$-bounded attacks, represented as the state-of-the-art defense method.
However, an obvious weakness of PGD adversarial training is its
highly-computational cost in generating adversarial samples, making it
computationally infeasible for large and high-resolution real datasets such as
the ImageNet dataset. In addition, recent work also has suggested a simple
"close-form" solution to a robust model on MNIST. Therefore, a natural question
raised is that is PGD adversarial training really necessary for robust defense?
In this paper, we give a negative answer by proposing a training paradigm that
is comparable to PGD adversarial training on several standard datasets, while
only using noisy-natural samples. Specifically, we reformulate the min-max
objective in PGD adversarial training by a problem to minimize the original
network loss plus $l_1$ norms of its gradients w.r.t. the inputs. For the
$l_1$-norm loss, we propose a computationally-feasible solution by embedding a
differentiable soft-quantization layer after the network input layer. We show
formally that the soft-quantization layer trained with noisy-natural samples is
an alternative approach to minimizing the $l_1$-gradient norms as in PGD
adversarial training. Extensive empirical evaluations on standard datasets show
that our proposed models are comparable to PGD-adversarially-trained models
under PGD and BPDA attacks. Remarkably, our method achieves a 24X speed-up on
MNIST while maintaining a comparable defensive ability, and for the first time
fine-tunes a robust Imagenet model within only two days. Code is provided on
\url{https://github.com/tianzheng4/Noisy-Training-Soft-Quantization}


http://arxiv.org/abs/1810.03913
Analyzing the Noise Robustness of Deep Neural Networks.
Mengchen Liu; Shixia Liu; Hang Su; Kelei Cao; Jun Zhu
  Deep neural networks (DNNs) are vulnerable to maliciously generated
adversarial examples. These examples are intentionally designed by making
imperceptible perturbations and often mislead a DNN into making an incorrect
prediction. This phenomenon means that there is significant risk in applying
DNNs to safety-critical applications, such as driverless cars. To address this
issue, we present a visual analytics approach to explain the primary cause of
the wrong predictions introduced by adversarial examples. The key is to analyze
the datapaths of the adversarial examples and compare them with those of the
normal examples. A datapath is a group of critical neurons and their
connections. To this end, we formulate the datapath extraction as a subset
selection problem and approximately solve it based on back-propagation. A
multi-level visualization consisting of a segmented DAG (layer level), an Euler
diagram (feature map level), and a heat map (neuron level), has been designed
to help experts investigate datapaths from the high-level layers to the
detailed neuron activations. Two case studies are conducted that demonstrate
the promise of our approach in support of explaining the working mechanism of
adversarial examples.


http://arxiv.org/abs/1810.03806
The Adversarial Attack and Detection under the Fisher Information Metric.
Chenxiao Zhao; P. Thomas Fletcher; Mixue Yu; Yaxin Peng; Guixu Zhang; Chaomin Shen
  Many deep learning models are vulnerable to the adversarial attack, i.e.,
imperceptible but intentionally-designed perturbations to the input can cause
incorrect output of the networks. In this paper, using information geometry, we
provide a reasonable explanation for the vulnerability of deep learning models.
By considering the data space as a non-linear space with the Fisher information
metric induced from a neural network, we first propose an adversarial attack
algorithm termed one-step spectral attack (OSSA). The method is described by a
constrained quadratic form of the Fisher information matrix, where the optimal
adversarial perturbation is given by the first eigenvector, and the model
vulnerability is reflected by the eigenvalues. The larger an eigenvalue is, the
more vulnerable the model is to be attacked by the corresponding eigenvector.
Taking advantage of the property, we also propose an adversarial detection
method with the eigenvalues serving as characteristics. Both our attack and
detection algorithms are numerically optimized to work efficiently on large
datasets. Our evaluations show superior performance compared with other
methods, implying that the Fisher information is a promising approach to
investigate the adversarial attacks and defenses.


http://arxiv.org/abs/1810.04065
Limitations of adversarial robustness: strong No Free Lunch Theorem.
Elvis Dohmatob
  This manuscript presents some new impossibility results on adversarial
robustness in machine learning, a very important yet largely open problem. We
show that if conditioned on a class label the data distribution satisfies the
$W_2$ Talagrand transportation-cost inequality (for example, this condition is
satisfied if the conditional distribution has density which is log-concave; is
the uniform measure on a compact Riemannian manifold with positive Ricci
curvature, any classifier can be adversarially fooled with high probability
once the perturbations are slightly greater than the natural noise level in the
problem. We call this result The Strong "No Free Lunch" Theorem as some recent
results (Tsipras et al. 2018, Fawzi et al. 2018, etc.) on the subject can be
immediately recovered as very particular cases. Our theoretical bounds are
demonstrated on both simulated and real data (MNIST). We conclude the
manuscript with some speculation on possible future research directions.


http://arxiv.org/abs/1810.03773
Average Margin Regularization for Classifiers.
Matt Olfat; Anil Aswani
  Adversarial robustness has become an important research topic given empirical
demonstrations on the lack of robustness of deep neural networks.
Unfortunately, recent theoretical results suggest that adversarial training
induces a strict tradeoff between classification accuracy and adversarial
robustness. In this paper, we propose and then study a new regularization for
any margin classifier or deep neural network. We motivate this regularization
by a novel generalization bound that shows a tradeoff in classifier accuracy
between maximizing its margin and average margin. We thus call our approach an
average margin (AM) regularization, and it consists of a linear term added to
the objective. We theoretically show that for certain distributions AM
regularization can both improve classifier accuracy and robustness to
adversarial attacks. We conclude by using both synthetic and real data to
empirically show that AM regularization can strictly improve both accuracy and
robustness for support vector machine's (SVM's) and deep neural networks,
relative to unregularized classifiers and adversarially trained classifiers.


http://arxiv.org/abs/1810.03739
Efficient Two-Step Adversarial Defense for Deep Neural Networks.
Ting-Jui Chang; Yukun He; Peng Li
  In recent years, deep neural networks have demonstrated outstanding
performance in many machine learning tasks. However, researchers have
discovered that these state-of-the-art models are vulnerable to adversarial
examples: legitimate examples added by small perturbations which are
unnoticeable to human eyes. Adversarial training, which augments the training
data with adversarial examples during the training process, is a well known
defense to improve the robustness of the model against adversarial attacks.
However, this robustness is only effective to the same attack method used for
adversarial training. Madry et al.(2017) suggest that effectiveness of
iterative multi-step adversarial attacks and particularly that projected
gradient descent (PGD) may be considered the universal first order adversary
and applying the adversarial training with PGD implies resistance against many
other first order attacks. However, the computational cost of the adversarial
training with PGD and other multi-step adversarial examples is much higher than
that of the adversarial training with other simpler attack techniques. In this
paper, we show how strong adversarial examples can be generated only at a cost
similar to that of two runs of the fast gradient sign method (FGSM), allowing
defense against adversarial attacks with a robustness level comparable to that
of the adversarial training with multi-step adversarial examples. We
empirically demonstrate the effectiveness of the proposed two-step defense
approach against different attack methods and its improvements over existing
defense strategies.


http://arxiv.org/abs/1810.03538
Combinatorial Attacks on Binarized Neural Networks.
Elias B. Khalil; Amrita Gupta; Bistra Dilkina
  Binarized Neural Networks (BNNs) have recently attracted significant interest
due to their computational efficiency. Concurrently, it has been shown that
neural networks may be overly sensitive to "attacks" - tiny adversarial changes
in the input - which may be detrimental to their use in safety-critical
domains. Designing attack algorithms that effectively fool trained models is a
key step towards learning robust neural networks. The discrete,
non-differentiable nature of BNNs, which distinguishes them from their
full-precision counterparts, poses a challenge to gradient-based attacks. In
this work, we study the problem of attacking a BNN through the lens of
combinatorial and integer optimization. We propose a Mixed Integer Linear
Programming (MILP) formulation of the problem. While exact and flexible, the
MILP quickly becomes intractable as the network and perturbation space grow. To
address this issue, we propose IProp, a decomposition-based algorithm that
solves a sequence of much smaller MILP problems. Experimentally, we evaluate
both proposed methods against the standard gradient-based attack (FGSM) on
MNIST and Fashion-MNIST, and show that IProp performs favorably compared to
FGSM, while scaling beyond the limits of the MILP.


http://arxiv.org/abs/1810.02424
Feature prioritization and regularization improve standard accuracy and adversarial robustness.
Chihuang Liu; Joseph JaJa
  Adversarial training has been successfully applied to build robust models at
a certain cost. While the robustness of a model increases, the standard
classification accuracy declines. This phenomenon is suggested to be an
inherent trade-off. We propose a model that employs feature prioritization by a
nonlinear attention module and $L_2$ feature regularization to improve the
adversarial robustness and the standard accuracy relative to adversarial
training. The attention module encourages the model to rely heavily on robust
features by assigning larger weights to them while suppressing non-robust
features. The regularizer encourages the model to extract similar features for
the natural and adversarial images, effectively ignoring the added
perturbation. In addition to evaluating the robustness of our model, we provide
justification for the attention module and propose a novel experimental
strategy that quantitatively demonstrates that our model is almost ideally
aligned with salient data characteristics. Additional experimental results
illustrate the power of our model relative to the state of the art methods.


http://arxiv.org/abs/1810.02180
Improved Generalization Bounds for Robust Learning.
Idan Attias; Aryeh Kontorovich; Yishay Mansour
  We consider a model of robust learning in an adversarial environment. The
learner gets uncorrupted training data with access to possible corruptions that
may be affected by the adversary during testing. The learner's goal is to build
a robust classifier that would be tested on future adversarial examples. We use
a zero-sum game between the learner and the adversary as our game theoretic
framework. The adversary is limited to $k$ possible corruptions for each input.
Our model is closely related to the adversarial examples model of Schmidt et
al. (2018); Madry et al. (2017).
  Our main results consist of generalization bounds for the binary and
multi-class classification, as well as the real-valued case (regression). For
the binary classification setting, we both tighten the generalization bound of
Feige, Mansour, and Schapire (2015), and also are able to handle an infinite
hypothesis class $H$. The sample complexity is improved from
$O(\frac{1}{\epsilon^4}\log(\frac{|H|}{\delta}))$ to
$O(\frac{1}{\epsilon^2}(k\log(k)VC(H)+\log\frac{1}{\delta}))$. Additionally, we
extend the algorithm and generalization bound from the binary to the multiclass
and real-valued cases. Along the way, we obtain results on fat-shattering
dimension and Rademacher complexity of $k$-fold maxima over function classes;
these may be of independent interest.
  For binary classification, the algorithm of Feige et al. (2015) uses a regret
minimization algorithm and an ERM oracle as a blackbox; we adapt it for the
multi-class and regression settings. The algorithm provides us with
near-optimal policies for the players on a given training sample.


http://arxiv.org/abs/1810.01407
Can Adversarially Robust Learning Leverage Computational Hardness?.
Saeed Mahloujifar; Mohammad Mahmoody
  Making learners robust to adversarial perturbation at test time (i.e.,
evasion attacks) or training time (i.e., poisoning attacks) has emerged as a
challenging task. It is known that for some natural settings, sublinear
perturbations in the training phase or the testing phase can drastically
decrease the quality of the predictions. These negative results, however, are
information theoretic and only prove the existence of such successful
adversarial perturbations. A natural question for these settings is whether or
not we can make classifiers computationally robust to polynomial-time attacks.
  In this work, we prove strong barriers against achieving such envisioned
computational robustness both for evasion and poisoning attacks. In particular,
we show that if the test instances come from a product distribution (e.g.,
uniform over $\{0,1\}^n$ or $[0,1]^n$, or isotropic $n$-variate Gaussian) and
that there is an initial constant error, then there exists a polynomial-time
attack that finds adversarial examples of Hamming distance $O(\sqrt n)$. For
poisoning attacks, we prove that for any learning algorithm with sample
complexity $m$ and any efficiently computable "predicate" defining some "bad"
property $B$ for the produced hypothesis (e.g., failing on a particular test)
that happens with an initial constant probability, there exist polynomial-time
online poisoning attacks that tamper with $O (\sqrt m)$ many examples, replace
them with other correctly labeled examples, and increases the probability of
the bad event $B$ to $\approx 1$.
  Both of our poisoning and evasion attacks are black-box in how they access
their corresponding components of the system (i.e., the hypothesis, the
concept, and the learning algorithm) and make no further assumptions about the
classifier or the learning algorithm producing the classifier.


http://arxiv.org/abs/1810.01185
Adversarial Examples - A Complete Characterisation of the Phenomenon.
Alexandru Constantin Serban; Erik Poll; Joost Visser
  We provide a complete characterisation of the phenomenon of adversarial
examples - inputs intentionally crafted to fool machine learning models. We aim
to cover all the important concerns in this field of study: (1) the conjectures
on the existence of adversarial examples, (2) the security, safety and
robustness implications, (3) the methods used to generate and (4) protect
against adversarial examples and (5) the ability of adversarial examples to
transfer between different machine learning models. We provide ample background
information in an effort to make this document self-contained. Therefore, this
document can be used as survey, tutorial or as a catalog of attacks and
defences using adversarial examples.


http://arxiv.org/abs/1810.01110
Link Prediction Adversarial Attack.
Jinyin Chen; Ziqiang Shi; Yangyang Wu; Xuanheng Xu; Haibin Zheng
  Deep neural network has shown remarkable performance in solving computer
vision and some graph evolved tasks, such as node classification and link
prediction. However, the vulnerability of deep model has also been revealed by
carefully designed adversarial examples generated by various adversarial attack
methods. With the wider application of deep model in complex network analysis,
in this paper we define and formulate the link prediction adversarial attack
problem and put forward a novel iterative gradient attack (IGA) based on the
gradient information in trained graph auto-encoder (GAE). To our best
knowledge, it is the first time link prediction adversarial attack problem is
defined and attack method is brought up. Not surprisingly, GAE was easily
fooled by adversarial network with only a few links perturbed on the clean
network. By conducting comprehensive experiments on different real-world data
sets, we can conclude that most deep model based and other state-of-art link
prediction algorithms cannot escape the adversarial attack just like GAE. We
can benefit the attack as an efficient privacy protection tool from link
prediction unknown violation, on the other hand, link prediction attack can be
a robustness evaluation metric for current link prediction algorithm in attack
defensibility.


http://arxiv.org/abs/1810.01279
Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network.
Xuanqing Liu; Yao Li; Chongruo Wu; Cho-Jui Hsieh
  We present a new algorithm to train a robust neural network against
adversarial attacks. Our algorithm is motivated by the following two ideas.
First, although recent work has demonstrated that fusing randomness can improve
the robustness of neural networks (Liu 2017), we noticed that adding noise
blindly to all the layers is not the optimal way to incorporate randomness.
Instead, we model randomness under the framework of Bayesian Neural Network
(BNN) to formally learn the posterior distribution of models in a scalable way.
Second, we formulate the mini-max problem in BNN to learn the best model
distribution under adversarial attacks, leading to an adversarial-trained
Bayesian neural net. Experiment results demonstrate that the proposed algorithm
achieves state-of-the-art performance under strong attacks. On CIFAR-10 with
VGG network, our model leads to 14\% accuracy improvement compared with
adversarial training (Madry 2017) and random self-ensemble (Liu 2017) under PGD
attack with $0.035$ distortion, and the gap becomes even larger on a subset of
ImageNet.


http://arxiv.org/abs/1810.01021
Large batch size training of neural networks with adversarial training and second-order information.
Zhewei Yao; Amir Gholami; Kurt Keutzer; Michael Mahoney
  The most straightforward method to accelerate Stochastic Gradient Descent
(SGD) is to distribute the randomly selected batch of inputs over multiple
processors. To keep the distributed processors fully utilized requires
commensurately growing the batch size; however, large batch training usually
leads to poor generalization. Existing solutions for large batch training
either significantly degrade accuracy or require massive hyper-parameter
tuning. To address this issue, we propose a novel large batch training method
which combines recent results in adversarial training and second order
information. We extensively evaluate our method on Cifar-10/100, SVHN,
TinyImageNet, and ImageNet datasets, using multiple NNs, including residual
networks as well as smaller networks such as SqueezeNext. Our new approach
exceeds the performance of the existing solutions in terms of both accuracy and
the number of SGD iterations (up to 1\% and $5\times$, respectively). We
emphasize that this is achieved without any additional hyper-parameter tuning
to tailor our proposed method in any of these experiments. With slight
hyper-parameter tuning, our method can reduce the number of SGD iterations of
ResNet18 on Cifar-10/ImageNet to $44.8\times$ and $28.8\times$, respectively.
We have open sourced the method including tools for computing Hessian spectrum.


http://arxiv.org/abs/1810.00953
Improved robustness to adversarial examples using Lipschitz regularization of the loss.
Chris Finlay; Adam Oberman; Bilal Abbasi
  We augment adversarial training (AT) with worst case adversarial training
(WCAT) which improves adversarial robustness by 11% over the current
state-of-the-art result in the $\ell_2$ norm on CIFAR-10. We obtain verifiable
average case and worst case robustness guarantees, based on the expected and
maximum values of the norm of the gradient of the loss. We interpret
adversarial training as Total Variation Regularization, which is a fundamental
tool in mathematical image processing, and WCAT as Lipschitz regularization.


http://arxiv.org/abs/1810.00740
Improving the Generalization of Adversarial Training with Domain Adaptation.
Chuanbiao Song; Kun He; Liwei Wang; John E. Hopcroft
  By injecting adversarial examples into training data, adversarial training is
promising for improving the robustness of deep learning models. However, most
existing adversarial training approaches are based on a specific type of
adversarial attack. It may not provide sufficiently representative samples from
the adversarial domain, leading to a weak generalization ability on adversarial
examples from other attacks. Moreover, during the adversarial training,
adversarial perturbations on inputs are usually crafted by fast single-step
adversaries so as to scale to large datasets. This work is mainly focused on
the adversarial training yet efficient FGSM adversary. In this scenario, it is
difficult to train a model with great generalization due to the lack of
representative adversarial samples, aka the samples are unable to accurately
reflect the adversarial domain. To alleviate this problem, we propose a novel
Adversarial Training with Domain Adaptation (ATDA) method. Our intuition is to
regard the adversarial training on FGSM adversary as a domain adaption task
with limited number of target domain samples. The main idea is to learn a
representation that is semantically meaningful and domain invariant on the
clean domain as well as the adversarial domain. Empirical evaluations on
Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly
improve the generalization of adversarial training and the smoothness of the
learned models, and outperforms state-of-the-art methods on standard benchmark
datasets. To show the transfer ability of our method, we also extend ATDA to
the adversarial training on iterative attacks such as PGD-Adversial Training
(PAT) and the defense performance is improved considerably.


http://arxiv.org/abs/1810.00470
Procedural Noise Adversarial Examples for Black-Box Attacks on Deep Neural Networks.
Kenneth T. Co; Luis Muñoz-González; Emil C. Lupu
  Deep neural networks have been shown to be vulnerable to adversarial
examples, perturbed inputs that are designed specifically to produce
intentional errors in the learning algorithms. However, existing attacks are
either computationally expensive or require extensive knowledge of the target
model and its dataset to succeed. Hence, these methods are not practical in a
deployed adversarial setting. In this paper we introduce an exploratory
approach for generating adversarial examples using procedural noise. We show
that it is possible to construct practical black-box attacks with low
computational cost against robust neural network architectures such as
Inception v3 and Inception ResNet v2 on the ImageNet dataset. We show that
these attacks successfully cause misclassification with a low number of
queries, significantly outperforming state-of-the-art black box attacks. Our
attack demonstrates the fragility of these neural networks to Perlin noise, a
type of procedural noise used for generating realistic textures. Perlin noise
attacks achieve at least 90% top 1 error across all classifiers. More
worryingly, we show that most Perlin noise perturbations are "universal" in
that they generalize, as adversarial examples, across large portions of the
dataset, with up to 73% of images misclassified using a single perturbation.
These findings suggest a systemic fragility of DNNs that needs to be explored
further. We also show the limitations of adversarial training, a technique used
to enhance the robustness against adversarial examples. Thus, the attacker just
needs to change the perspective to generate the adversarial examples to craft
successful attacks and, for the defender, it is difficult to foresee a priori
all possible types of adversarial perturbations.


http://arxiv.org/abs/1810.01268
CAAD 2018: Generating Transferable Adversarial Examples.
Yash Sharma; Tien-Dung Le; Moustafa Alzantot
  Deep neural networks (DNNs) are vulnerable to adversarial examples,
perturbations carefully crafted to fool the targeted DNN, in both the
non-targeted and targeted case. In the non-targeted case, the attacker simply
aims to induce misclassification. In the targeted case, the attacker aims to
induce classification to a specified target class. In addition, it has been
observed that strong adversarial examples can transfer to unknown models,
yielding a serious security concern. The NIPS 2017 competition was organized to
accelerate research in adversarial attacks and defenses, taking place in the
realistic setting where submitted adversarial attacks attempt to transfer to
submitted defenses. The CAAD 2018 competition took place with nearly identical
rules to the NIPS 2017 one. Given the requirement that the NIPS 2017
submissions were to be open-sourced, participants in the CAAD 2018 competition
were able to directly build upon previous solutions, and thus improve the
state-of-the-art in this setting. Our team participated in the CAAD 2018
competition, and won 1st place in both attack subtracks, non-targeted and
targeted adversarial attacks, and 3rd place in defense. We outline our
solutions and development results in this article. We hope our results can
inform researchers in both generating and defending against adversarial
examples.


http://arxiv.org/abs/1810.00208
To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compression.
Yiren Zhao; Ilia Shumailov; Robert Mullins; Ross Anderson
  As deep neural networks (DNNs) become widely used, pruned and quantised
models are becoming ubiquitous on edge devices; such compressed DNNs are
popular for lowering computational requirements. Meanwhile, recent studies show
that adversarial samples can be effective at making DNNs misclassify. We,
therefore, investigate the extent to which adversarial samples are transferable
between uncompressed and compressed DNNs. We find that adversarial samples
remain transferable for both pruned and quantised models. For pruning, the
adversarial samples generated from heavily pruned models remain effective on
uncompressed models. For quantisation, we find the transferability of
adversarial samples is highly sensitive to integer precision.


http://arxiv.org/abs/1810.00144
Interpreting Adversarial Robustness: A View from Decision Surface in Input Space.
Fuxun Yu; Chenchen Liu; Yanzhi Wang; Liang Zhao; Xiang Chen
  One popular hypothesis of neural network generalization is that the flat
local minima of loss surface in parameter space leads to good generalization.
However, we demonstrate that loss surface in parameter space has no obvious
relationship with generalization, especially under adversarial settings.
Through visualizing decision surfaces in both parameter space and input space,
we instead show that the geometry property of decision surface in input space
correlates well with the adversarial robustness. We then propose an adversarial
robustness indicator, which can evaluate a neural network's intrinsic
robustness property without testing its accuracy under adversarial attacks.
Guided by it, we further propose our robust training method. Without involving
adversarial training, our method could enhance network's intrinsic adversarial
robustness against various adversarial attacks.


http://arxiv.org/abs/1809.10875
Characterizing Audio Adversarial Examples Using Temporal Dependency.
Zhuolin Yang; Bo Li; Pin-Yu Chen; Dawn Song
  Recent studies have highlighted adversarial examples as a ubiquitous threat
to different neural network models and many downstream applications.
Nonetheless, as unique data properties have inspired distinct and powerful
learning principles, this paper aims to explore their potentials towards
mitigating adversarial inputs. In particular, our results reveal the importance
of using the temporal dependency in audio data to gain discriminate power
against adversarial examples. Tested on the automatic speech recognition (ASR)
tasks and three recent audio adversarial attacks, we find that (i) input
transformation developed from image adversarial defense provides limited
robustness improvement and is subtle to advanced attacks; (ii) temporal
dependency can be exploited to gain discriminative power against audio
adversarial examples and is resistant to adaptive attacks considered in our
experiments. Our results not only show promising means of improving the
robustness of ASR systems, but also offer novel insights in exploiting
domain-specific data properties to mitigate negative effects of adversarial
examples.


http://arxiv.org/abs/1810.00069
Adversarial Attacks and Defences: A Survey.
Anirban Chakraborty; Manaar Alam; Vishal Dey; Anupam Chattopadhyay; Debdeep Mukhopadhyay
  Deep learning has emerged as a strong and efficient framework that can be
applied to a broad spectrum of complex learning problems which were difficult
to solve using the traditional machine learning techniques in the past. In the
last few years, deep learning has advanced radically in such a way that it can
surpass human-level performance on a number of tasks. As a consequence, deep
learning is being extensively used in most of the recent day-to-day
applications. However, security of deep learning systems are vulnerable to
crafted adversarial examples, which may be imperceptible to the human eye, but
can lead the model to misclassify the output. In recent times, different types
of adversaries based on their threat model leverage these vulnerabilities to
compromise a deep learning system where adversaries have high incentives.
Hence, it is extremely important to provide robustness to deep learning
algorithms against these adversaries. However, there are only a few strong
countermeasures which can be used in all types of attack scenarios to design a
robust deep learning system. In this paper, we attempt to provide a detailed
discussion on different types of adversarial attacks with various threat models
and also elaborate the efficiency and challenges of recent countermeasures
against them.


http://arxiv.org/abs/1810.00024
Explainable Black-Box Attacks Against Model-based Authentication.
Washington Garcia; Joseph I. Choi; Suman K. Adari; Somesh Jha; Kevin R. B. Butler
  Establishing unique identities for both humans and end systems has been an
active research problem in the security community, giving rise to innovative
machine learning-based authentication techniques. Although such techniques
offer an automated method to establish identity, they have not been vetted
against sophisticated attacks that target their core machine learning
technique. This paper demonstrates that mimicking the unique signatures
generated by host fingerprinting and biometric authentication systems is
possible. We expose the ineffectiveness of underlying machine learning
classification models by constructing a blind attack based around the query
synthesis framework and utilizing Explainable-AI (XAI) techniques. We launch an
attack in under 130 queries on a state-of-the-art face authentication system,
and under 100 queries on a host authentication system. We examine how these
attacks can be defended against and explore their limitations. XAI provides an
effective means for adversaries to infer decision boundaries and provides a new
way forward in constructing attacks against systems using machine learning
models for authentication.


http://arxiv.org/abs/1810.07242
Adversarial Attacks on Cognitive Self-Organizing Networks: The Challenge and the Way Forward.
Muhammad Usama; Junaid Qadir; Ala Al-Fuqaha
  Future communications and data networks are expected to be largely cognitive
self-organizing networks (CSON). Such networks will have the essential property
of cognitive self-organization, which can be achieved using machine learning
techniques (e.g., deep learning). Despite the potential of these techniques,
these techniques in their current form are vulnerable to adversarial attacks
that can cause cascaded damages with detrimental consequences for the whole
network. In this paper, we explore the effect of adversarial attacks on CSON.
Our experiments highlight the level of threat that CSON have to deal with in
order to meet the challenges of next-generation networks and point out
promising directions for future work.


http://arxiv.org/abs/1809.09262
Neural Networks with Structural Resistance to Adversarial Attacks.
Alfaro Luca de
  In adversarial attacks to machine-learning classifiers, small perturbations
are added to input that is correctly classified. The perturbations yield
adversarial examples, which are virtually indistinguishable from the
unperturbed input, and yet are misclassified. In standard neural networks used
for deep learning, attackers can craft adversarial examples from most input to
cause a misclassification of their choice.
  We introduce a new type of network units, called RBFI units, whose non-linear
structure makes them inherently resistant to adversarial attacks. On
permutation-invariant MNIST, in absence of adversarial attacks, networks using
RBFI units match the performance of networks using sigmoid units, and are
slightly below the accuracy of networks with ReLU units. When subjected to
adversarial attacks, networks with RBFI units retain accuracies above 90% for
attacks that degrade the accuracy of networks with ReLU or sigmoid units to
below 2%. RBFI networks trained with regular input are superior in their
resistance to adversarial attacks even to ReLU and sigmoid networks trained
with the help of adversarial examples.
  The non-linear structure of RBFI units makes them difficult to train using
standard gradient descent. We show that networks of RBFI units can be
efficiently trained to high accuracies using pseudogradients, computed using
functions especially crafted to facilitate learning instead of their true
derivatives. We show that the use of pseudogradients makes training deep RBFI
networks practical, and we compare several structural alternatives of RBFI
networks for their accuracy.


http://arxiv.org/abs/1809.08999
Fast Geometrically-Perturbed Adversarial Faces.
Ali Dabouei; Sobhan Soleymani; Jeremy Dawson; Nasser M. Nasrabadi
  The state-of-the-art performance of deep learning algorithms has led to a
considerable increase in the utilization of machine learning in
security-sensitive and critical applications. However, it has recently been
shown that a small and carefully crafted perturbation in the input space can
completely fool a deep model. In this study, we explore the extent to which
face recognition systems are vulnerable to geometrically-perturbed adversarial
faces. We propose a fast landmark manipulation method for generating
adversarial faces, which is approximately 200 times faster than the previous
geometric attacks and obtains 99.86% success rate on the state-of-the-art face
recognition models. To further force the generated samples to be natural, we
introduce a second attack constrained on the semantic structure of the face
which has the half speed of the first attack with the success rate of 99.96%.
Both attacks are extremely robust against the state-of-the-art defense methods
with the success rate of equal or greater than 53.59%. Code is available at
https://github.com/alldbi/FLM


http://arxiv.org/abs/1809.08986
On The Utility of Conditional Generation Based Mutual Information for Characterizing Adversarial Subspaces.
Chia-Yi Hsu; Pei-Hsuan Lu; Pin-Yu Chen; Chia-Mu Yu
  Recent studies have found that deep learning systems are vulnerable to
adversarial examples; e.g., visually unrecognizable adversarial images can
easily be crafted to result in misclassification. The robustness of neural
networks has been studied extensively in the context of adversary detection,
which compares a metric that exhibits strong discriminate power between natural
and adversarial examples. In this paper, we propose to characterize the
adversarial subspaces through the lens of mutual information (MI) approximated
by conditional generation methods. We use MI as an information-theoretic metric
to strengthen existing defenses and improve the performance of adversary
detection. Experimental results on MagNet defense demonstrate that our proposed
MI detector can strengthen its robustness against powerful adversarial attacks.


http://arxiv.org/abs/1809.08758
Low Frequency Adversarial Perturbation.
Chuan Guo; Jared S. Frank; Kilian Q. Weinberger
  Recently, machine learning security has received significant attention. Many
computer vision and speech recognition systems have been compromised by
adversarially but imperceptibly perturbed input. To identify potential
perturbations, attackers search the high dimensional input space to find
directions in which the model lacks robustness. The exponential number of such
directions makes the existence of these adversarial perturbations likely, but
also creates significant challenges in the black-box setting: First, in the
absence of gradient information the search problem becomes expensive, resulting
in high query complexity. Second, the constructed perturbations are typically
high-frequency in nature and can be successfully defended against through
denoising transformations. In this paper we propose to restrict the search for
adversarial images to a low frequency domain. This approach is compatible with
existing white-box and black-box attacks, and has remarkable benefits in the
latter setting. In particular, we achieve state-of-the-art black-box query
efficiency and improve over prior work by an order of magnitude. Further, we
can circumvent image transformation defenses even when both the model and the
defense strategy are unknown. Finally, we demonstrate the efficacy of this
technique by fooling the Google Cloud Vision platform with an unprecedented low
number of model queries.


http://arxiv.org/abs/1809.08706
Is Ordered Weighted $\ell_1$ Regularized Regression Robust to Adversarial Perturbation? A Case Study on OSCAR.
Pin-Yu Chen; Bhanukiran Vinzamuri; Sijia Liu
  Many state-of-the-art machine learning models such as deep neural networks
have recently shown to be vulnerable to adversarial perturbations, especially
in classification tasks. Motivated by adversarial machine learning, in this
paper we investigate the robustness of sparse regression models with strongly
correlated covariates to adversarially designed measurement noises.
Specifically, we consider the family of ordered weighted $\ell_1$ (OWL)
regularized regression methods and study the case of OSCAR (octagonal shrinkage
clustering algorithm for regression) in the adversarial setting. Under a
norm-bounded threat model, we formulate the process of finding a maximally
disruptive noise for OWL-regularized regression as an optimization problem and
illustrate the steps towards finding such a noise in the case of OSCAR.
Experimental results demonstrate that the regression performance of grouping
strongly correlated features can be severely degraded under our adversarial
setting, even when the noise budget is significantly smaller than the
ground-truth signals.


http://arxiv.org/abs/1809.08516
Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization.
Bao Wang; Alex T. Lin; Zuoqiang Shi; Wei Zhu; Penghang Yin; Andrea L. Bertozzi; Stanley J. Osher
  We improve the robustness of deep neural nets to adversarial attacks by using
an interpolating function as the output activation. This data-dependent
activation remarkably improves both generalization and robustness towards
adversarial attacks. In the CIFAR10 benchmark, we raise the accuracy of the
Projected Gradient Descent adversarial training from $\sim 46\%$ to $\sim 69\%$
for ResNet20. When we combine this data-dependent activation with total
variation minimization on adversarial images and training data augmentation, we
achieve an improvement in accuracy by 38.9$\%$ for ResNet56 under the strongest
attack of the Iterative Fast Gradient Sign Method. We further provide an
intuitive explanation of our defense by analyzing the geometry of the feature
space. For reproducibility, the code is made available at
\url{https://github.com/BaoWangMath/DNN-DataDependentActivation}.


http://arxiv.org/abs/1809.08352
Unrestricted Adversarial Examples.
Tom B. Brown; Nicholas Carlini; Chiyuan Zhang; Catherine Olsson; Paul Christiano; Ian Goodfellow
  We introduce a two-player contest for evaluating the safety and robustness of
machine learning systems, with a large prize pool. Unlike most prior work in ML
robustness, which studies norm-constrained adversaries, we shift our focus to
unconstrained adversaries. Defenders submit machine learning models, and try to
achieve high accuracy and coverage on non-adversarial data while making no
confident mistakes on adversarial inputs. Attackers try to subvert defenses by
finding arbitrary unambiguous inputs where the model assigns an incorrect label
with high confidence. We propose a simple unambiguous dataset ("bird-or-
bicycle") to use as part of this contest. We hope this contest will help to
more comprehensively evaluate the worst-case adversarial risk of machine
learning models.


http://arxiv.org/abs/1809.08316
Adversarial Binaries for Authorship Identification.
Xiaozhu Meng; Barton P. Miller; Somesh Jha
  Binary code authorship identification determines authors of a binary program.
Existing techniques have used supervised machine learning for this task. In
this paper, we look this problem from an attacker's perspective. We aim to
modify a test binary, such that it not only causes misprediction but also
maintains the functionality of the original input binary. Attacks against
binary code are intrinsically more difficult than attacks against domains such
as computer vision, where attackers can change each pixel of the input image
independently and still maintain a valid image. For binary code, even flipping
one bit of a binary may cause the binary to be invalid, to crash at the
run-time, or to lose the original functionality. We investigate two types of
attacks: untargeted attacks, causing misprediction to any of the incorrect
authors, and targeted attacks, causing misprediction to a specific one among
the incorrect authors. We develop two key attack capabilities: feature vector
modification, generating an adversarial feature vector that both corresponds to
a real binary and causes the required misprediction, and input binary
modification, modifying the input binary to match the adversarial feature
vector while maintaining the functionality of the input binary. We evaluated
our attack against classifiers trained with a state-of-the-art method for
authorship attribution. The classifiers for authorship identification have 91%
accuracy on average. Our untargeted attack has a 96% success rate on average,
showing that we can effectively suppress authorship signal. Our targeted attack
has a 46% success rate on average, showing that it is possible, but
significantly more difficult to impersonate a specific programmer's style. Our
attack reveals that existing binary code authorship identification techniques
rely on code features that are easy to modify, and thus are vulnerable to
attacks.


http://arxiv.org/abs/1809.07802
Playing the Game of Universal Adversarial Perturbations.
Julien Perolat; Mateusz Malinowski; Bilal Piot; Olivier Pietquin
  We study the problem of learning classifiers robust to universal adversarial
perturbations. While prior work approaches this problem via robust
optimization, adversarial training, or input transformation, we instead phrase
it as a two-player zero-sum game. In this new formulation, both players
simultaneously play the same game, where one player chooses a classifier that
minimizes a classification loss whilst the other player creates an adversarial
perturbation that increases the same loss when applied to every sample in the
training set. By observing that performing a classification (respectively
creating adversarial samples) is the best response to the other player, we
propose a novel extension of a game-theoretic algorithm, namely fictitious
play, to the domain of training robust classifiers. Finally, we empirically
show the robustness and versatility of our approach in two defence scenarios
where universal attacks are performed on several image classification datasets
-- CIFAR10, CIFAR100 and ImageNet.


http://arxiv.org/abs/1809.08098
Efficient Formal Safety Analysis of Neural Networks.
Shiqi Wang; Kexin Pei; Justin Whitehouse; Junfeng Yang; Suman Jana
  Neural networks are increasingly deployed in real-world safety-critical
domains such as autonomous driving, aircraft collision avoidance, and malware
detection. However, these networks have been shown to often mispredict on
inputs with minor adversarial or even accidental perturbations. Consequences of
such errors can be disastrous and even potentially fatal as shown by the recent
Tesla autopilot crash. Thus, there is an urgent need for formal analysis
systems that can rigorously check neural networks for violations of different
safety properties such as robustness against adversarial perturbations within a
certain $L$-norm of a given image. An effective safety analysis system for a
neural network must be able to either ensure that a safety property is
satisfied by the network or find a counterexample, i.e., an input for which the
network will violate the property. Unfortunately, most existing techniques for
performing such analysis struggle to scale beyond very small networks and the
ones that can scale to larger networks suffer from high false positives and
cannot produce concrete counterexamples in case of a property violation. In
this paper, we present a new efficient approach for rigorously checking
different safety properties of neural networks that significantly outperforms
existing approaches by multiple orders of magnitude. Our approach can check
different safety properties and find concrete counterexamples for networks that
are 10$\times$ larger than the ones supported by existing analysis techniques.
We believe that our approach to estimating tight output bounds of a network for
a given input range can also help improve the explainability of neural networks
and guide the training process of more robust neural networks.


http://arxiv.org/abs/1809.07062
Adversarial Training Towards Robust Multimedia Recommender System.
Jinhui Tang; Xiaoyu Du; Xiangnan He; Fajie Yuan; Qi Tian; Tat-Seng Chua
  With the prevalence of multimedia content on the Web, developing recommender
solutions that can effectively leverage the rich signal in multimedia data is
in urgent need. Owing to the success of deep neural networks in representation
learning, recent advance on multimedia recommendation has largely focused on
exploring deep learning methods to improve the recommendation accuracy. To
date, however, there has been little effort to investigate the robustness of
multimedia representation and its impact on the performance of multimedia
recommendation.
  In this paper, we shed light on the robustness of multimedia recommender
system. Using the state-of-the-art recommendation framework and deep image
features, we demonstrate that the overall system is not robust, such that a
small (but purposeful) perturbation on the input image will severely decrease
the recommendation accuracy. This implies the possible weakness of multimedia
recommender system in predicting user preference, and more importantly, the
potential of improvement by enhancing its robustness. To this end, we propose a
novel solution named Adversarial Multimedia Recommendation (AMR), which can
lead to a more robust multimedia recommender model by using adversarial
learning. The idea is to train the model to defend an adversary, which adds
perturbations to the target image with the purpose of decreasing the model's
accuracy. We conduct experiments on two representative multimedia
recommendation tasks, namely, image recommendation and visually-aware product
recommendation. Extensive results verify the positive effect of adversarial
learning and demonstrate the effectiveness of our AMR method. Source codes are
available in https://github.com/duxy-me/AMR.


http://arxiv.org/abs/1809.07016
Generating 3D Adversarial Point Clouds.
Chong Xiang; Charles R. Qi; Bo Li
  Deep neural networks are known to be vulnerable to adversarial examples which
are carefully crafted instances to cause the models to make wrong predictions.
While adversarial examples for 2D images and CNNs have been extensively
studied, less attention has been paid to 3D data such as point clouds. Given
many safety-critical 3D applications such as autonomous driving, it is
important to study how adversarial point clouds could affect current deep 3D
models. In this work, we propose several novel algorithms to craft adversarial
point clouds against PointNet, a widely used deep neural network for point
cloud processing. Our algorithms work in two ways: adversarial point
perturbation and adversarial point generation. For point perturbation, we shift
existing points negligibly. For point generation, we generate either a set of
independent and scattered points or a small number (1-3) of point clusters with
meaningful shapes such as balls and airplanes which could be hidden in the
human psyche. In addition, we formulate six perturbation measurement metrics
tailored to the attacks in point clouds and conduct extensive experiments to
evaluate the proposed algorithms on the ModelNet40 3D shape classification
dataset. Overall, our attack algorithms achieve a success rate higher than 99%
for all targeted attacks


http://arxiv.org/abs/1809.06498
HashTran-DNN: A Framework for Enhancing Robustness of Deep Neural Networks against Adversarial Malware Samples.
Deqiang Li; Ramesh Baral; Tao Li; Han Wang; Qianmu Li; Shouhuai Xu
  Adversarial machine learning in the context of image processing and related
applications has received a large amount of attention. However, adversarial
machine learning, especially adversarial deep learning, in the context of
malware detection has received much less attention despite its apparent
importance. In this paper, we present a framework for enhancing the robustness
of Deep Neural Networks (DNNs) against adversarial malware samples, dubbed
Hashing Transformation Deep Neural Networks} (HashTran-DNN). The core idea is
to use hash functions with a certain locality-preserving property to transform
samples to enhance the robustness of DNNs in malware classification. The
framework further uses a Denoising Auto-Encoder (DAE) regularizer to
reconstruct the hash representations of samples, making the resulting DNN
classifiers capable of attaining the locality information in the latent space.
We experiment with two concrete instantiations of the HashTran-DNN framework to
classify Android malware. Experimental results show that four known attacks can
render standard DNNs useless in classifying Android malware, that known
defenses can at most defend three of the four attacks, and that HashTran-DNN
can effectively defend against all of the four attacks.


http://arxiv.org/abs/1809.06452
Robustness Guarantees for Bayesian Inference with Gaussian Processes.
Luca Cardelli; Marta Kwiatkowska; Luca Laurenti; Andrea Patane
  Bayesian inference and Gaussian processes are widely used in applications
ranging from robotics and control to biological systems. Many of these
applications are safety-critical and require a characterization of the
uncertainty associated with the learning model and formal guarantees on its
predictions. In this paper we define a robustness measure for Bayesian
inference against input perturbations, given by the probability that, for a
test point and a compact set in the input space containing the test point, the
prediction of the learning model will remain $\delta-$close for all the points
in the set, for $\delta>0.$ Such measures can be used to provide formal
guarantees for the absence of adversarial examples. By employing the theory of
Gaussian processes, we derive tight upper bounds on the resulting robustness by
utilising the Borell-TIS inequality, and propose algorithms for their
computation. We evaluate our techniques on two examples, a GP regression
problem and a fully-connected deep neural network, where we rely on weak
convergence to GPs to study adversarial examples on the MNIST dataset.


http://arxiv.org/abs/1809.05962
Robust Adversarial Perturbation on Deep Proposal-based Models.
Yuezun Li; Daniel Tian; Mingching-Chang; Xiao Bian; Siwei Lyu
  Adversarial noises are useful tools to probe the weakness of deep learning
based computer vision algorithms. In this paper, we describe a robust
adversarial perturbation (R-AP) method to attack deep proposal-based object
detectors and instance segmentation algorithms. Our method focuses on attacking
the common component in these algorithms, namely Region Proposal Network (RPN),
to universally degrade their performance in a black-box fashion. To do so, we
design a loss function that combines a label loss and a novel shape loss, and
optimize it with respect to image using a gradient based iterative algorithm.
Evaluations are performed on the MS COCO 2014 dataset for the adversarial
attacking of 6 state-of-the-art object detectors and 2 instance segmentation
algorithms. Experimental results demonstrate the efficacy of the proposed
method.


http://arxiv.org/abs/1809.05966
Exploring the Vulnerability of Single Shot Module in Object Detectors via Imperceptible Background Patches.
Yuezun Li; Xiao Bian; Ming-ching Chang; Siwei Lyu
  Recent works succeeded to generate adversarial perturbations on the entire
image or the object of interests to corrupt CNN based object detectors. In this
paper, we focus on exploring the vulnerability of the Single Shot Module (SSM)
commonly used in recent object detectors, by adding small perturbations to
patches in the background outside the object. The SSM is referred to the Region
Proposal Network used in a two-stage object detector or the single-stage object
detector itself. The SSM is typically a fully convolutional neural network
which generates output in a single forward pass. Due to the excessive
convolutions used in SSM, the actual receptive field is larger than the object
itself. As such, we propose a novel method to corrupt object detectors by
generating imperceptible patches only in the background. Our method can find a
few background patches for perturbation, which can effectively decrease true
positives and dramatically increase false positives. Efficacy is demonstrated
on 5 two-stage object detectors and 8 single-stage object detectors on the MS
COCO 2014 dataset. Results indicate that perturbations with small distortions
outside the bounding box of object region can still severely damage the
detection performance.


http://arxiv.org/abs/1809.05165
Defensive Dropout for Hardening Deep Neural Networks under Adversarial Attacks.
Siyue Wang; Xiao Wang; Pu Zhao; Wujie Wen; David Kaeli; Peter Chin; Xue Lin
  Deep neural networks (DNNs) are known vulnerable to adversarial attacks. That
is, adversarial examples, obtained by adding delicately crafted distortions
onto original legal inputs, can mislead a DNN to classify them as any target
labels. This work provides a solution to hardening DNNs under adversarial
attacks through defensive dropout. Besides using dropout during training for
the best test accuracy, we propose to use dropout also at test time to achieve
strong defense effects. We consider the problem of building robust DNNs as an
attacker-defender two-player game, where the attacker and the defender know
each others' strategies and try to optimize their own strategies towards an
equilibrium. Based on the observations of the effect of test dropout rate on
test accuracy and attack success rate, we propose a defensive dropout algorithm
to determine an optimal test dropout rate given the neural network model and
the attacker's strategy for generating adversarial examples.We also investigate
the mechanism behind the outstanding defense effects achieved by the proposed
defensive dropout. Comparing with stochastic activation pruning (SAP), another
defense method through introducing randomness into the DNN model, we find that
our defensive dropout achieves much larger variances of the gradients, which is
the key for the improved defense effects (much lower attack success rate). For
example, our defensive dropout can reduce the attack success rate from 100% to
13.89% under the currently strongest attack i.e., C&W attack on MNIST dataset.


http://arxiv.org/abs/1809.04913
Query-Efficient Black-Box Attack by Active Learning.
Pengcheng Li; Jinfeng Yi; Lijun Zhang
  Deep neural network (DNN) as a popular machine learning model is found to be
vulnerable to adversarial attack. This attack constructs adversarial examples
by adding small perturbations to the raw input, while appearing unmodified to
human eyes but will be misclassified by a well-trained classifier. In this
paper, we focus on the black-box attack setting where attackers have almost no
access to the underlying models. To conduct black-box attack, a popular
approach aims to train a substitute model based on the information queried from
the target DNN. The substitute model can then be attacked using existing
white-box attack approaches, and the generated adversarial examples will be
used to attack the target DNN. Despite its encouraging results, this approach
suffers from poor query efficiency, i.e., attackers usually needs to query a
huge amount of times to collect enough information for training an accurate
substitute model. To this end, we first utilize state-of-the-art white-box
attack methods to generate samples for querying, and then introduce an active
learning strategy to significantly reduce the number of queries needed.
Besides, we also propose a diversity criterion to avoid the sampling bias. Our
extensive experimental results on MNIST and CIFAR-10 show that the proposed
method can reduce more than $90\%$ of queries while preserve attacking success
rates and obtain an accurate substitute model which is more than $85\%$ similar
with the target oracle.


http://arxiv.org/abs/1809.04790
Adversarial Examples: Opportunities and Challenges.
Jiliang Zhang; Chen Li
  With the advent of the era of artificial intelligence (AI), deep neural
networks (DNNs) have shown huge superiority over human in image recognition,
speech processing, autonomous vehicles and medical diagnosis. However, recent
studies indicate that DNNs are vulnerable to adversarial examples (AEs) which
are designed by attackers to fool deep learning models. Different from real
examples, AEs can hardly be distinguished by human eyes, but mislead the model
to predict incorrect outputs and therefore threaten security critical
deep-learning applications. In recent years, the generation and defense of AEs
have become a research hotspot in the field of AI security. This article
reviews the latest research progress of AEs. First, we introduce the concept,
cause, characteristics and evaluation metrics of AEs, then give a survey on the
state-of-the-art AE generation methods with the discussion of advantages and
disadvantages. After that, we review the existing defenses and discuss their
limitations. Finally, the future research opportunities and challenges on AEs
are prospected.


http://arxiv.org/abs/1809.04098
On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions.
Yusuke Tsuzuku; Issei Sato
  Data-agnostic quasi-imperceptible perturbations on inputs are known to
degrade recognition accuracy of deep convolutional networks severely. This
phenomenon is considered to be a potential security issue. Moreover, some
results on statistical generalization guarantees indicate that the phenomenon
can be a key to improve the networks' generalization. However, the
characteristics of the shared directions of such harmful perturbations remain
unknown. Our primal finding is that convolutional networks are sensitive to the
directions of Fourier basis functions. We derived the property by specializing
a hypothesis of the cause of the sensitivity, known as the linearity of neural
networks, to convolutional networks and empirically validated it. As a
by-product of the analysis, we propose an algorithm to create shift-invariant
universal adversarial perturbations available in black-box settings.


http://arxiv.org/abs/1809.04397
Isolated and Ensemble Audio Preprocessing Methods for Detecting Adversarial Examples against Automatic Speech Recognition.
Krishan Rajaratnam; Kunal Shah; Jugal Kalita
  An adversarial attack is an exploitative process in which minute alterations
are made to natural inputs, causing the inputs to be misclassified by neural
models. In the field of speech recognition, this has become an issue of
increasing significance. Although adversarial attacks were originally
introduced in computer vision, they have since infiltrated the realm of speech
recognition. In 2017, a genetic attack was shown to be quite potent against the
Speech Commands Model. Limited-vocabulary speech classifiers, such as the
Speech Commands Model, are used in a variety of applications, particularly in
telephony; as such, adversarial examples produced by this attack pose as a
major security threat. This paper explores various methods of detecting these
adversarial examples with combinations of audio preprocessing. One particular
combined defense incorporating compressions, speech coding, filtering, and
audio panning was shown to be quite effective against the attack on the Speech
Commands Model, detecting audio adversarial examples with 93.5% precision and
91.2% recall.


http://arxiv.org/abs/1809.04120
Humans can decipher adversarial images.
Zhenglong Zhou; Chaz Firestone
  How similar is the human mind to the sophisticated machine-learning systems
that mirror its performance? Models of object categorization based on
convolutional neural networks (CNNs) have achieved human-level benchmarks in
assigning known labels to novel images. These advances promise to support
transformative technologies such as autonomous vehicles and machine diagnosis;
beyond this, they also serve as candidate models for the visual system itself
-- not only in their output but perhaps even in their underlying mechanisms and
principles. However, unlike human vision, CNNs can be "fooled" by adversarial
examples -- carefully crafted images that appear as nonsense patterns to humans
but are recognized as familiar objects by machines, or that appear as one
object to humans and a different object to machines. This seemingly extreme
divergence between human and machine classification challenges the promise of
these new advances, both as applied image-recognition systems and also as
models of the human mind. Surprisingly, however, little work has empirically
investigated human classification of such adversarial stimuli: Does human and
machine performance fundamentally diverge? Or could humans decipher such images
and predict the machine's preferred labels? Here, we show that human and
machine classification of adversarial stimuli are robustly related: In eight
experiments on five prominent and diverse adversarial imagesets, human subjects
reliably identified the machine's chosen label over relevant foils. This
pattern persisted for images with strong antecedent identities, and even for
images described as "totally unrecognizable to human eyes". We suggest that
human intuition may be a more reliable guide to machine (mis)classification
than has typically been imagined, and we explore the consequences of this
result for minds and machines alike.


http://arxiv.org/abs/1809.03113
Second-Order Adversarial Attack and Certifiable Robustness.
Bai Li; Changyou Chen; Wenlin Wang; Lawrence Carin
  We propose a powerful second-order attack method that outperforms existing
attack methods on reducing the accuracy of state-of-the-art defense models
based on adversarial training. The effectiveness of our attack method motivates
an investigation of provable robustness of a defense model. To this end, we
introduce a framework that allows one to obtain a certifiable lower bound on
the prediction accuracy against adversarial examples. We conduct experiments to
show the effectiveness of our attack method. At the same time, our defense
models obtain higher accuracies compared to previous works under our proposed
attack.


http://arxiv.org/abs/1809.03063
The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure.
Saeed Mahloujifar; Dimitrios I. Diochnos; Mohammad Mahmoody
  Many modern machine learning classifiers are shown to be vulnerable to
adversarial perturbations of the instances. Despite a massive amount of work
focusing on making classifiers robust, the task seems quite challenging. In
this work, through a theoretical study, we investigate the adversarial risk and
robustness of classifiers and draw a connection to the well-known phenomenon of
concentration of measure in metric measure spaces. We show that if the metric
probability space of the test instance is concentrated, any classifier with
some initial constant error is inherently vulnerable to adversarial
perturbations.
  One class of concentrated metric probability spaces are the so-called Levy
families that include many natural distributions. In this special case, our
attacks only need to perturb the test instance by at most $O(\sqrt n)$ to make
it misclassified, where $n$ is the data dimension. Using our general result
about Levy instance spaces, we first recover as special case some of the
previously proved results about the existence of adversarial examples. However,
many more Levy families are known (e.g., product distribution under the Hamming
distance) for which we immediately obtain new attacks that find adversarial
examples of distance $O(\sqrt n)$.
  Finally, we show that concentration of measure for product spaces implies the
existence of forms of "poisoning" attacks in which the adversary tampers with
the training data with the goal of degrading the classifier. In particular, we
show that for any learning algorithm that uses $m$ training examples, there is
an adversary who can increase the probability of any "bad property" (e.g.,
failing on a particular test instance) that initially happens with
non-negligible probability to $\approx 1$ by substituting only $\tilde{O}(\sqrt
m)$ of the examples with other (still correctly labeled) examples.


http://arxiv.org/abs/1809.03008
Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability.
Kai Y. Xiao; Vincent Tjeng; Nur Muhammad Shafiullah; Aleksander Madry
  We explore the concept of co-design in the context of neural network
verification. Specifically, we aim to train deep neural networks that not only
are robust to adversarial perturbations but also whose robustness can be
verified more easily. To this end, we identify two properties of network models
- weight sparsity and so-called ReLU stability - that turn out to significantly
impact the complexity of the corresponding verification task. We demonstrate
that improving weight sparsity alone already enables us to turn computationally
intractable verification problems into tractable ones. Then, improving ReLU
stability leads to an additional 4-13x speedup in verification times. An
important feature of our methodology is its "universality," in the sense that
it can be used with a broad range of training procedures and verification
approaches.


http://arxiv.org/abs/1809.02918
Towards Query Efficient Black-box Attacks: An Input-free Perspective.
Yali Du; Meng Fang; Jinfeng Yi; Jun Cheng; Dacheng Tao
  Recent studies have highlighted that deep neural networks (DNNs) are
vulnerable to adversarial attacks, even in a black-box scenario. However, most
of the existing black-box attack algorithms need to make a huge amount of
queries to perform attacks, which is not practical in the real world. We note
one of the main reasons for the massive queries is that the adversarial example
is required to be visually similar to the original image, but in many cases,
how adversarial examples look like does not matter much. It inspires us to
introduce a new attack called \emph{input-free} attack, under which an
adversary can choose an arbitrary image to start with and is allowed to add
perceptible perturbations on it. Following this approach, we propose two
techniques to significantly reduce the query complexity. First, we initialize
an adversarial example with a gray color image on which every pixel has roughly
the same importance for the target model. Then we shrink the dimension of the
attack space by perturbing a small region and tiling it to cover the input
image. To make our algorithm more effective, we stabilize a projected gradient
ascent algorithm with momentum, and also propose a heuristic approach for
region size selection. Through extensive experiments, we show that with only
1,701 queries on average, we can perturb a gray image to any target class of
ImageNet with a 100\% success rate on InceptionV3. Besides, our algorithm has
successfully defeated two real-world systems, the Clarifai food detection API
and the Baidu Animal Identification API.


http://arxiv.org/abs/1809.02797
Fast Gradient Attack on Network Embedding.
Jinyin Chen; Yangyang Wu; Xuanheng Xu; Yixian Chen; Haibin Zheng; Qi Xuan
  Network embedding maps a network into a low-dimensional Euclidean space, and
thus facilitate many network analysis tasks, such as node classification, link
prediction and community detection etc, by utilizing machine learning methods.
In social networks, we may pay special attention to user privacy, and would
like to prevent some target nodes from being identified by such network
analysis methods in certain cases. Inspired by successful adversarial attack on
deep learning models, we propose a framework to generate adversarial networks
based on the gradient information in Graph Convolutional Network (GCN). In
particular, we extract the gradient of pairwise nodes based on the adversarial
network, and select the pair of nodes with maximum absolute gradient to realize
the Fast Gradient Attack (FGA) and update the adversarial network. This process
is implemented iteratively and terminated until certain condition is satisfied,
i.e., the number of modified links reaches certain predefined value.
Comprehensive attacks, including unlimited attack, direct attack and indirect
attack, are performed on six well-known network embedding methods. The
experiments on real-world networks suggest that our proposed FGA behaves better
than some baseline methods, i.e., the network embedding can be easily disturbed
using FGA by only rewiring few links, achieving state-of-the-art attack
performance.


http://arxiv.org/abs/1809.02786
Structure-Preserving Transformation: Generating Diverse and Transferable Adversarial Examples.
Dan Peng; Zizhan Zheng; Xiaofeng Zhang
  Adversarial examples are perturbed inputs designed to fool machine learning
models. Most recent works on adversarial examples for image classification
focus on directly modifying pixels with minor perturbations. A common
requirement in all these works is that the malicious perturbations should be
small enough (measured by an L_p norm for some p) so that they are
imperceptible to humans. However, small perturbations can be unnecessarily
restrictive and limit the diversity of adversarial examples generated. Further,
an L_p norm based distance metric ignores important structure patterns hidden
in images that are important to human perception. Consequently, even the minor
perturbation introduced in recent works often makes the adversarial examples
less natural to humans. More importantly, they often do not transfer well and
are therefore less effective when attacking black-box models especially for
those protected by a defense mechanism. In this paper, we propose a
structure-preserving transformation (SPT) for generating natural and diverse
adversarial examples with extremely high transferability. The key idea of our
approach is to allow perceptible deviation in adversarial examples while
keeping structure patterns that are central to a human classifier. Empirical
results on the MNIST and the fashion-MNIST datasets show that adversarial
examples generated by our approach can easily bypass strong adversarial
training. Further, they transfer well to other target models with no loss or
little loss of successful attack rate.


http://arxiv.org/abs/1809.02861
Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks.
Ambra Demontis; Marco Melis; Maura Pintor; Matthew Jagielski; Battista Biggio; Alina Oprea; Cristina Nita-Rotaru; Fabio Roli
  Transferability captures the ability of an attack against a machine-learning
model to be effective against a different, potentially unknown, model.
Empirical evidence for transferability has been shown in previous work, but the
underlying reasons why an attack transfers or not are not yet well understood.
In this paper, we present a comprehensive analysis aimed to investigate the
transferability of both test-time evasion and training-time poisoning attacks.
We provide a unifying optimization framework for evasion and poisoning attacks,
and a formal definition of transferability of such attacks. We highlight two
main factors contributing to attack transferability: the intrinsic adversarial
vulnerability of the target model, and the complexity of the surrogate model
used to optimize the attack. Based on these insights, we define three metrics
that impact an attack's transferability. Interestingly, our results derived
from theoretical analysis hold for both evasion and poisoning attacks, and are
confirmed experimentally using a wide range of linear and non-linear
classifiers and datasets.


http://arxiv.org/abs/1809.02681
Open Set Adversarial Examples.
Zhedong Zheng; Liang Zheng; Zhilan Hu; Yi Yang
  Adversarial examples in recent works target at closed set recognition
systems, in which the training and testing classes are identical. In real-world
scenarios, however, the testing classes may have limited, if any, overlap with
the training classes, a problem named open set recognition. To our knowledge,
the community does not have a specific design of adversarial examples targeting
at this practical setting. Arguably, the new setting compromises traditional
closed set attack methods in two aspects. First, closed set attack methods are
based on classification and target at classification as well, but the open set
problem suggests a different task, \emph{i.e.,} retrieval. It is undesirable
that the generation mechanism of closed set recognition is different from the
aim of open set recognition. Second, given that the query image is usually of
an unseen class, predicting its category from the training classes is not
reasonable, which leads to an inferior adversarial gradient. In this work, we
view open set recognition as a retrieval task and propose a new approach,
Opposite-Direction Feature Attack (ODFA), to generate adversarial examples /
queries. When using an attacked example as query, we aim that the true matches
be ranked as low as possible. In addressing the two limitations of closed set
attack methods, ODFA directly works on the features for retrieval. The idea is
to push away the feature of the adversarial query in the opposite direction of
the original feature. Albeit simple, ODFA leads to a larger drop in Recall@K
and mAP than the close-set attack methods on two open set recognition datasets,
\emph{i.e.,} Market-1501 and CUB-200-2011. We also demonstrate that the attack
performance of ODFA is not evidently superior to the state-of-the-art methods
under closed set recognition (Cifar-10), suggesting its specificity for open
set problems.


http://arxiv.org/abs/1809.02560
A Deeper Look at 3D Shape Classifiers.
Jong-Chyi Su; Matheus Gadelha; Rui Wang; Subhransu Maji
  We investigate the role of representations and architectures for classifying
3D shapes in terms of their computational efficiency, generalization, and
robustness to adversarial transformations. By varying the number of training
examples and employing cross-modal transfer learning we study the role of
initialization of existing deep architectures for 3D shape classification. Our
analysis shows that multiview methods continue to offer the best generalization
even without pretraining on large labeled image datasets, and even when trained
on simplified inputs such as binary silhouettes. Furthermore, the performance
of voxel-based 3D convolutional networks and point-based architectures can be
improved via cross-modal transfer from image representations. Finally, we
analyze the robustness of 3D shape classifiers to adversarial transformations
and present a novel approach for generating adversarial perturbations of a 3D
shape for multiview classifiers using a differentiable renderer. We find that
point-based networks are more robust to point position perturbations while
voxel-based and multiview networks are easily fooled with the addition of
imperceptible noise to the input.


http://arxiv.org/abs/1809.02444
Metamorphic Relation Based Adversarial Attacks on Differentiable Neural Computer.
Alvin Chan; Lei Ma; Felix Juefei-Xu; Xiaofei Xie; Yang Liu; Yew Soon Ong
  Deep neural networks (DNN), while becoming the driving force of many novel
technology and achieving tremendous success in many cutting-edge applications,
are still vulnerable to adversarial attacks. Differentiable neural computer
(DNC) is a novel computing machine with DNN as its central controller operating
on an external memory module for data processing. The unique architecture of
DNC contributes to its state-of-the-art performance in tasks which requires the
ability to represent variables and data structure as well as to store data over
long timescales. However, there still lacks a comprehensive study on how
adversarial examples affect DNC in terms of robustness. In this paper, we
propose metamorphic relation based adversarial techniques for a range of tasks
described in the natural processing language domain. We show that the
near-perfect performance of the DNC in bAbI logical question answering tasks
can be degraded by adversarially injected sentences. We further perform
in-depth study on the role of DNC's memory size in its robustness and analyze
the potential reason causing why DNC fails. Our study demonstrates the current
challenges and potential opportunities towards constructing more robust DNCs.


http://arxiv.org/abs/1809.02701
Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering.
Eric Wallace; Pedro Rodriguez; Shi Feng; Ikuya Yamada; Jordan Boyd-Graber
  Adversarial evaluation stress tests a model's understanding of natural
language. While past approaches expose superficial patterns, the resulting
adversarial examples are limited in complexity and diversity. We propose
human-in-the-loop adversarial generation, where human authors are guided to
break models. We aid the authors with interpretations of model predictions
through an interactive user interface. We apply this generation framework to a
question answering task called Quizbowl, where trivia enthusiasts craft
adversarial questions. The resulting questions are validated via live
human--computer matches: although the questions appear ordinary to humans, they
systematically stump neural and information retrieval models. The adversarial
questions cover diverse phenomena from multi-hop reasoning to entity type
distractors, exposing open challenges in robust question answering.


http://arxiv.org/abs/1809.02104
Are adversarial examples inevitable?.
Ali Shafahi; W. Ronny Huang; Christoph Studer; Soheil Feizi; Tom Goldstein
  A wide range of defenses have been proposed to harden neural networks against
adversarial attacks. However, a pattern has emerged in which the majority of
adversarial defenses are quickly broken by new attacks. Given the lack of
success at generating robust defenses, we are led to ask a fundamental
question: Are adversarial attacks inevitable? This paper analyzes adversarial
examples from a theoretical perspective, and identifies fundamental bounds on
the susceptibility of a classifier to adversarial attacks. We show that, for
certain classes of problems, adversarial examples are inescapable. Using
experiments, we explore the implications of theoretical guarantees for
real-world problems and discuss how factors such as dimensionality and image
complexity limit a classifier's robustness against adversarial examples.


http://arxiv.org/abs/1809.02079
Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models.
Tong Niu; Mohit Bansal
  We present two categories of model-agnostic adversarial strategies that
reveal the weaknesses of several generative, task-oriented dialogue models:
Should-Not-Change strategies that evaluate over-sensitivity to small and
semantics-preserving edits, as well as Should-Change strategies that test if a
model is over-stable against subtle yet semantics-changing modifications. We
next perform adversarial training with each strategy, employing a max-margin
approach for negative generative examples. This not only makes the target
dialogue model more robust to the adversarial inputs, but also helps it perform
significantly better on the original inputs. Moreover, training on all
strategies combined achieves further improvements, achieving a new
state-of-the-art performance on the original task (also verified via human
evaluation). In addition to adversarial training, we also address the
robustness task at the model-level, by feeding it subword units as both inputs
and outputs, and show that the resulting model is equally competitive, requires
only 1/4 of the original vocabulary size, and is robust to one of the
adversarial strategies (to which the original model is vulnerable) even without
adversarial training.


http://arxiv.org/abs/1809.01829
Adversarial Reprogramming of Sequence Classification Neural Networks.
Paarth Neekhara; Shehzeen Hussain; Shlomo Dubnov; Farinaz Koushanfar
  Adversarial Reprogramming has demonstrated success in utilizing pre-trained
neural network classifiers for alternative classification tasks without
modification to the original network. An adversary in such an attack scenario
trains an additive contribution to the inputs to repurpose the neural network
for the new classification task. While this reprogramming approach works for
neural networks with a continuous input space such as that of images, it is not
directly applicable to neural networks trained for tasks such as text
classification, where the input space is discrete. Repurposing such
classification networks would require the attacker to learn an adversarial
program that maps inputs from one discrete space to the other. In this work, we
introduce a context-based vocabulary remapping model to reprogram neural
networks trained on a specific sequence classification task, for a new sequence
classification task desired by the adversary. We propose training procedures
for this adversarial program in both white-box and black-box settings. We
demonstrate the application of our model by adversarially repurposing various
text-classification models including LSTM, bi-directional LSTM and CNN for
alternate classification tasks.


http://arxiv.org/abs/1809.02077
IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detection.
Zilong Lin; Yong Shi; Zhi Xue
  As an important tool in security, the intrusion detection system bears the
responsibility of the defense to network attacks performed by malicious
traffic. Nowadays, with the help of machine learning algorithms, the intrusion
detection system develops rapidly. However, the robustness of this system is
questionable when it faces the adversarial attacks. To improve the detection
system, more potential attack approaches should be researched. In this paper, a
framework of the generative adversarial networks, IDSGAN, is proposed to
generate the adversarial attacks, which can deceive and evade the intrusion
detection system. Considering that the internal structure of the detection
system is unknown to attackers, adversarial attack examples perform the
black-box attacks against the detection system. IDSGAN leverages a generator to
transform original malicious traffic into adversarial malicious traffic. A
discriminator classifies traffic examples and simulates the black-box detection
system. More significantly, we only modify part of the attacks' nonfunctional
features to guarantee the validity of the intrusion. Based on the dataset
NSL-KDD, the feasibility of the model is demonstrated to attack many detection
systems with different attacks and the excellent results are achieved.
Moreover, the robustness of IDSGAN is verified by changing the amount of the
unmodified features.


http://arxiv.org/abs/1809.01715
Bridging machine learning and cryptography in defence against adversarial attacks.
Olga Taran; Shideh Rezaeifar; Slava Voloshynovskiy
  In the last decade, deep learning algorithms have become very popular thanks
to the achieved performance in many machine learning and computer vision tasks.
However, most of the deep learning architectures are vulnerable to so called
adversarial examples. This questions the security of deep neural networks (DNN)
for many security- and trust-sensitive domains. The majority of the proposed
existing adversarial attacks are based on the differentiability of the DNN cost
function.Defence strategies are mostly based on machine learning and signal
processing principles that either try to detect-reject or filter out the
adversarial perturbations and completely neglect the classical cryptographic
component in the defence. In this work, we propose a new defence mechanism
based on the second Kerckhoffs's cryptographic principle which states that the
defence and classification algorithm are supposed to be known, but not the key.
To be compliant with the assumption that the attacker does not have access to
the secret key, we will primarily focus on a gray-box scenario and do not
address a white-box one. More particularly, we assume that the attacker does
not have direct access to the secret block, but (a) he completely knows the
system architecture, (b) he has access to the data used for training and
testing and (c) he can observe the output of the classifier for each given
input. We show empirically that our system is efficient against most famous
state-of-the-art attacks in black-box and gray-box scenarios.


http://arxiv.org/abs/1809.01093
Adversarial Attacks on Node Embeddings.
Aleksandar Bojchevski; Stephan Günnemann
  The goal of network representation learning is to learn low-dimensional node
embeddings that capture the graph structure and are useful for solving
downstream tasks. However, despite the proliferation of such methods there is
currently no study of their robustness to adversarial attacks. We provide the
first adversarial vulnerability analysis on the widely used family of methods
based on random walks. We derive efficient adversarial perturbations that
poison the network structure and have a negative effect on both the quality of
the embeddings and the downstream tasks. We further show that our attacks are
transferable -- they generalize to many models -- and are successful even when
the attacker has restricted actions.


http://arxiv.org/abs/1809.00594
Adversarial Attack Type I: Generating False Positives.
Sanli Tang; Xiaolin Huang; Mingjian Chen; Jie Yang
  False positive and false negative rates are equally important for evaluating
the performance of a classifier. Adversarial examples by increasing false
negative rate have been studied in recent years. However, harming a classifier
by increasing false positive rate is almost blank, since it is much more
difficult to generate a new and meaningful positive than the negative. To
generate false positives, a supervised generative framework is proposed in this
paper. Experiment results show that our method is practical and effective to
generate those adversarial examples on large-scale image datasets.


http://arxiv.org/abs/1809.01697
HASP: A High-Performance Adaptive Mobile Security Enhancement Against Malicious Speech Recognition.
Zirui Xu; Fuxun Yu; Chenchen Liu; Xiang Chen
  Nowadays, machine learning based Automatic Speech Recognition (ASR) technique
has widely spread in smartphones, home devices, and public facilities. As
convenient as this technology can be, a considerable security issue also raises
-- the users' speech content might be exposed to malicious ASR monitoring and
cause severe privacy leakage. In this work, we propose HASP -- a
high-performance security enhancement approach to solve this security issue on
mobile devices. Leveraging ASR systems' vulnerability to the adversarial
examples, HASP is designed to cast human imperceptible adversarial noises to
real-time speech and effectively perturb malicious ASR monitoring by increasing
the Word Error Rate (WER). To enhance the practical performance on mobile
devices, HASP is also optimized for effective adaptation to the human speech
characteristics, environmental noises, and mobile computation scenarios. The
experiments show that HASP can achieve optimal real-time security enhancement:
it can lead an average WER of 84.55% for perturbing the malicious ASR
monitoring, and the data processing speed is 15x to 40x faster compared to the
state-of-the-art methods. Moreover, HASP can effectively perturb various ASR
systems, demonstrating a strong transferability.


http://arxiv.org/abs/1809.00065
MULDEF: Multi-model-based Defense Against Adversarial Examples for Neural Networks.
Siwakorn Srisakaokul; Yuhao Zhang; Zexuan Zhong; Wei Yang; Tao Xie
  Despite being popularly used in many application domains, neural network
models have been found to be vulnerable to adversarial examples, examples
formed by applying imperceptible perturbation on legitimate examples from the
datasets. Adversarial examples can pose potential risks on safety and security
of real-world applications. However, existing defense approaches are still
vulnerable to adversarial examples, especially in a white-box attack scenario.
To address this problem, we propose a new defense approach, named MULDEF, based
on robustness diversity. Our approach consists of (1) a general defense
framework based on multiple models and (2) a technique for generating these
multiple models to achieve high defense capability. In particular, given a
target model to defend, our framework includes multiple models (constructed
from the target model) to form a model family. The model family is designed to
achieve robustness diversity (i.e., an adversarial example successfully
attacking one model cannot succeed in attacking other models in the family). At
runtime, a model is randomly selected from the family to be applied on each
input example. Our general framework can inspire rich future research to
construct a desirable model family achieving higher robustness diversity. Our
evaluation results show that MULDEF (with only up to 5 models in the family)
can already substantially improve the target model's accuracy on adversarial
examples by 35-74% in a white-box attack scenario, while maintaining similar
accuracy on legitimate examples as the target model.


http://arxiv.org/abs/1808.09413
DLFuzz: Differential Fuzzing Testing of Deep Learning Systems.
Jianmin Guo; Yu Jiang; Yue Zhao; Quan Chen; Jiaguang Sun
  Deep learning (DL) systems are increasingly applied to safety-critical
domains such as autonomous driving cars. It is of significant importance to
ensure the reliability and robustness of DL systems. Existing testing
methodologies always fail to include rare inputs in the testing dataset and
exhibit low neuron coverage. In this paper, we propose DLFuzz, the frst
differential fuzzing testing framework to guide DL systems exposing incorrect
behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron
coverage and the prediction difference between the original input and the
mutated input, without manual labeling effort or cross-referencing oracles from
other DL systems with the same functionality. We present empirical evaluations
on two well-known datasets to demonstrate its efficiency. Compared with
DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not
require extra efforts to find similar functional DL systems for
cross-referencing check, but could generate 338.59% more adversarial inputs
with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron
coverage, and save 20.11% time consumption.


http://arxiv.org/abs/1808.09115
All You Need is "Love": Evading Hate-speech Detection.
Tommi Gröndahl; Luca Pajola; Mika Juuti; Mauro Conti; N. Asokan
  With the spread of social networks and their unfortunate use for hate speech,
automatic detection of the latter has become a pressing problem. In this paper,
we reproduce seven state-of-the-art hate speech detection models from prior
work, and show that they perform well only when tested on the same type of data
they were trained on. Based on these results, we argue that for successful hate
speech detection, model architecture is less important than the type of data
and labeling criteria. We further show that all proposed detection techniques
are brittle against adversaries who can (automatically) insert typos, change
word boundaries or add innocuous words to the original hate speech. A
combination of these methods is also effective against Google Perspective -- a
cutting-edge solution from industry. Our experiments demonstrate that
adversarial training does not completely mitigate the attacks, and using
character-level features makes the models systematically more attack-resistant
than using word-level features.


http://arxiv.org/abs/1809.00958
Targeted Nonlinear Adversarial Perturbations in Images and Videos.
Roberto Rey-de-Castro; Herschel Rabitz
  We introduce a method for learning adversarial perturbations targeted to
individual images or videos. The learned perturbations are found to be sparse
while at the same time containing a high level of feature detail. Thus, the
extracted perturbations allow a form of object or action recognition and
provide insights into what features the studied deep neural network models
consider important when reaching their classification decisions. From an
adversarial point of view, the sparse perturbations successfully confused the
models into misclassifying, although the perturbed samples still belonged to
the same original class by visual examination. This is discussed in terms of a
prospective data augmentation scheme. The sparse yet high-quality perturbations
may also be leveraged for image or video compression.


http://arxiv.org/abs/1808.08750
Generalisation in humans and deep neural networks.
Robert Geirhos; Carlos R. Medina Temme; Jonas Rauber; Heiko H. Schütt; Matthias Bethge; Felix A. Wichmann
  We compare the robustness of humans and current convolutional deep neural
networks (DNNs) on object recognition under twelve different types of image
degradations. First, using three well known DNNs (ResNet-152, VGG-19,
GoogLeNet) we find the human visual system to be more robust to nearly all of
the tested image manipulations, and we observe progressively diverging
classification error-patterns between humans and DNNs when the signal gets
weaker. Secondly, we show that DNNs trained directly on distorted images
consistently surpass human performance on the exact distortion types they were
trained on, yet they display extremely poor generalisation abilities when
tested on other distortion types. For example, training on salt-and-pepper
noise does not imply robustness on uniform white noise and vice versa. Thus,
changes in the noise distribution between training and testing constitutes a
crucial challenge to deep learning vision systems that can be systematically
addressed in a lifelong machine learning approach. Our new dataset consisting
of 83K carefully measured human psychophysical trials provide a useful
reference for lifelong robustness against image degradations set by the human
visual system.


http://arxiv.org/abs/1808.08609
Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge.
Pasquale Minervini; Sebastian Riedel
  Adversarial examples are inputs to machine learning models designed to cause
the model to make a mistake. They are useful for understanding the shortcomings
of machine learning models, interpreting their results, and for regularisation.
In NLP, however, most example generation strategies produce input text by using
known, pre-specified semantic transformations, requiring significant manual
effort and in-depth understanding of the problem and domain. In this paper, we
investigate the problem of automatically generating adversarial examples that
violate a set of given First-Order Logic constraints in Natural Language
Inference (NLI). We reduce the problem of identifying such adversarial examples
to a combinatorial optimisation problem, by maximising a quantity measuring the
degree of violation of such constraints and by using a language model for
generating linguistically-plausible examples. Furthermore, we propose a method
for adversarially regularising neural NLI models for incorporating background
knowledge. Our results show that, while the proposed method does not always
improve results on the SNLI and MultiNLI datasets, it significantly and
consistently increases the predictive accuracy on adversarially-crafted
datasets -- up to a 79.6% relative improvement -- while drastically reducing
the number of background knowledge violations. Furthermore, we show that
adversarial examples transfer among model architectures, and that the proposed
adversarial training procedure improves the robustness of NLI models to
adversarial examples.


http://arxiv.org/abs/1808.08444
Guiding Deep Learning System Testing using Surprise Adequacy.
Jinhan Kim; Robert Feldt; Shin Yoo
  Deep Learning (DL) systems are rapidly being adopted in safety and security
critical domains, urgently calling for ways to test their correctness and
robustness. Testing of DL systems has traditionally relied on manual collection
and labelling of data. Recently, a number of coverage criteria based on neuron
activation values have been proposed. These criteria essentially count the
number of neurons whose activation during the execution of a DL system
satisfied certain properties, such as being above predefined thresholds.
However, existing coverage criteria are not sufficiently fine grained to
capture subtle behaviours exhibited by DL systems. Moreover, evaluations have
focused on showing correlation between adversarial examples and proposed
criteria rather than evaluating and guiding their use for actual testing of DL
systems. We propose a novel test adequacy criterion for testing of DL systems,
called Surprise Adequacy for Deep Learning Systems (SADL), which is based on
the behaviour of DL systems with respect to their training data. We measure the
surprise of an input as the difference in DL system's behaviour between the
input and the training data (i.e., what was learnt during training), and
subsequently develop this as an adequacy criterion: a good test input should be
sufficiently but not overtly surprising compared to training data. Empirical
evaluation using a range of DL systems from simple image classifiers to
autonomous driving car platforms shows that systematic sampling of inputs based
on their surprise can improve classification accuracy of DL systems against
adversarial examples by up to 77.5% via retraining.


http://arxiv.org/abs/1808.08426
Analysis of adversarial attacks against CNN-based image forgery detectors.
Diego Gragnaniello; Francesco Marra; Giovanni Poggi; Luisa Verdoliva
  With the ubiquitous diffusion of social networks, images are becoming a
dominant and powerful communication channel. Not surprisingly, they are also
increasingly subject to manipulations aimed at distorting information and
spreading fake news. In recent years, the scientific community has devoted
major efforts to contrast this menace, and many image forgery detectors have
been proposed. Currently, due to the success of deep learning in many
multimedia processing tasks, there is high interest towards CNN-based
detectors, and early results are already very promising. Recent studies in
computer vision, however, have shown CNNs to be highly vulnerable to
adversarial attacks, small perturbations of the input data which drive the
network towards erroneous classification. In this paper we analyze the
vulnerability of CNN-based image forensics methods to adversarial attacks,
considering several detectors and several types of attack, and testing
performance on a wide range of common manipulations, both easily and hardly
detectable.


http://arxiv.org/abs/1808.08197
Is Machine Learning in Power Systems Vulnerable?.
Yize Chen; Yushi Tan; Deepjyoti Deka
  Recent advances in Machine Learning(ML) have led to its broad adoption in a
series of power system applications, ranging from meter data analytics,
renewable/load/price forecasting to grid security assessment. Although these
data-driven methods yield state-of-the-art performances in many tasks, the
robustness and security of applying such algorithms in modern power grids have
not been discussed. In this paper, we attempt to address the issues regarding
the security of ML applications in power systems. We first show that most of
the current ML algorithms proposed in power systems are vulnerable to
adversarial examples, which are maliciously crafted input data. We then adopt
and extend a simple yet efficient algorithm for finding subtle perturbations,
which could be used for generating adversaries for both categorical(e.g., user
load profile classification) and sequential applications(e.g., renewables
generation forecasting). Case studies on classification of power quality
disturbances and forecast of building loads demonstrate the vulnerabilities of
current ML algorithms in power networks under our adversarial designs. These
vulnerabilities call for design of robust and secure ML algorithms for real
world applications.


http://arxiv.org/abs/1808.07945
Maximal Jacobian-based Saliency Map Attack.
Rey Wiyatno; Anqi Xu
  The Jacobian-based Saliency Map Attack is a family of adversarial attack
methods for fooling classification models, such as deep neural networks for
image classification tasks. By saturating a few pixels in a given image to
their maximum or minimum values, JSMA can cause the model to misclassify the
resulting adversarial image as a specified erroneous target class. We propose
two variants of JSMA, one which removes the requirement to specify a target
class, and another that additionally does not need to specify whether to only
increase or decrease pixel intensities. Our experiments highlight the
competitive speeds and qualities of these variants when applied to datasets of
hand-written digits and natural scenes.


http://arxiv.org/abs/1808.07713
Adversarial Attacks on Deep-Learning Based Radio Signal Classification.
Meysam Sadeghi; Erik G. Larsson
  Deep learning (DL), despite its enormous success in many computer vision and
language processing applications, is exceedingly vulnerable to adversarial
attacks. We consider the use of DL for radio signal (modulation) classification
tasks, and present practical methods for the crafting of white-box and
universal black-box adversarial attacks in that application. We show that these
attacks can considerably reduce the classification performance, with extremely
small perturbations of the input. In particular, these attacks are
significantly more powerful than classical jamming attacks, which raises
significant security and robustness concerns in the use of DL-based algorithms
for the wireless physical layer.


http://arxiv.org/abs/1808.08282
Controlling Over-generalization and its Effect on Adversarial Examples Generation and Detection.
Mahdieh Abbasi; Arezoo Rajabi; Azadeh Sadat Mozafari; Rakesh B. Bobba; Christian Gagne
  Convolutional Neural Networks (CNNs) significantly improve the
state-of-the-art for many applications, especially in computer vision. However,
CNNs still suffer from a tendency to confidently classify out-distribution
samples from unknown classes into pre-defined known classes. Further, they are
also vulnerable to adversarial examples. We are relating these two issues
through the tendency of CNNs to over-generalize for areas of the input space
not covered well by the training set. We show that a CNN augmented with an
extra output class can act as a simple yet effective end-to-end model for
controlling over-generalization. As an appropriate training set for the extra
class, we introduce two resources that are computationally efficient to obtain:
a representative natural out-distribution set and interpolated in-distribution
samples. To help select a representative natural out-distribution set among
available ones, we propose a simple measurement to assess an out-distribution
set's fitness. We also demonstrate that training such an augmented CNN with
representative out-distribution natural datasets and some interpolated samples
allows it to better handle a wide range of unseen out-distribution samples and
black-box adversarial examples without training it on any adversaries. Finally,
we show that generation of white-box adversarial attacks using our proposed
augmented CNN can become harder, as the attack algorithms have to get around
the rejection regions when generating actual adversaries.


http://arxiv.org/abs/1808.06645
Stochastic Combinatorial Ensembles for Defending Against Adversarial Examples.
George A. Adam; Petr Smirnov; David Duvenaud; Benjamin Haibe-Kains; Anna Goldenberg
  Many deep learning algorithms can be easily fooled with simple adversarial
examples. To address the limitations of existing defenses, we devised a
probabilistic framework that can generate an exponentially large ensemble of
models from a single model with just a linear cost. This framework takes
advantage of neural network depth and stochastically decides whether or not to
insert noise removal operators such as VAEs between layers. We show empirically
the important role that model gradients have when it comes to determining
transferability of adversarial examples, and take advantage of this result to
demonstrate that it is possible to train models with limited adversarial attack
transferability. Additionally, we propose a detection method based on metric
learning in order to detect adversarial examples that have no hope of being
cleaned of maliciously engineered noise.


http://arxiv.org/abs/1808.05770
Reinforcement Learning for Autonomous Defence in Software-Defined Networking.
Yi Han; Benjamin I. P. Rubinstein; Tamas Abraham; Tansu Alpcan; Vel Olivier De; Sarah Erfani; David Hubczenko; Christopher Leckie; Paul Montague
  Despite the successful application of machine learning (ML) in a wide range
of domains, adaptability---the very property that makes machine learning
desirable---can be exploited by adversaries to contaminate training and evade
classification. In this paper, we investigate the feasibility of applying a
specific class of machine learning algorithms, namely, reinforcement learning
(RL) algorithms, for autonomous cyber defence in software-defined networking
(SDN). In particular, we focus on how an RL agent reacts towards different
forms of causative attacks that poison its training process, including
indiscriminate and targeted, white-box and black-box attacks. In addition, we
also study the impact of the attack timing, and explore potential
countermeasures such as adversarial training.


http://arxiv.org/abs/1808.05705
Mitigation of Adversarial Attacks through Embedded Feature Selection.
Ziyi Bao; Luis Muñoz-González; Emil C. Lupu
  Machine learning has become one of the main components for task automation in
many application domains. Despite the advancements and impressive achievements
of machine learning, it has been shown that learning algorithms can be
compromised by attackers both at training and test time. Machine learning
systems are especially vulnerable to adversarial examples where small
perturbations added to the original data points can produce incorrect or
unexpected outputs in the learning algorithms at test time. Mitigation of these
attacks is hard as adversarial examples are difficult to detect. Existing
related work states that the security of machine learning systems against
adversarial examples can be weakened when feature selection is applied to
reduce the systems' complexity. In this paper, we empirically disprove this
idea, showing that the relative distortion that the attacker has to introduce
to succeed in the attack is greater when the target is using a reduced set of
features. We also show that the minimal adversarial examples differ
statistically more strongly from genuine examples with a lower number of
features. However, reducing the feature count can negatively impact the
system's performance. We illustrate the trade-off between security and accuracy
with specific examples. We propose a design methodology to evaluate the
security of machine learning classifiers with embedded feature selection
against adversarial examples crafted using different attack strategies.


http://arxiv.org/abs/1808.05665
Adversarial Attacks Against Automatic Speech Recognition Systems via Psychoacoustic Hiding.
Lea Schönherr; Katharina Kohls; Steffen Zeiler; Thorsten Holz; Dorothea Kolossa
  Voice interfaces are becoming accepted widely as input methods for a diverse
set of devices. This development is driven by rapid improvements in automatic
speech recognition (ASR), which now performs on par with human listening in
many tasks. These improvements base on an ongoing evolution of DNNs as the
computational core of ASR. However, recent research results show that DNNs are
vulnerable to adversarial perturbations, which allow attackers to force the
transcription into a malicious output.
  In this paper, we introduce a new type of adversarial examples based on
psychoacoustic hiding. Our attack exploits the characteristics of DNN-based ASR
systems, where we extend the original analysis procedure by an additional
backpropagation step. We use this backpropagation to learn the degrees of
freedom for the adversarial perturbation of the input signal, i.e., we apply a
psychoacoustic model and manipulate the acoustic signal below the thresholds of
human perception. To further minimize the perceptibility of the perturbations,
we use forced alignment to find the best fitting temporal alignment between the
original audio sample and the malicious target transcription. These extensions
allow us to embed an arbitrary audio input with a malicious voice command that
is then transcribed by the ASR system, with the audio signal remaining barely
distinguishable from the original signal. In an experimental evaluation, we
attack the state-of-the-art speech recognition system Kaldi and determine the
best performing parameter and analysis setup for different types of input. Our
results show that we are successful in up to 98% of cases with a computational
effort of fewer than two minutes for a ten-second audio file. Based on user
studies, we found that none of our target transcriptions were audible to human
listeners, who still understand the original speech content with unchanged
accuracy.


http://arxiv.org/abs/1808.05537
Distributionally Adversarial Attack.
Tianhang Zheng; Changyou Chen; Kui Ren
  Recent work on adversarial attack has shown that Projected Gradient Descent
(PGD) Adversary is a universal first-order adversary, and the classifier
adversarially trained by PGD is robust against a wide range of first-order
attacks. It is worth noting that the original objective of an attack/defense
model relies on a data distribution $p(\mathbf{x})$, typically in the form of
risk maximization/minimization, e.g.,
$\max/\min\mathbb{E}_{p(\mathbf(x))}\mathcal{L}(\mathbf{x})$ with
$p(\mathbf{x})$ some unknown data distribution and $\mathcal{L}(\cdot)$ a loss
function. However, since PGD generates attack samples independently for each
data sample based on $\mathcal{L}(\cdot)$, the procedure does not necessarily
lead to good generalization in terms of risk optimization. In this paper, we
achieve the goal by proposing distributionally adversarial attack (DAA), a
framework to solve an optimal {\em adversarial-data distribution}, a perturbed
distribution that satisfies the $L_\infty$ constraint but deviates from the
original data distribution to increase the generalization risk maximally.
Algorithmically, DAA performs optimization on the space of potential data
distributions, which introduces direct dependency between all data points when
generating adversarial samples. DAA is evaluated by attacking state-of-the-art
defense models, including the adversarially-trained models provided by {\em MIT
MadryLab}. Notably, DAA ranks {\em the first place} on MadryLab's white-box
leaderboards, reducing the accuracy of their secret MNIST model to $88.79\%$
(with $l_\infty$ perturbations of $\epsilon = 0.3$) and the accuracy of their
secret CIFAR model to $44.71\%$ (with $l_\infty$ perturbations of $\epsilon =
8.0$). Code for the experiments is released on
\url{https://github.com/tianzheng4/Distributionally-Adversarial-Attack}.


http://arxiv.org/abs/1808.04218
Android HIV: A Study of Repackaging Malware for Evading Machine-Learning Detection.
Xiao Chen; Chaoran Li; Derui Wang; Sheng Wen; Jun Zhang; Surya Nepal; Yang Xiang; Kui Ren
  Machine learning based solutions have been successfully employed for
automatic detection of malware in Android applications. However, machine
learning models are known to lack robustness against inputs crafted by an
adversary. So far, the adversarial examples can only deceive Android malware
detectors that rely on syntactic features, and the perturbations can only be
implemented by simply modifying Android manifest. While recent Android malware
detectors rely more on semantic features from Dalvik bytecode rather than
manifest, existing attacking/defending methods are no longer effective. In this
paper, we introduce a new highly-effective attack that generates adversarial
examples of Android malware and evades being detected by the current models. To
this end, we propose a method of applying optimal perturbations onto Android
APK using a substitute model. Based on the transferability concept, the
perturbations that successfully deceive the substitute model are likely to
deceive the original models as well. We develop an automated tool to generate
the adversarial examples without human intervention to apply the attacks. In
contrast to existing works, the adversarial examples crafted by our method can
also deceive recent machine learning based detectors that rely on semantic
features such as control-flow-graph. The perturbations can also be implemented
directly onto APK's Dalvik bytecode rather than Android manifest to evade from
recent detectors. We evaluated the proposed manipulation methods for
adversarial examples by using the same datasets that Drebin and MaMadroid (5879
malware samples) used. Our results show that, the malware detection rates
decreased from 96% to 1% in MaMaDroid, and from 97% to 1% in Drebin, with just
a small distortion generated by our adversarial examples manipulation method.


http://arxiv.org/abs/1808.03601
Using Randomness to Improve Robustness of Machine-Learning Models Against Evasion Attacks.
Fan Yang; Zhiyuan Chen
  Machine learning models have been widely used in security applications such
as intrusion detection, spam filtering, and virus or malware detection.
However, it is well-known that adversaries are always trying to adapt their
attacks to evade detection. For example, an email spammer may guess what
features spam detection models use and modify or remove those features to avoid
detection. There has been some work on making machine learning models more
robust to such attacks. However, one simple but promising approach called {\em
randomization} is underexplored. This paper proposes a novel
randomization-based approach to improve robustness of machine learning models
against evasion attacks. The proposed approach incorporates randomization into
both model training time and model application time (meaning when the model is
used to detect attacks). We also apply this approach to random forest, an
existing ML method which already has some degree of randomness. Experiments on
intrusion detection and spam filtering data show that our approach further
improves robustness of random-forest method. We also discuss how this approach
can be applied to other ML models.


http://arxiv.org/abs/1808.02651
Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer.
Hsueh-Ti Derek Liu; Michael Tao; Chun-Liang Li; Derek Nowrouzezahrai; Alec Jacobson
  Many machine learning image classifiers are vulnerable to adversarial
attacks, inputs with perturbations designed to intentionally trigger
misclassification. Current adversarial methods directly alter pixel colors and
evaluate against pixel norm-balls: pixel perturbations smaller than a specified
magnitude, according to a measurement norm. This evaluation, however, has
limited practical utility since perturbations in the pixel space do not
correspond to underlying real-world phenomena of image formation that lead to
them and has no security motivation attached. Pixels in natural images are
measurements of light that has interacted with the geometry of a physical
scene. As such, we propose the direct perturbation of physical parameters that
underly image formation: lighting and geometry. As such, we propose a novel
evaluation measure, parametric norm-balls, by directly perturbing physical
parameters that underly image formation. One enabling contribution we present
is a physically-based differentiable renderer that allows us to propagate pixel
gradients to the parametric space of lighting and geometry. Our approach
enables physically-based adversarial attacks, and our differentiable renderer
leverages models from the interactive rendering literature to balance the
performance and accuracy trade-offs necessary for a memory-efficient and
scalable adversarial data augmentation workflow.


http://arxiv.org/abs/1808.02455
Data augmentation using synthetic data for time series classification with deep residual networks.
Hassan Ismail Fawaz; Germain Forestier; Jonathan Weber; Lhassane Idoumghar; Pierre-Alain Muller
  Data augmentation in deep neural networks is the process of generating
artificial data in order to reduce the variance of the classifier with the goal
to reduce the number of errors. This idea has been shown to improve deep neural
network's generalization capabilities in many computer vision tasks such as
image recognition and object localization. Apart from these applications, deep
Convolutional Neural Networks (CNNs) have also recently gained popularity in
the Time Series Classification (TSC) community. However, unlike in image
recognition problems, data augmentation techniques have not yet been
investigated thoroughly for the TSC task. This is surprising as the accuracy of
deep learning models for TSC could potentially be improved, especially for
small datasets that exhibit overfitting, when a data augmentation method is
adopted. In this paper, we fill this gap by investigating the application of a
recently proposed data augmentation technique based on the Dynamic Time Warping
distance, for a deep learning model for TSC. To evaluate the potential of
augmenting the training set, we performed extensive experiments using the UCR
TSC benchmark. Our preliminary experiments reveal that data augmentation can
drastically increase deep CNN's accuracy on some datasets and significantly
improve the deep model's accuracy when the method is used in an ensemble
approach.


http://arxiv.org/abs/1808.01976
Adversarial Vision Challenge.
Wieland Brendel; Jonas Rauber; Alexey Kurakin; Nicolas Papernot; Behar Veliqi; Marcel Salathé; Sharada P. Mohanty; Matthias Bethge
  The NIPS 2018 Adversarial Vision Challenge is a competition to facilitate
measurable progress towards robust machine vision models and more generally
applicable adversarial attacks. This document is an updated version of our
competition proposal that was accepted in the competition track of 32nd
Conference on Neural Information Processing Systems (NIPS 2018).


http://arxiv.org/abs/1808.01785
Defense Against Adversarial Attacks with Saak Transform.
Sibo Song; Yueru Chen; Ngai-Man Cheung; C. -C. Jay Kuo
  Deep neural networks (DNNs) are known to be vulnerable to adversarial
perturbations, which imposes a serious threat to DNN-based decision systems. In
this paper, we propose to apply the lossy Saak transform to adversarially
perturbed images as a preprocessing tool to defend against adversarial attacks.
Saak transform is a recently-proposed state-of-the-art for computing the
spatial-spectral representations of input images. Empirically, we observe that
outputs of the Saak transform are very discriminative in differentiating
adversarial examples from clean ones. Therefore, we propose a Saak transform
based preprocessing method with three steps: 1) transforming an input image to
a joint spatial-spectral representation via the forward Saak transform, 2)
apply filtering to its high-frequency components, and, 3) reconstructing the
image via the inverse Saak transform. The processed image is found to be robust
against adversarial perturbations. We conduct extensive experiments to
investigate various settings of the Saak transform and filtering functions.
Without harming the decision performance on clean images, our method
outperforms state-of-the-art adversarial defense methods by a substantial
margin on both the CIFAR-10 and ImageNet datasets. Importantly, our results
suggest that adversarial perturbations can be effectively and efficiently
defended using state-of-the-art frequency analysis.


http://arxiv.org/abs/1808.01753
Gray-box Adversarial Training.
Vivek B. S.; Konda Reddy Mopuri; R. Venkatesh Babu
  Adversarial samples are perturbed inputs crafted to mislead the machine
learning systems. A training mechanism, called adversarial training, which
presents adversarial samples along with clean samples has been introduced to
learn robust models. In order to scale adversarial training for large datasets,
these perturbations can only be crafted using fast and simple methods (e.g.,
gradient ascent). However, it is shown that adversarial training converges to a
degenerate minimum, where the model appears to be robust by generating weaker
adversaries. As a result, the models are vulnerable to simple black-box
attacks. In this paper we, (i) demonstrate the shortcomings of existing
evaluation policy, (ii) introduce novel variants of white-box and black-box
attacks, dubbed gray-box adversarial attacks" based on which we propose novel
evaluation method to assess the robustness of the learned models, and (iii)
propose a novel variant of adversarial training, named Graybox Adversarial
Training" that uses intermediate versions of the models to seed the
adversaries. Experimental evaluation demonstrates that the models trained using
our method exhibit better robustness compared to both undefended and
adversarially trained model


http://arxiv.org/abs/1808.01688
Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models.
Dong Su; Huan Zhang; Hongge Chen; Jinfeng Yi; Pin-Yu Chen; Yupeng Gao
  The prediction accuracy has been the long-lasting and sole standard for
comparing the performance of different image classification models, including
the ImageNet competition. However, recent studies have highlighted the lack of
robustness in well-trained deep neural networks to adversarial examples.
Visually imperceptible perturbations to natural images can easily be crafted
and mislead the image classifiers towards misclassification. To demystify the
trade-offs between robustness and accuracy, in this paper we thoroughly
benchmark 18 ImageNet models using multiple robustness metrics, including the
distortion, success rate and transferability of adversarial examples between
306 pairs of models. Our extensive experimental results reveal several new
insights: (1) linear scaling law - the empirical $\ell_2$ and $\ell_\infty$
distortion metrics scale linearly with the logarithm of classification error;
(2) model architecture is a more critical factor to robustness than model size,
and the disclosed accuracy-robustness Pareto frontier can be used as an
evaluation criterion for ImageNet model designers; (3) for a similar network
architecture, increasing network depth slightly improves robustness in
$\ell_\infty$ distortion; (4) there exist models (in VGG family) that exhibit
high adversarial transferability, while most adversarial examples crafted from
one model can only be transferred within the same family. Experiment code is
publicly available at \url{https://github.com/huanzhang12/Adversarial_Survey}.


http://arxiv.org/abs/1808.01664
Structured Adversarial Attack: Towards General Implementation and Better Interpretability.
Kaidi Xu; Sijia Liu; Pu Zhao; Pin-Yu Chen; Huan Zhang; Quanfu Fan; Deniz Erdogmus; Yanzhi Wang; Xue Lin
  When generating adversarial examples to attack deep neural networks (DNNs),
Lp norm of the added perturbation is usually used to measure the similarity
between original image and adversarial example. However, such adversarial
attacks perturbing the raw input spaces may fail to capture structural
information hidden in the input. This work develops a more general attack
model, i.e., the structured attack (StrAttack), which explores group sparsity
in adversarial perturbations by sliding a mask through images aiming for
extracting key spatial structures. An ADMM (alternating direction method of
multipliers)-based framework is proposed that can split the original problem
into a sequence of analytically solvable subproblems and can be generalized to
implement other attacking methods. Strong group sparsity is achieved in
adversarial perturbations even with the same level of Lp norm distortion as the
state-of-the-art attacks. We demonstrate the effectiveness of StrAttack by
extensive experimental results onMNIST, CIFAR-10, and ImageNet. We also show
that StrAttack provides better interpretability (i.e., better correspondence
with discriminative image regions)through adversarial saliency map (Papernot et
al., 2016b) and class activation map(Zhou et al., 2016).


http://arxiv.org/abs/1808.01546
Adversarial Examples: Attacks on Machine Learning-based Malware Visualization Detection Methods.
Xinbo Liu; Yapin Lin; He Li; Jiliang Zhang
  As the threat of malicious software (malware) becomes urgently serious,
automatic malware detection techniques have received increasing attention
recently, where the machine learning (ML)-based visualization detection plays a
significant role.However, this leads to a fundamental problem whether such
detection methods can be robust enough against various potential attacks.Even
though ML algorithms show superiority to conventional ones in malware detection
in terms of high efficiency and accuracy, this paper demonstrates that such
ML-based malware detection methods are vulnerable to adversarial examples (AE)
attacks.We propose the first AE-based attack framework, named Adversarial
Texture Malware Perturbation Attacks (ATMPA), based on the gradient descent or
L-norm optimization method.By introducing tiny perturbations on the transformed
dataset, ML-based malware detection methods completely fail.The experimental
results on the MS BIG malware dataset show that a small interference can reduce
the detection rate of convolutional neural network (CNN), support vector
machine (SVM) and random forest(RF)-based malware detectors down to 0 and the
attack transferability can achieve up to 88.7% and 74.1% on average in
different ML-based detection methods.


http://arxiv.org/abs/1808.01452
Traits & Transferability of Adversarial Examples against Instance Segmentation & Object Detection.
Raghav Gurbaxani; Shivank Mishra
  Despite the recent advancements in deploying neural networks for image
classification, it has been found that adversarial examples are able to fool
these models leading them to misclassify the images. Since these models are now
being widely deployed, we provide an insight on the threat of these adversarial
examples by evaluating their characteristics and transferability to more
complex models that utilize Image Classification as a subtask.
  We demonstrate the ineffectiveness of adversarial examples when applied to
Instance Segmentation & Object Detection models. We show that this
ineffectiveness arises from the inability of adversarial examples to withstand
transformations such as scaling or a change in lighting conditions. Moreover,
we show that there exists a small threshold below which the adversarial
property is retained while applying these input transformations.
  Additionally, these attacks demonstrate weak cross-network transferability
across neural network architectures, e.g. VGG16 and ResNet50, however, the
attack may fool both the networks if passed sequentially through networks
during its formation.
  The lack of scalability and transferability challenges the question of how
adversarial images would be effective in the real world.


http://arxiv.org/abs/1808.01352
DeepCloak: Adversarial Crafting As a Defensive Measure to Cloak Processes.
Mehmet Sinan Inci; Thomas Eisenbarth; Berk Sunar
  Over the past decade, side-channels have proven to be significant and
practical threats to modern computing systems. Recent attacks have all
exploited the underlying shared hardware. While practical, mounting such a
complicated attack is still akin to listening on a private conversation in a
crowded train station. The attacker has to either perform significant manual
labor or use AI systems to automate the process. The recent academic literature
points to the latter option. With the abundance of cheap computing power and
the improvements made in AI, it is quite advantageous to automate such tasks.
By using AI systems however, malicious parties also inherit their weaknesses.
One such weakness is undoubtedly the vulnerability to adversarial samples.
  In contrast to the previous literature, for the first time, we propose the
use of adversarial learning as a defensive tool to obfuscate and mask private
information. We demonstrate the viability of this approach by first training
CNNs and other machine learning classifiers on leakage trace of different
processes. After training highly accurate models (99+% accuracy), we
investigate their resolve against adversarial learning methods. By applying
minimal perturbations to input traces, the adversarial traffic by the defender
can run as an attachment to the original process and cloak it against a
malicious classifier.
  Finally, we investigate whether an attacker can protect her classifier model
by employing adversarial defense methods, namely adversarial re-training and
defensive distillation. Our results show that even in the presence of an
intelligent adversary that employs such techniques, all 10 of the tested
adversarial learning methods still manage to successfully craft adversarial
perturbations and the proposed cloaking methodology succeeds.


http://arxiv.org/abs/1808.01153
Ask, Acquire, and Attack: Data-free UAP Generation using Class Impressions.
Konda Reddy Mopuri; Phani Krishna Uppala; R. Venkatesh Babu
  Deep learning models are susceptible to input specific noise, called
adversarial perturbations. Moreover, there exist input-agnostic noise, called
Universal Adversarial Perturbations (UAP) that can affect inference of the
models over most input samples. Given a model, there exist broadly two
approaches to craft UAPs: (i) data-driven: that require data, and (ii)
data-free: that do not require data samples. Data-driven approaches require
actual samples from the underlying data distribution and craft UAPs with high
success (fooling) rate. However, data-free approaches craft UAPs without
utilizing any data samples and therefore result in lesser success rates. In
this paper, for data-free scenarios, we propose a novel approach that emulates
the effect of data samples with class impressions in order to craft UAPs using
data-driven objectives. Class impression for a given pair of category and model
is a generic representation (in the input space) of the samples belonging to
that category. Further, we present a neural network based generative model that
utilizes the acquired class impressions to learn crafting UAPs. Experimental
evaluation demonstrates that the learned generative model, (i) readily crafts
UAPs via simple feed-forwarding through neural network layers, and (ii)
achieves state-of-the-art success rates for data-free scenario and closer to
that for data-driven setting without actually utilizing any data samples.


http://arxiv.org/abs/1808.00123
EagleEye: Attack-Agnostic Defense against Adversarial Inputs (Technical Report).
Yujie Ji; Xinyang Zhang; Ting Wang
  Deep neural networks (DNNs) are inherently vulnerable to adversarial inputs:
such maliciously crafted samples trigger DNNs to misbehave, leading to
detrimental consequences for DNN-powered systems. The fundamental challenges of
mitigating adversarial inputs stem from their adaptive and variable nature.
Existing solutions attempt to improve DNN resilience against specific attacks;
yet, such static defenses can often be circumvented by adaptively engineered
inputs or by new attack variants.
  Here, we present EagleEye, an attack-agnostic adversarial tampering analysis
engine for DNN-powered systems. Our design exploits the {\em minimality
principle} underlying many attacks: to maximize the attack's evasiveness, the
adversary often seeks the minimum possible distortion to convert genuine inputs
to adversarial ones. We show that this practice entails the distinct
distributional properties of adversarial inputs in the input space. By
leveraging such properties in a principled manner, EagleEye effectively
discriminates adversarial inputs and even uncovers their correct classification
outputs. Through extensive empirical evaluation using a range of benchmark
datasets and DNN models, we validate EagleEye's efficacy. We further
investigate the adversary's possible countermeasures, which implies a difficult
dilemma for her: to evade EagleEye's detection, excessive distortion is
necessary, thereby significantly reducing the attack's evasiveness regarding
other detection mechanisms.


http://arxiv.org/abs/1807.10454
Rob-GAN: Generator, Discriminator, and Adversarial Attacker.
Xuanqing Liu; Cho-Jui Hsieh
  We study two important concepts in adversarial deep learning---adversarial
training and generative adversarial network (GAN). Adversarial training is the
technique used to improve the robustness of discriminator by combining
adversarial attacker and discriminator in the training phase. GAN is commonly
used for image generation by jointly optimizing discriminator and generator. We
show these two concepts are indeed closely related and can be used to
strengthen each other---adding a generator to the adversarial training
procedure can improve the robustness of discriminators, and adding an
adversarial attack to GAN training can improve the convergence speed and lead
to better generators. Combining these two insights, we develop a framework
called Rob-GAN to jointly optimize generator and discriminator in the presence
of adversarial attacks---the generator generates fake images to fool
discriminator; the adversarial attacker perturbs real images to fool the
discriminator, and the discriminator wants to minimize loss under fake and
adversarial images. Through this end-to-end training procedure, we are able to
simultaneously improve the convergence speed of GAN training, the quality of
synthetic images, and the robustness of discriminator under strong adversarial
attacks. Experimental results demonstrate that the obtained classifier is more
robust than the state-of-the-art adversarial training approach, and the
generator outperforms SN-GAN on ImageNet-143.


http://arxiv.org/abs/1807.10335
A general metric for identifying adversarial images.
Siddharth Krishna Kumar
  It is well known that a determined adversary can fool a neural network by
making imperceptible adversarial perturbations to an image. Recent studies have
shown that these perturbations can be detected even without information about
the neural network if the strategy taken by the adversary is known beforehand.
Unfortunately, these studies suffer from the generalization limitation -- the
detection method has to be recalibrated every time the adversary changes his
strategy. In this study, we attempt to overcome the generalization limitation
by deriving a metric which reliably identifies adversarial images even when the
approach taken by the adversary is unknown. Our metric leverages key
differences between the spectra of clean and adversarial images when an image
is treated as a matrix. Our metric is able to detect adversarial images across
different datasets and attack strategies without any additional re-calibration.
In addition, our approach provides geometric insights into several unanswered
questions about adversarial perturbations.


http://arxiv.org/abs/1807.10272
Evaluating and Understanding the Robustness of Adversarial Logit Pairing.
Logan Engstrom; Andrew Ilyas; Anish Athalye
  We evaluate the robustness of Adversarial Logit Pairing, a recently proposed
defense against adversarial examples. We find that a network trained with
Adversarial Logit Pairing achieves 0.6% accuracy in the threat model in which
the defense is considered. We provide a brief overview of the defense and the
threat models/claims considered, as well as a discussion of the methodology and
results of our attack, which may offer insights into the reasons underlying the
vulnerability of ALP to adversarial attack.


http://arxiv.org/abs/1807.09937
HiDDeN: Hiding Data With Deep Networks.
Jiren Zhu; Russell Kaplan; Justin Johnson; Li Fei-Fei
  Recent work has shown that deep neural networks are highly sensitive to tiny
perturbations of input images, giving rise to adversarial examples. Though this
property is usually considered a weakness of learned models, we explore whether
it can be beneficial. We find that neural networks can learn to use invisible
perturbations to encode a rich amount of useful information. In fact, one can
exploit this capability for the task of data hiding. We jointly train encoder
and decoder networks, where given an input message and cover image, the encoder
produces a visually indistinguishable encoded image, from which the decoder can
recover the original message. We show that these encodings are competitive with
existing data hiding algorithms, and further that they can be made robust to
noise: our models learn to reconstruct hidden information in an encoded image
despite the presence of Gaussian blurring, pixel-wise dropout, cropping, and
JPEG compression. Even though JPEG is non-differentiable, we show that a robust
model can be trained using differentiable approximations. Finally, we
demonstrate that adversarial training improves the visual quality of encoded
images.


http://arxiv.org/abs/1807.09705
Limitations of the Lipschitz constant as a defense against adversarial examples.
Todd Huster; Cho-Yu Jason Chiang; Ritu Chadha
  Several recent papers have discussed utilizing Lipschitz constants to limit
the susceptibility of neural networks to adversarial examples. We analyze
recently proposed methods for computing the Lipschitz constant. We show that
the Lipschitz constant may indeed enable adversarially robust neural networks.
However, the methods currently employed for computing it suffer from
theoretical and practical limitations. We argue that addressing this
shortcoming is a promising direction for future research into certified
adversarial defenses.


http://arxiv.org/abs/1807.09443
Unbounded Output Networks for Classification.
Stefan Elfwing; Eiji Uchibe; Kenji Doya
  We proposed the expected energy-based restricted Boltzmann machine (EE-RBM)
as a discriminative RBM method for classification. Two characteristics of the
EE-RBM are that the output is unbounded and that the target value of correct
classification is set to a value much greater than one. In this study, by
adopting features of the EE-RBM approach to feed-forward neural networks, we
propose the UnBounded output network (UBnet) which is characterized by three
features: (1) unbounded output units; (2) the target value of correct
classification is set to a value much greater than one; and (3) the models are
trained by a modified mean-squared error objective. We evaluate our approach
using the MNIST, CIFAR-10, and CIFAR-100 benchmark datasets. We first
demonstrate, for shallow UBnets on MNIST, that a setting of the target value
equal to the number of hidden units significantly outperforms a setting of the
target value equal to one, and it also outperforms standard neural networks by
about 25\%. We then validate our approach by achieving high-level
classification performance on the three datasets using unbounded output
residual networks. We finally use MNIST to analyze the learned features and
weights, and we demonstrate that UBnets are much more robust against
adversarial examples than the standard approach of using a softmax output layer
and training the networks by a cross-entropy objective.


http://arxiv.org/abs/1807.09380
Learning Discriminative Video Representations Using Adversarial Perturbations.
Jue Wang; Anoop Cherian
  Adversarial perturbations are noise-like patterns that can subtly change the
data, while failing an otherwise accurate classifier. In this paper, we propose
to use such perturbations for improving the robustness of video
representations. To this end, given a well-trained deep model for per-frame
video recognition, we first generate adversarial noise adapted to this model.
Using the original data features from the full video sequence and their
perturbed counterparts, as two separate bags, we develop a binary
classification problem that learns a set of discriminative hyperplanes -- as a
subspace -- that will separate the two bags from each other. This subspace is
then used as a descriptor for the video, dubbed discriminative subspace
pooling. As the perturbed features belong to data classes that are likely to be
confused with the original features, the discriminative subspace will
characterize parts of the feature space that are more representative of the
original data, and thus may provide robust video representations. To learn such
descriptors, we formulate a subspace learning objective on the Stiefel manifold
and resort to Riemannian optimization methods for solving it efficiently. We
provide experiments on several video datasets and demonstrate state-of-the-art
results.


http://arxiv.org/abs/1807.08108
Simultaneous Adversarial Training - Learn from Others Mistakes.
Zukang Liao
  Adversarial examples are maliciously tweaked images that can easily fool
machine learning techniques, such as neural networks, but they are normally not
visually distinguishable for human beings. One of the main approaches to solve
this problem is to retrain the networks using those adversarial examples,
namely adversarial training. However, standard adversarial training might not
actually change the decision boundaries but cause the problem of gradient
masking, resulting in a weaker ability to generate adversarial examples.
Therefore, it cannot alleviate the problem of black-box attacks, where
adversarial examples generated from other networks can transfer to the targeted
one. In order to reduce the problem of black-box attacks, we propose a novel
method that allows two networks to learn from each others' adversarial examples
and become resilient to black-box attacks. We also combine this method with a
simple domain adaptation to further improve the performance.


http://arxiv.org/abs/1807.07978
Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors.
Andrew Ilyas; Logan Engstrom; Aleksander Madry
  We study the problem of generating adversarial examples in a black-box
setting in which only loss-oracle access to a model is available. We introduce
a framework that conceptually unifies much of the existing work on black-box
attacks, and we demonstrate that the current state-of-the-art methods are
optimal in a natural sense. Despite this optimality, we show how to improve
black-box attacks by bringing a new element into the problem: gradient priors.
We give a bandit optimization-based algorithm that allows us to seamlessly
integrate any such priors, and we explicitly identify and incorporate two
examples. The resulting methods use two to four times fewer queries and fail
two to five times less often than the current state-of-the-art.


http://arxiv.org/abs/1807.07769
Physical Adversarial Examples for Object Detectors.
Kevin Eykholt; Ivan Evtimov; Earlence Fernandes; Bo Li; Amir Rahmati; Florian Tramer; Atul Prakash; Tadayoshi Kohno; Dawn Song
  Deep neural networks (DNNs) are vulnerable to adversarial
examples-maliciously crafted inputs that cause DNNs to make incorrect
predictions. Recent work has shown that these attacks generalize to the
physical domain, to create perturbations on physical objects that fool image
classifiers under a variety of real-world conditions. Such attacks pose a risk
to deep learning models used in safety-critical cyber-physical systems. In this
work, we extend physical attacks to more challenging object detection models, a
broader class of deep learning algorithms widely used to detect and label
multiple objects within a scene. Improving upon a previous physical attack on
image classifiers, we create perturbed physical objects that are either ignored
or mislabeled by object detection models. We implement a Disappearance Attack,
in which we cause a Stop sign to "disappear" according to the detector-either
by covering thesign with an adversarial Stop sign poster, or by adding
adversarial stickers onto the sign. In a video recorded in a controlled lab
environment, the state-of-the-art YOLOv2 detector failed to recognize these
adversarial Stop signs in over 85% of the video frames. In an outdoor
experiment, YOLO was fooled by the poster and sticker attacks in 72.5% and
63.5% of the video frames respectively. We also use Faster R-CNN, a different
object detection model, to demonstrate the transferability of our adversarial
perturbations. The created poster perturbation is able to fool Faster R-CNN in
85.9% of the video frames in a controlled lab environment, and 40.2% of the
video frames in an outdoor environment. Finally, we present preliminary results
with a new Creation Attack, where in innocuous physical stickers fool a model
into detecting nonexistent objects.


http://arxiv.org/abs/1807.10590
Harmonic Adversarial Attack Method.
Wen Heng; Shuchang Zhou; Tingting Jiang
  Adversarial attacks find perturbations that can fool models into
misclassifying images. Previous works had successes in generating
noisy/edge-rich adversarial perturbations, at the cost of degradation of image
quality. Such perturbations, even when they are small in scale, are usually
easily spottable by human vision. In contrast, we propose Harmonic Adversar-
ial Attack Methods (HAAM), that generates edge-free perturbations by using
harmonic functions. The property of edge-free guarantees that the generated
adversarial images can still preserve visual quality, even when perturbations
are of large magnitudes. Experiments also show that adversaries generated by
HAAM often have higher rates of success when transferring between models. In
addition, we find harmonic perturbations can simulate natural phenomena like
natural lighting and shadows. It would then be possible to help find corner
cases for given models, as a first step to improving them.


http://arxiv.org/abs/1807.06752
Gradient Band-based Adversarial Training for Generalized Attack Immunity of A3C Path Finding.
Tong Chen; Wenjia Niu; Yingxiao Xiang; Xiaoxuan Bai; Jiqiang Liu; Zhen Han; Gang Li
  As adversarial attacks pose a serious threat to the security of AI system in
practice, such attacks have been extensively studied in the context of computer
vision applications. However, few attentions have been paid to the adversarial
research on automatic path finding. In this paper, we show dominant adversarial
examples are effective when targeting A3C path finding, and design a Common
Dominant Adversarial Examples Generation Method (CDG) to generate dominant
adversarial examples against any given map. In addition, we propose Gradient
Band-based Adversarial Training, which trained with a single randomly choose
dominant adversarial example without taking any modification, to realize the
"1:N" attack immunity for generalized dominant adversarial examples. Extensive
experimental results show that, the lowest generation precision for CDG
algorithm is 91.91%, and the lowest immune precision for Gradient Band-based
Adversarial Training is 93.89%, which can prove that our method can realize the
generalized attack immunity of A3C path finding with a high confidence.


http://arxiv.org/abs/1807.06732
Motivating the Rules of the Game for Adversarial Example Research.
Justin Gilmer; Ryan P. Adams; Ian Goodfellow; David Andersen; George E. Dahl
  Advances in machine learning have led to broad deployment of systems with
impressive performance on important problems. Nonetheless, these systems can be
induced to make errors on data that are surprisingly similar to examples the
learned system handles correctly. The existence of these errors raises a
variety of questions about out-of-sample generalization and whether bad actors
might use such examples to abuse deployed systems. As a result of these
security concerns, there has been a flurry of recent papers proposing
algorithms to defend against such malicious perturbations of correctly handled
examples. It is unclear how such misclassifications represent a different kind
of security problem than other errors, or even other attacker-produced examples
that have no specific relationship to an uncorrupted input. In this paper, we
argue that adversarial example defense papers have, to date, mostly considered
abstract, toy games that do not relate to any specific security concern.
Furthermore, defense papers have not yet precisely described all the abilities
and limitations of attackers that would be relevant in practical security.
Towards this end, we establish a taxonomy of motivations, constraints, and
abilities for more plausible adversaries. Finally, we provide a series of
recommendations outlining a path forward for future work to more clearly
articulate the threat model and perform more meaningful evaluation.


http://arxiv.org/abs/1807.06714
Defend Deep Neural Networks Against Adversarial Examples via Fixed andDynamic Quantized Activation Functions.
Adnan Siraj Rakin; Jinfeng Yi; Boqing Gong; Deliang Fan
  Recent studies have shown that deep neural networks (DNNs) are vulnerable to
adversarial attacks. To this end, many defense approaches that attempt to
improve the robustness of DNNs have been proposed. In a separate and yet
related area, recent works have explored to quantize neural network weights and
activation functions into low bit-width to compress model size and reduce
computational complexity. In this work,we find that these two different tracks,
namely the pursuit of network compactness and robustness, can bemerged into one
and give rise to networks of both advantages. To the best of our knowledge,
this is the first work that uses quantization of activation functions to defend
against adversarial examples. We also propose to train robust neural networks
by using adaptive quantization techniques for the activation functions. Our
proposed Dynamic Quantized Activation (DQA) is verified through a wide range of
experiments with the MNIST and CIFAR-10 datasets under different white-box
attack methods, including FGSM, PGD, andC&W attacks. Furthermore, Zeroth Order
Optimization and substitute model based black-box attacks are also considered
in this work. The experimental results clearly show that the robustness of DNNs
could be greatly improved using the proposed DQA.


http://arxiv.org/abs/1807.06064
Online Robust Policy Learning in the Presence of Unknown Adversaries.
Aaron J. Havens; Zhanhong Jiang; Soumik Sarkar
  The growing prospect of deep reinforcement learning (DRL) being used in
cyber-physical systems has raised concerns around safety and robustness of
autonomous agents. Recent work on generating adversarial attacks have shown
that it is computationally feasible for a bad actor to fool a DRL policy into
behaving sub optimally. Although certain adversarial attacks with specific
attack models have been addressed, most studies are only interested in off-line
optimization in the data space (e.g., example fitting, distillation). This
paper introduces a Meta-Learned Advantage Hierarchy (MLAH) framework that is
attack model-agnostic and more suited to reinforcement learning, via handling
the attacks in the decision space (as opposed to data space) and directly
mitigating learned bias introduced by the adversary. In MLAH, we learn separate
sub-policies (nominal and adversarial) in an online manner, as guided by a
supervisory master agent that detects the presence of the adversary by
leveraging the advantage function for the sub-policies. We demonstrate that the
proposed algorithm enables policy learning with significantly lower bias as
compared to the state-of-the-art policy learning approaches even in the
presence of heavy state information attacks. We present algorithm analysis and
simulation results using popular OpenAI Gym environments.


http://arxiv.org/abs/1807.05832
Manifold Adversarial Learning.
Shufei Zhang; Kaizhu Huang; Jianke Zhu; Yang Liu
  The recently proposed adversarial training methods show the robustness to
both adversarial and original examples and achieve state-of-the-art results in
supervised and semi-supervised learning. All the existing adversarial training
methods con- sider only how the worst perturbed examples (i.e., adversarial
examples) could affect the model output. Despite their success, we argue that
such setting may be in lack of generalization, since the output space (or label
space) is apparently less informative. In this paper, we propose a novel
method, called Manifold Adver- sarial Training (MAT). MAT manages to build an
adversarial framework based on how the worst perturbation could affect the
distributional manifold rather than the output space. Particularly, a latent
data space with the Gaussian Mixture Model (GMM) will be first derived. On one
hand, MAT tries to perturb the input samples in the way that would rough the
distributional manifold the worst. On the other hand, the deep learning model
is trained trying to promote in the latent space the manifold smoothness,
measured by the variation of Gaussian mixtures (given the local perturbation
around the data point). Importantly, since the latent space is more informative
than the output space, the proposed MAT can learn better a ro- bust and compact
data representation, leading to further performance improvemen- t. The proposed
MAT is important in that it can be considered as a superset of one
recently-proposed discriminative feature learning approach called center loss.
We conducted a series of experiments in both supervised and semi-supervised
learn- ing on three benchmark data sets, showing that the proposed MAT can
achieve remarkable performance, much better than those of the state-of-the-art
adversarial approaches.


http://arxiv.org/abs/1807.05620
NEUZZ: Efficient Fuzzing with Neural Program Smoothing. (11%)
Dongdong She; Kexin Pei; Dave Epstein; Junfeng Yang; Baishakhi Ray; Suman Jana
  Fuzzing has become the de facto standard technique for finding software
vulnerabilities. However, even state-of-the-art fuzzers are not very efficient
at finding hard-to-trigger software bugs. Most popular fuzzers use evolutionary
guidance to generate inputs that can trigger different bugs. Such evolutionary
algorithms, while fast and simple to implement, often get stuck in fruitless
sequences of random mutations. Gradient-guided optimization presents a
promising alternative to evolutionary guidance. Gradient-guided techniques have
been shown to significantly outperform evolutionary algorithms at solving
high-dimensional structured optimization problems in domains like machine
learning by efficiently utilizing gradients or higher-order derivatives of the
underlying function. However, gradient-guided approaches are not directly
applicable to fuzzing as real-world program behaviors contain many
discontinuities, plateaus, and ridges where the gradient-based methods often
get stuck. We observe that this problem can be addressed by creating a smooth
surrogate function approximating the discrete branching behavior of target
program. In this paper, we propose a novel program smoothing technique using
surrogate neural network models that can incrementally learn smooth
approximations of a complex, real-world program's branching behaviors. We
further demonstrate that such neural network models can be used together with
gradient-guided input generation schemes to significantly improve the fuzzing
efficiency. Our extensive evaluations demonstrate that NEUZZ significantly
outperforms 10 state-of-the-art graybox fuzzers on 10 real-world programs both
at finding new bugs and achieving higher edge coverage. NEUZZ found 31 unknown
bugs that other fuzzers failed to find in 10 real world programs and achieved
3X more edge coverage than all of the tested graybox fuzzers for 24 hours
running.


http://arxiv.org/abs/1807.04457
Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach.
Minhao Cheng; Thong Le; Pin-Yu Chen; Jinfeng Yi; Huan Zhang; Cho-Jui Hsieh
  We study the problem of attacking a machine learning model in the hard-label
black-box setting, where no model information is revealed except that the
attacker can make queries to probe the corresponding hard-label decisions. This
is a very challenging problem since the direct extension of state-of-the-art
white-box attacks (e.g., CW or PGD) to the hard-label black-box setting will
require minimizing a non-continuous step function, which is combinatorial and
cannot be solved by a gradient-based optimizer. The only current approach is
based on random walk on the boundary, which requires lots of queries and lacks
convergence guarantees. We propose a novel way to formulate the hard-label
black-box attack as a real-valued optimization problem which is usually
continuous and can be solved by any zeroth order optimization algorithm. For
example, using the Randomized Gradient-Free method, we are able to bound the
number of iterations needed for our algorithm to achieve stationary points. We
demonstrate that our proposed method outperforms the previous random walk
approach to attacking convolutional neural networks on MNIST, CIFAR, and
ImageNet datasets. More interestingly, we show that the proposed algorithm can
also be used to attack other discrete and non-continuous machine learning
models, such as Gradient Boosting Decision Trees (GBDT).


http://arxiv.org/abs/1807.04200
With Friends Like These, Who Needs Adversaries?.
Saumya Jetley; Nicholas A. Lord; Philip H. S. Torr
  The vulnerability of deep image classification networks to adversarial attack
is now well known, but less well understood. Via a novel experimental analysis,
we illustrate some facts about deep convolutional networks for image
classification that shed new light on their behaviour and how it connects to
the problem of adversaries. In short, the celebrated performance of these
networks and their vulnerability to adversarial attack are simply two sides of
the same coin: the input image-space directions along which the networks are
most vulnerable to attack are the same directions which they use to achieve
their classification performance in the first place. We develop this result in
two main steps. The first uncovers the fact that classes tend to be associated
with specific image-space directions. This is shown by an examination of the
class-score outputs of nets as functions of 1D movements along these
directions. This provides a novel perspective on the existence of universal
adversarial perturbations. The second is a clear demonstration of the tight
coupling between classification performance and vulnerability to adversarial
attack within the spaces spanned by these directions. Thus, our analysis
resolves the apparent contradiction between accuracy and vulnerability. It
provides a new perspective on much of the prior art and reveals profound
implications for efforts to construct neural nets that are both accurate and
robust to adversarial attack.


http://arxiv.org/abs/1807.03888
A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks.
Kimin Lee; Kibok Lee; Honglak Lee; Jinwoo Shin
  Detecting test samples drawn sufficiently far away from the training
distribution statistically or adversarially is a fundamental requirement for
deploying a good classifier in many real-world machine learning applications.
However, deep neural networks with the softmax classifier are known to produce
highly overconfident posterior distributions even for such abnormal samples. In
this paper, we propose a simple yet effective method for detecting any abnormal
samples, which is applicable to any pre-trained softmax neural classifier. We
obtain the class conditional Gaussian distributions with respect to (low- and
upper-level) features of the deep models under Gaussian discriminant analysis,
which result in a confidence score based on the Mahalanobis distance. While
most prior methods have been evaluated for detecting either out-of-distribution
or adversarial samples, but not both, the proposed method achieves the
state-of-the-art performances for both cases in our experiments. Moreover, we
found that our proposed method is more robust in harsh cases, e.g., when the
training dataset has noisy labels or small number of samples. Finally, we show
that the proposed method enjoys broader usage by applying it to
class-incremental learning: whenever out-of-distribution samples are detected,
our classification rule can incorporate new classes well without further
training deep models.


http://arxiv.org/abs/1807.03571
A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees.
Min Wu; Matthew Wicker; Wenjie Ruan; Xiaowei Huang; Marta Kwiatkowska
  Despite the improved accuracy of deep neural networks, the discovery of
adversarial examples has raised serious safety concerns. In this paper, we
study two variants of pointwise robustness, the maximum safe radius problem,
which for a given input sample computes the minimum distance to an adversarial
example, and the feature robustness problem, which aims to quantify the
robustness of individual features to adversarial perturbations. We demonstrate
that, under the assumption of Lipschitz continuity, both problems can be
approximated using finite optimisation by discretising the input space, and the
approximation has provable guarantees, i.e., the error is bounded. We then show
that the resulting optimisation problems can be reduced to the solution of
two-player turn-based games, where the first player selects features and the
second perturbs the image within the feature. While the second player aims to
minimise the distance to an adversarial example, depending on the optimisation
objective the first player can be cooperative or competitive. We employ an
anytime approach to solve the games, in the sense of approximating the value of
a game by monotonically improving its upper and lower bounds. The Monte Carlo
tree search algorithm is applied to compute upper bounds for both games, and
the Admissible A* and the Alpha-Beta Pruning algorithms are, respectively, used
to compute lower bounds for the maximum safety radius and feature robustness
games. When working on the upper bound of the maximum safe radius problem, our
tool demonstrates competitive performance against existing adversarial example
crafting algorithms. Furthermore, we show how our framework can be deployed to
evaluate pointwise robustness of neural networks in safety-critical
applications such as traffic sign recognition in self-driving cars.


http://arxiv.org/abs/1807.04270
Attack and defence in cellular decision-making: lessons from machine learning.
Thomas J. Rademaker; Emmanuel Bengio; Paul François
  Machine learning algorithms can be fooled by small well-designed adversarial
perturbations. This is reminiscent of cellular decision-making where ligands
(called antagonists) prevent correct signalling, like in early immune
recognition. We draw a formal analogy between neural networks used in machine
learning and models of cellular decision-making (adaptive proofreading). We
apply attacks from machine learning to simple decision-making models, and show
explicitly the correspondence to antagonism by weakly bound ligands. Such
antagonism is absent in more nonlinear models, which inspired us to implement a
biomimetic defence in neural networks filtering out adversarial perturbations.
We then apply a gradient-descent approach from machine learning to different
cellular decision-making models, and we reveal the existence of two regimes
characterized by the presence or absence of a critical point for the gradient.
This critical point causes the strongest antagonists to lie close to the
decision boundary. This is validated in the loss landscapes of robust neural
networks and cellular decision-making models, and observed experimentally for
immune cells. For both regimes, we explain how associated defence mechanisms
shape the geometry of the loss landscape, and why different adversarial attacks
are effective in different regimes. Our work connects evolved cellular
decision-making to machine learning, and motivates the design of a general
theory of adversarial perturbations, both for in vivo and in silico systems.


http://arxiv.org/abs/1807.03326
Adaptive Adversarial Attack on Scene Text Recognition.
Xiaoyong Yuan; Pan He; Xiaolin Andy Li
  Recent studies have shown that state-of-the-art deep learning models are
vulnerable to the inputs with small perturbations (adversarial examples). We
observe two critical obstacles in adversarial examples: (i) Strong adversarial
attacks (e.g., C&W attack) require manually tuning hyper-parameters and take a
long time to construct an adversarial example, making it impractical to attack
real-time systems; (ii) Most of the studies focus on non-sequential tasks, such
as image classification, yet only a few consider sequential tasks. In this
work, we speed up adversarial attacks, especially on sequential learning tasks.
By leveraging the uncertainty of each task, we directly learn the adaptive
multi-task weightings, without manually searching hyper-parameters. A unified
architecture is developed and evaluated for both non-sequential tasks and
sequential ones. To validate the effectiveness, we take the scene text
recognition task as a case study. To our best knowledge, our proposed method is
the first attempt to adversarial attack for scene text recognition. Adaptive
Attack achieves over 99.9\% success rate with 3-6X speedup compared to
state-of-the-art adversarial attacks.


http://arxiv.org/abs/1807.02905
Vulnerability Analysis of Chest X-Ray Image Classification Against Adversarial Attacks.
Saeid Asgari Taghanaki; Arkadeep Das; Ghassan Hamarneh
  Recently, there have been several successful deep learning approaches for
automatically classifying chest X-ray images into different disease categories.
However, there is not yet a comprehensive vulnerability analysis of these
models against the so-called adversarial perturbations/attacks, which makes
deep models more trustful in clinical practices. In this paper, we extensively
analyzed the performance of two state-of-the-art classification deep networks
on chest X-ray images. These two networks were attacked by three different
categories (ten methods in total) of adversarial methods (both white- and
black-box), namely gradient-based, score-based, and decision-based attacks.
Furthermore, we modified the pooling operations in the two classification
networks to measure their sensitivities against different attacks, on the
specific task of chest X-ray classification.


http://arxiv.org/abs/1807.02188
Implicit Generative Modeling of Random Noise during Training for Adversarial Robustness.
Priyadarshini Panda; Kaushik Roy
  We introduce a Noise-based prior Learning (NoL) approach for training neural
networks that are intrinsically robust to adversarial attacks. We find that the
implicit generative modeling of random noise with the same loss function used
during posterior maximization, improves a model's understanding of the data
manifold furthering adversarial robustness. We evaluate our approach's efficacy
and provide a simplistic visualization tool for understanding adversarial data,
using Principal Component Analysis. Our analysis reveals that adversarial
robustness, in general, manifests in models with higher variance along the
high-ranked principal components. We show that models learnt with our approach
perform remarkably well against a wide-range of attacks. Furthermore, combining
NoL with state-of-the-art adversarial training extends the robustness of a
model, even beyond what it is adversarially trained for, in both white-box and
black-box attack scenarios.


http://arxiv.org/abs/1807.01697
Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations.
Dan Hendrycks; Thomas G. Dietterich
  In this paper we establish rigorous benchmarks for image classifier
robustness. Our first benchmark, ImageNet-C, standardizes and expands the
corruption robustness topic, while showing which classifiers are preferable in
safety-critical applications. Unlike recent robustness research, this benchmark
evaluates performance on commonplace corruptions not worst-case adversarial
corruptions. We find that there are negligible changes in relative corruption
robustness from AlexNet to ResNet classifiers, and we discover ways to enhance
corruption robustness. Then we propose a new dataset called Icons-50 which
opens research on a new kind of robustness, surface variation robustness. With
this dataset we evaluate the frailty of classifiers on new styles of known
objects and unexpected instances of known classes. We also demonstrate two
methods that improve surface variation robustness. Together our benchmarks may
aid future work toward networks that learn fundamental class structure and also
robustly generalize.


http://arxiv.org/abs/1807.01216
Local Gradients Smoothing: Defense against localized adversarial attacks.
Muzammal Naseer; Salman H. Khan; Fatih Porikli
  Deep neural networks (DNNs) have shown vulnerability to adversarial attacks,
i.e., carefully perturbed inputs designed to mislead the network at inference
time. Recently introduced localized attacks, Localized and Visible Adversarial
Noise (LaVAN) and Adversarial patch, pose a new challenge to deep learning
security by adding adversarial noise only within a specific region without
affecting the salient objects in an image. Driven by the observation that such
attacks introduce concentrated high-frequency changes at a particular image
location, we have developed an effective method to estimate noise location in
gradient domain and transform those high activation regions caused by
adversarial noise in image domain while having minimal effect on the salient
object that is important for correct classification. Our proposed Local
Gradients Smoothing (LGS) scheme achieves this by regularizing gradients in the
estimated noisy region before feeding the image to DNN for inference. We have
shown the effectiveness of our method in comparison to other defense methods
including Digital Watermarking, JPEG compression, Total Variance Minimization
(TVM) and Feature squeezing on ImageNet dataset. In addition, we systematically
study the robustness of the proposed defense mechanism against Back Pass
Differentiable Approximation (BPDA), a state of the art attack recently
developed to break defenses that transform an input sample to minimize the
adversarial effect. Compared to other defense mechanisms, LGS is by far the
most resistant to BPDA in localized adversarial attack setting.


http://arxiv.org/abs/1807.01069
Adversarial Robustness Toolbox v0.4.0.
Maria-Irina Nicolae; Mathieu Sinn; Minh Ngoc Tran; Ambrish Rawat; Martin Wistuba; Valentina Zantedeschi; Nathalie Baracaldo; Bryant Chen; Heiko Ludwig; Ian M. Molloy; Ben Edwards
  Adversarial examples have become an indisputable threat to the security of
modern AI systems based on deep neural networks (DNNs). The Adversarial
Robustness Toolbox (ART) is a Python library designed to support researchers
and developers in creating novel defence techniques, as well as in deploying
practical defences of real-world AI systems. Researchers can use ART to
benchmark novel defences against the state-of-the-art. For developers, the
library provides interfaces which support the composition of comprehensive
defence systems using individual methods as building blocks. The Adversarial
Robustness Toolbox supports machine learning models (and deep neural networks
(DNNs) specifically) implemented in any of the most popular deep learning
frameworks (TensorFlow, Keras, PyTorch and MXNet). Currently, the library is
primarily intended to improve the adversarial robustness of visual recognition
systems, however, future releases that will comprise adaptations to other data
modes (such as speech, text or time series) are envisioned. The ART source code
is released (https://github.com/IBM/adversarial-robustness-toolbox) under an
MIT license. The release includes code examples and extensive documentation
(http://adversarial-robustness-toolbox.readthedocs.io) to help researchers and
developers get quickly started.


http://arxiv.org/abs/1807.00458
Adversarial Perturbations Against Real-Time Video Classification Systems.
Shasha Li; Ajaya Neupane; Sujoy Paul; Chengyu Song; Srikanth V. Krishnamurthy; Amit K. Roy Chowdhury; Ananthram Swami
  Recent research has demonstrated the brittleness of machine learning systems
to adversarial perturbations. However, the studies have been mostly limited to
perturbations on images and more generally, classification that does not deal
with temporally varying inputs. In this paper we ask "Are adversarial
perturbations possible in real-time video classification systems and if so,
what properties must they satisfy?" Such systems find application in
surveillance applications, smart vehicles, and smart elderly care and thus,
misclassification could be particularly harmful (e.g., a mishap at an elderly
care facility may be missed). We show that accounting for temporal structure is
key to generating adversarial examples in such systems. We exploit recent
advances in generative adversarial network (GAN) architectures to account for
temporal correlations and generate adversarial samples that can cause
misclassification rates of over 80% for targeted activities. More importantly,
the samples also leave other activities largely unaffected making them
extremely stealthy. Finally, we also surprisingly find that in many scenarios,
the same perturbation can be applied to every frame in a video clip that makes
the adversary's ability to achieve misclassification relatively easy.


http://arxiv.org/abs/1807.00340
Towards Adversarial Training with Moderate Performance Improvement for Neural Network Classification.
Xinhan Di; Pengqian Yu; Meng Tian
  It has been demonstrated that deep neural networks are prone to noisy
examples particular adversarial samples during inference process. The gap
between robust deep learning systems in real world applications and vulnerable
neural networks is still large. Current adversarial training strategies improve
the robustness against adversarial samples. However, these methods lead to
accuracy reduction when the input examples are clean thus hinders the
practicability. In this paper, we investigate an approach that protects the
neural network classification from the adversarial samples and improves its
accuracy when the input examples are clean. We demonstrate the versatility and
effectiveness of our proposed approach on a variety of different networks and
datasets.


http://arxiv.org/abs/1807.00051
Adversarial Examples in Deep Learning: Characterization and Divergence.
Wenqi Wei; Ling Liu; Margaret Loper; Stacey Truex; Lei Yu; Mehmet Emre Gursoy; Yanzhao Wu
  The burgeoning success of deep learning has raised the security and privacy
concerns as more and more tasks are accompanied with sensitive data.
Adversarial attacks in deep learning have emerged as one of the dominating
security threat to a range of mission-critical deep learning systems and
applications. This paper takes a holistic and principled approach to perform
statistical characterization of adversarial examples in deep learning. We
provide a general formulation of adversarial examples and elaborate on the
basic principle for adversarial attack algorithm design. We introduce easy and
hard categorization of adversarial attacks to analyze the effectiveness of
adversarial examples in terms of attack success rate, degree of change in
adversarial perturbation, average entropy of prediction qualities, and fraction
of adversarial examples that lead to successful attacks. We conduct extensive
experimental study on adversarial behavior in easy and hard attacks under deep
learning models with different hyperparameters and different deep learning
frameworks. We show that the same adversarial attack behaves differently under
different hyperparameters and across different frameworks due to the different
features learned under different deep learning model training process. Our
statistical characterization with strong empirical evidence provides a
transformative enlightenment on mitigation strategies towards effective
countermeasures against present and future adversarial attacks.


http://arxiv.org/abs/1806.11146
Adversarial Reprogramming of Neural Networks.
Gamaleldin F. Elsayed; Ian Goodfellow; Jascha Sohl-Dickstein
  Deep neural networks are susceptible to \emph{adversarial} attacks. In
computer vision, well-crafted perturbations to images can cause neural networks
to make mistakes such as confusing a cat with a computer. Previous adversarial
attacks have been designed to degrade performance of models or cause machine
learning models to produce specific outputs chosen ahead of time by the
attacker. We introduce attacks that instead {\em reprogram} the target model to
perform a task chosen by the attacker---without the attacker needing to specify
or compute the desired output for each test-time input. This attack finds a
single adversarial perturbation, that can be added to all test-time inputs to a
machine learning model in order to cause the model to perform a task chosen by
the adversary---even if the model was not trained to do this task. These
perturbations can thus be considered a program for the new task. We demonstrate
adversarial reprogramming on six ImageNet classification models, repurposing
these models to perform a counting task, as well as classification tasks:
classification of MNIST and CIFAR-10 examples presented as inputs to the
ImageNet model.


http://arxiv.org/abs/1806.10707
Gradient Similarity: An Explainable Approach to Detect Adversarial Attacks against Deep Learning.
Jasjeet Dhaliwal; Saurabh Shintre
  Deep neural networks are susceptible to small-but-specific adversarial
perturbations capable of deceiving the network. This vulnerability can lead to
potentially harmful consequences in security-critical applications. To address
this vulnerability, we propose a novel metric called \emph{Gradient Similarity}
that allows us to capture the influence of training data on test inputs. We
show that \emph{Gradient Similarity} behaves differently for normal and
adversarial inputs, and enables us to detect a variety of adversarial attacks
with a near perfect ROC-AUC of 95-100\%. Even white-box adversaries equipped
with perfect knowledge of the system cannot bypass our detector easily. On the
MNIST dataset, white-box attacks are either detected with a high ROC-AUC of
87-96\%, or require very high distortion to bypass our detector.


http://arxiv.org/abs/1806.10496
Customizing an Adversarial Example Generator with Class-Conditional GANs.
Shih-hong Tsai
  Adversarial examples are intentionally crafted data with the purpose of
deceiving neural networks into misclassification. When we talk about strategies
to create such examples, we usually refer to perturbation-based methods that
fabricate adversarial examples by applying invisible perturbations onto normal
data. The resulting data reserve their visual appearance to human observers,
yet can be totally unrecognizable to DNN models, which in turn leads to
completely misleading predictions. In this paper, however, we consider crafting
adversarial examples from existing data as a limitation to example diversity.
We propose a non-perturbation-based framework that generates native adversarial
examples from class-conditional generative adversarial networks.As such, the
generated data will not resemble any existing data and thus expand example
diversity, raising the difficulty in adversarial defense. We then extend this
framework to pre-trained conditional GANs, in which we turn an existing
generator into an "adversarial-example generator". We conduct experiments on
our approach for MNIST and CIFAR10 datasets and have satisfactory results,
showing that this approach can be a potential alternative to previous attack
strategies.


http://arxiv.org/abs/1806.09410
Exploring Adversarial Examples: Patterns of One-Pixel Attacks.
David Kügler; Alexander Distergoft; Arjan Kuijper; Anirban Mukhopadhyay
  Failure cases of black-box deep learning, e.g. adversarial examples, might
have severe consequences in healthcare. Yet such failures are mostly studied in
the context of real-world images with calibrated attacks. To demystify the
adversarial examples, rigorous studies need to be designed. Unfortunately,
complexity of the medical images hinders such study design directly from the
medical images. We hypothesize that adversarial examples might result from the
incorrect mapping of image space to the low dimensional generation manifold by
deep networks. To test the hypothesis, we simplify a complex medical problem
namely pose estimation of surgical tools into its barest form. An analytical
decision boundary and exhaustive search of the one-pixel attack across multiple
image dimensions let us localize the regions of frequent successful one-pixel
attacks at the image space.


http://arxiv.org/abs/1806.09035
Defending Malware Classification Networks Against Adversarial Perturbations with Non-Negative Weight Restrictions.
Alex Kouzemtchenko
  There is a growing body of literature showing that deep neural networks are
vulnerable to adversarial input modification. Recently this work has been
extended from image classification to malware classification over boolean
features. In this paper we present several new methods for training restricted
networks in this specific domain that are highly effective at preventing
adversarial perturbations. We start with a fully adversarially resistant neural
network that has hard non-negative weight restrictions and is equivalent to
learning a monotonic boolean function and then attempt to relax the constraints
to improve classifier accuracy.


http://arxiv.org/abs/1806.09030
On Adversarial Examples for Character-Level Neural Machine Translation.
Javid Ebrahimi; Daniel Lowd; Dejing Dou
  Evaluating on adversarial examples has become a standard procedure to measure
robustness of deep learning models. Due to the difficulty of creating white-box
adversarial examples for discrete text input, most analyses of the robustness
of NLP models have been done through black-box adversarial examples. We
investigate adversarial examples for character-level neural machine translation
(NMT), and contrast black-box adversaries with a novel white-box adversary,
which employs differentiable string-edit operations to rank adversarial
changes. We propose two novel types of attacks which aim to remove or change a
word in a translation, rather than simply break the NMT. We demonstrate that
white-box adversarial examples are significantly stronger than their black-box
counterparts in different attack scenarios, which show more serious
vulnerabilities than previously known. In addition, after performing
adversarial training, which takes only 3 times longer than regular training, we
can improve the model's robustness significantly.


http://arxiv.org/abs/1806.08970
Evaluation of Momentum Diverse Input Iterative Fast Gradient Sign Method (M-DI2-FGSM) Based Attack Method on MCS 2018 Adversarial Attacks on Black Box Face Recognition System.
Md Ashraful Alam Milton
  The convolutional neural network is the crucial tool for the recent success
of deep learning based methods on various computer vision tasks like
classification, segmentation, and detection. Convolutional neural networks
achieved state-of-the-art performance in these tasks and every day pushing the
limit of computer vision and AI. However, adversarial attack on computer vision
systems is threatening their application in the real life and in
safety-critical applications. Necessarily, Finding adversarial examples are
important to detect susceptible models to attack and take safeguard measures to
overcome the adversarial attacks. In this regard, MCS 2018 Adversarial Attacks
on Black Box Face Recognition challenge aims to facilitate the research of
finding new adversarial attack techniques and their effectiveness in generating
adversarial examples. In this challenge, the attack"s nature is targeted-attack
on the black-box neural network where we have no knowledge about black-block"s
inner structure. The attacker must modify a set of five images of a single
person so that the neural network miss-classify them as target image which is a
set of five images of another person. In this competition, we applied Momentum
Diverse Input Iterative Fast Gradient Sign Method (M-DI2-FGSM) to make an
adversarial attack on black-box face recognition system. We tested our method
on MCS 2018 Adversarial Attacks on Black Box Face Recognition challenge and
found competitive result. Our solution got validation score 1.404 which better
than baseline score 1.407 and stood 14 place among 132 teams in the
leader-board. Further improvement can be achieved by finding improved feature
extraction from source image, carefully chosen hyper-parameters, finding
improved substitute model of the black-box and better optimization method.


http://arxiv.org/abs/1806.09186
Detection based Defense against Adversarial Examples from the Steganalysis Point of View.
Jiayang Liu; Weiming Zhang; Yiwei Zhang; Dongdong Hou; Yujia Liu; Hongyue Zha; Nenghai Yu
  Deep Neural Networks (DNNs) have recently led to significant improvements in
many fields. However, DNNs are vulnerable to adversarial examples which are
samples with imperceptible perturbations while dramatically misleading the
DNNs. Moreover, adversarial examples can be used to perform an attack on
various kinds of DNN based systems, even if the adversary has no access to the
underlying model. Many defense methods have been proposed, such as obfuscating
gradients of the networks or detecting adversarial examples. However it is
proved out that these defense methods are not effective or cannot resist
secondary adversarial attacks. In this paper, we point out that steganalysis
can be applied to adversarial examples detection, and propose a method to
enhance steganalysis features by estimating the probability of modifications
caused by adversarial attacks. Experimental results show that the proposed
method can accurately detect adversarial examples. Moreover, secondary
adversarial attacks cannot be directly performed to our method because our
method is not based on a neural network but based on high-dimensional
artificial features and FLD (Fisher Linear Discriminant) ensemble.


http://arxiv.org/abs/1806.08028
Gradient Adversarial Training of Neural Networks.
Ayan Sinha; Zhao Chen; Vijay Badrinarayanan; Andrew Rabinovich
  We propose gradient adversarial training, an auxiliary deep learning
framework applicable to different machine learning problems. In gradient
adversarial training, we leverage a prior belief that in many contexts,
simultaneous gradient updates should be statistically indistinguishable from
each other. We enforce this consistency using an auxiliary network that
classifies the origin of the gradient tensor, and the main network serves as an
adversary to the auxiliary network in addition to performing standard
task-based training. We demonstrate gradient adversarial training for three
different scenarios: (1) as a defense to adversarial examples we classify
gradient tensors and tune them to be agnostic to the class of their
corresponding example, (2) for knowledge distillation, we do binary
classification of gradient tensors derived from the student or teacher network
and tune the student gradient tensor to mimic the teacher's gradient tensor;
and (3) for multi-task learning we classify the gradient tensors derived from
different task loss functions and tune them to be statistically
indistinguishable. For each of the three scenarios we show the potential of
gradient adversarial training procedure. Specifically, gradient adversarial
training increases the robustness of a network to adversarial attacks, is able
to better distill the knowledge from a teacher network to a student network
compared to soft targets, and boosts multi-task learning by aligning the
gradient tensors derived from the task specific loss functions. Overall, our
experiments demonstrate that gradient tensors contain latent information about
whatever tasks are being trained, and can support diverse machine learning
problems when intelligently guided through adversarialization using a auxiliary
network.


http://arxiv.org/abs/1806.07723
Combinatorial Testing for Deep Learning Systems.
Lei Ma; Fuyuan Zhang; Minhui Xue; Bo Li; Yang Liu; Jianjun Zhao; Yadong Wang
  Deep learning (DL) has achieved remarkable progress over the past decade and
been widely applied to many safety-critical applications. However, the
robustness of DL systems recently receives great concerns, such as adversarial
examples against computer vision systems, which could potentially result in
severe consequences. Adopting testing techniques could help to evaluate the
robustness of a DL system and therefore detect vulnerabilities at an early
stage. The main challenge of testing such systems is that its runtime state
space is too large: if we view each neuron as a runtime state for DL, then a DL
system often contains massive states, rendering testing each state almost
impossible. For traditional software, combinatorial testing (CT) is an
effective testing technique to reduce the testing space while obtaining
relatively high defect detection abilities. In this paper, we perform an
exploratory study of CT on DL systems. We adapt the concept in CT and propose a
set of coverage criteria for DL systems, as well as a CT coverage guided test
generation technique. Our evaluation demonstrates that CT provides a promising
avenue for testing DL systems. We further pose several open questions and
interesting directions for combinatorial testing of DL systems.


http://arxiv.org/abs/1806.07492
On the Learning of Deep Local Features for Robust Face Spoofing Detection.
Souza Gustavo Botelho de; João Paulo Papa; Aparecido Nilceu Marana
  Biometrics emerged as a robust solution for security systems. However, given
the dissemination of biometric applications, criminals are developing
techniques to circumvent them by simulating physical or behavioral traits of
legal users (spoofing attacks). Despite face being a promising characteristic
due to its universality, acceptability and presence of cameras almost
everywhere, face recognition systems are extremely vulnerable to such frauds
since they can be easily fooled with common printed facial photographs.
State-of-the-art approaches, based on Convolutional Neural Networks (CNNs),
present good results in face spoofing detection. However, these methods do not
consider the importance of learning deep local features from each facial
region, even though it is known from face recognition that each facial region
presents different visual aspects, which can also be exploited for face
spoofing detection. In this work we propose a novel CNN architecture trained in
two steps for such task. Initially, each part of the neural network learns
features from a given facial region. Afterwards, the whole model is fine-tuned
on the whole facial images. Results show that such pre-training step allows the
CNN to learn different local spoofing cues, improving the performance and the
convergence speed of the final model, outperforming the state-of-the-art
approaches.


http://arxiv.org/abs/1806.07409
Built-in Vulnerabilities to Imperceptible Adversarial Perturbations.
Thomas Tanay; Jerone T. A. Andrews; Lewis D. Griffin
  Designing models that are robust to small adversarial perturbations of their
inputs has proven remarkably difficult. In this work we show that the reverse
problem---making models more vulnerable---is surprisingly easy. After
presenting some proofs of concept on MNIST, we introduce a generic tilting
attack that injects vulnerabilities into the linear layers of pre-trained
networks by increasing their sensitivity to components of low variance in the
training data without affecting their performance on test data. We illustrate
this attack on a multilayer perceptron trained on SVHN and use it to design a
stand-alone adversarial module which we call a steganogram decoder. Finally, we
show on CIFAR-10 that a poisoning attack with a poisoning rate as low as 0.1%
can induce vulnerabilities to chosen imperceptible backdoor signals in
state-of-the-art networks. Beyond their practical implications, these different
results shed new light on the nature of the adversarial example phenomenon.


http://arxiv.org/abs/1806.06108
Non-Negative Networks Against Adversarial Attacks.
William Fleshman; Edward Raff; Jared Sylvester; Steven Forsyth; Mark McLean
  Adversarial attacks against neural networks are a problem of considerable
importance, for which effective defenses are not yet readily available. We make
progress toward this problem by showing that non-negative weight constraints
can be used to improve resistance in specific scenarios. In particular, we show
that they can provide an effective defense for binary classification problems
with asymmetric cost, such as malware or spam detection. We also show the
potential for non-negativity to be helpful to non-binary problems by applying
it to image classification.


http://arxiv.org/abs/1806.05476
Copycat CNN: Stealing Knowledge by Persuading Confession with Random Non-Labeled Data.
Jacson Rodrigues Correia-Silva; Rodrigo F. Berriel; Claudine Badue; Souza Alberto F. de; Thiago Oliveira-Santos
  In the past few years, Convolutional Neural Networks (CNNs) have been
achieving state-of-the-art performance on a variety of problems. Many companies
employ resources and money to generate these models and provide them as an API,
therefore it is in their best interest to protect them, i.e., to avoid that
someone else copies them. Recent studies revealed that state-of-the-art CNNs
are vulnerable to adversarial examples attacks, and this weakness indicates
that CNNs do not need to operate in the problem domain (PD). Therefore, we
hypothesize that they also do not need to be trained with examples of the PD in
order to operate in it.
  Given these facts, in this paper, we investigate if a target black-box CNN
can be copied by persuading it to confess its knowledge through random
non-labeled data. The copy is two-fold: i) the target network is queried with
random data and its predictions are used to create a fake dataset with the
knowledge of the network; and ii) a copycat network is trained with the fake
dataset and should be able to achieve similar performance as the target
network.
  This hypothesis was evaluated locally in three problems (facial expression,
object, and crosswalk classification) and against a cloud-based API. In the
copy attacks, images from both non-problem domain and PD were used. All copycat
networks achieved at least 93.7% of the performance of the original models with
non-problem domain data, and at least 98.6% using additional data from the PD.
Additionally, the copycat CNN successfully copied at least 97.3% of the
performance of the Microsoft Azure Emotion API. Our results show that it is
possible to create a copycat CNN by simply querying a target network as
black-box with random non-labeled data.


http://arxiv.org/abs/1806.05337
Hierarchical interpretations for neural network predictions.
Chandan Singh; W. James Murdoch; Bin Yu
  Deep neural networks (DNNs) have achieved impressive predictive performance
due to their ability to learn complex, non-linear relationships between
variables. However, the inability to effectively visualize these relationships
has led to DNNs being characterized as black boxes and consequently limited
their applications. To ameliorate this problem, we introduce the use of
hierarchical interpretations to explain DNN predictions through our proposed
method, agglomerative contextual decomposition (ACD). Given a prediction from a
trained DNN, ACD produces a hierarchical clustering of the input features,
along with the contribution of each cluster to the final prediction. This
hierarchy is optimized to identify clusters of features that the DNN learned
are predictive. Using examples from Stanford Sentiment Treebank and ImageNet,
we show that ACD is effective at diagnosing incorrect predictions and
identifying dataset bias. Through human experiments, we demonstrate that ACD
enables users both to identify the more accurate of two DNNs and to better
trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to
adversarial perturbations, implying that it captures fundamental aspects of the
input and ignores spurious noise.


http://arxiv.org/abs/1806.05236
Manifold Mixup: Better Representations by Interpolating Hidden States.
Vikas Verma; Alex Lamb; Christopher Beckham; Amir Najafi; Ioannis Mitliagkas; Aaron Courville; David Lopez-Paz; Yoshua Bengio
  Deep neural networks excel at learning the training data, but often provide
incorrect and confident predictions when evaluated on slightly different test
examples. This includes distribution shifts, outliers, and adversarial
examples. To address these issues, we propose Manifold Mixup, a simple
regularizer that encourages neural networks to predict less confidently on
interpolations of hidden representations. Manifold Mixup leverages semantic
interpolations as additional training signal, obtaining neural networks with
smoother decision boundaries at multiple levels of representation. As a result,
neural networks trained with Manifold Mixup learn class-representations with
fewer directions of variance. We prove theory on why this flattening happens
under ideal conditions, validate it on practical situations, and connect it to
previous works on information theory and generalization. In spite of incurring
no significant computation and being implemented in a few lines of code,
Manifold Mixup improves strong baselines in supervised learning, robustness to
single-step adversarial attacks, and test log-likelihood.


http://arxiv.org/abs/1806.04646
Adversarial Attacks on Variational Autoencoders.
George Gondim-Ribeiro; Pedro Tabacof; Eduardo Valle
  Adversarial attacks are malicious inputs that derail machine-learning models.
We propose a scheme to attack autoencoders, as well as a quantitative
evaluation framework that correlates well with the qualitative assessment of
the attacks. We assess --- with statistically validated experiments --- the
resistance to attacks of three variational autoencoders (simple, convolutional,
and DRAW) in three datasets (MNIST, SVHN, CelebA), showing that both DRAW's
recurrence and attention mechanism lead to better resistance. As autoencoders
are proposed for compressing data --- a scenario in which their safety is
paramount --- we expect more attention will be given to adversarial attacks on
them.


http://arxiv.org/abs/1806.04425
Ranking Robustness Under Adversarial Document Manipulations.
Gregory Goren; Oren Kurland; Moshe Tennenholtz; Fiana Raiber
  For many queries in the Web retrieval setting there is an on-going ranking
competition: authors manipulate their documents so as to promote them in
rankings. Such competitions can have unwarranted effects not only in terms of
retrieval effectiveness, but also in terms of ranking robustness. A case in
point, rankings can (rapidly) change due to small indiscernible perturbations
of documents. While there has been a recent growing interest in analyzing the
robustness of classifiers to adversarial manipulations, there has not yet been
a study of the robustness of relevance-ranking functions. We address this
challenge by formally analyzing different definitions and aspects of the
robustness of learning-to-rank-based ranking functions. For example, we
formally show that increased regularization of linear ranking functions
increases ranking robustness. This finding leads us to conjecture that
decreased variance of any ranking function results in increased robustness. We
propose several measures for quantifying ranking robustness and use them to
analyze ranking competitions between documents' authors. The empirical findings
support our formal analysis and conjecture for both RankSVM and LambdaMART.


http://arxiv.org/abs/1806.04169
Defense Against the Dark Arts: An overview of adversarial example security research and future research directions.
Ian Goodfellow
  This article presents a summary of a keynote lecture at the Deep Learning
Security workshop at IEEE Security and Privacy 2018. This lecture summarizes
the state of the art in defenses against adversarial examples and provides
recommendations for future research directions on this topic.


http://arxiv.org/abs/1806.02977
Monge blunts Bayes: Hardness Results for Adversarial Training.
Zac Cranko; Aditya Krishna Menon; Richard Nock; Cheng Soon Ong; Zhan Shi; Christian Walder
  The last few years have seen a staggering number of empirical studies of the
robustness of neural networks in a model of adversarial perturbations of their
inputs. Most rely on an adversary which carries out local modifications within
prescribed balls. None however has so far questioned the broader picture: how
to frame a resource-bounded adversary so that it can be severely detrimental to
learning, a non-trivial problem which entails at a minimum the choice of loss
and classifiers.
  We suggest a formal answer for losses that satisfy the minimal statistical
requirement of being proper. We pin down a simple sufficient property for any
given class of adversaries to be detrimental to learning, involving a central
measure of "harmfulness" which generalizes the well-known class of integral
probability metrics. A key feature of our result is that it holds for all
proper losses, and for a popular subset of these, the optimisation of this
central measure appears to be independent of the loss. When classifiers are
Lipschitz -- a now popular approach in adversarial training --, this
optimisation resorts to optimal transport to make a low-budget compression of
class marginals. Toy experiments reveal a finding recently separately observed:
training against a sufficiently budgeted adversary of this kind improves
generalization.


http://arxiv.org/abs/1806.02924
Revisiting Adversarial Risk.
Arun Sai Suggala; Adarsh Prasad; Vaishnavh Nagarajan; Pradeep Ravikumar
  Recent works on adversarial perturbations show that there is an inherent
trade-off between standard test accuracy and adversarial accuracy.
Specifically, they show that no classifier can simultaneously be robust to
adversarial perturbations and achieve high standard test accuracy. However,
this is contrary to the standard notion that on tasks such as image
classification, humans are robust classifiers with low error rate. In this
work, we show that the main reason behind this confusion is the inexact
definition of adversarial perturbation that is used in the literature. To fix
this issue, we propose a slight, yet important modification to the existing
definition of adversarial perturbation. Based on the modified definition, we
show that there is no trade-off between adversarial and standard accuracies;
there exist classifiers that are robust and achieve high standard accuracy. We
further study several properties of this new definition of adversarial risk and
its relation to the existing definition.


http://arxiv.org/abs/1806.02782
Training Augmentation with Adversarial Examples for Robust Speech Recognition.
Sining Sun; Ching-Feng Yeh; Mari Ostendorf; Mei-Yuh Hwang; Lei Xie
  This paper explores the use of adversarial examples in training speech
recognition systems to increase robustness of deep neural network acoustic
models. During training, the fast gradient sign method is used to generate
adversarial examples augmenting the original training data. Different from
conventional data augmentation based on data transformations, the examples are
dynamically generated based on current acoustic model parameters. We assess the
impact of adversarial data augmentation in experiments on the Aurora-4 and
CHiME-4 single-channel tasks, showing improved robustness against noise and
channel variation. Further improvement is obtained when combining adversarial
examples with teacher/student training, leading to a 23% relative word error
rate reduction on Aurora-4.


http://arxiv.org/abs/1806.02371
Adversarial Attack on Graph Structured Data.
Hanjun Dai; Hui Li; Tian Tian; Xin Huang; Lin Wang; Jun Zhu; Le Song
  Deep learning on graph structures has shown exciting results in various
applications. However, few attentions have been paid to the robustness of such
models, in contrast to numerous research work for image or text adversarial
attack and defense. In this paper, we focus on the adversarial attacks that
fool the model by modifying the combinatorial structure of data. We first
propose a reinforcement learning based attack method that learns the
generalizable attack policy, while only requiring prediction labels from the
target classifier. Also, variants of genetic algorithms and gradient methods
are presented in the scenario where prediction confidence or gradients are
available. We use both synthetic and real-world data to show that, a family of
Graph Neural Network models are vulnerable to these attacks, in both
graph-level and node-level classification tasks. We also show such attacks can
be used to diagnose the learned classifiers.


http://arxiv.org/abs/1806.02256
Adversarial Regression with Multiple Learners.
Liang Tong; Sixie Yu; Scott Alfeld; Yevgeniy Vorobeychik
  Despite the considerable success enjoyed by machine learning techniques in
practice, numerous studies demonstrated that many approaches are vulnerable to
attacks. An important class of such attacks involves adversaries changing
features at test time to cause incorrect predictions. Previous investigations
of this problem pit a single learner against an adversary. However, in many
situations an adversary's decision is aimed at a collection of learners, rather
than specifically targeted at each independently. We study the problem of
adversarial linear regression with multiple learners. We approximate the
resulting game by exhibiting an upper bound on learner loss functions, and show
that the resulting game has a unique symmetric equilibrium. We present an
algorithm for computing this equilibrium, and show through extensive
experiments that equilibrium models are significantly more robust than
conventional regularized linear regression.


http://arxiv.org/abs/1806.02032
Killing Four Birds with one Gaussian Process: Analyzing Test-Time Attack Vectors on Classification.
Kathrin Grosse; Michael T. Smith; Michael Backes
  The wide usage of Machine Learning (ML) leads to direct security threats, as
ML algorithms are vulnerable to a plethora of attacks themselves. Different
attack vectors are known, and target for example the training phase using
manipulated data. Alternatively, they take place at test time and aim for
miss-classification, the leakage of the training data or extraction of the
model. Previous works studied different test time attacks individually. We show
that using an ML model enabling formal analysis and allowing control over the
decision surface curvature, interesting insights can be gained when attack
vectors are not studied in isolation but in relation to each pother. We show
for example how we can secure Gaussian Process Classification against empirical
membership inference by properly configuring the algorithm. In this
configuration, however, the model's parameters are leaked. This allows an
analytic computation of the the training data, which is thus leaked, against
the original intention of protecting the data. We extend our study to evasion
attacks, and find that analogously, hardening the model against one attack
boils down to enabling a different attacker.


http://arxiv.org/abs/1806.02299
DPatch: An Adversarial Patch Attack on Object Detectors.
Xin Liu; Huanrui Yang; Ziwei Liu; Linghao Song; Hai Li; Yiran Chen
  Object detectors have emerged as an indispensable module in modern computer
vision systems. In this work, we propose DPatch -- a black-box
adversarial-patch-based attack towards mainstream object detectors (i.e. Faster
R-CNN and YOLO). Unlike the original adversarial patch that only manipulates
image-level classifier, our DPatch simultaneously attacks the bounding box
regression and object classification so as to disable their predictions.
Compared to prior works, DPatch has several appealing properties: (1) DPatch
can perform both untargeted and targeted effective attacks, degrading the mAP
of Faster R-CNN and YOLO from 75.10% and 65.7% down to below 1%, respectively.
(2) DPatch is small in size and its attacking effect is location-independent,
making it very practical to implement real-world attacks. (3) DPatch
demonstrates great transferability among different detectors as well as
training datasets. For example, DPatch that is trained on Faster R-CNN can
effectively attack YOLO, and vice versa. Extensive evaluations imply that
DPatch can perform effective attacks under black-box setup, i.e., even without
the knowledge of the attacked network's architectures and parameters.
Successful realization of DPatch also illustrates the intrinsic vulnerability
of the modern detector architectures to such patch-based adversarial attacks.


http://arxiv.org/abs/1806.02190
Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noise.
Vahid Behzadan; Arslan Munir
  Recent developments have established the vulnerability of deep reinforcement
learning to policy manipulation attacks via intentionally perturbed inputs,
known as adversarial examples. In this work, we propose a technique for
mitigation of such attacks based on addition of noise to the parameter space of
deep reinforcement learners during training. We experimentally verify the
effect of parameter-space noise in reducing the transferability of adversarial
examples, and demonstrate the promising performance of this technique in
mitigating the impact of whitebox and blackbox attacks at both test and
training times.


http://arxiv.org/abs/1806.01477
An Explainable Adversarial Robustness Metric for Deep Learning Neural Networks.
Chirag Agarwal; Bo Dong; Dan Schonfeld; Anthony Hoogs
  Deep Neural Networks(DNN) have excessively advanced the field of computer
vision by achieving state of the art performance in various vision tasks. These
results are not limited to the field of vision but can also be seen in speech
recognition and machine translation tasks. Recently, DNNs are found to poorly
fail when tested with samples that are crafted by making imperceptible changes
to the original input images. This causes a gap between the validation and
adversarial performance of a DNN. An effective and generalizable robustness
metric for evaluating the performance of DNN on these adversarial inputs is
still missing from the literature. In this paper, we propose Noise Sensitivity
Score (NSS), a metric that quantifies the performance of a DNN on a specific
input under different forms of fix-directional attacks. An insightful
mathematical explanation is provided for deeply understanding the proposed
metric. By leveraging the NSS, we also proposed a skewness based dataset
robustness metric for evaluating a DNN's adversarial performance on a given
dataset. Extensive experiments using widely used state of the art architectures
along with popular classification datasets, such as MNIST, CIFAR-10, CIFAR-100,
and ImageNet, are used to validate the effectiveness and generalization of our
proposed metrics. Instead of simply measuring a DNN's adversarial robustness in
the input domain, as previous works, the proposed NSS is built on top of
insightful mathematical understanding of the adversarial attack and gives a
more explicit explanation of the robustness.


http://arxiv.org/abs/1806.01471
PAC-learning in the presence of evasion adversaries.
Daniel Cullina; Arjun Nitin Bhagoji; Prateek Mittal
  The existence of evasion attacks during the test phase of machine learning
algorithms represents a significant challenge to both their deployment and
understanding. These attacks can be carried out by adding imperceptible
perturbations to inputs to generate adversarial examples and finding effective
defenses and detectors has proven to be difficult. In this paper, we step away
from the attack-defense arms race and seek to understand the limits of what can
be learned in the presence of an evasion adversary. In particular, we extend
the Probably Approximately Correct (PAC)-learning framework to account for the
presence of an adversary. We first define corrupted hypothesis classes which
arise from standard binary hypothesis classes in the presence of an evasion
adversary and derive the Vapnik-Chervonenkis (VC)-dimension for these, denoted
as the adversarial VC-dimension. We then show that sample complexity upper
bounds from the Fundamental Theorem of Statistical learning can be extended to
the case of evasion adversaries, where the sample complexity is controlled by
the adversarial VC-dimension. We then explicitly derive the adversarial
VC-dimension for halfspace classifiers in the presence of a sample-wise
norm-constrained adversary of the type commonly studied for evasion attacks and
show that it is the same as the standard VC-dimension, closing an open
question. Finally, we prove that the adversarial VC-dimension can be either
larger or smaller than the standard VC-dimension depending on the hypothesis
class and adversary, making it an interesting object of study in its own right.


http://arxiv.org/abs/1806.00667
Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks.
Yarin Gal; Lewis Smith
  We prove, under two sufficient conditions, that idealised models can have no
adversarial examples. We discuss which idealised models satisfy our conditions,
and show that idealised Bayesian neural networks (BNNs) satisfy these. We
continue by studying near-idealised BNNs using HMC inference, demonstrating the
theoretical ideas in practice. We experiment with HMC on synthetic data derived
from MNIST for which we know the ground-truth image density, showing that
near-perfect epistemic uncertainty correlates to density under image manifold,
and that adversarial images lie off the manifold in our setting. This suggests
why MC dropout, which can be seen as performing approximate inference, has been
observed to be an effective defence against adversarial examples in practice;
We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting
a new attack for dropout models and a new defence as well. Lastly, we
demonstrate the defence on a cats-vs-dogs image classification task with a
VGG13 variant.


http://arxiv.org/abs/1806.00580
Detecting Adversarial Examples via Key-based Network.
Pinlong Zhao; Zhouyu Fu; Ou wu; Qinghua Hu; Jun Wang
  Though deep neural networks have achieved state-of-the-art performance in
visual classification, recent studies have shown that they are all vulnerable
to the attack of adversarial examples. Small and often imperceptible
perturbations to the input images are sufficient to fool the most powerful deep
neural networks. Various defense methods have been proposed to address this
issue. However, they either require knowledge on the process of generating
adversarial examples, or are not robust against new attacks specifically
designed to penetrate the existing defense. In this work, we introduce
key-based network, a new detection-based defense mechanism to distinguish
adversarial examples from normal ones based on error correcting output codes,
using the binary code vectors produced by multiple binary classifiers applied
to randomly chosen label-sets as signatures to match normal images and reject
adversarial examples. In contrast to existing defense methods, the proposed
method does not require knowledge of the process for generating adversarial
examples and can be applied to defend against different types of attacks. For
the practical black-box and gray-box scenarios, where the attacker does not
know the encoding scheme, we show empirically that key-based network can
effectively detect adversarial examples generated by several state-of-the-art
attacks.


http://arxiv.org/abs/1806.00088
PeerNets: Exploiting Peer Wisdom Against Adversarial Attacks.
Jan Svoboda; Jonathan Masci; Federico Monti; Michael M. Bronstein; Leonidas Guibas
  Deep learning systems have become ubiquitous in many aspects of our lives.
Unfortunately, it has been shown that such systems are vulnerable to
adversarial attacks, making them prone to potential unlawful uses. Designing
deep neural networks that are robust to adversarial attacks is a fundamental
step in making such systems safer and deployable in a broader variety of
applications (e.g. autonomous driving), but more importantly is a necessary
step to design novel and more advanced architectures built on new computational
paradigms rather than marginally building on the existing ones. In this paper
we introduce PeerNets, a novel family of convolutional networks alternating
classical Euclidean convolutions with graph convolutions to harness information
from a graph of peer samples. This results in a form of non-local forward
propagation in the model, where latent features are conditioned on the global
structure induced by the graph, that is up to 3 times more robust to a variety
of white- and black-box adversarial attacks compared to conventional
architectures with almost no drop in accuracy.


http://arxiv.org/abs/1806.00081
Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders.
Partha Ghosh; Arpan Losalka; Michael J Black
  Susceptibility of deep neural networks to adversarial attacks poses a major
theoretical and practical challenge. All efforts to harden classifiers against
such attacks have seen limited success. Two distinct categories of samples to
which deep networks are vulnerable, "adversarial samples" and "fooling
samples", have been tackled separately so far due to the difficulty posed when
considered together. In this work, we show how one can address them both under
one unified framework. We tie a discriminative model with a generative model,
rendering the adversarial objective to entail a conflict. Our model has the
form of a variational autoencoder, with a Gaussian mixture prior on the latent
vector. Each mixture component of the prior distribution corresponds to one of
the classes in the data. This enables us to perform selective classification,
leading to the rejection of adversarial samples instead of misclassification.
Our method inherently provides a way of learning a selective classifier in a
semi-supervised scenario as well, which can resist adversarial attacks. We also
show how one can reclassify the rejected adversarial samples.


http://arxiv.org/abs/1805.12514
Scaling provable adversarial defenses.
Eric Wong; Frank R. Schmidt; Jan Hendrik Metzen; J. Zico Kolter
  Recent work has developed methods for learning deep network classifiers that
are provably robust to norm-bounded adversarial perturbation; however, these
methods are currently only possible for relatively small feedforward networks.
In this paper, in an effort to scale these approaches to substantially larger
models, we extend previous work in three main directions. First, we present a
technique for extending these training procedures to much more general
networks, with skip connections (such as ResNets) and general nonlinearities;
the approach is fully modular, and can be implemented automatically (analogous
to automatic differentiation). Second, in the specific case of $\ell_\infty$
adversarial perturbations and networks with ReLU nonlinearities, we adopt a
nonlinear random projection for training, which scales linearly in the number
of hidden units (previous approaches scaled quadratically). Third, we show how
to further improve robust error through cascade models. On both MNIST and CIFAR
data sets, we train classifiers that improve substantially on the state of the
art in provable robust adversarial error bounds: from 5.8% to 3.1% on MNIST
(with $\ell_\infty$ perturbations of $\epsilon=0.1$), and from 80% to 36.4% on
CIFAR (with $\ell_\infty$ perturbations of $\epsilon=2/255$). Code for all
experiments in the paper is available at
https://github.com/locuslab/convex_adversarial/.


http://arxiv.org/abs/1805.12487
Sequential Attacks on Agents for Long-Term Adversarial Goals.
Edgar Tretschk; Seong Joon Oh; Mario Fritz
  Reinforcement learning (RL) has advanced greatly in the past few years with
the employment of effective deep neural networks (DNNs) on the policy networks.
With the great effectiveness came serious vulnerability issues with DNNs that
small adversarial perturbations on the input can change the output of the
network. Several works have pointed out that learned agents with a DNN policy
network can be manipulated against achieving the original task through a
sequence of small perturbations on the input states. In this paper, we
demonstrate furthermore that it is also possible to impose an arbitrary
adversarial reward on the victim policy network through a sequence of attacks.
Our method involves the latest adversarial attack technique, Adversarial
Transformer Network (ATN), that learns to generate the attack and is easy to
integrate into the policy network. As a result of our attack, the victim agent
is misguided to optimise for the adversarial reward over time. Our results
expose serious security threats for RL applications in safety-critical systems
including drones, medical analysis, and self-driving cars.


http://arxiv.org/abs/1805.12316
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data.
Puyudi Yang; Jianbo Chen; Cho-Jui Hsieh; Jane-Ling Wang; Michael I. Jordan
  We present a probabilistic framework for studying adversarial attacks on
discrete data. Based on this framework, we derive a perturbation-based method,
Greedy Attack, and a scalable learning-based method, Gumbel Attack, that
illustrate various tradeoffs in the design of attacks. We demonstrate the
effectiveness of these methods using both quantitative metrics and human
evaluation on various state-of-the-art models for text classification,
including a word-based CNN, a character-based CNN and an LSTM. As as example of
our results, we show that the accuracy of character-based convolutional
networks drops to the level of random selection by modifying only five
characters through Greedy Attack.


http://arxiv.org/abs/1805.12302
Adversarial Attacks on Face Detectors using Neural Net based Constrained Optimization.
Avishek Joey Bose; Parham Aarabi
  Adversarial attacks involve adding, small, often imperceptible, perturbations
to inputs with the goal of getting a machine learning model to misclassifying
them. While many different adversarial attack strategies have been proposed on
image classification models, object detection pipelines have been much harder
to break. In this paper, we propose a novel strategy to craft adversarial
examples by solving a constrained optimization problem using an adversarial
generator network. Our approach is fast and scalable, requiring only a forward
pass through our trained generator network to craft an adversarial sample.
Unlike in many attack strategies, we show that the same trained generator is
capable of attacking new images without explicitly optimizing on them. We
evaluate our attack on a trained Faster R-CNN face detector on the cropped
300-W face dataset where we manage to reduce the number of detected faces to
$0.5\%$ of all originally detected faces. In a different experiment, also on
300-W, we demonstrate the robustness of our attack to a JPEG compression based
defense typical JPEG compression level of $75\%$ reduces the effectiveness of
our attack from only $0.5\%$ of detected faces to a modest $5.0\%$.


http://arxiv.org/abs/1805.12152
Robustness May Be at Odds with Accuracy.
Dimitris Tsipras; Shibani Santurkar; Logan Engstrom; Alexander Turner; Aleksander Madry
  We show that there exists an inherent tension between the goal of adversarial
robustness and that of standard generalization. Specifically, training robust
models may not only be more resource-consuming, but also lead to a reduction of
standard accuracy. We demonstrate that this trade-off between the standard
accuracy of a model and its robustness to adversarial perturbations provably
exists even in a fairly simple and natural setting. These findings also
corroborate a similar phenomenon observed in practice. Further, we argue that
this phenomenon is a consequence of robust classifiers learning fundamentally
different feature representations than standard classifiers. These differences,
in particular, seem to result in unexpected benefits: the representations
learned by robust models tend to align better with salient data characteristics
and human perception.


http://arxiv.org/abs/1805.11852
ADAGIO: Interactive Experimentation with Adversarial Attack and Defense for Audio.
Nilaksh Das; Madhuri Shanbhogue; Shang-Tse Chen; Li Chen; Michael E. Kounavis; Duen Horng Chau
  Adversarial machine learning research has recently demonstrated the
feasibility to confuse automatic speech recognition (ASR) models by introducing
acoustically imperceptible perturbations to audio samples. To help researchers
and practitioners gain better understanding of the impact of such attacks, and
to provide them with tools to help them more easily evaluate and craft strong
defenses for their models, we present ADAGIO, the first tool designed to allow
interactive experimentation with adversarial attacks and defenses on an ASR
model in real time, both visually and aurally. ADAGIO incorporates AMR and MP3
audio compression techniques as defenses, which users can interactively apply
to attacked audio samples. We show that these techniques, which are based on
psychoacoustic principles, effectively eliminate targeted attacks, reducing the
attack success rate from 92.5% to 0%. We will demonstrate ADAGIO and invite the
audience to try it on the Mozilla Common Voice dataset.


http://arxiv.org/abs/1805.12017
Robustifying Models Against Adversarial Attacks by Langevin Dynamics.
Vignesh Srinivasan; Arturo Marban; Klaus-Robert Müller; Wojciech Samek; Shinichi Nakajima
  Adversarial attacks on deep learning models have compromised their
performance considerably. As remedies, a lot of defense methods were proposed,
which however, have been circumvented by newer attacking strategies. In the
midst of this ensuing arms race, the problem of robustness against adversarial
attacks still remains unsolved. This paper proposes a novel, simple yet
effective defense strategy where adversarial samples are relaxed onto the
underlying manifold of the (unknown) target class distribution. Specifically,
our algorithm drives off-manifold adversarial samples towards high density
regions of the data generating distribution of the target class by the
Metroplis-adjusted Langevin algorithm (MALA) with perceptual boundary taken
into account. Although the motivation is similar to projection methods, e.g.,
Defense-GAN, our algorithm, called MALA for DEfense (MALADE), is equipped with
significant dispersion - projection is distributed broadly, and therefore any
whitebox attack cannot accurately align the input so that the MALADE moves it
to a targeted untrained spot where the model predicts a wrong label. In our
experiments, MALADE exhibited state-of-the-art performance against various
elaborate attacking strategies.


http://arxiv.org/abs/1805.11770
AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks.
Chun-Chen Tu; Paishun Ting; Pin-Yu Chen; Sijia Liu; Huan Zhang; Jinfeng Yi; Cho-Jui Hsieh; Shin-Ming Cheng
  Recent studies have shown that adversarial examples in state-of-the-art image
classifiers trained by deep neural networks (DNN) can be easily generated when
the target model is transparent to an attacker, known as the white-box setting.
However, when attacking a deployed machine learning service, one can only
acquire the input-output correspondences of the target model; this is the
so-called black-box attack setting. The major drawback of existing black-box
attacks is the need for excessive model queries, which may give a false sense
of model robustness due to inefficient query designs. To bridge this gap, we
propose a generic framework for query-efficient black-box attacks. Our
framework, AutoZOOM, which is short for Autoencoder-based Zeroth Order
Optimization Method, has two novel building blocks towards efficient black-box
attacks: (i) an adaptive random gradient estimation strategy to balance query
counts and distortion, and (ii) an autoencoder that is either trained offline
with unlabeled data or a bilinear resizing operation for attack acceleration.
Experimental results suggest that, by applying AutoZOOM to a state-of-the-art
black-box attack (ZOO), a significant reduction in model queries can be
achieved without sacrificing the attack success rate and the visual quality of
the resulting adversarial examples. In particular, when compared to the
standard ZOO method, AutoZOOM can consistently reduce the mean query counts in
finding successful adversarial examples (or reaching the same distortion level)
by at least 93% on MNIST, CIFAR-10 and ImageNet datasets, leading to novel
insights on adversarial robustness.


http://arxiv.org/abs/1805.11596
Adversarial Noise Attacks of Deep Learning Architectures - Stability Analysis via Sparse Modeled Signals.
Yaniv Romano; Aviad Aberdam; Jeremias Sulam; Michael Elad
  Despite their impressive performance, deep convolutional neural networks
(CNNs) have been shown to be sensitive to small adversarial perturbations.
These nuisances, which one can barely notice, are powerful enough to fool
sophisticated and well performing classifiers, leading to ridiculous
misclassification results. In this paper we analyze the stability of
state-of-the-art deep-learning classification machines to adversarial
perturbations, where we assume that the signals belong to the (possibly
multi-layer) sparse representation model. We start with convolutional sparsity
and then proceed to its multi-layered version, which is tightly connected to
CNNs. Our analysis links between the stability of the classification to noise
and the underlying structure of the signal, quantified by the sparsity of its
representation under a fixed dictionary. In addition, we offer similar
stability theorems for two practical pursuit algorithms, which are posed as two
different deep-learning architectures - the layered Thresholding and the
layered Basis Pursuit. Our analysis establishes the better robustness of the
later to adversarial attacks. We corroborate these theoretical results by
numerical experiments on three datasets: MNIST, CIFAR-10 and CIFAR-100.


http://arxiv.org/abs/1805.11666
Why Botnets Work: Distributed Brute-Force Attacks Need No Synchronization.
Salman Salamatian; Wasim Huleihel; Ahmad Beirami; Asaf Cohen; Muriel Médard
  In September 2017, McAffee Labs quarterly report estimated that brute force
attacks represent 20\% of total network attacks, making them the most prevalent
type of attack ex-aequo with browser based vulnerabilities. These attacks have
sometimes catastrophic consequences, and understanding their fundamental limits
may play an important role in the risk assessment of password-secured systems,
and in the design of better security protocols. While some solutions exist to
prevent online brute-force attacks that arise from one single IP address,
attacks performed by botnets are more challenging. In this paper, we analyze
these distributed attacks by using a simplified model. Our aim is to understand
the impact of distribution and asynchronization on the overall computational
effort necessary to breach a system. Our result is based on Guesswork, a
measure of the number of queries (guesses) required of an adversary before a
correct sequence, such as a password, is found in an optimal attack. Guesswork
is a direct surrogate for time and computational effort of guessing a sequence
from a set of sequences with associated likelihoods. We model the lack of
synchronization by a worst-case optimization in which the queries made by
multiple adversarial agents are received in the worst possible order for the
adversary, resulting in a min-max formulation. We show that, even without
synchronization, and for sequences of growing length, the asymptotic optimal
performance is achievable by using randomized guesses drawn from an appropriate
distribution. Therefore, randomization is key for distributed asynchronous
attacks. In other words, asynchronous guessers can asymptotically perform
brute-force attacks as efficiently as synchronized guessers.


http://arxiv.org/abs/1805.10997
Adversarial Examples in Remote Sensing.
Wojciech Czaja; Neil Fendley; Michael Pekala; Christopher Ratto; I-Jeng Wang
  This paper considers attacks against machine learning algorithms used in
remote sensing applications, a domain that presents a suite of challenges that
are not fully addressed by current research focused on natural image data such
as ImageNet. In particular, we present a new study of adversarial examples in
the context of satellite image classification problems. Using a recently
curated data set and associated classifier, we provide a preliminary analysis
of adversarial examples in settings where the targeted classifier is permitted
multiple observations of the same location over time. While our experiments to
date are purely digital, our problem setup explicitly incorporates a number of
practical considerations that a real-world attacker would need to take into
account when mounting a physical attack. We hope this work provides a useful
starting point for future studies of potential vulnerabilities in this setting.


http://arxiv.org/abs/1805.11090
GenAttack: Practical Black-box Attacks with Gradient-Free Optimization.
Moustafa Alzantot; Yash Sharma; Supriyo Chakraborty; Huan Zhang; Cho-Jui Hsieh; Mani Srivastava
  Deep neural networks are vulnerable to adversarial examples, even in the
black-box setting, where the attacker is restricted solely to query access.
Existing black-box approaches to generating adversarial examples typically
require a significant number of queries, either for training a substitute
network or performing gradient estimation. We introduce GenAttack, a
gradient-free optimization technique that uses genetic algorithms for
synthesizing adversarial examples in the black-box setting. Our experiments on
different datasets (MNIST, CIFAR-10, and ImageNet) show that GenAttack can
successfully generate visually imperceptible adversarial examples against
state-of-the-art image recognition models with orders of magnitude fewer
queries than previous approaches. Against MNIST and CIFAR-10 models, GenAttack
required roughly 2,126 and 2,568 times fewer queries respectively, than ZOO,
the prior state-of-the-art black-box attack. In order to scale up the attack to
large-scale high-dimensional ImageNet models, we perform a series of
optimizations that further improve the query efficiency of our attack leading
to 237 times fewer queries against the Inception-v3 model than ZOO.
Furthermore, we show that GenAttack can successfully attack some
state-of-the-art ImageNet defenses, including ensemble adversarial training and
non-differentiable or randomized input transformations. Our results suggest
that evolutionary algorithms open up a promising area of research into
effective black-box attacks.


http://arxiv.org/abs/1805.10652
Defending Against Adversarial Attacks by Leveraging an Entire GAN.
Gokula Krishnan Santhanam; Paulina Grnarova
  Recent work has shown that state-of-the-art models are highly vulnerable to
adversarial perturbations of the input. We propose cowboy, an approach to
detecting and defending against adversarial attacks by using both the
discriminator and generator of a GAN trained on the same dataset. We show that
the discriminator consistently scores the adversarial samples lower than the
real samples across multiple attacks and datasets. We provide empirical
evidence that adversarial samples lie outside of the data manifold learned by
the GAN. Based on this, we propose a cleaning method which uses both the
discriminator and generator of the GAN to project the samples back onto the
data manifold. This cleaning procedure is independent of the classifier and
type of attack and thus can be deployed in existing systems.


http://arxiv.org/abs/1805.10265
Training verified learners with learned verifiers.
Krishnamurthy Dvijotham; Sven Gowal; Robert Stanforth; Relja Arandjelovic; Brendan O'Donoghue; Jonathan Uesato; Pushmeet Kohli
  This paper proposes a new algorithmic framework, predictor-verifier training,
to train neural networks that are verifiable, i.e., networks that provably
satisfy some desired input-output properties. The key idea is to simultaneously
train two networks: a predictor network that performs the task at hand,e.g.,
predicting labels given inputs, and a verifier network that computes a bound on
how well the predictor satisfies the properties being verified. Both networks
can be trained simultaneously to optimize a weighted combination of the
standard data-fitting loss and a term that bounds the maximum violation of the
property. Experiments show that not only is the predictor-verifier architecture
able to train networks to achieve state of the art verified robustness to
adversarial examples with much shorter training times (outperforming previous
algorithms on small datasets like MNIST and SVHN), but it can also be scaled to
produce the first known (to the best of our knowledge) verifiably robust
networks for CIFAR-10.


http://arxiv.org/abs/1805.10204
Adversarial examples from computational constraints.
Sébastien Bubeck; Eric Price; Ilya Razenshteyn
  Why are classifiers in high dimension vulnerable to "adversarial"
perturbations? We show that it is likely not due to information theoretic
limitations, but rather it could be due to computational constraints.
  First we prove that, for a broad set of classification tasks, the mere
existence of a robust classifier implies that it can be found by a possibly
exponential-time algorithm with relatively few training examples. Then we give
a particular classification task where learning a robust classifier is
computationally intractable. More precisely we construct a binary
classification task in high dimensional space which is (i) information
theoretically easy to learn robustly for large perturbations, (ii) efficiently
learnable (non-robustly) by a simple linear separator, (iii) yet is not
efficiently robustly learnable, even for small perturbations, by any algorithm
in the statistical query (SQ) model. This example gives an exponential
separation between classical learning and robust learning in the statistical
query model. It suggests that adversarial examples may be an unavoidable
byproduct of computational limitations of learning algorithms.


http://arxiv.org/abs/1805.10133
Laplacian Networks: Bounding Indicator Function Smoothness for Neural Network Robustness.
Carlos Eduardo Rosar Kos Lassance; Vincent Gripon; Antonio Ortega
  For the past few years, Deep Neural Network (DNN) robustness has become a
question of paramount importance. As a matter of fact, in sensitive settings
misclassification can lead to dramatic consequences. Such misclassifications
are likely to occur when facing adversarial attacks, hardware failures or
limitations, and imperfect signal acquisition. To address this question,
authors have proposed different approaches aiming at increasing the robustness
of DNNs, such as adding regularizers or training using noisy examples. In this
paper we propose a new regularizer built upon the Laplacian of similarity
graphs obtained from the representation of training data at each layer of the
DNN architecture. This regularizer penalizes large changes (across consecutive
layers in the architecture) in the distance between examples of different
classes, and as such enforces smooth variations of the class boundaries. Since
it is agnostic to the type of deformations that are expected when predicting
with the DNN, the proposed regularizer can be combined with existing ad-hoc
methods. We provide theoretical justification for this regularizer and
demonstrate its effectiveness to improve robustness of DNNs on classical
supervised learning vision datasets.


http://arxiv.org/abs/1805.09380
Anonymizing k-Facial Attributes via Adversarial Perturbations.
Saheb Chhabra; Richa Singh; Mayank Vatsa; Gaurav Gupta
  A face image not only provides details about the identity of a subject but
also reveals several attributes such as gender, race, sexual orientation, and
age. Advancements in machine learning algorithms and popularity of sharing
images on the World Wide Web, including social media websites, have increased
the scope of data analytics and information profiling from photo collections.
This poses a serious privacy threat for individuals who do not want to be
profiled. This research presents a novel algorithm for anonymizing selective
attributes which an individual does not want to share without affecting the
visual quality of images. Using the proposed algorithm, a user can select
single or multiple attributes to be surpassed while preserving identity
information and visual content. The proposed adversarial perturbation based
algorithm embeds imperceptible noise in an image such that attribute prediction
algorithm for the selected attribute yields incorrect classification result,
thereby preserving the information according to user's choice. Experiments on
three popular databases i.e. MUCT, LFWcrop, and CelebA show that the proposed
algorithm not only anonymizes k-attributes, but also preserves image quality
and identity information.


http://arxiv.org/abs/1805.09370
Towards Robust Training of Neural Networks by Regularizing Adversarial Gradients.
Fuxun Yu; Zirui Xu; Yanzhi Wang; Chenchen Liu; Xiang Chen
  In recent years, neural networks have demonstrated outstanding effectiveness
in a large amount of applications.However, recent works have shown that neural
networks are susceptible to adversarial examples, indicating possible flaws
intrinsic to the network structures. To address this problem and improve the
robustness of neural networks, we investigate the fundamental mechanisms behind
adversarial examples and propose a novel robust training method via regulating
adversarial gradients. The regulation effectively squeezes the adversarial
gradients of neural networks and significantly increases the difficulty of
adversarial example generation.Without any adversarial example involved, the
robust training method could generate naturally robust networks, which are
near-immune to various types of adversarial examples. Experiments show the
naturally robust networks can achieve optimal accuracy against Fast Gradient
Sign Method (FGSM) and C\&W attacks on MNIST, Cifar10, and Google Speech
Command dataset. Moreover, our proposed method also provides neural networks
with consistent robustness against transferable attacks.


http://arxiv.org/abs/1805.09190
Towards the first adversarially robust neural network model on MNIST.
Lukas Schott; Jonas Rauber; Matthias Bethge; Wieland Brendel
  Despite much effort, deep neural networks remain highly susceptible to tiny
input perturbations and even for MNIST, one of the most common toy datasets in
computer vision, no neural network model exists for which adversarial
perturbations are large and make semantic sense to humans. We show that even
the widely recognized and by far most successful defense by Madry et al. (1)
overfits on the L-infinity metric (it's highly susceptible to L2 and L0
perturbations), (2) classifies unrecognizable images with high certainty, (3)
performs not much better than simple input binarization and (4) features
adversarial perturbations that make little sense to humans. These results
suggest that MNIST is far from being solved in terms of adversarial robustness.
We present a novel robust classification model that performs analysis by
synthesis using learned class-conditional data distributions. We derive bounds
on the robustness and go to great length to empirically evaluate our model
using maximally effective adversarial attacks by (a) applying decision-based,
score-based, gradient-based and transfer-based attacks for several different Lp
norms, (b) by designing a new attack that exploits the structure of our
defended model and (c) by devising a novel decision-based attack that seeks to
minimize the number of perturbed pixels (L0). The results suggest that our
approach yields state-of-the-art robustness on MNIST against L0, L2 and
L-infinity perturbations and we demonstrate that most adversarial examples are
strongly perturbed towards the perceptual boundary between the original and the
adversarial class.


http://arxiv.org/abs/1805.08736
Adversarially Robust Training through Structured Gradient Regularization.
Kevin Roth; Aurelien Lucchi; Sebastian Nowozin; Thomas Hofmann
  We propose a novel data-dependent structured gradient regularizer to increase
the robustness of neural networks vis-a-vis adversarial perturbations. Our
regularizer can be derived as a controlled approximation from first principles,
leveraging the fundamental link between training with noise and regularization.
It adds very little computational overhead during learning and is simple to
implement generically in standard deep learning frameworks. Our experiments
provide strong evidence that structured gradient regularization can act as an
effective first line of defense against attacks based on low-level signal
corruption.


http://arxiv.org/abs/1805.08006
Bidirectional Learning for Robust Neural Networks.
Sidney Pontes-Filho; Marcus Liwicki
  A multilayer perceptron can behave as a generative classifier by applying
bidirectional learning (BL). It consists of training an undirected neural
network to map input to output and vice-versa; therefore it can produce a
classifier in one direction, and a generator in the opposite direction for the
same data. The learning process of BL tries to reproduce the neuroplasticity
stated in Hebbian theory using only backward propagation of errors. In this
paper, two novel learning techniques are introduced which use BL for improving
robustness to white noise static and adversarial examples. The first method is
bidirectional propagation of errors, which the error propagation occurs in
backward and forward directions. Motivated by the fact that its generative
model receives as input a constant vector per class, we introduce as a second
method the hybrid adversarial networks (HAN). Its generative model receives a
random vector as input and its training is based on generative adversarial
networks (GAN). To assess the performance of BL, we perform experiments using
several architectures with fully and convolutional layers, with and without
bias. Experimental results show that both methods improve robustness to white
noise static and adversarial examples, and even increase accuracy, but have
different behavior depending on the architecture and task, being more
beneficial to use the one or the other. Nevertheless, HAN using a convolutional
architecture with batch normalization presents outstanding robustness, reaching
state-of-the-art accuracy on adversarial examples of hand-written digits.


http://arxiv.org/abs/1805.08000
Adversarial Noise Layer: Regularize Neural Network By Adding Noise.
Zhonghui You; Jinmian Ye; Kunming Li; Zenglin Xu; Ping Wang
  In this paper, we introduce a novel regularization method called Adversarial
Noise Layer (ANL) and its efficient version called Class Adversarial Noise
Layer (CANL), which are able to significantly improve CNN's generalization
ability by adding carefully crafted noise into the intermediate layer
activations. ANL and CANL can be easily implemented and integrated with most of
the mainstream CNN-based models. We compared the effects of the different types
of noise and visually demonstrate that our proposed adversarial noise instruct
CNN models to learn to extract cleaner feature maps, which further reduce the
risk of over-fitting. We also conclude that models trained with ANL or CANL are
more robust to the adversarial examples generated by FGSM than the traditional
adversarial training approaches.


http://arxiv.org/abs/1805.07984
Adversarial Attacks on Neural Networks for Graph Data.
Daniel Zügner; Amir Akbarnejad; Stephan Günnemann
  Deep learning models for graphs have achieved strong performance for the task
of node classification. Despite their proliferation, currently there is no
study of their robustness to adversarial attacks. Yet, in domains where they
are likely to be used, e.g. the web, adversaries are common. Can deep learning
models for graphs be easily fooled? In this work, we introduce the first study
of adversarial attacks on attributed graphs, specifically focusing on models
exploiting ideas of graph convolutions. In addition to attacks at test time, we
tackle the more challenging class of poisoning/causative attacks, which focus
on the training phase of a machine learning model. We generate adversarial
perturbations targeting the node's features and the graph structure, thus,
taking the dependencies between instances in account. Moreover, we ensure that
the perturbations remain unnoticeable by preserving important data
characteristics. To cope with the underlying discrete domain we propose an
efficient algorithm Nettack exploiting incremental computations. Our
experimental study shows that accuracy of node classification significantly
drops even when performing only few perturbations. Even more, our attacks are
transferable: the learned attacks generalize to other state-of-the-art node
classification models and unsupervised approaches, and likewise are successful
even when only limited knowledge about the graph is given.


http://arxiv.org/abs/1805.07894
Constructing Unrestricted Adversarial Examples with Generative Models.
Yang Song; Rui Shu; Nate Kushman; Stefano Ermon
  Adversarial examples are typically constructed by perturbing an existing data
point within a small matrix norm, and current defense methods are focused on
guarding against this type of attack. In this paper, we propose unrestricted
adversarial examples, a new threat model where the attackers are not restricted
to small norm-bounded perturbations. Different from perturbation-based attacks,
we propose to synthesize unrestricted adversarial examples entirely from
scratch using conditional generative models. Specifically, we first train an
Auxiliary Classifier Generative Adversarial Network (AC-GAN) to model the
class-conditional distribution over data samples. Then, conditioned on a
desired class, we search over the AC-GAN latent space to find images that are
likely under the generative model and are misclassified by a target classifier.
We demonstrate through human evaluation that unrestricted adversarial examples
generated this way are legitimate and belong to the desired class. Our
empirical results on the MNIST, SVHN, and CelebA datasets show that
unrestricted adversarial examples can bypass strong adversarial training and
certified defense methods designed for traditional adversarial attacks.


http://arxiv.org/abs/1805.07862
Featurized Bidirectional GAN: Adversarial Defense via Adversarially Learned Semantic Inference.
Ruying Bao; Sihang Liang; Qingcan Wang
  Deep neural networks have been demonstrated to be vulnerable to adversarial
attacks, where small perturbations intentionally added to the original inputs
can fool the classifier. In this paper, we propose a defense method, Featurized
Bidirectional Generative Adversarial Networks (FBGAN), to extract the semantic
features of the input and filter the non-semantic perturbation. FBGAN is
pre-trained on the clean dataset in an unsupervised manner, adversarially
learning a bidirectional mapping between the high-dimensional data space and
the low-dimensional semantic space; also mutual information is applied to
disentangle the semantically meaningful features. After the bidirectional
mapping, the adversarial data can be reconstructed to denoised data, which
could be fed into any pre-trained classifier. We empirically show the quality
of reconstruction images and the effectiveness of defense.


http://arxiv.org/abs/1805.07820
Targeted Adversarial Examples for Black Box Audio Systems.
Rohan Taori; Amog Kamsetty; Brenton Chu; Nikita Vemuri
  The application of deep recurrent networks to audio transcription has led to
impressive gains in automatic speech recognition (ASR) systems. Many have
demonstrated that small adversarial perturbations can fool deep neural networks
into incorrectly predicting a specified target with high confidence. Current
work on fooling ASR systems have focused on white-box attacks, in which the
model architecture and parameters are known. In this paper, we adopt a
black-box approach to adversarial generation, combining the approaches of both
genetic algorithms and gradient estimation to solve the task. We achieve a
89.25% targeted attack similarity after 3000 generations while maintaining
94.6% audio file similarity.


http://arxiv.org/abs/1805.07816
Towards Understanding Limitations of Pixel Discretization Against Adversarial Attacks.
Jiefeng Chen; Xi Wu; Vaibhav Rastogi; Yingyu Liang; Somesh Jha
  Wide adoption of artificial neural networks in various domains has led to an
increasing interest in defending adversarial attacks against them.
Preprocessing defense methods such as pixel discretization are particularly
attractive in practice due to their simplicity, low computational overhead, and
applicability to various systems. It is observed that such methods work well on
simple datasets like MNIST, but break on more complicated ones like ImageNet
under recently proposed strong white-box attacks. To understand the conditions
for success and potentials for improvement, we study the pixel discretization
defense method, including more sophisticated variants that take into account
the properties of the dataset being discretized. Our results again show poor
resistance against the strong attacks. We analyze our results in a theoretical
framework and offer strong evidence that pixel discretization is unlikely to
work on all but the simplest of the datasets. Furthermore, our arguments
present insights why some other preprocessing defenses may be insecure.


http://arxiv.org/abs/1805.06605
Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models.
Pouya Samangouei; Maya Kabkab; Rama Chellappa
  In recent years, deep neural network approaches have been widely adopted for
machine learning tasks, including classification. However, they were shown to
be vulnerable to adversarial perturbations: carefully crafted small
perturbations can cause misclassification of legitimate images. We propose
Defense-GAN, a new framework leveraging the expressive capability of generative
models to defend deep neural networks against such attacks. Defense-GAN is
trained to model the distribution of unperturbed images. At inference time, it
finds a close output to a given image which does not contain the adversarial
changes. This output is then fed to the classifier. Our proposed method can be
used with any classification model and does not modify the classifier structure
or training procedure. It can also be used as a defense against any attack as
it does not assume knowledge of the process for generating the adversarial
examples. We empirically show that Defense-GAN is consistently effective
against different attack methods and improves on existing defense strategies.
Our code has been made publicly available at
https://github.com/kabkabm/defensegan


http://arxiv.org/abs/1805.06130
Towards Robust Neural Machine Translation.
Yong Cheng; Zhaopeng Tu; Fandong Meng; Junjie Zhai; Yang Liu
  Small perturbations in the input can severely distort intermediate
representations and thus impact translation quality of neural machine
translation (NMT) models. In this paper, we propose to improve the robustness
of NMT models with adversarial stability training. The basic idea is to make
both the encoder and decoder in NMT models robust against input perturbations
by enabling them to behave similarly for the original input and its perturbed
counterpart. Experimental results on Chinese-English, English-German and
English-French translation tasks show that our approaches can not only achieve
significant improvements over strong NMT systems but also improve the
robustness of NMT models.


http://arxiv.org/abs/1805.05010
Detecting Adversarial Samples for Deep Neural Networks through Mutation Testing.
Jingyi Wang; Jun Sun; Peixin Zhang; Xinyu Wang
  Recently, it has been shown that deep neural networks (DNN) are subject to
attacks through adversarial samples. Adversarial samples are often crafted
through adversarial perturbation, i.e., manipulating the original sample with
minor modifications so that the DNN model labels the sample incorrectly. Given
that it is almost impossible to train perfect DNN, adversarial samples are
shown to be easy to generate. As DNN are increasingly used in safety-critical
systems like autonomous cars, it is crucial to develop techniques for defending
such attacks. Existing defense mechanisms which aim to make adversarial
perturbation challenging have been shown to be ineffective. In this work, we
propose an alternative approach. We first observe that adversarial samples are
much more sensitive to perturbations than normal samples. That is, if we impose
random perturbations on a normal and an adversarial sample respectively, there
is a significant difference between the ratio of label change due to the
perturbations. Observing this, we design a statistical adversary detection
algorithm called nMutant (inspired by mutation testing from software
engineering community). Our experiments show that nMutant effectively detects
most of the adversarial samples generated by recently proposed attacking
methods. Furthermore, we provide an error bound with certain statistical
significance along with the detection.


http://arxiv.org/abs/1805.04810
AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning.
Jinyuan Jia; Neil Zhenqiang Gong
  Users in various web and mobile applications are vulnerable to attribute
inference attacks, in which an attacker leverages a machine learning classifier
to infer a target user's private attributes (e.g., location, sexual
orientation, political view) from its public data (e.g., rating scores, page
likes). Existing defenses leverage game theory or heuristics based on
correlations between the public data and attributes. These defenses are not
practical. Specifically, game-theoretic defenses require solving intractable
optimization problems, while correlation-based defenses incur large utility
loss of users' public data.
  In this paper, we present AttriGuard, a practical defense against attribute
inference attacks. AttriGuard is computationally tractable and has small
utility loss. Our AttriGuard works in two phases. Suppose we aim to protect a
user's private attribute. In Phase I, for each value of the attribute, we find
a minimum noise such that if we add the noise to the user's public data, then
the attacker's classifier is very likely to infer the attribute value for the
user. We find the minimum noise via adapting existing evasion attacks in
adversarial machine learning. In Phase II, we sample one attribute value
according to a certain probability distribution and add the corresponding noise
found in Phase I to the user's public data. We formulate finding the
probability distribution as solving a constrained convex optimization problem.
We extensively evaluate AttriGuard and compare it with existing methods using a
real-world dataset. Our results show that AttriGuard substantially outperforms
existing methods. Our work is the first one that shows evasion attacks can be
used as defensive techniques for privacy protection.


http://arxiv.org/abs/1805.04807
Curriculum Adversarial Training.
Qi-Zhi Cai; Min Du; Chang Liu; Dawn Song
  Recently, deep learning has been applied to many security-sensitive
applications, such as facial authentication. The existence of adversarial
examples hinders such applications. The state-of-the-art result on defense
shows that adversarial training can be applied to train a robust model on MNIST
against adversarial examples; but it fails to achieve a high empirical
worst-case accuracy on a more complex task, such as CIFAR-10 and SVHN. In our
work, we propose curriculum adversarial training (CAT) to resolve this issue.
The basic idea is to develop a curriculum of adversarial examples generated by
attacks with a wide range of strengths. With two techniques to mitigate the
forgetting and the generalization issues, we demonstrate that CAT can improve
the prior art's empirical worst-case accuracy by a large margin of 25% on
CIFAR-10 and 35% on SVHN. At the same, the model's performance on
non-adversarial inputs is comparable to the state-of-the-art models.


http://arxiv.org/abs/1805.04613
Breaking Transferability of Adversarial Samples with Randomness.
Yan Zhou; Murat Kantarcioglu; Bowei Xi
  We investigate the role of transferability of adversarial attacks in the
observed vulnerabilities of Deep Neural Networks (DNNs). We demonstrate that
introducing randomness to the DNN models is sufficient to defeat adversarial
attacks, given that the adversary does not have an unlimited attack budget.
Instead of making one specific DNN model robust to perfect knowledge attacks
(a.k.a, white box attacks), creating randomness within an army of DNNs
completely eliminates the possibility of perfect knowledge acquisition,
resulting in a significantly more robust DNN ensemble against the strongest
form of attacks. We also show that when the adversary has an unlimited budget
of data perturbation, all defensive techniques would eventually break down as
the budget increases. Therefore, it is important to understand the game saddle
point where the adversary would not further pursue this endeavor.
  Furthermore, we explore the relationship between attack severity and decision
boundary robustness in the version space. We empirically demonstrate that by
simply adding a small Gaussian random noise to the learned weights, a DNN model
can increase its resilience to adversarial attacks by as much as 74.2%. More
importantly, we show that by randomly activating/revealing a model from a pool
of pre-trained DNNs at each query request, we can put a tremendous strain on
the adversary's attack strategies. We compare our randomization techniques to
the Ensemble Adversarial Training technique and show that our randomization
techniques are superior under different attack budget constraints.


http://arxiv.org/abs/1805.03553
On Visual Hallmarks of Robustness to Adversarial Malware.
Alex Huang; Abdullah Al-Dujaili; Erik Hemberg; Una-May O'Reilly
  A central challenge of adversarial learning is to interpret the resulting
hardened model. In this contribution, we ask how robust generalization can be
visually discerned and whether a concise view of the interactions between a
hardened decision map and input samples is possible. We first provide a means
of visually comparing a hardened model's loss behavior with respect to the
adversarial variants generated during training versus loss behavior with
respect to adversarial variants generated from other sources. This allows us to
confirm that the association of observed flatness of a loss landscape with
generalization that is seen with naturally trained models extends to
adversarially hardened models and robust generalization. To complement these
means of interpreting model parameter robustness we also use self-organizing
maps to provide a visual means of superimposing adversarial and natural
variants on a model's decision space, thus allowing the model's global
robustness to be comprehensively examined.


http://arxiv.org/abs/1805.03438
Robust Classification with Convolutional Prototype Learning.
Hong-Ming Yang; Xu-Yao Zhang; Fei Yin; Cheng-Lin Liu
  Convolutional neural networks (CNNs) have been widely used for image
classification. Despite its high accuracies, CNN has been shown to be easily
fooled by some adversarial examples, indicating that CNN is not robust enough
for pattern classification. In this paper, we argue that the lack of robustness
for CNN is caused by the softmax layer, which is a totally discriminative model
and based on the assumption of closed world (i.e., with a fixed number of
categories). To improve the robustness, we propose a novel learning framework
called convolutional prototype learning (CPL). The advantage of using
prototypes is that it can well handle the open world recognition problem and
therefore improve the robustness. Under the framework of CPL, we design
multiple classification criteria to train the network. Moreover, a prototype
loss (PL) is proposed as a regularization to improve the intra-class
compactness of the feature representation, which can be viewed as a generative
model based on the Gaussian assumption of different classes. Experiments on
several datasets demonstrate that CPL can achieve comparable or even better
results than traditional CNN, and from the robustness perspective, CPL shows
great advantages for both the rejection and incremental category learning
tasks.


http://arxiv.org/abs/1805.02917
Interpretable Adversarial Perturbation in Input Embedding Space for Text.
Motoki Sato; Jun Suzuki; Hiroyuki Shindo; Yuji Matsumoto
  Following great success in the image processing field, the idea of
adversarial training has been applied to tasks in the natural language
processing (NLP) field. One promising approach directly applies adversarial
training developed in the image processing field to the input word embedding
space instead of the discrete input space of texts. However, this approach
abandons such interpretability as generating adversarial texts to significantly
improve the performance of NLP tasks. This paper restores interpretability to
such methods by restricting the directions of perturbations toward the existing
words in the input embedding space. As a result, we can straightforwardly
reconstruct each input with perturbations to an actual text by considering the
perturbations to be the replacement of words in the sentence while maintaining
or even improving the task performance.


http://arxiv.org/abs/1805.02131
A Counter-Forensic Method for CNN-Based Camera Model Identification.
David Güera; Yu Wang; Luca Bondi; Paolo Bestagini; Stefano Tubaro; Edward J. Delp
  An increasing number of digital images are being shared and accessed through
websites, media, and social applications. Many of these images have been
modified and are not authentic. Recent advances in the use of deep
convolutional neural networks (CNNs) have facilitated the task of analyzing the
veracity and authenticity of largely distributed image datasets. We examine in
this paper the problem of identifying the camera model or type that was used to
take an image and that can be spoofed. Due to the linear nature of CNNs and the
high-dimensionality of images, neural networks are vulnerable to attacks with
adversarial examples. These examples are imperceptibly different from correctly
classified images but are misclassified with high confidence by CNNs. In this
paper, we describe a counter-forensic method capable of subtly altering images
to change their estimated camera model when they are analyzed by any CNN-based
camera model detector. Our method can use both the Fast Gradient Sign Method
(FGSM) or the Jacobian-based Saliency Map Attack (JSMA) to craft these
adversarial images and does not require direct access to the CNN. Our results
show that even advanced deep learning architectures trained to analyze images
and obtain camera model information are still vulnerable to our proposed
method.


http://arxiv.org/abs/1805.01431
Siamese networks for generating adversarial examples.
Mandar Kulkarni; Aria Abubakar
  Machine learning models are vulnerable to adversarial examples. An adversary
modifies the input data such that humans still assign the same label, however,
machine learning models misclassify it. Previous approaches in the literature
demonstrated that adversarial examples can even be generated for the remotely
hosted model. In this paper, we propose a Siamese network based approach to
generate adversarial examples for a multiclass target CNN. We assume that the
adversary do not possess any knowledge of the target data distribution, and we
use an unlabeled mismatched dataset to query the target, e.g., for the
ResNet-50 target, we use the Food-101 dataset as the query. Initially, the
target model assigns labels to the query dataset, and a Siamese network is
trained on the image pairs derived from these multiclass labels. We learn the
\emph{adversarial perturbations} for the Siamese model and show that these
perturbations are also adversarial w.r.t. the target model. In experimental
results, we demonstrate effectiveness of our approach on MNIST, CIFAR-10 and
ImageNet targets with TinyImageNet/Food-101 query datasets.


http://arxiv.org/abs/1805.00089
Concolic Testing for Deep Neural Networks.
Youcheng Sun; Min Wu; Wenjie Ruan; Xiaowei Huang; Marta Kwiatkowska; Daniel Kroening
  Concolic testing combines program execution and symbolic analysis to explore
the execution paths of a software program. This paper presents the first
concolic testing approach for Deep Neural Networks (DNNs). More specifically,
we formalise coverage criteria for DNNs that have been studied in the
literature, and then develop a coherent method for performing concolic testing
to increase test coverage. Our experimental results show the effectiveness of
the concolic testing approach in both achieving high coverage and finding
adversarial examples.


http://arxiv.org/abs/1804.11313
How Robust are Deep Neural Networks?.
Biswa Sengupta; Karl J. Friston
  Convolutional and Recurrent, deep neural networks have been successful in
machine learning systems for computer vision, reinforcement learning, and other
allied fields. However, the robustness of such neural networks is seldom
apprised, especially after high classification accuracy has been attained. In
this paper, we evaluate the robustness of three recurrent neural networks to
tiny perturbations, on three widely used datasets, to argue that high accuracy
does not always mean a stable and a robust (to bounded perturbations,
adversarial attacks, etc.) system. Especially, normalizing the spectrum of the
discrete recurrent network to bound the spectrum (using power method, Rayleigh
quotient, etc.) on a unit disk produces stable, albeit highly non-robust neural
networks. Furthermore, using the $\epsilon$-pseudo-spectrum, we show that
training of recurrent networks, say using gradient-based methods, often result
in non-normal matrices that may or may not be diagonalizable. Therefore, the
open problem lies in constructing methods that optimize not only for accuracy
but also for the stability and the robustness of the underlying neural network,
a criterion that is distinct from the other.


http://arxiv.org/abs/1804.11285
Adversarially Robust Generalization Requires More Data.
Ludwig Schmidt; Shibani Santurkar; Dimitris Tsipras; Kunal Talwar; Aleksander Mądry
  Machine learning models are often susceptible to adversarial perturbations of
their inputs. Even small perturbations can cause state-of-the-art classifiers
with high "standard" accuracy to produce an incorrect prediction with high
confidence. To better understand this phenomenon, we study adversarially robust
learning from the viewpoint of generalization. We show that already in a simple
natural data model, the sample complexity of robust learning can be
significantly larger than that of "standard" learning. This gap is information
theoretic and holds irrespective of the training algorithm or the model family.
We complement our theoretical results with experiments on popular image
classification datasets and show that a similar gap exists here as well. We
postulate that the difficulty of training robust classifiers stems, at least
partially, from this inherently larger sample complexity.


http://arxiv.org/abs/1804.11022
Adversarial Regression for Detecting Attacks in Cyber-Physical Systems.
Amin Ghafouri; Yevgeniy Vorobeychik; Xenofon Koutsoukos
  Attacks in cyber-physical systems (CPS) which manipulate sensor readings can
cause enormous physical damage if undetected. Detection of attacks on sensors
is crucial to mitigate this issue. We study supervised regression as a means to
detect anomalous sensor readings, where each sensor's measurement is predicted
as a function of other sensors. We show that several common learning approaches
in this context are still vulnerable to \emph{stealthy attacks}, which
carefully modify readings of compromised sensors to cause desired damage while
remaining undetected. Next, we model the interaction between the CPS defender
and attacker as a Stackelberg game in which the defender chooses detection
thresholds, while the attacker deploys a stealthy attack in response. We
present a heuristic algorithm for finding an approximately optimal threshold
for the defender in this game, and show that it increases system resilience to
attacks without significantly increasing the false alarm rate.


http://arxiv.org/abs/1804.10829
Formal Security Analysis of Neural Networks using Symbolic Intervals.
Shiqi Wang; Kexin Pei; Justin Whitehouse; Junfeng Yang; Suman Jana
  Due to the increasing deployment of Deep Neural Networks (DNNs) in real-world
security-critical domains including autonomous vehicles and collision avoidance
systems, formally checking security properties of DNNs, especially under
different attacker capabilities, is becoming crucial. Most existing security
testing techniques for DNNs try to find adversarial examples without providing
any formal security guarantees about the non-existence of such adversarial
examples. Recently, several projects have used different types of
Satisfiability Modulo Theory (SMT) solvers to formally check security
properties of DNNs. However, all of these approaches are limited by the high
overhead caused by the solver.
  In this paper, we present a new direction for formally checking security
properties of DNNs without using SMT solvers. Instead, we leverage interval
arithmetic to compute rigorous bounds on the DNN outputs. Our approach, unlike
existing solver-based approaches, is easily parallelizable. We further present
symbolic interval analysis along with several other optimizations to minimize
overestimations of output bounds.
  We design, implement, and evaluate our approach as part of ReluVal, a system
for formally checking security properties of Relu-based DNNs. Our extensive
empirical results show that ReluVal outperforms Reluplex, a state-of-the-art
solver-based system, by 200 times on average. On a single 8-core machine
without GPUs, within 4 hours, ReluVal is able to verify a security property
that Reluplex deemed inconclusive due to timeout after running for more than 5
days. Our experiments demonstrate that symbolic interval analysis is a
promising new direction towards rigorously analyzing different security
properties of DNNs.


http://arxiv.org/abs/1804.09699
Towards Fast Computation of Certified Robustness for ReLU Networks.
Tsui-Wei Weng; Huan Zhang; Hongge Chen; Zhao Song; Cho-Jui Hsieh; Duane Boning; Inderjit S. Dhillon; Luca Daniel
  Verifying the robustness property of a general Rectified Linear Unit (ReLU)
network is an NP-complete problem [Katz, Barrett, Dill, Julian and Kochenderfer
CAV17]. Although finding the exact minimum adversarial distortion is hard,
giving a certified lower bound of the minimum distortion is possible. Current
available methods of computing such a bound are either time-consuming or
delivering low quality bounds that are too loose to be useful. In this paper,
we exploit the special structure of ReLU networks and provide two
computationally efficient algorithms Fast-Lin and Fast-Lip that are able to
certify non-trivial lower bounds of minimum distortions, by bounding the ReLU
units with appropriate linear functions Fast-Lin, or by bounding the local
Lipschitz constant Fast-Lip. Experiments show that (1) our proposed methods
deliver bounds close to (the gap is 2-3X) exact minimum distortion found by
Reluplex in small MNIST networks while our algorithms are more than 10,000
times faster; (2) our methods deliver similar quality of bounds (the gap is
within 35% and usually around 10%; sometimes our bounds are even better) for
larger networks compared to the methods based on solving linear programming
problems but our algorithms are 33-14,000 times faster; (3) our method is
capable of solving large MNIST and CIFAR networks up to 7 layers with more than
10,000 neurons within tens of seconds on a single CPU core.
  In addition, we show that, in fact, there is no polynomial time algorithm
that can approximately find the minimum $\ell_1$ adversarial distortion of a
ReLU network with a $0.99\ln n$ approximation ratio unless
$\mathsf{NP}$=$\mathsf{P}$, where $n$ is the number of neurons in the network.


http://arxiv.org/abs/1804.08794
Towards Dependable Deep Convolutional Neural Networks (CNNs) with Out-distribution Learning.
Mahdieh Abbasi; Arezoo Rajabi; Christian Gagné; Rakesh B. Bobba
  Detection and rejection of adversarial examples in security sensitive and
safety-critical systems using deep CNNs is essential. In this paper, we propose
an approach to augment CNNs with out-distribution learning in order to reduce
misclassification rate by rejecting adversarial examples. We empirically show
that our augmented CNNs can either reject or classify correctly most
adversarial examples generated using well-known methods ( >95% for MNIST and
>75% for CIFAR-10 on average). Furthermore, we achieve this without requiring
to train using any specific type of adversarial examples and without
sacrificing the accuracy of models on clean samples significantly (< 4%).


http://arxiv.org/abs/1804.08757
Siamese Generative Adversarial Privatizer for Biometric Data.
Witold Oleszkiewicz; Peter Kairouz; Karol Piczak; Ram Rajagopal; Tomasz Trzcinski
  State-of-the-art machine learning algorithms can be fooled by carefully
crafted adversarial examples. As such, adversarial examples present a concrete
problem in AI safety. In this work we turn the tables and ask the following
question: can we harness the power of adversarial examples to prevent malicious
adversaries from learning identifying information from data while allowing
non-malicious entities to benefit from the utility of the same data? For
instance, can we use adversarial examples to anonymize biometric dataset of
faces while retaining usefulness of this data for other purposes, such as
emotion recognition? To address this question, we propose a simple yet
effective method, called Siamese Generative Adversarial Privatizer (SGAP), that
exploits the properties of a Siamese neural network to find discriminative
features that convey identifying information. When coupled with a generative
model, our approach is able to correctly locate and disguise identifying
information, while minimally reducing the utility of the privatized dataset.
Extensive evaluation on a biometric dataset of fingerprints and cartoon faces
confirms usefulness of our simple yet effective method.


http://arxiv.org/abs/1804.08598
Black-box Adversarial Attacks with Limited Queries and Information.
Andrew Ilyas; Logan Engstrom; Anish Athalye; Jessy Lin
  Current neural network-based classifiers are susceptible to adversarial
examples even in the black-box setting, where the attacker only has query
access to the model. In practice, the threat model for real-world systems is
often more restrictive than the typical black-box model where the adversary can
observe the full output of the network on arbitrarily many chosen inputs. We
define three realistic threat models that more accurately characterize many
real-world classifiers: the query-limited setting, the partial-information
setting, and the label-only setting. We develop new attacks that fool
classifiers under these more restrictive threat models, where previous methods
would be impractical or ineffective. We demonstrate that our methods are
effective against an ImageNet classifier under our proposed threat models. We
also demonstrate a targeted black-box attack against a commercial classifier,
overcoming the challenges of limited query access, partial information, and
other practical issues to break the Google Cloud Vision API.


http://arxiv.org/abs/1804.08529
VectorDefense: Vectorization as a Defense to Adversarial Examples.
Vishaal Munusamy Kabilan; Brandon Morris; Anh Nguyen
  Training deep neural networks on images represented as grids of pixels has
brought to light an interesting phenomenon known as adversarial examples.
Inspired by how humans reconstruct abstract concepts, we attempt to codify the
input bitmap image into a set of compact, interpretable elements to avoid being
fooled by the adversarial structures. We take the first step in this direction
by experimenting with image vectorization as an input transformation step to
map the adversarial examples back into the natural manifold of MNIST
handwritten digits. We compare our method vs. state-of-the-art input
transformations and further discuss the trade-offs between a hand-designed and
a learned transformation defense.


http://arxiv.org/abs/1804.08778
Query-Efficient GAN Based Black-Box Attack Against Sequence Based Classifiers.
Ishai Rosenberg; Asaf Shabtai; Yuval Elovici; Lior Rokach
  In this paper we present a generic and query-efficient black-box attack
demonstrated against API call based machine learning malware classifiers. We
generate adversarial examples combining sequences (API call sequences) and
other features (e.g., printable strings) that will be misclassified by the
classifier without affecting the malware functionality. Opposed to previous
studies, our attack minimizes the number of target recurrent neural network
(RNN) classifier queries and only requires access to the predicted label of the
attacked model (with or without its confidence score). We evaluate the attack's
effectiveness against a variety of classifiers, including RNN variants, deep
neural networks, support vector machines, and gradient-boosted decision trees.
Our attack success rate is about 98% when the confidence score is known and 88%
when only the label is known. We implement four state of the art
query-efficient attacks for sequence input and show that our attack requires
fewer queries and less knowledge about the attacked model's architecture than
other existing query-efficient attacks, making it practical for attacking cloud
based models at a minimal cost.


http://arxiv.org/abs/1804.07998
Generating Natural Language Adversarial Examples.
Moustafa Alzantot; Yash Sharma; Ahmed Elgohary; Bo-Jhang Ho; Mani Srivastava; Kai-Wei Chang
  Deep neural networks (DNNs) are vulnerable to adversarial examples,
perturbations to correctly classified examples which can cause the model to
misclassify. In the image domain, these perturbations are often virtually
indistinguishable to human perception, causing humans and state-of-the-art
models to disagree. However, in the natural language domain, small
perturbations are clearly perceptible, and the replacement of a single word can
drastically alter the semantics of the document. Given these challenges, we use
a black-box population-based optimization algorithm to generate semantically
and syntactically similar adversarial examples that fool well-trained sentiment
analysis and textual entailment models with success rates of 97% and 70%,
respectively. We additionally demonstrate that 92.3% of the successful
sentiment analysis adversarial examples are classified to their original label
by 20 human annotators, and that the examples are perceptibly quite similar.
Finally, we discuss an attempt to use adversarial training as a defense, but
fail to yield improvement, demonstrating the strength and diversity of our
adversarial examples. We hope our findings encourage researchers to pursue
improving the robustness of DNNs in the natural language domain.


http://arxiv.org/abs/1804.07870
Gradient Masking Causes CLEVER to Overestimate Adversarial Perturbation Size.
Ian Goodfellow
  A key problem in research on adversarial examples is that vulnerability to
adversarial examples is usually measured by running attack algorithms. Because
the attack algorithms are not optimal, the attack algorithms are prone to
overestimating the size of perturbation needed to fool the target model. In
other words, the attack-based methodology provides an upper-bound on the size
of a perturbation that will fool the model, but security guarantees require a
lower bound. CLEVER is a proposed scoring method to estimate a lower bound.
Unfortunately, an estimate of a bound is not a bound. In this report, we show
that gradient masking, a common problem that causes attack methodologies to
provide only a very loose upper bound, causes CLEVER to overestimate the size
of perturbation needed to fool the model. In other words, CLEVER does not
resolve the key problem with the attack-based methodology, because it fails to
provide a lower bound.


http://arxiv.org/abs/1804.07757
Learning More Robust Features with Adversarial Training.
Shuangtao Li; Yuanke Chen; Yanlin Peng; Lin Bai
  In recent years, it has been found that neural networks can be easily fooled
by adversarial examples, which is a potential safety hazard in some
safety-critical applications. Many researchers have proposed various method to
make neural networks more robust to white-box adversarial attacks, but an
effective method have not been found so far. In this short paper, we focus on
the robustness of the features learned by neural networks. We show that the
features learned by neural networks are not robust, and find that the
robustness of the learned features is closely related to the resistance against
adversarial examples of neural networks. We also find that adversarial training
against fast gradients sign method (FGSM) does not make the leaned features
very robust, even if it can make the trained networks very resistant to FGSM
attack. Then we propose a method, which can be seen as an extension of
adversarial training, to train neural networks to learn more robust features.
We perform experiments on MNIST and CIFAR-10 to evaluate our method, and the
experiment results show that this method greatly improves the robustness of the
learned features and the resistance to adversarial attacks.


http://arxiv.org/abs/1804.07729
ADef: an Iterative Algorithm to Construct Adversarial Deformations.
Rima Alaifari; Giovanni S. Alberti; Tandri Gauksson
  While deep neural networks have proven to be a powerful tool for many
recognition and classification tasks, their stability properties are still not
well understood. In the past, image classifiers have been shown to be
vulnerable to so-called adversarial attacks, which are created by additively
perturbing the correctly classified image. In this paper, we propose the ADef
algorithm to construct a different kind of adversarial attack created by
iteratively applying small deformations to the image, found through a gradient
descent step. We demonstrate our results on MNIST with convolutional neural
networks and on ImageNet with Inception-v3 and ResNet-101.


http://arxiv.org/abs/1804.07062
Attacking Convolutional Neural Network using Differential Evolution.
Jiawei Su; Danilo Vasconcellos Vargas; Kouichi Sakurai
  The output of Convolutional Neural Networks (CNN) has been shown to be
discontinuous which can make the CNN image classifier vulnerable to small
well-tuned artificial perturbations. That is, images modified by adding such
perturbations(i.e. adversarial perturbations) that make little difference to
human eyes, can completely alter the CNN classification results. In this paper,
we propose a practical attack using differential evolution(DE) for generating
effective adversarial perturbations. We comprehensively evaluate the
effectiveness of different types of DEs for conducting the attack on different
network structures. The proposed method is a black-box attack which only
requires the miracle feedback of the target CNN systems. The results show that
under strict constraints which simultaneously control the number of pixels
ch