# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
We want f to have the property that   C(x + δ) = t ⟺ f(x + δ) ≤ 0. That is, if the network classifies x + δ as target t then (and only then) the chosen f should be ≤ 0. Let’s check each equation. 1. fâ‚(x′) = –loss_{F,t}(x′) + 1  • The cross-entropy loss loss_{F,t}(x′) for the target is, by definition, –log(F(x′)_t). For a sample that is very confidently classified as t, F(x′)_t is near 1 so loss is near 0, and then f₠≈ 1 > 0. In contrast, the “if and only if†condition requires f(x′) ≤ 0 when the classification is t. Thus, fâ‚ does not meet the specification. 2. fâ‚‚(x′) = (max_{i≠t}( F(x′)_i) – F(x′)_t)⺠ • When the target probability is larger than every other (i.e. F(x′)_t > max_{i≠t}F(x′)_i), then the difference is negative and the (â‹…)⺠(which means max(e,0)) yields 0. If any other class wins, the term is positive. So fâ‚‚ = 0 exactly in the “correct†case and > 0 when not. (The borderline case when F(x′)_t equals the max is acceptable.) This matches the criterion. 3. f₃(x′) = softplus(max_{i≠t}(F(x′)_i) – F(x′)_t) – log 2  • Note that softplus(x) = log(1 + exp(x)) and has the property that softplus(0) = log 2, and softplus(x) < log 2 for x < 0. Thus, if F(x′)_t is strictly larger than all other F(x′)_i (x < 0) then f₃ < 0; if F(x′)_t equals the highest other entry then (x=0) f₃ = 0; and if it is lower then f₃ > 0. This fits the specification. 4. fâ‚„(x′) = (0.5 – F(x′)_t)⺠ • Here the condition fâ‚„(x′) ≤ 0 amounts to requiring 0.5 – F(x′)_t ≤ 0, i.e. F(x′)_t ≥ 0.5. But a network’s output might be the correct (largest) class even if its probability is below 0.5. In other words, the decision “F(x′)_t ≥ 0.5†is not equivalent to “t is the argmax class.†Hence, fâ‚„ is not correct. 5. fâ‚…(x′) = –log(2 F(x′)_t – 2)  • Rewrite the inside: 2 F(x′)_t – 2 = 2 (F(x′)_t – 1). Since F(x′)_t is in [0,1] the term F(x′)_t – 1 is non-positive (and 0 only when F(x′)_t = 1). This means that for almost every x′ the argument of log is ≤ 0 (and log is undefined for non-positive values in the reals). Thus, fâ‚… is not a valid objective according to the specification. 6. f₆(x′) = (max_{i≠t}( Z(x′)_i) – Z(x′)_t)⺠ • Recall that the softmax is monotonic with respect to its inputs (the logits), so the argmax of F(x′) is the same as the argmax of Z(x′). Therefore, when Z(x′)_t is greater than every other component it will yield (max_{i≠t}(Z(x′)_i) – Z(x′)_t) < 0 and then (â‹…)⺠gives 0, exactly as desired. Otherwise it is positive. This f₆ meets the property. 7. f₇(x′) = softplus(max_{i≠t}(Z(x′)_i) – Z(x′)_t) – log 2  • The same reasoning as for f₃ applies here, now with logits in place of probabilities. When the target logit is highest the inner margin is < 0 (or 0 in the tie case), so softplus yields a value ≤ log 2, and hence f₇ ≤ 0 exactly when t is the selected class. This is correct. To summarize, the functions that are inconsistent with the requirement “f(x+δ) ≤ 0 if and only if C(x+δ) = t†are equations 1, 4, and 5. Thus, the final answer is: [1, 4, 5]
Resulting in output: False
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
We require that f be such that C(x′) = t (i.e. x′ is classified as target t) if and only if f(x′) ≤ 0. Let’s examine each function: 1. fâ‚(x′) = –loss_F,t(x′) + 1 Here the crossâ€entropy loss is defined as loss_F,t(x′) = –log(F(x′)_t). So   fâ‚(x′) = log(F(x′)_t) + 1. For fâ‚(x′) ≤ 0 we need log(F(x′)_t) ≤ –1, i.e. F(x′)_t ≤ exp(–1) ≈ 0.3679. However, correct classification means that F(x′)_t is merely the largest component among all classes – it can easily be below 0.3679. Thus, the condition fâ‚(x′) ≤ 0 is not equivalent to “x′ is classified as t.†⇒ fâ‚ is incorrect. 2. fâ‚‚(x′) = (max_{i≠t}(F(x′)_i) – F(x′)_t)^+ If t is the class with the maximum probability then F(x′)_t is greater than any F(x′)_i for i ≠t, making the difference negative and the positive part (denoted by “^+â€) equal to 0. If some other class has a higher probability then the difference is positive. ⇒ fâ‚‚ meets the specification. 3. f₃(x′) = softplus(max_{i≠t}(F(x′)_i) – F(x′)_t) – log(2) softplus(x) = log(1+exp(x)). Note that softplus(0) = log2, so at the decision boundary (when max_{i≠t}(F(x′)_i) = F(x′)_t) f₃ = 0. When F(x′)_t is larger (correct classification) the softplus term is less than log2 (giving a negative value), and otherwise positive. ⇒ f₃ is correct. 4. fâ‚„(x′)= (0.5 – F(x′)_t)^+ This function forces a threshold of 0.5 on F(x′)_t. However, correct classification only requires that F(x′)_t be the largest output, not necessarily at least 0.5. For instance, in a 3‑class problem the correct target might have probability 0.4 while the others are lower (say 0.35 and 0.25), yet fâ‚„ would be (0.5–0.4) = 0.1 > 0. ⇒ fâ‚„ is incorrect. 5. fâ‚…(x′) = –log(2 F(x′)_t – 2) Notice that 2 F(x′)_t – 2 = 2(F(x′)_t – 1). Since F(x′)_t ∈ [0,1], the argument inside the log is non-positive except possibly when F(x′)_t = 1 (in which case it is 0 and log 0 is undefined). In almost all cases fâ‚… is either undefined or does not meet the specification. ⇒ fâ‚… is incorrect. 6. f₆(x′) = (max_{i≠t}(Z(x′)_i) – Z(x′)_t)^+ Here the function uses the logits Z(x′), and since the ordering is preserved under softmax the condition Z(x′)_t > Z(x′)_i for all i ≠t is equivalent to correct classification. If that holds, the inside difference is negative and f₆ = 0 (after taking the positive part); otherwise it is positive. ⇒ f₆ is correct. 7. f₇(x′) = softplus(max_{i≠t}(Z(x′)_i) – Z(x′)_t) – log(2) As with f₃ but using logits instead of probabilities, note that when max_{i≠t}(Z(x′)_i) – Z(x′)_t = 0, softplus returns log2 (so f₇ = 0). For correct classification the difference is negative (yielding f₇ < 0) and vice versa. ⇒ f₇ is correct. Thus, the functions that do not meet the specification are fâ‚, fâ‚„, and fâ‚…. Final answer: [1, 4, 5]
Resulting in output: False