# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Analysis of Equations Against Specification I'll examine each equation to determine if it correctly meets the specification: $C(x+\delta) = t$ if and only if $f(x+\delta) \leq 0$. ## Equation 1: $f_{1}(x') = -\loss_{F,t}(x') + 1$ - Cross-entropy loss $\loss_{F,t}(x')$ approaches 0 when the model confidently predicts class $t$ - For high confidence predictions where $F(x')_t$ is large (e.g., 0.8), we get $f_1(x') > 0$ - However, for correct but low-confidence predictions (e.g., $F(x')_t = 0.2$ being the highest probability), we get $f_1(x') < 0$ - This means $f_1(x') \leq 0$ doesn't correspond exactly to when $C(x') = t$ - **Incorrect** ## Equation 2: $f_{2}(x') = (\max_{i \ne t}( F(x')_i) - F(x')_t)^+$ - This computes the difference between the highest non-t probability and the probability of class t - When $t$ has the highest probability: $\max_{i \ne t}(F(x')_i) < F(x')_t$, so $f_2(x') = 0$ - When any other class has higher probability: $f_2(x') > 0$ - Therefore, $f_2(x') \leq 0$ if and only if $C(x') = t$ - **Correct** ## Equation 3: $f_{3}(x') = \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)$ - When $t$ has the highest probability: $\max_{i \ne t}(F(x')_i) - F(x')_t < 0$ - This makes $\text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t) < \log(2)$, so $f_3(x') < 0$ - When $t$ doesn't have highest probability: $f_3(x') > 0$ - At the decision boundary when probabilities are equal: $f_3(x') = 0$ - Therefore, $f_3(x') \leq 0$ if and only if $C(x') = t$ - **Correct** ## Equation 4: $f_{4}(x') = (0.5 - F(x')_t)^+$ - This function is 0 when $F(x')_t \geq 0.5$ and positive when $F(x')_t < 0.5$ - In multiclass classification with many classes, $t$ could be the predicted class (highest probability) without reaching 0.5 - For example, with probabilities [0.4, 0.3, 0.3], class 0 is predicted but $f_4(x') > 0$ - Therefore, $f_4(x') \leq 0$ doesn't correspond exactly to when $C(x') = t$ - **Incorrect** ## Equation 5: $f_{5}(x') = -\log(2 F(x')_t - 2)$ - Since $F(x')_t \in [0,1]$ (probability), $2F(x')_t - 2 \in [-2,0]$ - The logarithm of a negative number is undefined - This function is mathematically invalid for all valid probability values - **Incorrect** ## Equation 6: $f_{6}(x') = (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+$ - Similar to $f_2$ but using logits instead of probabilities - Since softmax preserves ordering, the class with highest logit is the predicted class - When $t$ has the highest logit: $f_6(x') = 0$ - When any other class has higher logit: $f_6(x') > 0$ - Therefore, $f_6(x') \leq 0$ if and only if $C(x') = t$ - **Correct** ## Equation 7: $f_{7}(x') = \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2)$ - Similar to $f_3$ but using logits instead of probabilities - When $t$ has the highest logit: $f_7(x') < 0$ - When $t$ doesn't have highest logit: $f_7(x') > 0$ - At the decision boundary: $f_7(x') = 0$ - Therefore, $f_7(x') \leq 0$ if and only if $C(x') = t$ - **Correct** The incorrect equations are: [1, 4, 5]
Resulting in output: False
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Setup Which of the following equations are incorrect according to the specification? # Notation A neural network is a function $F(x) = y$ that accepts an input $x \in \mathbb{R}^n$ and produces an output $y \in \mathbb{R}^m$. The model $F$ also implicitly depends on some model parameters $\theta$; in our work the model is fixed, so for convenience we don't show the dependence on $\theta$. In this paper we focus on neural networks used as an $m$-class classifier. The output of the network is computed using the softmax function, which ensures that the output vector $y$ satisfies $0 \le y_i \le 1$ and $y_1 + \dots + y_m = 1$. The output vector $y$ is thus treated as a probability distribution, i.e., $y_i$ is treated as the probability that input $x$ has class $i$. The classifier assigns the label $C(x) = \arg\max_i F(x)_i$ to the input $x$. Let $C^*(x)$ be the correct label of $x$. The inputs to the softmax function are called \emph{logits}. We use the notation from Papernot et al. \cite{distillation}: define $F$ to be the full neural network including the softmax function, $Z(x) = z$ to be the output of all layers except the softmax (so $z$ are the logits), and \begin{equation*} F(x) = \softmax(Z(x)) = y. \end{equation*} A neural network typically \footnote{Most simple networks have this simple linear structure, however other more sophisticated networks have more complicated structures (e.g., ResNet \cite{he2016deep} and Inception \cite{szegedy2015rethinking}). The network architecture does not impact our attacks.} consists of layers \begin{equation*} F = \softmax \circ F_n \circ F_{n-1} \circ \cdots \circ F_1 \end{equation*} where \begin{equation*} F_i(x) = \sigma(\theta_i \cdot x) + \hat\theta_i \end{equation*} for some non-linear activation function $\sigma$, some matrix $\theta_i$ of model weights, and some vector $\hat\theta_i$ of model biases. Together $\theta$ and $\hat\theta$ make up the model parameters. Common choices of $\sigma$ are tanh \cite{mishkin2015all}, sigmoid, ReLU \cite{maas2013rectifier}, or ELU \cite{clevert2015fast}. In this paper we focus primarily on networks that use a ReLU activation function, as it currently is the most widely used activation function \cite{szegedy2015rethinking,springenberg2014striving,mishkin2015all,distillation}. # Specification: We define an objective function $f$ such that $C(x+\delta) = t$ if and only if $f(x+\delta) \le 0$. There are many possible choices for $f$: # The equations \begin{align*} f_{1}(x') &= -\loss_{F,t}(x') + 1\\ f_{2}(x') &= (\max_{i \ne t}( F(x')_i) - F(x')_t)^+\\ f_{3}(x') &= \text{softplus}(\max_{i \ne t}(F(x')_i) - F(x')_t)-\log(2)\\ f_{4}(x') &= (0.5 - F(x')_t)^+\\ f_{5}(x') &= -\log(2 F(x')_t - 2)\\ f_{6}(x') &= (\max_{i \ne t}( Z(x')_i) - Z(x')_t)^+\\ f_{7}(x') &= \text{softplus}(\max_{i \ne t}(Z(x')_i) - Z(x')_t)-\log(2) \end{align*} where $s$ is the correct classification, $(e)^+$ is short-hand for $\max(e,0)$, $\text{softplus}(x) = \log(1+\exp(x))$, and $\loss_{F,s}(x)$ is the cross entropy loss for $x$. # Your task Consider each equation one by one. End your answer with a python list of numbers [1,2,3,4,5,6,7] for those that are wrong.
# Analysis of Each Equation I'll examine each equation to see if it satisfies the specification that $C(x+\delta) = t$ if and only if $f(x+\delta) \leq 0$. For brevity, I'll use $x'$ in place of $x+\delta$. ## Equation 1: $f_1(x') = -\text{loss}_{F,t}(x') + 1$ Cross-entropy loss is typically defined as $\text{loss}_{F,t}(x') = -\log(F(x')_t)$. This gives us $f_1(x') = \log(F(x')_t) + 1$. This function is $\leq 0$ when $\log(F(x')_t) \leq -1$, which means $F(x')_t \leq e^{-1} \approx 0.368$. However, $C(x') = t$ means that $F(x')_t$ is the highest probability. These conditions don't match. For example, if $F(x')_t = 0.4$ and all other probabilities are $< 0.4$, then $C(x') = t$ but $f_1(x') > 0$. **Verdict: Incorrect** ## Equation 2: $f_2(x') = (\max_{i \neq t}(F(x')_i) - F(x')_t)^+$ This function is $\leq 0$ if and only if $\max_{i \neq t}(F(x')_i) \leq F(x')_t$, which is precisely the condition for $C(x') = t$. **Verdict: Correct** ## Equation 3: $f_3(x') = \text{softplus}(\max_{i \neq t}(F(x')_i) - F(x')_t) - \log(2)$ Since $\text{softplus}(x) = \log(1+\exp(x))$, this function is $\leq 0$ when: $\log(1+\exp(\max_{i \neq t}(F(x')_i) - F(x')_t)) \leq \log(2)$ Simplifying: $1+\exp(\max_{i \neq t}(F(x')_i) - F(x')_t) \leq 2$ $\exp(\max_{i \neq t}(F(x')_i) - F(x')_t) \leq 1$ $\max_{i \neq t}(F(x')_i) - F(x')_t \leq 0$ $\max_{i \neq t}(F(x')_i) \leq F(x')_t$ This is exactly the condition for $C(x') = t$. **Verdict: Correct** ## Equation 4: $f_4(x') = (0.5 - F(x')_t)^+$ This function is $\leq 0$ when $F(x')_t \geq 0.5$. For a binary classifier (m=2), this is equivalent to $C(x') = t$ since $F(x')_t \geq 0.5$ implies $F(x')_t \geq F(x')_{1-t}$. However, for multi-class classification (m>2), it's possible to have $C(x') = t$ even when $F(x')_t < 0.5$. For example, if $F(x')_t = 0.4$ and all other classes have probabilities < 0.4. **Verdict: Incorrect** (for general m-class classification) ## Equation 5: $f_5(x') = -\log(2F(x')_t - 2)$ For this function to be well-defined, we need $2F(x')_t - 2 > 0$, which means $F(x')_t > 1$. However, since the output of softmax is a probability distribution, we always have $F(x')_t \leq 1$. This means the function is not well-defined for any valid input. **Verdict: Incorrect** ## Equation 6: $f_6(x') = (\max_{i \neq t}(Z(x')_i) - Z(x')_t)^+$ This function is $\leq 0$ if and only if $Z(x')_t \geq Z(x')_i$ for all $i \neq t$. Since softmax is a monotonic function, the ordering of logits ($Z$ values) is the same as the ordering of probabilities ($F$ values). Therefore, $Z(x')_t \geq Z(x')_i$ for all $i \neq t$ if and only if $F(x')_t \geq F(x')_i$ for all $i \neq t$, which is exactly the condition for $C(x') = t$. **Verdict: Correct** ## Equation 7: $f_7(x') = \text{softplus}(\max_{i \neq t}(Z(x')_i) - Z(x')_t) - \log(2)$ Using the same logic as equation 3, but with logits instead of probabilities, this function is $\leq 0$ if and only if $\max_{i \neq t}(Z(x')_i) \leq Z(x')_t$, which is equivalent to $C(x') = t$. **Verdict: Correct** The equations that are incorrect are [1, 4, 5].
Resulting in output: False