Initial Query

From the following data extract the best performing defense each year, in the format {year: robust accuracy} So for example the answer for {"2024": 69.71, "2023": ..., ...}, now fill it in for every other year. Return the answer as a JSON dict. Rank Method Standard accuracy AutoAttack robust accuracy Best known robust accuracy AA eval. potentially unreliable Extra data Architecture Venue 1 Robust Principles: Architectural Design Principles for Adversarially Robust CNNs It uses additional 50M synthetic images in training. 93.27% 71.07% 71.07% × × RaWideResNet-70-16 BMVC 2023 2 Better Diffusion Models Further Improve Adversarial Training It uses additional 50M synthetic images in training. 93.25% 70.69% 70.69% × × WideResNet-70-16 ICML 2023 3 MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers It uses an ensemble of networks. The robust base classifier uses 50M synthetic images. 69.71% robust accuracy is due to the original evaluation (Adaptive AutoAttack) 95.19% 70.08% 69.71% × ☑ ResNet-152 + WideResNet-70-16 arXiv, Feb 2024 4 Improving the Accuracy-Robustness Trade-off of Classifiers via Adaptive Smoothing It uses an ensemble of networks. The robust base classifier uses 50M synthetic images. 95.23% 68.06% 68.06% × ☑ ResNet-152 + WideResNet-70-16 + mixing network SIMODS 2024 5 Decoupled Kullback-Leibler Divergence Loss It uses additional 20M synthetic images in training. 92.16% 67.73% 67.73% × × WideResNet-28-10 arXiv, May 2023 6 Better Diffusion Models Further Improve Adversarial Training It uses additional 20M synthetic images in training. 92.44% 67.31% 67.31% × × WideResNet-28-10 ICML 2023 7 Fixing Data Augmentation to Improve Adversarial Robustness 66.56% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 92.23% 66.58% 66.56% × ☑ WideResNet-70-16 arXiv, Mar 2021 8 Improving Robustness using Generated Data It uses additional 100M synthetic images in training. 66.10% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 88.74% 66.11% 66.10% × × WideResNet-70-16 NeurIPS 2021 9 Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples 65.87% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 91.10% 65.88% 65.87% × ☑ WideResNet-70-16 arXiv, Oct 2020 10 Revisiting Residual Networks for Adversarial Robustness: An Architectural Perspective 91.58% 65.79% 65.79% × ☑ WideResNet-A4 arXiv, Dec. 2022 11 Fixing Data Augmentation to Improve Adversarial Robustness It uses additional 1M synthetic images in training. 64.58% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 88.50% 64.64% 64.58% × × WideResNet-106-16 arXiv, Mar 2021 12 Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks Based on the model Rebuffi2021Fixing_70_16_cutmix_extra. 64.20% robust accuracy is due to AutoAttack + transfer APGD from Rebuffi2021Fixing_70_16_cutmix_extra 93.73% 71.28% 64.20% ☑ ☑ WideResNet-70-16, Neural ODE block NeurIPS 2021 13 Fixing Data Augmentation to Improve Adversarial Robustness It uses additional 1M synthetic images in training. 64.20% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 88.54% 64.25% 64.20% × × WideResNet-70-16 arXiv, Mar 2021 14 Exploring and Exploiting Decision Boundary Dynamics for Adversarial Robustness It uses additional 10M synthetic images in training. 93.69% 63.89% 63.89% × × WideResNet-28-10 ICLR 2023 15 Improving Robustness using Generated Data It uses additional 100M synthetic images in training. 63.38% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 87.50% 63.44% 63.38% × × WideResNet-28-10 NeurIPS 2021 16 Robustness and Accuracy Could Be Reconcilable by (Proper) Definition It uses additional 1M synthetic images in training. 89.01% 63.35% 63.35% × × WideResNet-70-16 ICML 2022 17 Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off 91.47% 62.83% 62.83% × ☑ WideResNet-34-10 OpenReview, Jun 2021 18 Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? It uses additional 10M synthetic images in training. 87.30% 62.79% 62.79% × × ResNest152 ICLR 2022 19 Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples 62.76% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 89.48% 62.80% 62.76% × ☑ WideResNet-28-10 arXiv, Oct 2020 20 Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks Uses exponential moving average (EMA) 91.23% 62.54% 62.54% × ☑ WideResNet-34-R NeurIPS 2021 21 Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks 90.56% 61.56% 61.56% × ☑ WideResNet-34-R NeurIPS 2021 22 Parameterizing Activation Functions for Adversarial Robustness It uses additional ~6M synthetic images in training. 87.02% 61.55% 61.55% × × WideResNet-28-10-PSSiLU arXiv, Oct 2021 23 Robustness and Accuracy Could Be Reconcilable by (Proper) Definition It uses additional 1M synthetic images in training. 88.61% 61.04% 61.04% × × WideResNet-28-10 ICML 2022 24 Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off It uses additional 1M synthetic images in training. 88.16% 60.97% 60.97% × × WideResNet-28-10 OpenReview, Jun 2021 25 Fixing Data Augmentation to Improve Adversarial Robustness It uses additional 1M synthetic images in training. 60.73% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 87.33% 60.75% 60.73% × × WideResNet-28-10 arXiv, Mar 2021 26 Do Wider Neural Networks Really Help Adversarial Robustness? 87.67% 60.65% 60.65% Unknown ☑ WideResNet-34-15 arXiv, Oct 2020 27 Improving Neural Network Robustness via Persistency of Excitation 86.53% 60.41% 60.41% × ☑ WideResNet-34-15 ACC 2022 28 Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? It uses additional 10M synthetic images in training. 86.68% 60.27% 60.27% × × WideResNet-34-10 ICLR 2022 29 Adversarial Weight Perturbation Helps Robust Generalization 88.25% 60.04% 60.04% × ☑ WideResNet-28-10 NeurIPS 2020 30 Improving Neural Network Robustness via Persistency of Excitation 89.46% 59.66% 59.66% × ☑ WideResNet-28-10 ACC 2022 31 Geometry-aware Instance-reweighted Adversarial Training Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255. 89.36% 59.64% 59.64% × ☑ WideResNet-28-10 ICLR 2021 32 Unlabeled Data Improves Adversarial Robustness 89.69% 59.53% 59.53% × ☑ WideResNet-28-10 NeurIPS 2019 33 Improving Robustness using Generated Data It uses additional 100M synthetic images in training. 58.50% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 87.35% 58.63% 58.50% × × PreActResNet-18 NeurIPS 2021 34 Data filtering for efficient adversarial training 86.10% 58.09% 58.09% × × WideResNet-34-20 Pattern Recognition 2024 35 Scaling Adversarial Training to Large Perturbation Bounds 85.32% 58.04% 58.04% × × WideResNet-34-10 ECCV 2022 36 Efficient and Effective Augmentation Strategy for Adversarial Training 88.71% 57.81% 57.81% × × WideResNet-34-10 NeurIPS 2022 37 LTD: Low Temperature Distillation for Robust Adversarial Training 86.03% 57.71% 57.71% × × WideResNet-34-20 arXiv, Nov 2021 38 Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off 89.02% 57.67% 57.67% × ☑ PreActResNet-18 OpenReview, Jun 2021 39 LAS-AT: Adversarial Training with Learnable Attack Strategy 85.66% 57.61% 57.61% × × WideResNet-70-16 arXiv, Mar 2022 40 A Light Recipe to Train Robust Vision Transformers 91.73% 57.58% 57.58% × ☑ XCiT-L12 arXiv, Sep 2022 41 Data filtering for efficient adversarial training 86.54% 57.30% 57.30% × × WideResNet-34-10 Pattern Recognition 2024 42 A Light Recipe to Train Robust Vision Transformers 91.30% 57.27% 57.27% × ☑ XCiT-M12 arXiv, Sep 2022 43 Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples 57.14% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 85.29% 57.20% 57.14% × × WideResNet-70-16 arXiv, Oct 2020 44 HYDRA: Pruning Adversarially Robust Neural Networks Compressed model 88.98% 57.14% 57.14% × ☑ WideResNet-28-10 NeurIPS 2020 45 Decoupled Kullback-Leibler Divergence Loss 85.31% 57.09% 57.09% × × WideResNet-34-10 arXiv, May 2023 46 Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off It uses additional 1M synthetic images in training. 86.86% 57.09% 57.09% × × PreActResNet-18 OpenReview, Jun 2021 47 LTD: Low Temperature Distillation for Robust Adversarial Training 85.21% 56.94% 56.94% × × WideResNet-34-10 arXiv, Nov 2021 48 Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples 56.82% robust accuracy is due to the original evaluation (AutoAttack + MultiTargeted) 85.64% 56.86% 56.82% × × WideResNet-34-20 arXiv, Oct 2020 49 Fixing Data Augmentation to Improve Adversarial Robustness It uses additional 1M synthetic images in training. 83.53% 56.66% 56.66% × × PreActResNet-18 arXiv, Mar 2021 50 Improving Adversarial Robustness Requires Revisiting Misclassified Examples 87.50% 56.29% 56.29% × ☑ WideResNet-28-10 ICLR 2020 51 LAS-AT: Adversarial Training with Learnable Attack Strategy 84.98% 56.26% 56.26% × × WideResNet-34-10 arXiv, Mar 2022 52 Adversarial Weight Perturbation Helps Robust Generalization 85.36% 56.17% 56.17% × × WideResNet-34-10 NeurIPS 2020 53 A Light Recipe to Train Robust Vision Transformers 90.06% 56.14% 56.14% × ☑ XCiT-S12 arXiv, Sep 2022 54 Are Labels Required for Improving Adversarial Robustness? 86.46% 56.03% 56.03% Unknown ☑ WideResNet-28-10 NeurIPS 2019 55 Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? It uses additional 10M synthetic images in training. 84.59% 55.54% 55.54% × × ResNet-18 ICLR 2022 56 Using Pre-Training Can Improve Model Robustness and Uncertainty 87.11% 54.92% 54.92% × ☑ WideResNet-28-10 ICML 2019 57 Bag of Tricks for Adversarial Training 86.43% 54.39% 54.39% Unknown × WideResNet-34-20 ICLR 2021 58 Boosting Adversarial Training with Hypersphere Embedding 85.14% 53.74% 53.74% × × WideResNet-34-20 NeurIPS 2020 59 Learnable Boundary Guided Adversarial Training Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255 88.70% 53.57% 53.57% × × WideResNet-34-20 ICCV 2021 60 Attacks Which Do Not Kill Training Make Adversarial Learning Stronger 84.52% 53.51% 53.51% × × WideResNet-34-10 ICML 2020 61 Overfitting in adversarially robust deep learning 85.34% 53.42% 53.42% × × WideResNet-34-20 ICML 2020 62 Self-Adaptive Training: beyond Empirical Risk Minimization Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255. 83.48% 53.34% 53.34% Unknown × WideResNet-34-10 NeurIPS 2020 63 Theoretically Principled Trade-off between Robustness and Accuracy Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255. 84.92% 53.08% 53.08% Unknown × WideResNet-34-10 ICML 2019 64 Learnable Boundary Guided Adversarial Training Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255 88.22% 52.86% 52.86% × × WideResNet-34-10 ICCV 2021 65 Adversarial Robustness through Local Linearization 86.28% 52.84% 52.84% Unknown × WideResNet-40-8 NeurIPS 2019 66 Efficient and Effective Augmentation Strategy for Adversarial Training 85.71% 52.48% 52.48% × × ResNet-18 NeurIPS 2022 67 Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning Uses ensembles of 3 models. 86.04% 51.56% 51.56% Unknown × ResNet-50 CVPR 2020 68 Efficient Robust Training via Backward Smoothing 85.32% 51.12% 51.12% Unknown × WideResNet-34-10 arXiv, Oct 2020 69 Scaling Adversarial Training to Large Perturbation Bounds 80.24% 51.06% 51.06% × × ResNet-18 ECCV 2022 70 Improving Adversarial Robustness Through Progressive Hardening 86.84% 50.72% 50.72% Unknown × WideResNet-34-10 arXiv, Mar 2020 71 Robustness library 87.03% 49.25% 49.25% Unknown × ResNet-50 GitHub, Oct 2019 72 Harnessing the Vulnerability of Latent Layers in Adversarially Trained Models 87.80% 49.12% 49.12% Unknown × WideResNet-34-10 IJCAI 2019 73 Metric Learning for Adversarial Robustness 86.21% 47.41% 47.41% Unknown × WideResNet-34-10 NeurIPS 2019 74 You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle Focuses on fast adversarial training. 87.20% 44.83% 44.83% Unknown × WideResNet-34-10 NeurIPS 2019 75 Towards Deep Learning Models Resistant to Adversarial Attacks 87.14% 44.04% 44.04% Unknown × WideResNet-34-10 ICLR 2018 76 Understanding and Improving Fast Adversarial Training Focuses on fast adversarial training. 79.84% 43.93% 43.93% Unknown × PreActResNet-18 NeurIPS 2020 77 Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness 80.89% 43.48% 43.48% Unknown × ResNet-32 ICLR 2020 78 Fast is better than free: Revisiting adversarial training Focuses on fast adversarial training. 83.34% 43.21% 43.21% Unknown × PreActResNet-18 ICLR 2020 79 Adversarial Training for Free! 86.11% 41.47% 41.47% Unknown × WideResNet-34-10 NeurIPS 2019 80 MMA Training: Direct Input Space Margin Maximization through Adversarial Training 84.36% 41.44% 41.44% Unknown × WideResNet-28-4 ICLR 2020 81 A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs Compressed model 87.32% 40.41% 40.41% × × ResNet-18 ASP-DAC 2021 82 Controlling Neural Level Sets Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255. 81.30% 40.22% 40.22% Unknown × ResNet-18 NeurIPS 2019 83 Robustness via Curvature Regularization, and Vice Versa 83.11% 38.50% 38.50% Unknown × ResNet-18 CVPR 2019 84 Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training 89.98% 36.64% 36.64% Unknown × WideResNet-28-10 NeurIPS 2019 85 Adversarial Interpolation Training: A Simple Approach for Improving Model Robustness 90.25% 36.45% 36.45% Unknown × WideResNet-28-10 OpenReview, Sep 2019 86 Adversarial Defense via Learning to Generate Diverse Attacks 78.91% 34.95% 34.95% Unknown × ResNet-20 ICCV 2019 87 Sensible adversarial learning 91.51% 34.22% 34.22% Unknown × WideResNet-34-10 OpenReview, Sep 2019 88 Towards Stable and Efficient Training of Verifiably Robust Neural Networks Verifiably robust model with 32.24% provable robust accuracy 44.73% 32.64% 32.64% Unknown × 5-layer-CNN ICLR 2020 89 Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks 92.80% 29.35% 29.35% Unknown × WideResNet-28-10 ICCV 2019 90 Enhancing Adversarial Defense by k-Winners-Take-All Uses ℓ ∞ = 0.031 ≈ 7.9/255 instead of 8/255. 7.40% robust accuracy is due to 1 restart of APGD-CE and 30 restarts of Square Attack Note: this adaptive evaluation (Section 5) reports 0.16% robust accuracy on a different model (adversarially trained ResNet-18). 79.28% 18.50% 7.40% ☑ × DenseNet-121 ICLR 2020 91 Manifold Regularization for Adversarial Robustness 90.84% 1.35% 1.35% Unknown × ResNet-18 arXiv, Mar 2020 92 Adversarial Defense by Restricting the Hidden Space of Deep Neural Networks 89.16% 0.28% 0.28% Unknown × ResNet-110 ICCV 2019 93 Jacobian Adversarially Regularized Networks for Robustness 93.79% 0.26% 0.26% Unknown × WideResNet-34-10 ICLR 2020 94 ClusTR: Clustering Training for Robustness 91.03% 0.00% 0.00% Unknown × WideResNet-28-10 arXiv, Jun 2020 95 Standardly trained model 94.78% 0.0% 0.0% Unknown × WideResNet-28-10 N/A

LLM Generation

Query

Output

Here is the JSON dict with the best performing defense each year in the specified format:

{
  "2020": 65.87,
  "2021": 66.56,
  "2022": 65.79,
  "2023": 71.07,
  "2024": 70.08
}

This dict is derived by selecting the highest "AutoAttack robust accuracy" for each year from the data provided.

Extract Json

I extracted the following JSON from that output:

{
  "2020": 65.87,
  "2021": 66.56,
  "2022": 65.79,
  "2023": 71.07,
  "2024": 70.08
}

JSON Subset Evaluator

Testing if the previous output matches the JSON: { "2024": 69.71, "2023": 71.07, "2022": 65.79, "2021": 66.56, "2020": 65.87, "2019": 59.53, "2018": 44.04 }: False

Initial Query

LLM Generation

Query

Output

Based on the provided data, here is the JSON dict of the best performing defense each year in the format {year: robust accuracy}:

{
  "2024": 70.08,
  "2023": 71.07,
  "2022": 65.79,
  "2021": 66.56,
  "2020": 65.87,
  "2019": 59.53,
  "2018": 44.04
}

This data reflects the highest robust accuracy reported for each year based on the given rankings.

Extract Json

I extracted the following JSON from that output:

{
  "2024": 70.08,
  "2023": 71.07,
  "2022": 65.79,
  "2021": 66.56,
  "2020": 65.87,
  "2019": 59.53,
  "2018": 44.04
}

JSON Subset Evaluator

Testing if the previous output matches the JSON: { "2024": 69.71, "2023": 71.07, "2022": 65.79, "2021": 66.56, "2020": 65.87, "2019": 59.53, "2018": 44.04 }: False

Initial Query

LLM Generation

Query

Output

{
    "2018": 44.04,
    "2019": 59.53,
    "2020": 65.87,
    "2021": 66.56,
    "2022": 65.79,
    "2023": 71.07,
    "2024": 69.71
}

Extract Json

I extracted the following JSON from that output:

{
    "2018": 44.04,
    "2019": 59.53,
    "2020": 65.87,
    "2021": 66.56,
    "2022": 65.79,
    "2023": 71.07,
    "2024": 69.71
}

JSON Subset Evaluator

Testing if the previous output matches the JSON: { "2024": 69.71, "2023": 71.07, "2022": 65.79, "2021": 66.56, "2020": 65.87, "2019": 59.53, "2018": 44.04 }: True

Initial Query

LLM Generation

Query

Output

Based on the provided data, the best-performing defenses each year in terms of robust accuracy are: - **2024:** 70.08% - **2023:** 71.07% - **2022:** 63.35% - **2021:** 66.56% - **2020:** 65.87% - **2019:** 59.53% - **2018:** 44.04% Here is the JSON dict:

{
  "2024": 70.08,
  "2023": 71.07,
  "2022": 63.35,
  "2021": 66.56,
  "2020": 65.87,
  "2019": 59.53,
  "2018": 44.04
}

Extract Json

I extracted the following JSON from that output:

{
  "2024": 70.08,
  "2023": 71.07,
  "2022": 63.35,
  "2021": 66.56,
  "2020": 65.87,
  "2019": 59.53,
  "2018": 44.04
}

JSON Subset Evaluator

Testing if the previous output matches the JSON: { "2024": 69.71, "2023": 71.07, "2022": 65.79, "2021": 66.56, "2020": 65.87, "2019": 59.53, "2018": 44.04 }: False

Initial Query

LLM Generation

Query

Output

To extract the best performing defense each year based on the given data, we need to identify the highest robust accuracy for each year. Here is the processed JSON dictionary with the best robust accuracy for each year:

{
    "2024": 70.08,
    "2023": 71.07,
    "2022": 65.79,
    "2021": 66.56,
    "2020": 65.87,
    "2019": 59.53,
    "2018": 44.04
}

This data is derived from the table by taking the highest robust accuracy for each respective year. Note that for years with multiple entries, the highest robust accuracy is selected.

Extract Json

I extracted the following JSON from that output:

{
    "2024": 70.08,
    "2023": 71.07,
    "2022": 65.79,
    "2021": 66.56,
    "2020": 65.87,
    "2019": 59.53,
    "2018": 44.04
}

JSON Subset Evaluator

Testing if the previous output matches the JSON: { "2024": 69.71, "2023": 71.07, "2022": 65.79, "2021": 66.56, "2020": 65.87, "2019": 59.53, "2018": 44.04 }: False