AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks

Li, Hao; Fu, Tianwen; Dai, Jifeng; Li, Hongsheng; Huang, Gao; Zhu, Xizhou

doi:10.48550/arxiv.2103.14026

Cited by 6 publications

(11 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared with these handcrafted losses, Parameterized AP Loss can yield 1.5 ∼ 3.0 AP score gain. We also compared with AutoLoss-Zero [19] and CSE AutoLoss [28], which are two AutoML-based losses. Our approach can obtain over 1.5 AP score improvement over them.…”

Section: Resultsmentioning

confidence: 99%

“…Searching Loss Functions for Object Detection. Recent works [28,19] have also tried to search suitable loss functions for object detection. In these methods, loss functions are formulated as computational graphs composed of basic mathematical operators.…”

Section: Related Workmentioning

confidence: 99%

“…Our proposed Parameterized AP Loss constitutes a compact search space, where different loss functions are represented by different parameters. Since the parameterized search space is continuous, the search process would be more effective than the discrete combinatorial optimization in [28,19]. Moreover, Parameterized AP Loss drives the localization and classification training with a single unified loss, which better mitigates the mis-alignment issue.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Searching Parameterized AP Loss for Object Detection

Tao¹,

Li²,

Zhu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Loss functions play an important role in training deep-network-based object detectors. The most widely used evaluation metric for object detection is Average Precision (AP), which captures the performance of localization and classification sub-tasks simultaneously. However, due to the non-differentiable nature of the AP metric, traditional object detectors adopt separate differentiable losses for the two sub-tasks. Such a mis-alignment issue may well lead to performance degradation. To address this, existing works seek to design surrogate losses for the AP metric manually, which requires expertise and may still be sub-optimal. In this paper, we propose Parameterized AP Loss, where parameterized functions are introduced to substitute the non-differentiable components in the AP calculation. Different AP approximations are thus represented by a family of parameterized functions in a unified formula. Automatic parameter search algorithm is then employed to search for the optimal parameters. Extensive experiments on the COCO benchmark with three different object detectors (i.e., RetinaNet, Faster R-CNN, and Deformable DETR) demonstrate that the proposed Parameterized AP Loss consistently outperforms existing handcrafted losses. Code is released at https://github.com/fundamentalvision/Parameterized-AP-Loss.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Searching Parameterized AP Loss for Object Detection

Tao¹,

Li²,

Zhu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Subsequently, Li et al [42] and Liu et al [43] introduced this idea into semantic segmentation and object detection, respectively. More recently, Li et al [44] proposed AutoLoss-Zero, a general framework for learning tasks. They verified the effectiveness of AutoLoss-Zero on four tasks, i.e., semantic segmentation, object detection, instance segmentation, and pose estimation.…”

Section: Auto Loss Function Searchmentioning

confidence: 99%

“…To solve the above problem, each loss function is expressed as a tree, and the genetic programming (GP) [14] is adopted for optimizing a suitable solution among the search space. Our method is similar to CSE-AutoLoss [43] and AutoLoss-Zero [44], and is dubbed as AutoLoss-AR, which is the abbreviation of auto loss function search for adversarial risk. The major difference is that AutoLoss-AR focuses on searching for a suitable loss for tightening the approximation error of adversarial risk, rather than training a model for a specific task with a better performance.…”

Section: A Overviewmentioning

confidence: 99%

Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search

Xia¹,

Li²,

Li³

2021

Preprint

View full text Add to dashboard Cite

Numerous studies have demonstrated that deep neural networks are easily misled by adversarial examples. Effectively evaluating the adversarial robustness of a model is important for its deployment in practical applications. Currently, a common type of evaluation is to approximate the adversarial risk of a model as a robustness indicator by constructing malicious instances and executing attacks. Unfortunately, there is an error (gap) between the approximate value and the true value. Previous studies manually design attack methods to achieve a smaller error, which is inefficient and may miss a better solution.In this paper, we establish the tightening of the approximation error as an optimization problem and try to solve it with an algorithm. More specifically, we first analyze that replacing the non-convex and discontinuous 0-1 loss with a surrogate loss, a necessary compromise in calculating the approximation, is one of the main reasons for the error. Then we propose AutoLoss-AR, the first method for searching loss functions for tightening the approximation error of adversarial risk. Extensive experiments are conducted in multiple settings. The results demonstrate the effectiveness of the proposed method: the best-discovered loss functions outperform the handcrafted baseline by 0.9%-2.9% and 0.7%-2.0% on MNIST and CIFAR-10, respectively. Besides, we also verify that the searched losses can be transferred to other settings and explore why they are better than the baseline by visualizing the local loss landscape.

show abstract

Fast and Efficient Local-Search for Genetic Programming Based Loss Function Learning

Raymond

Chen

Xue

et al. 2023

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

In this paper, we develop upon the topic of loss function learning, an emergent meta-learning paradigm that aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. The framework first uses genetic programming to find a set of symbolic loss functions. Second, the set of learned loss functions is subsequently parameterized and optimized via unrolled differentiation. The versatility and performance of the proposed framework are empirically validated on a diverse set of supervised learning tasks. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems, using a variety of task-specific neural network architectures. CCS Concepts• Computing methodologies → Machine learning algorithms.

show abstract

AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks

Cited by 6 publications

References 41 publications

Searching Parameterized AP Loss for Object Detection

Searching Parameterized AP Loss for Object Detection

Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search

Fast and Efficient Local-Search for Genetic Programming Based Loss Function Learning

Contact Info

Product

Resources

About