It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" -linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.

In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A primary insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures to replicate this work will be made available upon publication.

No abstract

No abstract

Simple Summary: Domestic dogs are abundant worldwide-as owned pets, unowned strays, and feral animals. High numbers of free-roaming dogs can be a concern because of the risks they pose to public health, animal welfare, and wildlife. Using a systematic review process, we investigated what the research published to date can tell us about the effectiveness of different dog population management methods. We found that management methods for dog populations have been researched in multiple countries worldwide, using a wide range of indicators to assess method effectiveness. We outline the results and suggest improvements to help guide future dog population management policy. Abstract:The worldwide population of domestic dogs is estimated at approximately 700 million, with around 75% classified as "free-roaming". Where free-roaming dogs exist in high densities, there are significant implications for public health, animal welfare, and wildlife. Approaches to manage dog populations include culling, fertility control, and sheltering. Understanding the effectiveness of each of these interventions is important in guiding future dog population management. We present the results of a systematic review of published studies investigating dog population management, to assess: (1) where and when studies were carried out;(2) what population management methods were used; and (3) what was the effect of the method. We evaluated the reporting quality of the published studies for strength of evidence assessment. The systematic review resulted in a corpus of 39 papers from 15 countries, reporting a wide disparity of approaches and measures of effect. We synthesised the management methods and reported effects. Fertility control was most investigated and had the greatest reported effect on dog population size. Reporting quality was low for power calculations (11%), sample size calculations (11%), and the use of control populations (17%). We provide recommendations for future studies to use common metrics and improve reporting quality, study design, and modelling approaches in order to allow better assessment of the true impact of dog population management. Author Contributions: Conceptualisation, L.M.S., L.M.C., and R.J.Q.; methodology, L.M.S., L.M.C., and R.J.Q.; formal analysis, L.M.S.; investigation, L.M.S.; data curation, L.M.S.; writing-original draft preparation, L.M.S.; writing-review and editing, L.M.S., L.M.C., S.H., A.M.M., P.D.V., and R.J.Q.; visualisation, L.M.S.; supervision, L.M.C. and R.J.Q.; and funding acquisition, L.M.C. Funding: This research was funded by VIER PFOTEN International.Acknowledgments: The authors would like to thank Steven Sait, Helen Gray, and Mary Friel for their useful comments on an earlier draft of this manuscript. We would also like to thank the three anonymous reviewers for providing valuable feedback on the manuscript. Conflicts of Interest:The authors declare no conflict of interest. The funders were involved in the reviewing of the manuscript.