Computer Vision News - December 2018

This month we shall review BOHB , a new tool for dealing with the hyperparameter optimization problem , presented at the ICML2018 conference. BOHB offers a significant speedup by combining the advantages from both of the optimization approaches currently in use: Hyperband and Bayesian optimization . BOHB converges much more quickly than either approach, moreover, finding several optimal hyperparameter value combinations. The last decade has seen an explosion in the successful application of Machine Learning, and Deep Learning in particular, to a very wide range of fields, these achievements depend to a large degree on correctly selecting hyperparameter values during the training stage . Choosing the wrong value for the most fundamental hyperparameter -- the learning rate -- will result in a failed training of the network. Of course, there are plenty of other hyperparameters that need to be set during network training, such as: network architecture, the regularization module, and many more. A common solution in current use is to simply employ a random search within a certain range of values considered relevant, train with each setting and pick the most precise result, but this is obviously extremely inefficient. Instead, the performance of deep learning algorithms can be defined as an objective-function of their hyperparameters. We will define the Hyperparameter Optimization (HPO) problem as finding the value for each hyperparameter such that it minimizes this objective-function’s value. Two methods for tackling the problem of minimization of the objective-function are currently popular: (1) Bayesian optimization (BO; Shahriari et al., 2016) (2) Hyperband (HB; Li et al., 2016) The Hyperband methods employ a Bandit-Based algorithm, which uses a random search, and therefore sometimes don’t quickly converge to the best configuration. Bayesian approaches, on the other hand, usually require immense computation power, nearly impossible to achieve. BOHB is an approach that combines the advantages of Bayesian optimization with the efficiency of using Bandit-Based methods , to achieve the best of both worlds: strong performance 24 Focus on: BOHB Focus on by Assaf Spanier A new tool for dealing with the hyperparameter optimization problem Computer Vision News