Computer Vision News - December 2018

and rapid convergence to optimal configuration. BOHB is a practical, innovative method for optimization, that for a wide range of different fields outperforms both optimization methods currently in use separately, by combining them. 1. Bayesian Optimization -- models the hyperparameter objective-function using a probabilistic model based on the set of observed data points. Using this model, BO proposes a new hyperparameter configuration setting. BO iterates over three steps: (a) select the point that optimizes for the current data point selection method; (b) evaluate the objective-function at this point; and (c) add the new observation to the data and refit the model. The selected data points determine a trade-off between exploration and exploitation. BO requires long computation times to build-up its model so as to find better configurations. With sufficient computational power the model acquires more data points and achieves high optimization. 2. Hyperband Optimization (Hyperband) is a method that uses low budgets of quick-and-fast approximations of the objective function. HB calls SuccessiveHalving (SH; Jamieson et al., 2016) to identify the best out of n randomly-sampled configurations. SH evaluates these n configurations with a small budget, keeps the best half and doubles their budget. Hyperband Optimization balances between quick training with a low budget and long training using a higher budget. This helps you conserve your resources, by using a low budget when only a short training time is needed. The literature reports Hyperband Optimization performs very well with a low to medium budget, better than either random search or Bayesian Optimization, however, its convergence is limited by dependence on randomly-drawn configurations: with higher budgets its advantage over random search decreases radically. Bayesian Optimization, on the other hand, exhibits results very similar to random search in early training iterations, which are equivalent to low to medium budgets, and does not achieve results as good as Hyperband. For higher budgets, however, Bayesian Optimization usually manages to scan much wider search areas, thus outperforming Hyperband. BOHB is a simple yet efficient approach to hyperparameter optimization , which offers the following advantages: it is strong, flexible and expandable (both in terms of handling more dimensions and parallel processing); and it achieves strong performance results under all conditions . BOHB showed impressive results in the series of experiments conducted that met the requirements and expectations set for it. But you don’t need to trust published papers blindly: we recommend you try to implement BOHB to see for yourself and we shall demonstrate a simple code for doing so. 25 Focus on Computer Vision News Focus on: BOHB