Computer Vision News - June 2018

cycles to get a “general feel” for the right ballpark for your hyper- parameters, making sure your loss term is going down. Then, later in the process, you fine-tune your hyper-parameters over larger numbers of cycles. 9. Make sure there are no nan or inf values throughout the network weights -- their presence usually indicates that something is wrong with the weights update, either the learning rate is too high or low, or there is a problem with the update computation. 10. Analyze the rate of weights update -- the rate of update between batches should be more or less constant and should range around 0.001. Now, let’s demonstrate the use of some of these tips. To do this, there are a number of debugging tools we shall get to know today: TensorBoard and tfdbg . Some of the capabilities that those tools allows you to follow are: 1. Loss value and hidden unit activations histograms monitoring capability. 2. Monitoring the rate of weights update. 3. Verifying there are no nan or inf values throughout the network, and that the fit output dimensions of each layer are appropriate to the input dimensions of the next layer. TensorBoard provides a suite of visualization tools to make it easier to understand and debug deep learning methods written in TensorFlow. For TensorBoard visualization you can use the following functions: tf.summary.scalar to trace a single statistic, and tf.summary.histogram for aggregate statistics. For instance, you can add the following code snippet to network layers constructed in TensorFlow to export its statistics to TensorBoard to trace distribution, variance, and max and min values of each parameter which we want to monitor. 12 Tool Computer Vision News def variable_summaries(var): """Attach a lot of summaries to a Tensor (for TensorBoard visualization).""" with tf.name_scope( 'summaries' ): mean = tf.reduce_mean(var) tf.summary.scalar( 'mean' , mean) with tf.name_scope( 'stddev' ): stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean))) tf.summary.scalar( 'stddev' , stddev) tf.summary.scalar( 'max' , tf.reduce_max(var)) tf.summary.scalar( 'min' , tf.reduce_min(var)) tf.summary.histogram( 'histogram' , var) Focus on: Debug and Analysis Mechanisms