Computer Vision News - January 2020

2 Summary Rese rch 6 To train the network, PyTorch gives a friendly API, which is similar to that of Keras. To use it, we define the trainable parameters in target, and then we set the optimizer (which can be chosen from a variety of optimizers). As most practitioners, we will choose Adam optimizer. The next step is to iterate over the epochs : at each epoch, to calculate the content loss which is the MSE of the difference between convolution layers and the style loss which is based on the weighted correlation matrix that we defined earlier. The total loss is a linear combination between the content loss and the style loss (with a hyperparameter that has to be tuned). The final step is to optimize the loss function: The last step before the actual training is to define an object which contains the features of each image. Then, we build the correlation between the features as well as weights for each feature. This will serve us during the training to define the objective: img1 = Image . open( "shoes.jpg" ) . convert( 'RGB' ) img2 = Image . open( "abstract.jpg" ) . convert( 'RGB' ) img1 = transformation(img1) . to(device) img2 = transformation(img2) . to(device) img1_features = get_features(img1, vgg) img2_features = get_features(img2, vgg) correlations = {l: correlation_matrix(img2_features[l]) for l in img2_features} weights = { 'conv1_1' : 0.9 , 'conv2_1' : 0.6 , 'conv3_1' : 0.2 , 'conv4_1' : 0.3 , 'conv5_1' : 0.1 } target = img1 . clone() . requires_grad_( True ) . to(device) optimizer = optim . Adam([target], lr = 0.3 ) for ii in range ( 1 , 2000 ): target_features = get_features(target, vgg) loss = target_features[ 'conv4_2' ] - img1_features[ 'conv4_2' ] content_loss = torch . mean((loss) ** 2 ) style_loss = 0 for layer in weights: target_feature = target_features[layer] target_corr = correlation_matrix(target_feature) style_corr = correlations[layer] layer_loss = torch . mean((target_corr - style_corr) ** 2 ) layer_loss *= weights[layer] _, d, h, w = target_feature . shape style_loss += layer_loss / (d * h * w) total_loss = 0.1 * style_loss + content_loss print (total_loss) optimizer . zero_grad() total_loss . backward() optimizer . step()