21 Prune Truong Computer Vision News Despite these advantages, when I started my PhD, dense matching methods had almost only been explored for optical flow, focusing on consecutive views of a video. In contrast, the more general dense correspondence problem under large appearance and viewpoint changes had received much less attention. Thus, I focused on this general dense correspondence probleminmy PhD. In particular, I tackled three main research questions, which I perceived to be the main bottlenecks in dense matching. What architecture is suitable for dense matching? Most existing dense correspondence architectures were specialized for small appearance changes and limited displacements. I proposed novel architectures capable of handling arbitrary large viewpoint and illumination changes, while still producing sub-pixel accurate predictions. I also introduced an online optimization-based matching module, to improve the network’s robustness to repetitive structures (such as windows) or low-textured areas (such as walls). How to train such a network? Obtaining dense correspondence ground truths for real-world image pairs is extremely challenging, if not impossible. To address this, I proposed two unsupervised training frameworks to train dense matching networks from single images or pairs of images, without any additional annotations. This enables large-scale training on real-world images, while also providing an easy way of customizing a model for new domains. How to select the “good” matches? Dense matching methods predict matches for every pixel, even in areas that are occluded or for which a match is ill-defined, such as in the sky. This greatly limits the usability of dense matching in downstream tasks like 3D reconstruction, which need highly accurate matches as input. I proposed a probabilistic formulation of the flow prediction, which pairs the matches with a confidence map, reflecting their accuracy and reliability. This confidence prediction enables the direct use of dense matching approaches in popular applications such as image-based localization or style transfer, by filtering out unreliable matches. Our code is open source on GitHub at PruneTruong/DenseMatching. I hope my thesis will inspire others towards the wonderful world of dense correspondences. Some applications of my work on dense matching
RkJQdWJsaXNoZXIy NTc3NzU=