Computer Vision News - December 2022

23 Fabio Pizzati priors easy to define thanks to human abstraction capabilities. In particular, in his latest paper (F. Pizzati et al., ManiFest, ECCV 2022) he shows results of few-shot or even one- shot generation of complex road scenarios such as nighttime or adverse weather, by exploiting a semantic consistency mechanism learned on additional domains. But the applications are not limited to autonomous driving: have a look at Fig. 1, that showcases erupting volcanoes and auroras, generated by only using four images for training! His thesis offers a second line of research, this time focused on physics-informed learning . Image-to-image translation networks, indeed, fail to be accurately controllable when applied to physics-based generation, including time of day modifications, or rendering of weather-related phenomena. With CoMoGAN (F.Pizzati et al., CoMoGAN, CVPR 2021) , the results of which are visible in Fig. 2, it is possible to process images acquired at daytime, and generate realistic timelapses inwhich the angle of the sun is explicitly controllable. The training is guidedby naivephysicalmodels that roughly describe the appearanceof daytime changes on images for different sun elevation values. In a previous ECCV paper (F. Pizzati et al., Model-based Disentanglement, ECCV 2020) it was also proposed to combine realistic physical models and generative networks in a disentangled manner . In fact, when generating images, it may be convenient to rely on the availability of realistic rendering for well-known traits, and generate the rest of the scene with neural networks . This makes the output images way more realistic and variable. In the future, Fabio plans to extend his research to text-driven diffusion models , while also exploring exciting new directions such as continual learning and adversarial robustness . Good luck!