Computer Vision News - December 2020

RL-CycleGAN 5 The authors make use of two important concepts from the ĞĞƉ >ĞĂƌŶŝŶŐ world which we are probably already familiar with: 1) the Domain Adaptation technique, and 2) Reinforcement Learning. Domain adaptation methods aim at training models using many examples from a source domain (here simulation) and few examples from a target domain (here reality). They can be based on a pixel-level adaptation, as does the LJĐůĞ' E , or on a feature-level adaptation. Let’s look at both of these techniques more in depth! ࣦ ீ஺ே ሺ ܩ ǡ ܦ ௒ ǡ ܺǡ ܻሻ ൌ ॱ ௬̱௒ ሾŽ‘‰ ܦ ௒ ሺ ݕ ሻሿ ൅ As per the original paper, CycleGAN includes 2 GANs, here called ^ŝŵϮZĞĂů and ZĞĂůϮ^ŝŵ . These learn the mappings from the simulation to real domains and vice versa. The architecture also includes 2 adversarial discriminators. Below we report the mathematical formulations, assuming that X and Y are the two image domains (simulation and real). Generator Sim2Real Generation Real2Sim Adversarial discriminator 1 Adversarial discriminator 2 Adversarial Loss for Sim2Real ॱ ௫̱௑ ሾŽ‘‰ ቀͳ െ ܦ ௒ ൫ ܩ ሺ ݔ ሻ൯ቁሿ Adversarial Loss for Real2Sim Cycle Consistency Loss ࣦ ீ஺ே ሺ ܨ ǡ ܦ ௑ ǡ ܻǡ ܺሻ ൌ ॱ ௫̱௑ ሾŽ‘‰ ܦ ௑ ሺ ݔ ሻሿ ൅ ॱ ௬̱௒ ሾŽ‘‰ ቀͳ െ ܦ ௫ ൫ ܨ ሺ ݕ ሻ൯ቁሿ ࣦ ௖௬௖ ሺ ܩ ǡ ܨ ሻ ൌ ॱ ௫̱஽ ೞ೔೘ ݀൫ ܨ ൫ ܩ ሺ ݔ ሻ൯ǡ ݔ ൯ ൅ ॱ ௬̱஽ ೝ೐ೌ೗ ݀൫ ܩ ൫ ܨ ሺ ݕ ሻ൯ǡ ݕ ൯ G: X -> Y F: Y -> X Dx: {x} from {F(y)} Dy: {y} from {G(x)} Sim2Real is trained by: Real2Sim is trained by: The last item in the table is the cycle consistency loss, used to ensure that, from the generated images, the original ones can always be recovered, i.e.: ݉݅݊ ீ ݉ܽ ݔ ஽ ೊ ࣦ ீ஺ே ሺ ܩ ǡ ܦ ௒ ǡ ܺǡ ܻሻ ݉݅݊ ி ݉ܽ ݔ ஽ ೉ ࣦ ீ஺ே ሺ ܨ ǡ ܦ ௑ ǡ ܻǡ ܺሻ ܩ ՜ ݔ ሺ ݔ ሻ ՜ ܨ ൫ ܩ ሺ ݔ ሻ൯ ൎ ݔ ǡ ܨ ՜ ݕ ሺ ݕ ሻ ՜ ܩ ൫ ܨ ሺ ݕ ሻ൯ ൎ ݕ More on CycleGAN