CVPR Daily - Wednesday

DAILY Wednesday Poster Presentation 12 ZekunHao is a second-year PhDstudent in theDepartment of Computer Science at Cornell University. His work is about 3D shape manipulation and he speaks to us ahead of his poster presentation this afternoon (Wednesday). There are many existing methods for image manipulation where you can utilize a powerful machine learning model to assist the user to modify a 2D image in a semantic, meaningful and realistic way, but there is not yet such a solution for 3D shapes . This was the motivation for Zekun’s work, which aims to achieve a similar result using semantic segmentation masks in 3D . “We had the idea to use a lower- resolution proxy shape as a proxy of manipulating a higher-resolution more detailed shape,” Zekun explains. “A lower-resolution proxy shape made up of shape primitives is easier for humans to interact with compared to a really complex shape. You can’t easily do any manual manipulation to a high-resolution mesh with millions of triangles.” He designed this originally with VR and AR applications in mind. One such application could be VR sculpting – building a 3D model with a VR headset and controller, which would be more intuitive than working on a 2D screen. “The work is an intersection of computer vision, computer graphics and deep learning ,” Zekun tells us. “It’s not pure computer vision but it shares a lot of common techniques with computer vision. It is mostly 3D machine learning. We learn two signed distance functions and they share the same latent space. We use conditioning augmentation and a variation auto-decoder objective to make the two models couple better. Then when you make any changes on one model, it will propagate to the second model. Eventually you will have rendered a shape to an image, so it’s roughly computer vision.” DualSDF: Semantic Shape Manipulation Using a Two-Level Representation “We use conditioning augmentation and a variation auto-decoder objective to make the two models couple better.”