Computer Vision News - September 2021

Kuan Fang recently completed his Ph.D. at Stanford University. His research interest lies at the intersection of robotics, computer vision, and machine learning, with the goal of enabling robots to learn effective physical interactions in the real world through visual reasoning. He received his B.S. degree from Tsinghua University and has spent time at Microsoft Research Asia, Google [x] Robotics, and Google Brain. He will start as a postdoctoral researcher at UC Berkeley in October 2021. Congrats, Doctor Kuan! Building robotic systems that can autonomously solve challenging tasks in the physical world is a long-standing goal in artificial intelligence. Despite the encouraging progress in robotics in the past decades, most successes occur in task-specific systems that are programmed to follow fixed routines in well- controlled environments. To enable robots to perform challenging tasks in unstructured environments, we investigate methods for learning generalizable perception and control from rich interactions . In particular, we focus on three essential themes: (i) Utilization of rich interactions : How do we design structured models and learning algorithms to endow robots with generalizable perception and control? (ii) Collection of rich interactions : How do we provide sufficient and suitable interaction data to scale up learning. (iii) Close the loop of utilization and collection : How do we jointly adapt the algorithms and the data without extensive human intervention? Learning Tool Use from Simulated Self-Supervision. We develop a method that learns to use novel objects as tools through simulated self-supervision . A neural network work policy is designed to jointly predict the task-oriented grasps and the manipulation actions given visual observations of the object. To improve the generalization capability of the policy, we collect large-scale data through self-supervision using procedurally generated tool objects in the simulation. We demonstrate that the learned policy can successfully perform tool use tasks with a variety of novel objects in both simulation and the real world. 16 Congrats, Doctor! Kuan would like to thank his advisors Fei-Fei Li and Silvio Savarese for their support and guidance during these years. This work wouldn’t have been possible without them.