Computer Vision News - July 2021

46 Congrats, Doctor! For modelling the patient scene, while pre-operative scans such as CTs capture the anatomy, they do not necessarily capture how organs would deform. Current methods for simulating soft-tissue deformation rely on accurate material parameters and boundary conditions, which are often unavailable in patients. We developed a machine learning method to correct the imperfect simulation based on observations. It is expensive to label data for learning deformations though since it is non-categorical: imagine the difference between labelling that this image contains a liver vs. labelling how the liver should deform if a robot interacts with it a certain way. Instead, we use observations of the scene as labels. In a very simplistic set up, we probe a soft-tissue phantom, shown in Figure 1, using a da Vinci robot and learn to correct simulations of how it deforms. Jie Ying Wu recently completed her PhD at the Johns Hopkins University. The goal of her research is to enable surgical robotic systems to better understand surgeries. This constitutes a step to create intelligent assistance to surgeons during operations. She will start as an assistant professor at Vanderbilt University in January 2022. Congrats, Jie Ying!!! Jie Ying separates a robot-assisted surgery (RAS) into 3 parts: the robot, the surgeon, and the patient scene. For each of the parts, she learns representations of the data through cross-modal self-supervised learning. Cross-modal self-supervised learning exploits the synchronicity of the signals inherent in the system and learns to map the signals from different modalities onto each other. This could be used to reveal underlying patterns, such as surgeon gestures or skill level such as in [Wu et al, IJCARS 2021] , or improve models of patient anatomy. We build a model of the phantom in simulation. We can then use finite-element method (FEM) to model how the phantom should deform using material parameters we had measured. Since the robot tracks its instrument Figure 1: Soft-tissue gel phantom (left) and a model of it for simulation (right) [Wu et al., IJCARS 2020] .