Computer Vision News - July 2021

47 Jie Ying Wu In order to use these simulations for intraoperativeguidance,weneedtoensure the simulations are not only accurate but can run at real time speeds. While FEMs are the gold standard for accuracy, they are generally slow. Since we already have a neural network predicting corrections, movements, we can replay the interaction in simulation and see how well it matches real observations of the scene. To observe the interaction with the phantom, we mounted a depth camera and recorded point clouds of scene, such as the one shown in Figure 2. We trained a network to correct for the difference between the point cloud and the model. In a real surgery, the base simulation parameters could come from atlases while we learn patient-specific characteristics from observation. we hypothesized that the network can also learn to predict the deformation step. We train graph neural network to learn to mimic the predictions from an FEM. This required the neural network to learn the dynamics of the scene and use it to predict successive deformations. simulator further for refinement of the predictions. Future goals for this project include learning to model more complex phantoms, such as the hysterectomy phantom in Figure 4, and using augmented reality to provide guidance to users through the real-time simulations. Jie Ying would like to thank her wonderful advisors Peter Kazanzides and Mathias Unberath for being ever supportive and encouraging. She would also like to acknowledge Adnan Munawar for contributing his simulators expertise for this work. We can see there is some accuracy trade off as the network’s predictions are less deep and smoother than the FEM’s. Nevertheless, the speedup realized by the graph neural network is considerable and we consider that it is worth pursuing a network-based Figure 2: Point cloud observing interactions between the robot and the phantom. Figure 3: Deformation as estimated by the FEM (left) vs. the network (right). For the same sequence, the FEM took hours to simulate while the network simulated the deformation in real time. Figure 4: Hysterectomy phantom in the workspace of a da Vinci Research Kit (left) and a model of the same scene (right). The simulation scene is available here.