This article was first published on Computer Vision News of December 2021
Professor Russell Taylor is the John C. Malone Professor of Computer Science at Johns Hopkins University, with joint appointments in mechanical engineering, radiology, and surgery. He is also director of JHU’s Laboratory for Computational Sensing and Robotics. With a career spanning over 50 illustrious years, he speaks to us about his experiences to date and his hopes for the future.
It is a pleasure to speak to you Professor Taylor. Can you tell us more about your work?
For over 30 years, both here at Johns Hopkins and before that at IBM Research, I have been focused primarily on developing a three-way partnership between humans, technology – machines, robots, sensors – and information, to improve surgery and interventional medicine. Through this partnership, you can improve the intervention for an individual patient by making it safer, more efficacious, and less invasive. You can save that information and relate what you did to outcomes, using statistical methods to improve your treatment processes for the next patient. It really is the synergy between this patient-specific control loop, and this broader machine learning-based use of the information that I think will be driving interventional medicine and the progress of medicine in the coming years.
What is the current status of works in this field?
In the last 10 years, there has been an exponential increase in the amount of research and commercial activity in this area. Particularly surgical robotics, which seems to get the most publicity. Personally, we are very active. We have a large center here at Johns Hopkins that grew out of a National Science Foundation funded Engineering Research Center that we established shortly after I moved to JHU. There are very, very large centers in this broad area of computer-enhanced interventional medicine around the world in Europe and Asia, as well as in North America now.
Are there any advancements that have made you go “wow” in the last few years?
What has been most impressive has been the strength of machine learning methods very broadly to improve the use of information in this field. We are beginning to see something I thought for a long time we would be seeing more of, which is that machines are beginning to be more actively involved in interventional procedures. The
first autonomous medical robots were radiation therapy machines that moved around the patient and shot beams of X-rays to kill a tumor – and you hoped not do too much damage elsewhere. The system we developed at IBM Research and which was then commercialized, Robodoc, did the machining step to prepare bones for orthopedic implants.
With surgical robots there are really two key questions. The first is, how can I tell the machine what it is supposed to do in a way it can understand? Second, how can you be sure it can do what you have told it to do safely and properly and it is not going to do something else? At extreme ends of the systems, these are fairly straightforward. The stereotactic systems, like radiation therapy and bone preparation systems, have planned everything out from a CT image or other set of medical images, and there may be levels of autonomy in the planning process – it is typically a human-machine partnership there – but the actual doing of it is very straightforward. The only thing you have left to do is to register all the images and the coordinate systems of these plans to the physical patient. Then the system just does it, perhaps with feedback during the process in case of patient motion or changes.
The dominant paradigm today in medical robots is teleoperation. The machine moves its instruments in the same way I am moving my control handles. The very successful da Vinci surgical robots are an example of that. The commanding and what the robot is supposed to do is very straightforward. There is still a lot that has to be done and the machine has to make a bunch of decisions internally, but they are very straightforward. Now, if I want the robot to take over and do a step while I am doing interactive surgery, then the robot and the surgeon both need to have an agreement as to what the environment of the patient is. You need richer situational awareness. You need things like real-time modelling, real-time computer vision, much better understanding of surgical tasks and how you interact with humans and all of that.
“… how can I tell the machine what it is supposed to do in a way it can understand?”
How can we achieve this?
This is where the machine learning comes in, and just generally the much greater power of computers, making it possible to have much richer environmental models for this situational awareness. There is a danger with that as well that machine learning methods need to learn more humility. They do not know what they do not know. One of the challenges is, how can the machine recognize when it is in a situation it has not been trained for? We are beginning to see some research in that area, but I think it is a key barrier.
What can you see happening next?
I see there being a gradual increase in the capabilities of these systems either remotely, like teleoperated systems, or another way of controlling these robots which is more like hand-over-hand control. If the robot and the surgeon are holding a tool and the surgeon pushes on the tool, the robot can feel that and move accordingly.
But because it is a robot, and it does not have hand tremor it can also refuse to move in a certain direction. It can enforce a safety barrier, or control can be shared in various other ways between the surgeon and the robot.
In some cases, the robot might be adding a palpating motion to some guidance a surgeon is using, say with a joystick or hand-over-hand control. In other cases, the surgeon might guide the robot to a place where it can start and take over and do something on its own, which is what happened with the system we developed at IBM all those years ago. The surgeon used hand-over-hand guiding to help get the robot into the position where first it could do some registration, but then they would hand-over-hand guide it to a position where it was safe for the robot to begin machining, and then it would take over and do that task on its own while the surgeon supervised.
I think you are going to see a lot of examples of that over time. There is a lot of interest in suturing, but personally I am a little less excited by that than with some of the other things you might ask a robot to do for you.
What would the other tasks be?
There are a number. Some are maybe easier. Retraction, placement of therapy, tissue resection and debridement, there are a whole bunch of things like that. Suturing is a complex skill, so if you can do that, you can do a lot of other things. There are also tools that help there. The obvious one is a stapler, where you do not suture, you staple. Intelligent retraction might be another one.
I suppose there still is a lot of space for advancement in R&D in this field…
Yes, of course. Some technology is just now beginning to be mature enough to be commercialized, but there is a huge development gap to go from something that is proof of concept to clinical deployment. There are also things we do not know how to do and that is across the board. From the point of view of academics, what I was just telling some students the other day is it is very important to understand the whole problem. To understand in this three-way partnership what is the advantage you are trying to provide to a surgeon, or to a patient? Then to achieve that there are some things that are simply good engineering that we must do. There are some things frankly we do not yet know how to do. It is at that boundary between not knowing how to do it and it being good engineering to do it that I think the most productive academic research occurs.
What is the biggest advancement that you hope in our lifetime we will be able to attain?
In my lifetime? Well, I am an old guy, but there are a number! If you could get a surgical robot that was able to be a really effective assistant, that would be one thing. Another would be where a robot could do tasks that currently you would expect a skilled resident to do. That is something I would really hope to see. Also, more on the engineering side, I would like the system to be able to look over a surgeon’s shoulder or another robot’s shoulder and be able to follow along what is being done, detecting when they are about to get into trouble and raising the
alarm.
How far are we from that?
Oh, at least four Einsteins! [he laughs] I would not want to speculate. When I was a graduate student at the Stanford AI Lab – and that was 50 years ago – I remember we were beginning to do research on robotics, and we had a hand-eye project. People were already talking somewhat derisively about the Summer Vision Project, where all of computer vision could be done in a summer, but we also had the notion with what we knew at the time that we could program a robot to assemble a chainsaw engine. That practical problem drove lots of research. Some of which led us to have a much better appreciation for manufacturing engineers and led to advances in understanding how to do programming to take advantage of sensing to make a precise, but not geometrically accurate robot able to do some of these tasks. That experience and many others since then have led me to be careful about predicting the future. Obviously, the past is much easier to predict!
“I have been focused primarily on developing a three-way partnership between humans, technology – machines, robots, sensors – and information, to improve surgery and interventional medicine.”
What advice would you give to a young student now, like you were at Stanford 50 years ago, who is starting his or her career and wants to have an impact?
If you want to get into my field, there are a few things. You obviously need to develop deep knowledge in an area of computer science, which is itself becoming somewhat more specialized, but it is also very important to develop broader knowledge as well. We are dealing with systems that have mechanical components, electronic components, sensors, information, programming, human interfaces. Also, get yourself into an environment where you can work with end users. Ideally, a place where you can go into the operating room fairly frequently to talk to physicians and try to understand their problems. You do not need to go to medical school in my opinion. You will quickly learn a lot of medicine is about vocabulary, but it is important to get some domain knowledge as well, if you can, and be open to asking questions. The other thing is what are sometimes referred to as the Heilmeier questions. At IBM Research, Ralph Gomory had a similar set of questions he would ask. What do you want to do? What is your goal? How will you know when you have succeeded? How will you measure success? Also, who cares? Why is that important? Then you can begin to worry about how you are going to go about it and what you do not know. That is general advice whatever field you are going into but understanding that is really important.
Finally, after a challenging couple of years where we have all been kept apart, do you have a message for the MICCAI community in advance of our planned reunion in Singapore next year?
I was one of the founders of the MICCAI Society and the growth has been remarkable. We have to be careful not to go off in being so enamored of one new technique, such as deep nets, that we lose sight of what our larger goals are. We need to begin to look at the complete path of information in medicine. It is medical image computing and computer-assisted intervention and often you see this tension between MIC and CAI communities (Medical Image Computing and Computer Assisted Interventions), but they are beginning to come together more. The CAI folks need to understand the information they can get from imaging and the imaging folks need to focus on what more they can do with information other than get a nice picture or a nice analysis or something. I am a great believer in MIC and CAI together. That was the whole point. MICCAI was formed from the union of three conferences, and it is wonderful that we are continuing that legacy.
Keep reading the Medical Imaging News section of our magazine.
Read about RSIP Vision’s R&D work on the surgical applications of AI.
