Segmentation in Orthopedics with Deep Learning

Segmentation is important both for examination and planning of knee replacement, hip replacement, shoulder surgery, lesion detection, osteotomy and many other orthopedic procedures. RSIP Vision’s CTO Ilya Kovler and Senior Algorithm Developer Arie Rond explain how to improve segmentation in orthopedics with deep learning.

Deep learning is being proven to be the most powerful framework for various tasks, and segmentation in orthopedics is no exception. Generic out-of-the-box solutions exist and can produce fair results, but carefully crafted and tailored solutions are needed to make the most out of a deep learning approach. Choosing the correct input, selecting the most suitable neural network architecture and incorporating task-specific prior knowledge into the model can all significantly improve results.

Some specific challenges we encountered in our work on segmentation for orthopedic applications at RSIP Vision include:

• Dealing with metal artifacts that cause severe degradation of medical imaging. Our solution was to dramatically reduce the degrading effect by developing an accurate, physical-based and robust simulation tool of these metal artifacts. This tool was used to produce synthetic data that was added to the training process and substantially improved results.
• Handling delicate issues like osteophytes, missing cartilage, merging bones, etc. Focusing on specific anatomical locations which are more prone to algorithmic errors can be of special benefit to the bone segmentation task. This can be done in the form of synthetic data generation and clever data tuning that facilitates the development of machine learning solutions.
• Combining semi-automatic and expert manual data annotation. Even though unsupervised techniques are available, the best performing deep-learning models require high quality annotated data. We have found that the most effective way to generate high-quality annotations is utilizing specialized algorithms to perform initial segmentation, then completing it manually using specially trained annotators, supervised by expert radiologists. This enables saving the expert work for the crucial points.
• Handling hardware and performance requirements. In typical cases it is necessary to process the image in parts. We recommend paying extra attention to include the diagnostic information in the size of the image part (patch) to ensure correct classification.
• There is no one network that fits all. Choosing the proper network architecture is crucial for producing good results. The platforms we use at RSIP Vision’s labs are variants of several segmentation-oriented architectures that we have developed, with a shape-aware loss that increases accuracy.
• The deep learning results are not always accurate on the pixel level. Using advanced classical algorithms for post-processing can significantly improve the precision. For example, by applying an optimization algorithm with boundary-aware constraints, bone segmentation can be significantly improved.

Additional challenges arise from different image modalities:
• MRI scans suffer from low resolution. For segmentation tasks it has been proven that deep learning exhibits greater robustness and better results.
• In CT scans some pathologies and metal objects can cause severe artifacts in the scan. As mentioned previously, specific deep learning models can be trained to handle these artifacts.
• X-rays produce 2D data that present multiple challenges as anatomical regions can be overlaid on top of each other. To overcome this challenge, we train neural networks to detect and use those complex features. To summarize, AI combined with domain expertise allows for faster, more accurate and robust solutions for bones and joints segmentation for orthopedic procedures.