35 Computer Vision News Computer Vision News Test-Time Adaptation with SaLIP to describe the lungs in a chest Xray. These prompts were then fed into CLIP’s text encoder, which calculated the similarity of all the SAM-generated region proposals and the text prompts and retrieved the relevant mask from the pool of SAM-generated masks. Finally, SAM was prompted by the retrieved region of interest to segment the lung. The SaLIP model has been tested on medical imaging datasets encompassing MRI scans, ultrasound, and X-ray images and diverse segmentation tasks, including brain, lung, and fetal head. “The lung segmentation was more challenging, as there were two regions of interest – the left and right lung,” she recalls. “As reported in the paper, the performance was really good!” A significant contribution of this work is that it employs both the ‘segment everything’ and ‘promptable’ modes of SAM connected through CLIP. This dualmode approach allows for more precise and varied applications in clinical settings. Previous work adapting SAM in medical imaging has used it to segment everything in the image. However, clinicians want to focus on specific regions of interest, which vary depending on the clinical need. “First, I utilized the segment everything mode of SAM to create region proposals for every region in the image,” Sidra explains. “Then, I used the promptable mode, where we use specific prompts to segment a specific region. To connect both these modes, I used CLIP as a bridge between them. To the best of my knowledge, none of the work in medical imaging has utilized both modes of SAM. The literature mainly focuses on finetuning either the segment everything or promptable mode.”
RkJQdWJsaXNoZXIy NTc3NzU=