Computer Vision News - May 2023

16 Congrats, Doctor Annika! Annika Reinke recently defended her PhD at the German Cancer Research Center (DKFZ). Her research focuses on the validation of biomedical image analysis algorithms - making sure that our algorithms do what they are supposed to do. After completing her PhD, she will continue working on the topic as a postdoc at the DKFZ. Congrats, Doctor Annika! Why do we need to care about algorithm validation? In fact, proper validation of AI algorithms is frequently neglected in favor of a strong focus on the development and exploration of new models. This research practice can, however, be really risky as it may propagate poorly validated algorithms that could cause adverse outcomes forpatients!Thus,athoroughandhigh-qualityvalidationiscrucial foranyalgorithm to potentially be used in clinical practice. This particularly holds true for image analysis competitions (challenges), which have emerged as the standard technique for comparative assessment of AI algorithms and determining which is the most effective in solving a certain research question. Given the tremendous importance of challenges for the research, it is surprising that hardly any attention has so far been given to quality control. Validation pitfalls During her PhD, Annika and team raised awareness of severe flaws in challenges and algorithm validation. She evidenced how effortlessly both challenge participants and organizers could, in theory, manipulate challenges by taking advantage of security holes in the challenge design. She also found that researchers typically favor common performance metrics without being aware of numerous pitfalls pertaining to their use. She even put together a comprehensive list