Computer Vision News - February 2022

53 AIROGS (Glaucoma Screening) “ Be careful not to focus on only one of the two tasks but consider there are two sides to the challenge. The training set and the test set are different. We have deliberately set aside this ungradabledata to simulate what would happen in the real world in a different setting. ” Looking ahead, Clarisa tells us there is a hot topic they will need to consider for future challenges. “ Everybody is talking about trustworthy AI . There are already European guidelines, and they’re also coming in the US, so we’re all looking at what’s needed. We’ll need to think about how we incorporate that into new challenges. ” Clarisa and Coen have enjoyed organizing this challenge, but with a mix of technical people and clinicians on board, it has not been without its obstacles. “ We’ve had to learn how to communicate with each other! ” Clarisa laughs. “ I remember once we had a long discussion, andwewereall talkingabout the same thing but in different ways. After that, we said, okay, we need to find a common language! That really helped going forward. ” Automated methods can fail in the real world due to unexpected out-of- distribution data and poor-quality images . These images are often kept out of curated evaluation datasets, but part of this challenge is evaluating how well solutions can distinguish between gradable and ungradable images . “ We realized that when AI solutions are applied in real clinical practice, they encounter situations they can’t cope with, and performance drops ,” Clarisa tells us. “ We wanted to focus on a challenge that represents real-world problems, so we’re trying to reproduce some of these problems to assess how the algorithms perform. ” Clarisa says you should keep in mind any issues you’ve encountered when developing algorithms in the past if you're taking part in the challenge. “ Think back to times you’ve processed datasets and faced problems due to bad quality or other errors, ” she advises. “ That happens in real practice, and maybe it appears here too. ” Coen adds: