AI Challenge and Global Participation
The RSNA Screening Mammography Breast Cancer Detection AI Challenge, held in 2023, drew more than 1,500 teams worldwide. Participants were tasked with developing artificial intelligence models capable of detecting breast cancer in screening mammograms.
The study was led by Yan Chen, Ph.D., professor in cancer screening at the University of Nottingham, UK, and assessed how the submitted algorithms performed in breast cancer detection.
“We were overwhelmed by the volume of contestants and the number of AI algorithms that were submitted as part of the Challenge,” said Prof. Chen. “It’s one of the most participated-in RSNA AI Challenges. We were also impressed by the performance of the algorithms given the relatively short window allowed for algorithm development and the requirement to source training data from open-sourced locations.”
How the Algorithms Were Evaluated
Chen’s team evaluated 1,537 algorithms using a test set of 10,830 single-breast exams that were confirmed by pathology as positive or negative for cancer. The training dataset included approximately 11,000 breast screening images provided by Emory University in Atlanta, Georgia, and BreastScreen Victoria in Australia. Participants could also use publicly available data.
Key performance outcomes included:
- Specificity: 98.7% median rate for confirming no cancer was present
- Sensitivity: 27.6% for positive identification of cancer
- Recall rate: 1.7% – the percentage of cases that AI judged positive
When researchers combined the top three and top 10 algorithms, sensitivity increased to 60.7% and 67.8%, respectively.
“When ensembling the top performing entries, we were surprised that different AI algorithms were so complementary, identifying different cancers,” Prof. Chen said.
The ensemble of the top 10 algorithms performed at a level comparable to an average screening radiologist in Europe or Australia. However, individual algorithm performance varied depending on cancer type, imaging equipment manufacturer, and clinical site.
Implications for Radiology
The study found that algorithms showed higher sensitivity for invasive cancers than for noninvasive cancers. Many of the submitted models are open source, which, according to Prof. Chen, could support both academic and commercial development in AI for mammography.
“By releasing the algorithms and a comprehensive imaging dataset to the public, participants provide valuable resources that can drive further research and enable the benchmarking that is required for the effective and safe integration of AI into clinical practice,” she said.
The research team plans additional studies to compare top challenge models with commercially available AI products and to test performance on smaller, more challenging datasets such as those developed under the UK-based PERFORMS scheme.
Next Steps
RSNA continues to host AI Challenges annually to advance machine learning in medical imaging. The 2025 competition seeks submissions for AI models that assist in detecting and localizing intracranial aneurysms.
