Artificial intelligence in radiology is becoming more capable, while clinical value depends on how these capabilities are used in practice. Generative AI can draft and structure reports, extract information, and support workflow tasks. Agentic systems can connect models, tools, and clinical data to coordinate more complex processes. Once AI enters routine practice, its real-world performance becomes equally important.
Three independent sessions at RSNA 2025 examined these questions from different perspectives: the practical use of generative AI, the possibilities and risks of agentic AI, and the evaluation of AI after deployment. Taken together, they highlight a broader challenge for radiology: making increasingly capable AI useful, trustworthy, governable, and measurable in clinical practice.
AI That Assists: Turning GenAI Into a Useful Clinical Tool
Generative AI has potential applications throughout the radiology workflow. Lisa C. Adams, MD, pointed to intelligent protocol selection, automated prescreening, and real-time appropriateness checks. Large language models can also structure and draft reports, identify inconsistencies or omissions, extract data for research and quality assessment, and generate patient-friendly summaries. Retrieval-augmented generation (RAG) can ground outputs in guidelines, publications, or internal documents, with relevance, traceability, curated sources, and continuous updates as key principles.
Foundation models provide much of this versatility. Isabel Molwitz, MD, described them as the “Swiss army knife of AI”: broadly trained models that can be adapted to a wide range of downstream tasks, including image denoising, sequence acceleration and applications with limited task-specific training data.
For radiologists working directly with large language models, prompting becomes a practical skill. “When you prompt a language model, you're not really asking, you're more directing,” Keno K. Bressem, MD, PhD, explained. Context, expertise, and desired style can substantially shape the response. His practical advice: “better to spend five minutes in your first interaction than to argue with the model.” Users also need to recognize sycophancy: “the model optimizes for your approval as a user and not for truthfulness.”
Generative imaging requires similar scrutiny. Diffusion models offer applications including reconstruction, modality translation, and image generation but can also generate abnormalities. “I would always encourage you to not trust this output blindly,” Anna Zapaishchykova, PhD, said. Ultimately, clinical relevance “must be measured by diagnostic utility, not by just image similarity metrics.”
These applications point toward what Adams called the "augmented radiologist": AI handles suitable routine processes in the background, while radiologists validate results, interpret complex cases, and communicate with patients. “The human judgment does not get automated.”
AI That Acts: Agentic Workflows
Agentic AI gives large language models a more active role within clinical workflows. George L. Shih, MD, described AI agents as “augmented LLMs with access to external tools and knowledge that can decide on its own how to solve a particular problem.” Multiple specialized agents can work together under a supervising agent, while technologies such as the Model Context Protocol (MCP) allow LLMs to communicate with external applications. This can, for example, connect an LLM with PACS and enable natural-language requests for imaging studies.
Potential applications extend beyond information retrieval. Shih presented an educational workflow using dedicated agents to compare residents’ draft reports with final reports and identify different types of revisions. Another project, RADFLOW, was designed to use several agents to extract aneurysm size and location from radiology reports and support handoff to neurosurgery. In preliminary work using synthetic validation data, Shih reported that a workflow combining specialized agents with smaller, self-hosted open-source models performed at approximately the level of a frontier model.
Other potential applications include tracking follow-up recommendations, orchestrating workflows across different systems, and preparing tumor boards by combining information from prior examinations, pathology, and patient records. Connecting agents to multiple clinical systems also introduces additional potential failure points involving hallucinations, integration, accountability, and documentation.
Paulo E. Kuriki, MD, emphasized human involvement, transparency and auditability, equity and continuous monitoring. His recommendation was pragmatic: “Start with something simple,” initially targeting workflow support and defining measurable outcomes. Radiologists also need to understand the agent, its underlying LLM, and the tools it can access so they can recognize and interpret failures. As Kuriki summarized, “humans will supervise and partner with agents.”
AI That Proves Itself: Measuring Performance in the Real World
Traditional quality improvement in radiology is often manual, retrospective, and sample based. AI creates opportunities for continuous monitoring because its outputs generate performance data. True positives, false positives, false negatives, reader-AI discordances, report inconsistencies, and workflow signals can reveal patterns across scanners, protocols, patient populations, and clinical settings. Patricia Balthazar, MD, described this as “AI as a quality sensor”: an always-on source of information that can make the radiology pipeline observable and identify emerging problems earlier.
Clinical performance also depends on what happens when radiologists and AI work together. In a study of 227 radiologists interpreting chest X-rays, the AI model on its own performed better than most of the participating radiologists. Yet with AI assistance, “the average effect on the performance of radiologists was zero.” When radiologists were uncertain and AI was confident, errors fell by around ten percentage points; when AI was uncertain, performance deteriorated by five to ten percentage points.
That average concealed substantial individual variation, including what Pranav Rajpurkar, PhD, called “extreme winners and extreme losers.” Years of experience, specialization, and previous AI experience could not reliably predict who would benefit. In this study, automation neglect also occurred more frequently than automation bias. The findings illustrate why standalone algorithm performance provides only part of the picture: human-AI interaction influences clinical performance as well.
Post-deployment evaluation can expose another gap. At Emory, validated LLM-generated labels were used to assess commercial triage algorithms across more than 101,000 intracranial hemorrhage cases, 40,000 pulmonary embolism cases, and 37,000 incidental PE cases. For intracranial hemorrhage, real-world sensitivity was approximately 82%, compared with 96% reported in the FDA clearance data presented, while subgroup analysis revealed additional weaknesses. Theodorus Dapamede captured the central question succinctly: after deployment, the focus shifts from “will it work” to “is it still working for us.”
Therefore, clinical evaluation has to encompass the algorithm, its performance across local populations and subgroups, and its interaction with radiologists.
Making AI Work in Clinical Practice
Across these different perspectives, technical capability is only one component of successful clinical AI. Effective use requires radiologists who understand the systems and their limitations, governance appropriate to the level of autonomy, and continuous evaluation of local performance and human-AI interaction.
The evidence also cautions against equating strong model performance with clinical benefit. A high-performing model may produce little average improvement when combined with radiologists, while performance after deployment may differ substantially across populations and settings.
As AI becomes more deeply embedded in radiology workflows, its value will depend increasingly on how well it is used, supervised, integrated, and measured. Radiology’s challenge is to build the skills, oversight, and evidence needed to make increasingly capable AI useful, trustworthy and accountable in everyday practice.
This article is based on presentations from the RSNA sessions “From Theory to Practice: GenAI Skills for Radiologists", "Agentic AI in Radiology: Definitions and Applications", and "Advances Quality Improvement Through AI: Automated Monitoring and Real-World Impact".
