AI Development and Evaluation (AIDE) Lab
Department of Radiology
We explore and leverage the latest machine learning techniques and architectures to develop AI algorithms addressing inefficiencies and challenges in maintaining high quality in imaging-based healthcare. Our goal is to utilize AI in quality control processes and alleviate healthcare staff of tasks that take them away from direct patient care.
Our lab is currently investigating the use of large language models to assess clinical histories accompanying imaging orders. Clinical histories that are accurate, relevant, complete, and concise have been shown to improve radiologists’ accuracy of detection and characterization of abnormalities; however, in practice, clinical histories are largely incomplete or unhelpful. We are leveraging LLMs to efficiently evaluate the completeness of clinical histories to quantify and raise awareness of this potential quality improvement opportunity.
Implementation and deployment of AI in the clinical environment can be a high cost, time, and resource effort. We develop methods to comprehensively assess AI beyond just the model and algorithm by focusing on dataset evaluation and real-world clinical variables to provide a thorough understanding of the model capabilities and limitations. We analyze how AI models are affected by differences in patient demographics, conditions, and care settings to ensure consistent and equitable performance.
Although there are 500+ FDA cleared AI products in radiology, only a handful are used routinely in clinical practice. One of the barriers for adoption is the gap in AI product performance observed at clinical sites compared to the performance results that are published and marketed. There is also limited data available for clinical sites to compare multiple vendor solutions. We are completing a comparative evaluation of FDA cleared AI applications for intracranial hemorrhage detection on non-contrast CT scans.
Establishing meaningful, relevant metrics and acceptable ranges of performance for AI applications are critical to ensure the safe and effective use of AI in routine clinical practice. We develop strategies and recommendations to conduct robust quality assessments of AI models.
Proper patient positioning and image quality are essential for diagnostic tasks, especially for measurement-based diagnoses. Rather than relying on qualitative methods of assessing image quality, we are developing an AI model to evaluate the quality of radiographs to be used for pediatric acetabular index (PAI) measurements in the assessment of hip dysplasia.
We build tools for monitoring AI performance to detect anomalies, deviations from expected performance, and potential risks associated with AI deployment. By employing proactive monitoring strategies, we aim to foster trust and confidence in AI technologies while ensuring timely intervention in the event of performance degradation or application failure.
