In a new study published in Radiology, our team successfully leveraged large language models (LLMs) to automatically assess the completeness of clinical histories accompanying imaging orders.

This research spans two of the AIDE Lab’s research foci: algorithm development, and quality assessment, in which we strive to use the latest machine learning techniques to not only maintain high-quality AI algorithms but also enable robust quality assessments.

The Problem: Incomplete Clinical Histories

Radiologists depend on concise, relevant, and complete clinical histories to accurately interpret imaging studies and provide a more useful radiology report. However, the clinical histories accompanying imaging orders are often incomplete, inconsistent, or missing key details. Previous efforts to improve the quality (completeness) of clinical histories have required extensive manual processes that are tedious and expensive. An automated approach could both speed up such assessments and make dedicate quality improvement efforts easier to adopt.

The Methodology

The AIDE Lab leveraged existing LLMs for this task. We adapted both smaller open-source and larger closed-source LLMs, including Mistral-7B, LLaMA-7B, and ChatGPT-4-Turbo. Model training and adaptation included in-context learning (all models) and fine-tuning through QLoRA (open-source models only) using a set of deidentified clinical histories accompanying imaging orders from the adult and pediatric emergency department of Stanford Medical Center.

To assess “completeness”, the models were trained to extract five clinical history elements: “past medical history,” “what,” “when,” “where,” and “clinical concern”.

Key Results and Impact

• Accuracy and Agreement: Our fine-tuned Mistral-7B model demonstrated substantial agreement with expert radiologists, achieving an impressive accuracy rate of 91%. Importantly, in agreement studies, the fine-tuned Mistral and adapted GPT models showed stronger agreement with radiologists for some clinical history elements than the two board-certified radiologists achieved between themselves. This also demonstrates the inherent subjectivity of how some clinical history elements are perceived.
• Scalability: The fine-tuned Mistral model was deployed on a large dataset of 48,942 clinical histories. We found that only 26.2% of the clinical histories contained all five essential elements. However, when weighting was applied based on perceived importance of each element, the weighted average inclusion rate of all relevant clinical history elements was 74.3%, which is an impressively high inclusion rate largely due to previous quality improvement efforts carried out by colleagues at Stanford Medicine.
• Open-Source Innovation: In this study, the fine-tuned open-source LLM rivaled the trained proprietary model. This can lead to several advantages, including lower compute resources required to run the smaller open-source model, as well as benefits in patient data security since open-source models do not require external servers. These factors not only enable similar quality assessments at other institutions but also show that automating such efforts may not require significant resources.

Why This Matters

Automating the assessment of clinical histories helps address many of the previous challenges associated with dedicated quality improvement efforts to improve the completeness of clinical histories. Our method provides a way to reliably compare and assess the quality of clinical histories at scale. Up until recently, there was no way of utilizing information from unstructured data sources like clinical histories to learn about communication quality. But with the advent of LLMs, this can be done automatically, and reliably at scale.

Along these lines, the study results also highlight the overall potential of AI tools for quality assessments of text, which may be more broadly generalizable and have wider applications across quality assurance and improvement in radiology.  

Our next steps include exploring the possibility of not only using the tool to measure and monitor quality of information shared within Stanford Medicine but also as an educational tool for clinical trainees to help demonstrate “complete” vs “incomplete” clinical histories. Additionally, this model could serve as an assessment and monitoring tool for AI-assisted clinical summary tools as they become available to assist in providing clinical histories to radiologists.

To help facilitate similar quality improvement projects, our model and code are fully open-sourced: https://github.com/stanfordaide/clinical-history-eval

Evaluate your own clinical histories through this interactive demo: https://huggingface.co/spaces/stanfordaide/clinic-hist-eval-demo-hf

Read the full paper here: https://pubs.rsna.org/eprint/N43E56EYFWGHSBHPZXIW/full

This work was previously featured in the RSNA 2024 Daily Bulletin: https://dailybulletin.rsna.org/en/2024/tue/tue07