Large Language Models for Clinical Summarization in Preventive Medicine

The Frontiers in Digital Health MEDAI-LLM-SUMM study found that large language models crossed a real capability threshold in clinical text summarization somewhere in the 2024-2025 window. The models can now match or beat human experts on specific, bounded summarization tasks. That claim carries a lot of weight, and this piece exists to test it against the actual evidence, starting with the study that anchors it and moving through where the performance holds, where it breaks, and what has to be true before any preventive medicine team should trust one of these tools with a patient record.

The core data point comes from Van Veen and colleagues, published in Nature Medicine: LLM-generated summaries of radiology reports and clinical records came out comparable to human experts in 45% of cases, and rated better than the human version in 36% of cases. Add those together and you get a majority of cases where the machine either matched or beat the clinician doing the same summarization task. That's a genuinely notable result. "Inflection point" means something narrower and more useful than parity everywhere, all the time. On tasks that are well-defined and bounded, a single document, a clear objective, a known format, the models perform. Stretch the task, and the performance changes. The most current systematic look at this question inside a single preventive medicine domain comes from Ferreira Santos and colleagues, published in Diagnostics on January 26, 2026, covering LLM evidence in cardiovascular prevention specifically. That review anchors most of the specialty-specific claims that follow.

What LLMs are doing in preventive medicine, three distinct task categories

Ferreira Santos and colleagues break the current use of LLMs in cardiovascular prevention into three buckets: patient-facing work, clinician-facing work, and system-level work. Each one carries a different risk profile, and lumping them together is how vendors oversell readiness.

On the patient side, LLMs write education material that reads as warm and largely follows clinical guidelines. That's a real, usable output. What they can't do is hold the nuance a human counselor brings to an unsupervised, personalized conversation. Telling a patient with a borderline cholesterol-marker number and a family history of early heart disease what their risk actually means, and what to do about it, is not a job for an LLM operating without a clinician checking the output. Autonomous patient interaction in a high-stakes preventive context, cardiovascular risk counseling being the clearest example, isn't safe territory yet.

On the clinician-facing side, the picture splits cleanly in two. Summarizing clinical notes and drafting documentation, the actual bread-and-butter of the job, is where these models earn their keep. That's the core use case, and it's the one backed by the strongest data. But ask the same model to compute something deterministic, a Framingham risk score, say, or to make an autonomous clinical call, and reliability drops. It cuts the time clinicians spend on documentation. The decision-support claim needs heavy qualification, and any vendor pitching that as similarly ready is getting ahead of the evidence.

System-level applications sit earliest on the adoption curve: automated phenotyping, registry construction, quality surveillance across a population. These carry high potential value for preventive programs run at scale, but they're less proven. Multimodal risk prediction, combining imaging, labs, and text into a single risk output, is the least validated of the three categories discussed here. A preventive medicine team reading this should draw one clear line: note summarization and documentation drafting are ready for near-term use. Individualized risk scoring and treatment recommendations are not, at least not without a clinician checking every output.

Where performance holds up and breaks down: measured error rates and benchmarks

The clearest error-rate data comes from Asgari and colleagues, published in npj Digital Medicine. The study ran across 49,590 transcript sentences and 12,999 clinical note sentences, spread over 18 experimental setups, which makes it one of the larger error-analysis efforts in this space. The overall hallucination rate landed at 1.47%, and of those hallucinations, 44% were judged "major." The omission rate came in at 3.45%, with 17% of those rated major. Through iterative prompting and workflow adjustments, the researchers pushed major error rates below previously reported rates for human note-taking errors.

That 1.47% number is genuinely good. It is also not representative of the field as a whole. The MEDAI-LLM-SUMM review, which looked across 216 published studies, found hallucination rates ranging from 1.47% up to 61.6%, depending on task complexity and how the evaluation was done. So the low figure is a best-case result achieved under careful conditions, not a floor the field can assume by default. What moves that number around so much? Task complexity, the specific prompting method, which model gets used, and how narrow or broad the clinical domain is. The same underlying model can turn in a tight, accurate summary of a chest X-ray report and then produce a much shakier summary of a cardiovascular risk history spanning several years of visits.

A 2026 benchmark comparing eleven LLMs against 50 discharge summaries drawn from the MIMIC-IV dataset makes the point concrete. Clinicians rated the summaries at a mean completeness of 79.0%. The models' own self-assessed completeness scores ranged from 54.9% for Qwen 2.5-7B up to 74.2% for Gemini 3 Flash. Every one of the eleven models under-rated its own completeness relative to what clinicians actually saw. Gemini came closest, landing 4.8 percentage points below the clinician score. Qwen missed by 20 points. The roster tested included models from Google DeepMind, DeepSeek, Anthropic, OpenAI, xAI, and Meta AI, among others. That spread across major labs tells you this isn't a quirk of one vendor's model, it's a pattern across the field.

There's also a well-documented "lost-in-the-middle" effect: when a summarization task spans multiple documents, models tend to drop details or scramble the chronology somewhere in the middle of the record. In preventive medicine, where a useful summary often needs to track a blood pressure trend or a lipid panel across years of visits, that's not a minor glitch. It's a structural risk sitting right where the specialty needs the most reliability.

The ICD coding failure documented in MEDAI-LLM-SUMM makes the cautionary case: GPT-4, tested on assigning ICD codes, barely cleared 50% accuracy. A model that writes fluent, clinically literate prose can still fail badly at a rigid, structured coding task. Strong performance in one register is specific to that register and does not transfer to another. The real question for anyone evaluating these tools is which specific tasks, on which specific models, under which specific prompting setup, produce an error rate low enough to trust. It's which specific tasks, on which specific models, under which specific prompting setup, produce an error rate low enough to trust. Right now, that answer changes depending on which corner of preventive medicine you're standing in.

The evaluation problem, why measuring LLM summarization quality in clinical settings is harder than it looks

Human review catches the errors that matter, but it doesn't scale. Reading every summary produced by one model against the source chart, sentence by sentence, is labor-intensive review that clinical operations can't sustain at volume. Automated text-similarity metrics scale fine, but they routinely miss the errors that actually matter clinically, a wrong dosage, a flipped lab value, because those metrics were built to measure fluency and overlap, not factual accuracy.

One practical bridge between the two has emerged: using an LLM itself as the judge of another LLM's output. Benchmarked against a validated instrument called the Provider Documentation Summarization Quality Instrument, or PDSQI, a model called GPT-o3-mini scored an intraclass correlation coefficient of 0.818 against human evaluators. Its median score difference from human raters was zero, and it finished its evaluations in 22 seconds. That's a strong agreement score for something that fast, and it points toward a workable quality-control layer for clinical deployment at scale, one that doesn't require a human to read every single summary.

Still, the underlying evidence base has a large hole in it. Of the 216 studies reviewed by MEDAI-LLM-SUMM, only 7% ran any kind of external validation. Only 3% performed a patient safety assessment. That means the overwhelming majority of published performance claims in this space rest entirely on internal validation, tested by the same team that built the tool, on the same data the tool was tuned against. That's a meaningful caveat to carry into any reading of a vendor's benchmark chart.

The MEDAI-LLM-SUMM authors respond to this gap with a 24-item checklist spread across six sections. It's built specifically to cover ground that existing frameworks, CONSORT-AI, SPIRIT-AI, TRIPOD-LLM, and DEAL among them, don't adequately address for summarization tasks: hallucination assessment protocols, how reference summaries get built in the first place, validation standards for LLM-as-judge setups, and pilot-testing requirements. Any tool marketed as clinically ready deserves to be measured against that checklist before it's taken at its word, because the published record shows external validation and safety assessment are still closer to the exception than the rule.

Specific failure modes that matter most in preventive medicine contexts

Certain failure types carry outsized weight in preventive medicine specifically, because the specialty depends so heavily on getting particular kinds of detail right.

A hallucinated medication name, an invented lab value, or a finding attributed to the wrong visit can bend a cardiovascular risk summary in a direction that steers a clinician's prevention decision. That's a direct line from a model's error to a patient's care plan, not an abstract data quality issue.

Omissions in longitudinal records carry similar weight. Preventive medicine runs on trend data, including blood pressure over years, a lipid panel history, and a BMI trajectory. The "lost-in-the-middle" degradation described earlier isn't an equal-opportunity risk across specialties. It lands hardest exactly where the clinical value depends on tracking change over a long record. Then there's temporal obsolescence: a model trained on an older guideline snapshot may summarize a patient's risk profile against thresholds that have since been revised. Ferreira Santos and colleagues name this directly as one of the constraints on safe deployment.

Automation bias deserves its own line. A clinician working under time pressure may simply accept a model's summary without giving it the scrutiny it needs, especially once the tool has built up a track record of being right most of the time. Ferreira Santos and colleagues list this alongside hallucination, temporal obsolescence, and data privacy as core constraints on deployment, and it's arguably the hardest one to fix with better engineering, because it's a human behavior problem sitting downstream of a technical one.

Personalization has its own ceiling. These models generate content that's accurate at the population level and follows the guideline. What they don't do well is adjust for the patient in front of them who has three comorbidities interacting in ways no guideline anticipated. Preventive medicine relies on ongoing, longitudinal patient records, which makes data privacy architecture a condition that has to be satisfied before deployment starts, not a feature bolted on afterward.

None of these failure modes scatter randomly across clinical tasks. They cluster tightly around precision on specific facts, chronological accuracy across long records, and individualized judgment, which happens to describe most of what preventive care actually requires.

What governance frameworks currently require before clinical deployment

Ferreira Santos and colleagues propose a framework built specifically as a roadmap for responsible LLM use in cardiovascular prevention. It breaks down into six requirements: clinical validation, auditability, risk stratification, data privacy, integration, and ongoing vigilance. Each letter is a gate a deployment has to pass through, not a box to check once.

The framework is organized around a single principle: the clinician stays in the loop. Ferreira Santos and colleagues are explicit that LLMs should run as supervised reasoning tools that support a clinician's judgment, not replace it. Clinicians remain the ones positioned to catch the specific kinds of errors these models make, because they hold context the model doesn't.

The Asgari framework from May 2025 offers a workable companion to this: pairing hallucination and omission measurement with an assessment of clinical safety impact gives an organization a template for both governance and safety review. Their iterative prompting approach pushed major error rates below rates reported for human note-taking, the strongest result in this whole body of evidence. That's a real, demonstrated ceiling on how good this can get with the right process wrapped around it.

MEDAI-LLM-SUMM's own checklist includes a Safety section with just two items: ethical approval and data anonymization. That's about as low a bar as a clinical AI deployment could clear, and yet only 3% of the reviewed studies included any patient safety risk analysis at all. On the deployment side, John Snow Labs released a Medical LLM suite in 2025, spanning models from a smaller scale up to a much larger one in terms of parameters, benchmarked specifically on clinical knowledge and medical reasoning, deployable on-premise rather than through a cloud API. That distinction matters directly for the data privacy piece of any governance framework, since on-premise deployment sidesteps a category of privacy risk that cloud deployment carries by default. The 70B version of that suite handles up to 32,000 words of context in a single pass, which speaks directly to the long-record problem that occurs throughout preventive medicine.

None of this closes every gap. Large-scale external validation, real-world patient safety assessment, and evaluation standards that hold consistent across institutions are all still open problems, framework or no framework.

Gaps in the evidence base for adopters today

Out of 216 studies reviewed by MEDAI-LLM-SUMM, only 7% ran external validation. The overwhelming majority of what's published is a research team testing its own tool on its own terms.

The 3% patient safety assessment figure is the more consequential number of the two. The field has spent enormous effort measuring how well these models perform on benchmark tasks and comparatively little effort answering the question that actually determines whether a deployment is safe: what happens to a patient when the model gets it wrong?

Coverage across the literature is narrow in ways that matter specifically for preventive medicine. MEDAI-LLM-SUMM found 57% of the reviewed studies focused on a single narrow task type, radiology reports being the common example. Half the studies drew on ICU data. And 87% worked exclusively in a single language. Preventive medicine lives in outpatient settings, tracks patients over years rather than days, and serves populations that span many languages. The existing evidence base barely touches that territory.

The gap between lab results and clinical reality is visible in deployed products, too. Microsoft's DAX Copilot, the Dragon Ambient eXperience tool, has been marketed on significant reductions in documentation time, and it's drawn real skepticism from outpatient physicians over the lack of independent efficacy evidence backing those claims. That gap between what a vendor states and what gets independently confirmed in practice is exactly the pattern any adopter needs to watch for.

Before adopting any tool, a preventive medicine practice should be asking a short set of pointed questions. Is the validation behind this tool external, or only internal? Has anyone run a patient safety impact assessment on it? What's the documented hallucination rate, and what share of those errors were rated major? Does the deployment run on-premise or through the cloud, and does that match the practice's data privacy requirements? And is there an actual defined workflow that keeps a clinician checking the output, not just a line of disclaimer text buried in the interface?

The MEDAI-LLM-SUMM checklist, all 24 items across its six sections, works as well for auditing a vendor's claims as it does for judging a published study. Adopters don't need to take anyone's marketing copy at face value when a structured, publicly available tool exists for checking the work.

How to assess readiness for LLM summarization deployment in a preventive medicine setting

Readiness comes down to three layers that have to be judged together: which task is being handed to the model, which model is doing the work, and whether the governance structure around the deployment is actually in place. Assessing any one of these in isolation gives a false sense of confidence.

On task selection, the evidence points in a fairly clear direction. Summarizing a single clinical note, drafting routine documentation, and pulling together a radiology report summary are at the high end of readiness, backed directly by the 45%/36% parity-and-superiority finding from Van Veen and colleagues. Generating patient education material sits in the middle: it produces genuinely useful, guideline-concordant drafts, but it needs a human reviewing the copy before it reaches a patient. At the low end sit deterministic risk calculations, individualized treatment recommendations, ICD coding, and summarization across long multi-document records. Each of those has documented, specific evidence of unreliability, not just a general note of caution.

Model selection has to follow from that same evidence rather than from a vendor's marketing sheet. The discharge summary benchmark discussed earlier showed meaningful, measurable differences between models on the exact same task, with completeness scores spread across a 20-point range. A practice choosing between models should be asking for that kind of task-specific completeness and hallucination data, not a general capability claim, and should weigh whether the deployment needs on-premise infrastructure to satisfy its own privacy requirements. Governance readiness, finally, is a working structure. It is a defined clinician-in-the-loop review step, a documented external validation trail, and a real answer to what happens when the model gets something wrong. Where any one of those three legs is missing, the deployment isn't ready, regardless of how good the underlying model's benchmark scores look on paper.