Algorithmic Bias in AI Diagnostic Tools for Underrepresented Populations

Algorithmic Bias in AI Diagnostic Tools for Underrepresented Populations.

Why AI diagnostics outpace understanding of their equity gaps

The FDA's list of AI-enabled medical devices hit 882 approvals as of May 13, 2024, with 191 of those added in a single update https://www.nature.com/articles/s41746-025-01503-7. That is not a slow, careful rollout. That is a pipeline running at speed, and the specialties receiving the bulk of these tools are the ones where a missed or delayed diagnosis costs the most: radiology takes 76% of approvals, cardiology 10%, neurology 4% https://www.nature.com/articles/s41746-025-01503-7. Distinguishing statistical bias (systematic error in data) from social bias (prejudiced patterns in historical practice) shows why in medical AI these two frequently compound each other. The volume of approval has outrun the work of checking whether these tools perform the same way across every patient who walks into the exam room, and the specialties absorbing the most new tools are exactly the ones where underdiagnosis means a tumor gets missed or a heart attack gets called something else.

Algorithmic bias in diagnostics takes several distinct forms.

Algorithmic bias, in plain terms, means a model that works well for some patients and poorly for others. It is not simply "a bad model." A sepsis prediction tool can post strong overall accuracy numbers and still fail one racial group badly enough to cost lives, because the model was never tested for that gap.

Two different kinds of bias underlie that failure, and medical AI tends to have both at once. Statistical bias is a data problem: systematic errors baked into how information gets collected, measured, or sampled. Social bias is a history problem: patterns of prejudice, embedded over decades of clinical practice, that got recorded into medical records as if they were neutral facts. When a model trains on records shaped by both, it learns the data's errors and the era's prejudices at the same time, and the two reinforce each other rather than canceling out.

Bias does not enter at one stage. Bias can appear during data collection, model development, evaluation, or after deployment, when the model meets a patient population it was never built to serve. A 2025 npj Digital Medicine study presented four LLMs (Claude, ChatGPT, Gemini, and NewMes-15, a local medical-focused LLaMA 3 variant) with 10 psychiatric patient cases representing 5 diagnoses under three conditions (race-neutral, race-implied, and race-explicitly stated). Representation bias happens when the training population does not look like the population that will eventually be diagnosed. Selection and sampling bias happens when certain groups are systematically over included or under included in the data. Measurement bias comes from inconsistent data collection across hospitals and clinics, where one site's blood test protocol does not match another's. Historical and proxy bias comes from the variables themselves, the targets a model is told to optimize, when those targets already carry old inequities inside them.

Skewed training data as the first layer of the problem

Homogeneous training data is where most of this starts. A model can only learn patterns that show up in the data fed to it, and if that data skews heavily toward one race, one age range, one gender, or one geographic and economic bracket, the model becomes very good at reading that group and unreliable at reading everyone else. This is a direct, mechanical result of how these systems learn. It's a direct, mechanical result of how these systems learn.

The University of Michigan uncovered a version of this that deserves close attention, because it shows how bias compounds rather than sits still. Researchers found that Black patients are less likely than white patients to receive the diagnostic tests doctors use to catch severe disease, sepsis among them. Testing rates for white patients run up to 4.5 percentage points higher than for Black patients with the same age, sex, complaint, and triage score at the emergency department door https://news.umich.edu/?p=196425. That gap does not stay contained to the exam room. A patient who never got tested shows up in the record as healthier than they actually were, and a model trained on that record learns the same false lesson, underestimating how sick Black patients are before they even arrive. The undertesting becomes the training signal, and the training signal becomes the diagnosis. The BMJ PROBAST+AI framework is a quality and risk-of-bias assessment tool for prediction models; it does not itself report the statistic that nearly 80% of AI risk tools were developed on homogeneous populations, and that figure along with the external validation claim require a different source.

Proxy variables, historical practice, and structural inequity in model design

Models rarely get told to predict "health" directly. That's too abstract to train on, so developers reach for stand-ins: cost of care, how often a patient gets readmitted, how frequently they get hospitalized. A patient who used fewer healthcare dollars might simply have less insurance, live farther from a specialist, or have good reason to distrust the system treating them, not less illness. A model trained to predict "need" from "past spending" will read all three of those situations as good health, and it will underserve every patient in them going forward.

Historical treatment patterns carry the same trap. If clinicians, over years of practice, undertreated a group of patients, that undertreatment is in the record as the norm, and a model trained on that record learns to repeat it rather than correct it. One framing of this problem, from the research collected under the name Beyond the Algorithm, describes what happens next with real precision: these systems embed inequality into digital infrastructure "under the guise of neutrality," because the output comes out of a math formula and math formulas look objective even when the inputs are not. That is the trap. A biased spreadsheet still looks like a spreadsheet.

Dermatology: what happens when training data excludes most human skin

Dermatology makes the abstract argument concrete, because the gap can be measured in a single statistic. Only 2.1% of images in public dermatology datasets include Fitzpatrick skin type data at all, which means the datasets training AI to spot skin cancer were built almost entirely on lighter skin, with darker tones barely represented in the record https://www.jaadreviews.org/article/S2950-1989(26)00065-6/fulltext.

The performance numbers that follow are stark. ModelDerm posts an AUROC of 0.94 on lighter skin, a strong number by any clinical standard, but that score falls to 0.65 on Fitzpatrick types V and VI, the darker end of the scale https://www.jaadreviews.org/article/S2950-1989(26)00065-6/fulltext. DeepDerm follows the same arc, dropping from 0.88 to 0.56 https://www.jaadreviews.org/article/S2950-1989(26)00065-6/fulltext. DeepDerm's published AUROC fell from 0.88 on lighter skin tones to 0.56 on Fitzpatrick types V to VI https://www.jaadreviews.org/article/S2950-1989(26)00065-6/fulltext. False negative rates on dark skin reached 77% to 88% across these tools, so the majority of actual cancers got labeled healthy https://www.jaadreviews.org/article/S2950-1989(26)00065-6/fulltext. A melanoma missed by a false negative does not just sit still. It grows.

This is not confined to one company's model or one generation of tool. It appears again when a general-purpose large language model gets asked to do dermatology, which suggests the problem sits in the training data itself rather than in any single product's engineering.

There's a more unsettling wrinkle when AI gets paired with a human clinician rather than working alone. A 2024 Nature Medicine study led by Groh and colleagues found that AI assistance raised diagnostic accuracy by 33% among dermatologists and by 69% among primary care physicians overall https://news.northwestern.edu/stories/2024/02/new-study-suggests-racial-bias-exists-in-photo-based-diagnosis-despite-assistance-from-fair-ai/. A 2024 Nature Medicine study (Groh et al.) found that AI assistance increased diagnostic accuracy by 33% among dermatologists and 69% among primary care physicians overall, but for primary care physicians the accuracy gain was larger on light skin than on dark skin, exacerbating accuracy disparities by 5 percentage points, a difference the researchers found statistically significant. The tool meant to help doctors treat patients more fairly ended up making the unfairness slightly worse for one group while helping the other more. Performance collapses when tested on Fitzpatrick types V–VI.

Chest radiology, sepsis prediction, and cardiac diagnostics: the same pattern across specialties

The underdiagnosis rate was higher for intersectional subgroups, with Hispanic female patients cited as an example. A separate review looking at chest radiography, breast ultrasound, and eye imaging together found the same story repeating: reduced diagnostic performance, lower sensitivity, and higher underdiagnosis specifically among Black and Hispanic patients. Underdiagnosis carries a particular kind of danger that a simple accuracy drop does not. Telling a sick person they're healthy closes the door to treatment entirely, rather than just making that treatment less precise.

Sepsis prediction tells a similar story through a different mechanism. The Epic Sepsis Model, a proprietary algorithm built into Epic's electronic health record system, exhibited poor performance in external validation in a US cohort study. Underrepresentation of racial groups in the training data has been named as a contributing cause. And the University of Michigan's undertesting finding connects directly here: if Black patients get tested less often for the markers of severe illness, a model trained on those records will underestimate how sick they are, following the exact mechanical path already outlined in the training data section above. The field has not slowed down to fix this before moving forward, either. In 2024, a different company received the FDA's first-ever clearance for an AI sepsis detection tool, proof that validation gaps and continued deployment are running on parallel tracks rather than one waiting for the other.

Cardiac diagnostics sit in a different position: less settled, more open. AI-powered ECG analysis can flag arrhythmias and heart failure faster than a clinician working alone, and that speed genuinely matters in acute cardiac events. A scoping review running through 2025 and into 2026 is still in the process of mapping how much performance varies by patient subgroup, with the manuscript expected to wrap by the end of February 2026 and submission for peer review to follow shortly after. This is a live case study, and the honest thing to say is that the evidence base is still being built. It's a live one, and the honest thing to say is that the evidence base is still being built. In chest radiology, Seyyed-Kalantari et al., published in Nature Medicine, identified the same pattern.

Pulse oximeters: a hardware bias that cascades into AI training data

Pulse oximeters are not AI tools. They matter here anyway, because the bias they carry in hardware becomes the bias AI inherits once that hardware's readings get folded into a training dataset. The device works by shining light through skin, and melanin absorbs light differently depending on how much of it is present. The readings themselves are not colorblind. Black patients experienced occult hypoxemia, low blood oxygen that the device failed to flag, at nearly three times the rate of white patients, and researchers have reported similar effects for Asian and Hispanic patients.

During COVID-19, this was not an abstract measurement error. Pulse oximetry served as a primary triage tool for deciding who needed hospital admission, and an oximeter reading oxygen saturation as higher than it actually was meant slower recognition that a Black or Hispanic patient was deteriorating, and slower admission as a result. State attorneys general have flagged the next step in this chain directly: as pulse oximetry data gets digitized at scale and folded into datasets that train healthcare AI, the color bias baked into the hardware does not stay contained to the device, it gets learned and reproduced by whatever model trains on that data next.

Regulators have started responding. The FDA released draft guidance raising the recommended data submission for pulse oximeter approval from 200 data points to 3,000, and raising the recommended study sample size from 10 people to 150 or more https://www.statnews.com/2025/01/06/pulse-oximeter-fda-draft-guidance-released/. A sample size of 10 was never going to catch a disparity that only becomes visible when you have enough patients of different skin tones to compare. Whether 150 is enough remains an open question, but it is a meaningfully larger net than the one that existed before.

Psychiatric and mental health AI: where race-explicit data makes recommendations worse

A 2025 study in npj Digital Medicine tested four large language models, Claude, ChatGPT, Gemini, and NewMes-15, a locally run, medically focused variant of LLaMA 3, against ten psychiatric patient cases spanning five diagnoses. Each case ran under three conditions: race left out entirely, race implied through context, and race stated outright.

The finding cuts against a common assumption about where bias in these tools would show up. Diagnostic decisions, the actual labeling of what condition a patient had, showed minimal bias across the three conditions. Treatment recommendations did not hold up nearly as well: models frequently proposed weaker treatment options once a patient's race was indicated, whether implied or stated directly. The models could still tell what was wrong. They got worse, measurably, at deciding what to do about it once race entered the picture.

Bias was not evenly distributed across the four models tested. NewMes-15 showed the highest degree of racial bias of the group, while Gemini showed the least. That range matters, because it means the outcome is not some fixed, unavoidable property of language models in general, it is a property of specific training choices that specific developers made, and it can apparently move a lot depending on those choices.

A study from Cedars-Sinai adds a sharper, more reproducible version of the same problem. Researchers tested two large language models against an ADHD case, and both models left out medication recommendations entirely when the patient's race was stated explicitly in the case, while including those same recommendations when race was left out of the description. That is a direct, repeatable demonstration that adding a single piece of demographic information to an otherwise identical case changed what treatment a patient was offered by the machine. It is a direct, repeatable demonstration that adding a single piece of demographic information to an otherwise identical case changed what treatment a patient was offered by the machine.

The cost bearers: populations absorbing compounded diagnostic failures

A 2024 systematic review covering 30 studies over a ten-year stretch found a significant link between AI use in healthcare and worsening racial disparities in outcomes, including longer waits for appointments, weaker success predicting mental health outcomes, and underdiagnosis of disease, with Black and Hispanic patients bearing the weight of it most consistently https://www.kff.org/racial-equity-and-health-policy/the-growing-use-of-artificial-intelligence-in-health-care-and-implications-for-disparities/. That is not one bad tool in one hospital system. That is a pattern replicated across three decades' worth of studies and multiple, unrelated clinical domains.

Underdiagnosis deserves to be named again as the harm that matters most here, because it is not a neutral kind of error the way a slightly-off reading might be. Telling a sick patient they are healthy closes off a referral, delays a treatment, and in a condition like sepsis or acute respiratory failure, that delay can be the difference between recovery and death.

Intersectionality sharpens the damage further rather than simply adding it up. The Nature Medicine chest X-ray study found underdiagnosis rates peaked for patients who sat at the overlap of more than one underrepresented category, Hispanic female patients again serving as the clearest example. A patient carrying two marginalized identities at once does not experience twice the risk in some simple additive sense; the model's failure compounds in ways that are worse than either category alone would predict.

And scale is what separates this from ordinary clinical error. A single overworked doctor having a bad day affects the patients in front of them that day, and then it's over. A biased model embedded in a hospital's electronic health record system touches every patient who passes through that system, applying the same skewed judgment identically, over and over, for as long as the tool stays in use. A sepsis model that underestimates severity in Black patients does not do it occasionally or by accident. It does it every time, for every Black patient the system processes, until someone rebuilds the model or takes it offline. The BMJ PROBAST+AI framework (Moons et al.) found that nearly eighty percent of published AI-driven risk models were developed on homogeneous populations that failed to reflect demographic diversity https://arxiv.org/pdf/2606.18289.. A study used 324 biopsy-confirmed dermatologic images to test ChatGPT-4o for melanoma detection https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12323556/. The DermaSensor pivotal trial enrolled 97.1% White participants https://www.sciencedirect.com/science/article/pii/S2950198926000656.