Three feedback loops — the Override/Adaptation Loop, the Clinician Learning Loop, and the Training Feedback Loop — can turn a deployed clinical AI system into a dynamic system. Once in use, the system begins to react to its environment, influencing clinician behavior and, in some cases, even creating the very drift it was meant to prevent.

These loops do more than create risk. They make that risk harder to see. Traditional incident review is not built to detect them. Individual clinicians cannot spot them from the bedside. They only become visible when you look across larger populations and over longer periods of time.

Issue 04 asks the next logical question.

If these loops cannot be avoided and routine monitoring cannot consistently detect them, what does an institution actually need to build?

The Governance Reality

This is not a pessimistic conclusion; it is a structural one.

Clinical AI systems are static tools placed into environments that are constantly changing. From the moment they are deployed, their outputs begin interacting with clinicians, workflows, incentives, and data systems that are also evolving. Some of that change comes from outside the model — shifting patient populations, updated protocols, or new disease patterns. Some of it comes from the model itself through the three feedback loops described in Issue 03.

Either way, the result is the same. The gap between the model’s training environment and the real clinical setting widens at deployment. That divergence is not a flaw. It is the predictable outcome of placing a fixed artifact inside a dynamic environment.

That is why initial validation cannot be the endpoint. Responsible AI deployment requires continuous surveillance, not as an optional quality-improvement activity but as a core operational function.

What a Minimum Viable Surveillance Dataset Really Needs

The literature is now clear enough to show what separates a truly useful surveillance dataset from the dashboards most health systems currently rely on. Three components are critical, and all remain largely missing from standard practice.

The first is data provenance markers: clear indicators of whether the training data have outcomes influenced by earlier model predictions. Without this information, every retraining cycle risks absorbing distortions created by the model’s own prior use. In one documented clinical example, a retrained model performed worse even after receiving six times as much training data because AI-influenced labels had contaminated the dataset.

The second is a baseline measure of clinician performance gathered in AI-off conditions before deployment. This is the only reliable way to understand what the Clinician Learning Loop is doing over time. Without a pre-deployment baseline, there is no meaningful reference point for identifying trust miscalibration, deskilling, or never-skilling.

The third is adherence flags: structured records showing whether clinicians followed the model’s recommendations on an encounter-by-encounter basis. Without adherence tracking, retraining cannot separate outcomes that reflect the model’s true predictive performance from those that reflect clinicians’ responses to its recommendations.

Two additional elements complete the minimum viable dataset:

Clinician identifiers for longitudinal tracking, so behavioral drift can be seen at the individual level rather than only in aggregate.

Cumulative alert exposure per clinician, measured over a rolling 90-day window, so institutions can monitor the dose-response relationship between alert burden and behavioral degradation before the effects become permanent.

Right now, no post-deployment monitoring system brings all of this together in real time. That is the real issue. The field does not lack ambition. It lacks infrastructure.

The Five Signal Types: What They Are, When They Appear, and Who Owns Them

To detect the three feedback loops, institutions need to distinguish among five distinct signal types that current systems often blur together. Each one appears at a different stage after deployment, depends on different infrastructure, and belongs to different stakeholders. Treating them as interchangeable is exactly how health systems end up with dashboards that seem comprehensive while missing the most important problems.

1. The Bedside Signal

The bedside signal reflects patient physiology directly — things like vital signs, clinical deterioration markers, and early warning scores. It is independent of the AI system and available in real time, making it useful for understanding what is happening with the patient in the moment.

But that is also its limitation. It tells you what is happening, not why. A worsening bedside signal cannot, on its own, implicate the algorithm. It only becomes meaningful during algorithmic surveillance when paired with the other four signal types.

Ownership: the clinical team.

2. The Clinician Behavior Signal

The clinician behavior signal captures how individual clinicians respond to AI recommendations over time. This includes override rates by clinician and alert type, time-to-decision after alert presentation, cumulative alert exposure over a rolling 90-day period, and agreement rates with AI recommendations. To make this signal useful, institutions need clinician-level longitudinal tracking rather than broad aggregate measures.

This signal commonly emerges within days to weeks of deployment, making it the earliest detectable indicator among the five. For example, a clinician whose override rate is climbing, whose decision-making is becoming faster, and whose alert exposure falls in the highest quartile may already be showing signs of automation bias before any patient outcome data are available.

Ownership: clinical informatics, with escalation to the department chief and the AI governance committee.

3. The Workflow Signal

The workflow signal reflects aggregate system performance. It includes unit-level override rates, unusual alert-firing patterns, irregularities in system response times, and the share of overrides submitted without a structured reason. It looks across clinicians and encounters to identify system-wide patterns rather than individual decisions.

Most health systems already collect the raw data needed for this signal through EHR audit logs. What they usually lack is the analytical capability to monitor those logs for meaningful anomalies.

Ownership: IT and informatics, with escalation to quality and safety leadership as well as operations.

4. The Subgroup Outcome Signal

The subgroup outcome signal tracks how algorithm performance differs across patient populations — by race, ethnicity, age, sex, comorbidity burden, and combinations of those factors. It is the most important signal for health equity, but also the slowest to appear, because it depends on outcome data that may take weeks or months to accumulate.

This is also the signal most likely to reveal endogenous bias amplification. Aggregate performance measures will not capture that. Only subgroup-level stratification will.

Ownership: quality and safety, with escalation to the AI governance committee and executive leadership.

5. The Retraining and Data Feedback Signal

The retraining and data feedback signal monitors how model inputs, outputs, and training data evolve over the course of deployment. It requires model-specific instrumentation, including feature distribution tracking with rolling windows, label contamination monitoring, adherence-weighted performance estimates, and AUROC tracking against pre-deployment benchmarks.

This is the signal that most directly detects the Training Feedback Loop. An AUROC drop of more than 0.05 from baseline should trigger investigation. A 9% to 39% decline in specificity after retraining is a quantified warning sign of feedback loop contamination.

Ownership: data science and ML operations, with escalation to both the governance committee and the vendor.

What This Means for Healthcare Institutions

If your institution has already deployed clinical AI, the real question is not whether the feedback loops described in Issue 03 exist.

The real question is whether you can see them.

Health executives and governance leaders should be asking:

1. Do we track override rates over time by clinician and alert type rather than only in aggregate?

2. Do we know if our retraining data has already been contaminated by prior model use?

3. Do we have a way to measure clinician performance without AI assistance?

4. Do we continuously monitor subgroup performance rather than only checking it at deployment?

5. Do we know which signals appear first and who is responsible for escalation when thresholds are crossed?

If the answer is no, then feedback loops may already be active in your institution without any meaningful surveillance signal in place.

That is not an individual failure. It is a field-wide blind spot that can no longer be defended as being invisible.

What Comes Next

Issue 03 introduced the mechanism—three feedback loops and the epistemological challenge they pose.

Issue 04 has now laid out the governance implications, the infrastructure requirements, and the five signal types that any real surveillance system must be able to distinguish.

Issue 05 will introduce the six-domain surveillance architecture, the operational framework that enables surveillance, detection, and mitigation at an institutional scale. These domains are not abstract ideas. They are the practical answer to the central question this issue raises: if feedback loops are inevitable, what does a health system actually need to build in order to see them?

Drift cannot be prevented. But it can be detected. And detection is where the discipline begins.

Subscribe to get each new issue of The Epidemiology of Algorithms delivered to your email inbox.

Subscribe now

“Trust in algorithmic systems should be continuously earned through rigorous, population-level surveillance rather than historically inherited from initial validation or deployment approval.” — Anne E. Burnley, MD, MHS, MS

In Issue 02, I introduced the Algorithm Exposure Model: a causal chain for understanding how clinical AI produces effects not one patient at a time, but across entire populations simultaneously.

That model was deliberately linear.

It began with the algorithm as agent, moved through algorithm output, clinician interpretation, clinical decision, patient, and patient outcome, and showed how the clinical environment modifies every step.

That structure was necessary.

It clarified the architecture of exposure.

But it was also incomplete.

A causal chain is useful because it imposes order. Inputs move toward outputs. Causes precede effects. The model acts, and the world responds.

Deployed clinical AI does not remain that kind of system for long.

Once it enters a real clinical environment, it does not simply generate outputs and wait passively for the world to react. Clinicians adapt to it. Institutions reorganize around it. And in some systems, decisions made in response to the model become part of the data used later to evaluate or retrain the next version.

The output begins to shape the input. The effect begins to modify the cause. That is what Figure 2 adds.

Figure 1 showed us the architecture of exposure. Figure 2 adds motion.

From causal chain to living system

The six nodes of the Algorithm Exposure Model are unchanged. The clinical environment continues to function as an effect modifier across all nodes and transitions.

What is new are three recursive arcs: feedback loops that connect downstream effects back to upstream system behavior.

Those loops turn a linear causal model into a living system.

And that distinction matters, because the problem is not only that clinical AI has downstream effects.

It is those effects that can return to the system, reshape its future behavior, and make the resulting harms harder to detect.

Figure 2. The Algorithm Exposure Model with Feedback Loops. Three recursive arcs connect downstream effects back to upstream system behavior, turning the linear causal chain from Figure 1 into a living system. The clinical environment functions as an effect modifier across all nodes and transitions.

The three loops are:

The Override/Adaptation Loop — from decision host back to algorithm output

The Clinician Learning Loop — from clinical decision back to decision host

The Training Feedback Loop — from patient outcome back to algorithm agent

Each has a distinct mechanism. Each has a distinct failure mode. And each operates in ways that are difficult to see in a single patient encounter.

That is the point.

The Override/Adaptation Loop: where automation bias lives

The most recognizable loop is the shortest one.

The Override/Adaptation Loop sits between the clinician node and the algorithm output node. Every time a clinician receives an alert, recommendation, score, or classification, they must decide whether to accept, modify, or override it.

That decision is never purely technical.

It is shaped by trust, fatigue, workload, recent experience with the system, alert burden, and the cognitive cost of checking whether the recommendation is correct.

Over time, what matters is not any single decision. It is the pattern of those decisions across hundreds or thousands of encounters.

This is where automation bias lives.

The evidence base here is already substantial. Across specialties, AI recommendations can influence clinical decision-making powerfully — sometimes appropriately, sometimes not. Experience may moderate that effect, but it does not eliminate it. Even highly experienced clinicians remain vulnerable when an output appears plausible enough, and the burden of verification is high.

Alert volume compounds the problem.

As exposure increases, adherence patterns degrade. What begins as decision support becomes background noise. The system continues to generate outputs. The clinician continues to adapt. And the aggregate pattern that emerges becomes the institution’s operational reality, regardless of what the original validation study claimed.

This loop often generates one of the earliest detectable signals in a deployed system. Changes in override rates, response rates, and alert fatigue can appear within days to weeks of deployment — long before a health system has enough outcome data to say anything definitive about patient harm.

That makes this loop the first place a surveillance architecture should look.

The Clinician Learning Loop: trust miscalibration over time

The second loop is slower and less visible.

The Clinician Learning Loop runs from the clinical decision node back to the clinician node. It reflects the fact that clinicians do not merely use AI systems.

They learn from them.

They update their internal model of the system’s reliability, when to trust it, and when to ignore it.

Sometimes that learning is appropriate. Sometimes it is not.

The problem is not simply overtrust or undertrust in the abstract. Repeated exposure to algorithmic output can reshape clinical reasoning itself. Trust can become miscalibrated either gradually over months of use or rapidly after brief exposure to outputs that appear consistently correct.

That miscalibration has a name in the literature — actually, three names: Deskilling, never-skilling, and mis-skilling.

Deskilling is the loss of previously acquired clinical ability through disuse.

Never-skilling is the failure to develop competency because the system scaffolds the task before the clinician can internalize it.

Mis-skilling is the reinforcement of incorrect behavior when algorithmic error becomes part of the clinician’s own reasoning pattern.

All three reduce the clinician’s ability to detect error independently, and that matters because, in the Algorithm Exposure Model, the clinician is not a passive recipient of output. The clinician is the decision host.

If the decision host is being reshaped by the system, then the system’s future behavior is being altered indirectly through the human-in-the-loop, meant to regulate it.

The clinician learning loop is also the least well-instrumented. Most health systems do not measure trust calibration in real time. They do not track independent baseline performance in AI-off conditions. They also do not routinely test agreement rates against challenge cases designed to reveal overreliance.

Without those measures, trust miscalibration remains not only unmanaged but largely unseen.

The Training Feedback Loop: the self-fulfilling system

The longest and most concerning loop runs from the patient outcome node back to the algorithm agent node itself. This is concerning because this is the loop in which clinical AI begins to learn from its own effects.

If algorithmic predictions influence clinical decisions, and those decisions influence outcomes, and those outcomes later enter the data used for evaluation or retraining, then the model is no longer learning from an independent world. It is learning from a world it has already helped shape.

That is a fundamentally different problem from simple model degradation over time. It is a self-fulfilling system.

Recent literature has begun to name this directly. In ICU prognostic models, researchers have described model-mediated intervention as a mechanism that changes the relationship between predictors and outcomes after deployment. In resuscitation science, self-fulfilling prophecies arise when treatment decisions are not adequately represented in training, when human-machine interaction compounds those decisions, and when historically sensible clinical choices become structurally misleading as medicine advances.

Simulation work has made the stakes concrete. Models retrained after deployment can lose substantial specificity precisely because the data no longer mean what they meant before deployment. A model that appears to be improving by incorporating more recent data may be introducing distortions from its prior use. In one documented clinical case, a retrained model performed worse despite having six times as much training data because AI-influenced labels had contaminated the training set.

This loop is not only hard to detect, but it also changes the meaning of the evidence used to detect it.

More data may not correct the model, in some cases, more data makes it worse.

The epistemology of drift

This is where the argument moves from mechanism to interpretation.

Standard drift literature often considers performance degradation as external to the model. The patient population evolves. Disease prevalence varies. Clinical protocols are updated. The environment changes. The model must adapt accordingly.

That framing is not only wrong but also incomplete.

The feedback loops in Figure 2 introduce a different category: endogenous drift.

This is not drift happening to the model from outside; it is drift the model helps create by its own deployment.

Endogenous drift emerges when algorithmic output alters clinician behavior, workflow patterns, data provenance, and the relationship between predictors and outcomes.

That distinction matters because it changes what can be known about error:

· A model that improves outcomes may appear to degrade because it has changed the world it was trained to predict.

· A model that appears to perform well may do so only because its outputs have shaped the outcomes now being counted as ground truth.

· A system may seem stable in the aggregate while subgroup harm quietly intensifies beneath the surface.

The loops do not merely produce errors; they also obscure them:

· A self-fulfilling prophecy can look like a successful prediction.

· Alert fatigue can look like clinician noncompliance.

· Trust miscalibration can look like individual variation.

· Retraining contamination can look like model adaptation.

That is the epistemic problem.

The system can alter the conditions under which it is observed, and in doing so, make its own failures harder to see.

The loops are not independent

These feedback loops do not operate independently.

The Override/Adaptation Loop can alter the data-generating process for the Training Feedback Loop. The Clinician Learning Loop can reduce the clinician’s ability to detect errors, allowing flawed outputs to pass uncorrected into future data. Behavioral shifts become retraining distortions. Retraining distortions make outputs less reliable. Less reliable outputs further degrade trust calibration.

This is how harm compounds.

Not usually as a single dramatic failure, but as a gradually tightening cycle.

A system that generates automation bias produces override patterns that contaminate training data; likewise, contaminated training data creates a model with more systematic error. More systematic error deepens trust miscalibration. More miscalibration reduces error detection. Reduced error detection allows the next round of labels and outcomes to carry the distortion forward.

The loops tighten around one another.

That is why this is not a series of isolated problems. It is a system problem.

At the bedside, none of these patterns are obvious; however, as they aggregate at the population level over time, they become detectable.

The reason why incident review cannot solve this problem

Individual incident reviews cannot solve this problem because they were never designed to do so.

Incident review works best when harm is discrete, visible, and attributable to a bounded event. Feedback-loop harms are different. They unfold gradually, distribute across populations, and can alter the evidentiary frame through which later events are interpreted.

In other words, the problem is not simply that we are missing isolated failures. We are using methods built for isolated failures to monitor recursive systems. That mismatch guarantees blind spots.

A surveillance architecture for deployed clinical AI must be able to detect early behavioral adaptation, monitor evolving trust calibration, and interrogate whether retraining data remain epistemically meaningful after deployment.

Without that infrastructure, health systems may continue to rely on approval, validation, or aggregate performance summaries long after those measures no longer reflect the true state of the system.

The governance problem after deployment

Clinical AI should not be understood as a tool acting on a static environment.

Once deployed, it enters into relationships with clinicians, institutions, workflows, and data systems that change in response to it.

Those responses loop back.

They affect future decisions, future measurements, and future models.

That is why trust in algorithmic systems cannot be permanently inherited from initial validation or deployment approval. It has to be continuously earned through rigorous, population-level surveillance capable of seeing the recursive dynamics that ordinary monitoring misses.

The key question is no longer just whether a model was valid when deployed. It is whether the system it evolves into after deployment remains understandable enough to govern safely.

“Trust in algorithmic systems should be continuously earned through rigorous, population-level surveillance rather than historically inherited from initial validation or deployment approval.”

What comes next:

Issue 03 identifies the problem. In issue 04, we will examine what it would take to monitor these loops in practice.

“Six Blind Men and an Elephant” is an ancient Indian fable, documented in the Tittha Sutta around 500 BC.

Six blind men encounter an elephant.

Each touches a different part and declares a different truth: A tree, a fan, a wall, a snake, a rope, a spear.

Each is absolutely certain. None is wholly wrong. But none can see the whole.

I thought about that fable one afternoon in our occupational health clinic.

I was reviewing audiograms for a workforce I had been watching for years. Halfway through that stack, I noticed something that shouldn’t have been there.

One third of the group had significant threshold shifts. Not one worker. Not a cluster. A third — distributed quietly across a population I happened to be watching in its entirety. No one had complained. The shift had aggregated in silence.

A worksite visit confirmed what I suspected. We contained the harm because we caught it early. Without that intervention, those workers would have presented years later with permanent hearing loss, entirely preventable.

What it required: One provider. One population. Complete visibility.

Remove any one of those three conditions, and the shift continues. Distribute that workforce across three clinics, assign them to five different providers, and the signal disappears into the noise of individual encounters. Each provider sees their fraction. Nobody sees the elephant.

“Clinical AI does not act on one patient. It acts on every patient to produce population-level effects.” — Anne E. Burnley, MD

Clinical AI is deployed across thousands of providers and hundreds of institutions. Every clinician sees their piece, their patient, their alert, their encounter. Nobody is standing back, watching the whole animal.

That is the problem this discipline exists to solve.

THE PROBLEM

When the Epic Sepsis Model was deployed at hundreds of hospitals beginning in 2018, no systematic surveillance infrastructure existed to monitor its performance. The model was proprietary. External validation was not required before deployment.

It took a research team at Michigan Medicine, working retrospectively, on their own data, with limited vendor cooperation, to find out what the model was actually doing. What they found: the algorithm missed two-thirds of sepsis patients while firing alerts on nearly one in five hospitalizations. Epic had reported performance that would correctly distinguish sepsis patients from non-sepsis patients roughly 8 times out of 10. The real number was barely better than flipping a coin.

The model ran unwatched.

Not because clinicians were careless. But because no architecture existed for watching. No surveillance infrastructure. No signal detection system. No adverse event taxonomy. No unique identifier to track the model across sites or versions.

That is not a technology problem. That is an epidemiological problem that has a solution.

THE ALGORITHM EXPOSURE MODEL

The framework begins with a reframe.

Clinical AI is not a software tool deployed in individual clinical encounters. It is a population-level intervention — operating simultaneously across thousands of providers and hundreds of institutions, producing effects that aggregate into outcomes no single clinician can see.

The Algorithm Exposure Model extends the classical epidemiological triad — agent, host, environment — to clinical AI.

The algorithm is the agent. Its outputs influence the clinician, the decision host, whose clinical decisions then affect the patient. Individual outcomes aggregate into population-level effects. The clinical environment modifies every step: staffing ratios, alert burden, workflow design, and institutional culture around AI trust.

The same algorithm, deployed in different environments, produces different outcomes. Not because the model changes. Because the environment changes its effect.

Algorithms are not inherently biased. But the data they are trained on may carry biases, and algorithms can amplify those biases across populations.

The Algorithm Exposure Model traces a causal pathway from algorithm to patient, six nodes, one chain, one environment that modifies everything. The algorithm generates an output. A clinician interprets it. A clinical decision follows. A patient is affected. Individual outcomes aggregate into population-level effects that no single provider can see at the bedside.

Figure 1. The Algorithm Exposure Model: Linear Causal Chain. The algorithm acts as the agent, generating outputs that influence the decision host (clinician), whose clinical decisions affect the patient host. The clinical environment: Staffing, alert burden, workflow, and institutional culture- acts as an effect modifier at every node. Individual patient outcomes aggregate across settings to produce population-level signals.

WHY CLINICIANS MUST LEAD

The clinician who sees the patient is the only person who can recognize when an algorithm’s recommendation doesn’t fit the clinical picture. The clinician who reviews a hundred alerts in a shift is the only person who can report when the signal-to-noise ratio has become unworkable. The clinician who knows their patient population is the only person who can notice when a subgroup is being systematically missed.

If clinicians do not take a prominent role in AI surveillance and governance, we will have ceded the system’s most important feedback loop.

Our patients will pay the price.

AN OPEN INVITATION

This framework is my best current thinking. It is not finished.

A surveillance architecture for clinical AI, one that actually works at scale, cannot be designed by one physician working alone.

It needs the cardiologist who has watched a risk algorithm miss her patients. The informaticist who knows where the data lives. The patient safety officer has been trying to report an algorithm-associated event, but has nowhere to send it. The health equity researcher who knows which populations are being missed. The nurse who has watched alert fatigue erode her colleagues’ responsiveness.

Post-deployment surveillance of clinical AI is not optional. It is a requirement.

But Figure 1 is not the whole picture. It shows how algorithm-associated events travel. It does not show how they compound.

In Issue 03 - The System Talks Back: Feedback Loops and the Epistemology of Drift, the model becomes dynamic with feedback loops, propagation, and aggregation. The linear model in Figure 1 becomes a living system, changing everything we think about drift.

The chain you saw here talks back. The next issue shows you how.

“Trust in algorithmic systems should be continuously earned through rigorous, population-level surveillance rather than historically inherited from initial validation or deployment approval.” — Anne E. Burnley, MD

In healthcare, particularly, trust must be:

• Measured through subgroup performance
• Maintained through transparent evaluation
• Monitored as models drift
• Re-earned every time a system is updated, retrained, or deployed in a new context

This is what The Epidemiology of Algorithms exists to build.

Next issue, April 16, 2026: The System Talks Back: Feedback Loops and the Epistemology of Drift

The Epidemiology of Algorithms
Anne E. Burnley, MD, MHS, MS
newsletter.epidemiologyofalgorithms.org

Thanks for reading The Epidemiology of Algorithms!
Subscribe to receive new posts and support my work.

Subscribe now

WHAT DO AI SYSTEMS AND INFLUENZA VIRUSES HAVE IN COMMON?

They both shift and drift, and we have built a global surveillance system for only one of them.

THE EXPOSURE

I asked my new Occupational Health Resident this question as a “joke” in the clinic a few weeks ago. “What do AI systems and influenza viruses have in common?” I had just taken a course on “AI in Healthcare,” and he had spent a year studying epidemiology for an MPH. I could see the wheels turning in his head, possibly wondering what he was missing or if it was a trick question, after all, we had just met. After a minute or so, he said, “I have no clue. What do they have in common?” Excitedly, I said, “They both shift and drift.” We both went quiet, the specific kind of quiet that happens when something meant as a joke turns out to be true.

Every clinician reading this has encountered a clinical decision support tool. An early warning score. A sepsis alert. A diagnostic algorithm. You’ve learned, over time, how much trust to put in each. You’ve calibrated your response. You’ve built a mental model of when it’s right and when it goes off randomly.

Here is what you were almost certainly never told: The mental model you so carefully curated over the years may no longer be valid.

The algorithm you use today may not be the algorithm you were trained on. It may have been silently updated. Its underlying model may have drifted as the population it was trained on diverged from the population you’re now treating. No alarm fired. No notification was sent. The interface looks identical. But something changed, and your calibration, built on the old version, is now obsolete.

THE SIGNAL

Antigenic shift and drift: A borrowed framework that just fits.

In virology, antigenic drift is the slow, incremental change in the surface proteins of viruses, especially influenza viruses, over time. These changes due to mutations are small, and the immune system, trained on a previous version of the virus, still recognizes it, but less effectively. Protection erodes gradually, invisibly, until it fails when it matters.

Antigenic shift is different. It is a sudden, discontinuous change — a reassortment of genetic material that produces a novel viral strain the immune system has never encountered. No prior immunity. No warning. The 1918 influenza pandemic and the 2009 H1N1 pandemic were significant events. In contrast, the 2009 COVID-19 pandemic was caused by a new virus, SARS-CoV-2, which was new to human populations.

The algorithmic parallels:

Algorithmic drift, or model drift, is the gradual decline in an algorithm’s performance over time. It happens when the data used to train the model diverges from the data it encounters in the real world, such as changes in the patient population or clinical practices.

The model still runs normally, and no error messages appear. Because the system looks the same, clinicians continue to use and trust it at the same rate as before.

However, as the gap between the training data and current conditions grows, the model’s predictions may become less accurate. Performance slowly erodes without immediate notice.

An algorithmic shift is a sudden, major change in how an algorithm behaves. It can happen when a system receives a major version update, a new vendor system is adopted, or a retrained model is deployed overnight.

Unlike gradual changes, an algorithmic shift is discontinuous. The system’s behavior can change immediately and significantly, leaving clinicians and users without time to adapt their expectations or update their clinical intuition.

Because these changes can occur without clear notice, there may be little “memory” of how the previous system behaved and no routine surveillance to detect what changed. As a result, decisions may be influenced by a tool that now behaves differently from the one users learned to trust.

The analogy is not merely poetic. It is structurally precise. In both cases, the change is invisible at the individual level; the harm is distributed across a population. Early detection requires systematic surveillance, and no single clinician can see the patterns within their own practice.

Algorithmic drift in clinical settings can only be detected by watching a population, which is, by definition, epidemiology or the study of disease distribution, causes, and control in populations.

THE EVIDENCE

We built a global surveillance system for influenza, but have almost nothing for algorithms in healthcare settings.

The World Health Organization’s Global Influenza Surveillance and Response System coordinates sentinel sites across 114 countries. It sequences circulating strains continuously. It detects drift in real time. It produces annual vaccine composition recommendations based on that surveillance. The entire architecture exists because we recognized after 1918 that a pathogen that shifts and drifts requires ongoing population-level monitoring, not just point-of-care response.

Now consider what exists for clinical AI. Most health systems have no formal process for detecting performance degradation in deployed algorithms. Vendors are not required to report model updates to clinical users. There is no equivalent of strain sequencing, no systematic comparison of how an algorithm behaves now versus six months ago. When a sepsis model starts missing more cases than before, no alarm goes off. The signal is buried in individual clinical outcomes, invisible without aggregation.

The gap in numbers:

The FDA has cleared over 700 AI/ML-based medical devices. Post-market surveillance requirements for algorithmic performance degradation are minimal. There is no mandatory registry of algorithm adverse events. There is no equivalent of MedWatch for clinical AI. We are literally flying with instruments we do not calibrate.

This is not an argument against clinical AI, just as influenza surveillance is not an argument against the influenza vaccine. It is the science that makes the vaccine work, because without surveillance, you cannot know what you are vaccinating against.

The Epidemiology of Algorithms is not a critique of AI in medicine or healthcare in general. It is the science that makes AI in medicine safe.

OPEN QUESTION

What would algorithmic surveillance actually look like?

If we take the influenza analogy seriously, the architecture almost writes itself: sentinel sites — hospitals that systematically monitor algorithm performance against outcomes. Strain sequencing: Version-control systems that track changes between model iterations. Signal detection: Statistical methods that distinguish random variation from true performance degradation. Governance: Who acts when a signal is detected, and how fast.

None of this exists at scale for algorithms. Building it is the work of this discipline. In the next issue, I will introduce the framework I have been developing, the five-component architecture for population-level algorithmic surveillance, and why the classical epidemiological triad of agent, host, and environment maps more precisely onto clinical AI than anything currently proposed in health informatics.

For now, I leave you with the question I want you to carry into your next clinical shift:

“Do you know if the algorithm you used today is the same algorithm you used six months ago?”

If you cannot answer that question with certainty, you are practicing in a surveillance gap. That gap now has a name, and closing it is why this newsletter exists.

The Epidemiology of Algorithms
Anne E. Burnley, MD, MHS, MS
Founder & Writer