Preserving Clinical Reasoning in the Age of AI: What Medical Training Needs to Measure 

Resuable components schematic

Preserving Clinical Reasoning in the Age of AI: What Medical Training Needs to Measure
Deskilling occurs when a clinician’s ability to perform a task independently declines after relying on technology to assist with it. For example, a clinician who routinely accepts an AI-generated differential diagnosis may become less practiced at developing one unaided. Never-skilling concerns trainees who begin using that assistance before they have developed the underlying clinical reasoning skills.

Never-skilling describes a failure to develop clinical reasoning skills. Evidence measuring it directly remains limited, and many proposed remedies have yet to be evaluated. Trials of several countermeasures have measured single-session accuracy in already-trained clinicians, leaving their effects on skill acquisition unresolved. Addressing never-skilling will require acquisition endpoints, unaided testing, and an instrument that could be built from data radiology already collects.

Deskilling occurs when a clinician’s ability to perform a task independently declines after relying on technology for assistance. For example, a clinician who routinely accepts an AI-generated differential diagnosis may become less practiced at developing one unaided. Never-skilling concerns trainees who begin using that assistance before they have developed the underlying clinical reasoning skills.

Researchers have begun to quantify deskilling. At four Polish endoscopy centers, investigators compared unassisted colonoscopy during the three months before and after AI detection tools arrived. Across 1,443 colonoscopies, the adenoma detection rate fell from 28.4% to 22.4%, a difference of 6.0 percentage points (95% CI −10.5 to −1.6, p=0.0089).1 Because the study was retrospective and observational, it cannot establish causality. It nevertheless documents a change in practicing clinicians’ performance on a quality measure that maps to missed cancer.

The available research provides less insight into never-skilling. The studies discussed here do not establish how trainees who complete an AI-rich training program perform when that assistance is removed. Researchers recognize this limitation, although the evidence behind proposed remedies also warrants closer examination. Many remain untested, and several relevant trials have addressed immediate performance. Evidence about how to prevent never-skilling is therefore limited, while existing credentials offer little assurance that it has not occurred.

Where the idea came from

Abdulnour, Gin and Boscardin trace the term to the Josiah Macy Jr. Foundation conference on AI and medical education, held 18 to 21 November 2024.2 Nine months later, in the New England Journal of Medicine, they defined three related problems: “deskilling” as the loss of previously acquired skills, “never-skilling” as the failure to develop essential competencies, and “mis-skilling” as the reinforcement of incorrect behavior due to AI errors or bias.3

In January 2026, Ong drew on aviation to propose minimum unaided practice, manual proficiency quotas, and unannounced “surprise breaks” from AI to test readiness. Ong also identified a prerequisite for assessing these approaches: “It is impossible to assess de-skilling without measurements.”4 In May, Ke and colleagues provided a detailed treatment of the problem, including a useful distinction between AI in answer-delivery mode and AI in learning mode.5 They describe their framework as a hypothesis, acknowledge the absence of causal evidence in trainees, and call for trials before adoption. The question is what those trials should measure.

What we do know

The founding review offers a finding that deserves closer attention. Reviewing human-AI combinations, Abdulnour and colleagues report that when AI outperformed clinicians, the combination performed worse than AI alone, and that “when clinicians outperformed AI, the combination of human and AI reasoning increased performance, which suggests that high baseline performance promotes effective and safe AI assistance.”

This finding describes the relative performance of clinicians and AI across studies; it does not establish an absolute threshold of clinical competence. It does suggest that the benefits of collaboration depend on what the clinician contributes. Recognizing when one’s judgment adds something the model lacks may therefore be an essential part of using AI effectively.

An objection to this concern follows from medicine’s history. Every generation of clinicians has delegated some work to technology: the stethoscope displaced palpation, imaging displaced much of the physical examination, and the calculator displaced mental arithmetic. Medicine improved through each transition, which raises the question of why delegating reasoning to AI should be different.

One difference concerns the visibility of failure. Earlier instruments could be broken, unavailable, or produce an image whose defects were apparent to the user. A language model delivers a fluent answer in the same form as the reasoning it has replaced, making its errors harder to recognize. The profession has also begun to specify the competence clinicians must retain. The ACR-SIIM practice parameter for imaging AI defines a qualified end-user as a licensed physician with the qualifications to independently provide the equivalent medical service without AI.6 Such a requirement was unnecessary for the stethoscope; its inclusion now suggests that independent performance can no longer be assumed.

What the trials actually measured

Proposed countermeasures include educational exercises and changes to system design. Educational approaches ask learners to practice without AI, complete matched tasks with and without assistance, identify seeded errors in adversarial cases, or commit to an answer before consulting the model. System-design approaches include withholding uncertain output, explaining the model’s reasoning, and training users to interrogate its answers.

Direct trial evidence for AI-free problem-based learning, matched task pairs, and seeded-error cases remains limited in the literature discussed here. These recommendations have a reasonable educational rationale, although their effectiveness requires further evaluation. Four relevant countermeasures have been tested, with outcomes that illustrate the limits of the available evidence.

Everett randomized 70 US-licensed physicians to receive AI as a first opinion or as a second opinion after the clinician had committed to an answer. Accuracy was 82% with commit-then-compare and 85% with AI-first, both above the 75% achieved with conventional resources.7 The study measured diagnostic accuracy on vignettes during a single session.

Buçinca found that cognitive forcing functions reduced overreliance on incorrect AI from 64% to 48% across 199 participants.8 The study also provides a concerning result for explanations as a remedy. When the model was wrong, participants shown the AI answer with an explanation found the correct answer 8% of the time, compared with 49% among those who received no AI. Forcing functions raised that figure to 27%, barely half the unassisted rate. These were lay participants completing a meal-nutrition task in one sitting.

In a vignette study of 259 clinicians across nine US states, inaccurate AI reduced treatment accuracy from 62% to 50%. Selective prediction, in which the model withholds output when it lacks confidence, restored the false positive rate to baseline, although the false negative rate remained elevated. Clinicians undertreated when told that the model had abstained.9 The study was published as a preprint and assessed credentialled clinicians in a single sitting.

Qazi gave 44 physicians 20 hours of training on model capabilities, prompting and critical appraisal, then exposed one arm to erroneous LLM output and the other to accurate output. The erroneous-output arm scored 14.0 adjusted percentage points lower.10 The study measured accuracy during one encounter.

These four trials assessed performance within a session, leaving longer-term competency acquisition unmeasured. Three involved clinicians who already possessed the skills under examination, making them deskilling designs; the fourth involved members of the public. Never-skilling concerns the development of reasoning during training, a process these single-session studies could not observe.

The available evidence therefore cannot establish whether the proposed remedies prevent never-skilling. Many remain untested, and the four examined here addressed immediate performance. Conclusions about skill acquisition have been drawn from experiments designed to answer a different question.

It is reasonable to ask why an intervention that fails to improve a trained clinician’s performance in one sitting would help a trainee develop competence over three years. For system-design remedies, this objection has force because selective prediction is intended to change reliance during the encounter. Jabbour’s result therefore represents a genuine partial failure. Educational interventions require a different assessment because their intended effects develop through practice. Commit-then-compare creates a record of the learner’s reasoning for subsequent comparison, and scoring the immediate diagnosis cannot establish whether that process improves learning. Everett’s null result concerns diagnostic performance and leaves skill formation unresolved.

The studies that did measure acquisition

Several studies have used a design suited to examining acquisition by testing learners after support was removed.

Bastani randomized roughly a thousand Turkish high school students to a control group, a plain ChatGPT interface, or a guardrailed tutor developed with teacher input and instructed to offer hints. During assisted practice, grades improved by 48% with the plain interface and 127% with the tutor. The AI was then removed, and all students completed an unaided exam. The plain-interface arm finished 17% below control; the tutor arm’s estimate was −0.004, with a standard error of 0.013.11

These results support a narrower conclusion than is often reported. Students using the guardrailed tutor performed at the same level as students who had never used AI, so the guardrails prevented the harm associated with the unguarded interface without producing additional learning. With the tool, task and school held constant, interaction design accounted for the difference between harm and no harm. The study is also widely described as showing that AI harmed weaker students most, although the authors report “limited to no statistically significant support for heterogeneous treatment effects with either treatment.”

A Montreal laboratory has examined AI coaching through randomized trials on the same neurosurgical simulator, and reading the studies together is more informative than considering either alone. Fazlollahi randomized 70 medical students to an AI tutor, remote expert instruction, or no feedback. The tutor exceeded the expert by 0.66 points on the automated expertise score.12 Yilmaz compared real-time AI instruction, in-person expert instruction and post hoc feedback across 97 analysed trainees. AI exceeded the expert on the practice composite by 0.332, with a 95% CI of 0.173 to 0.491.13 The blinded human ratings of the realistic task showed little of this advantage: Fazlollahi’s AI and expert arms scored 4.63 and 4.40, respectively, p = .78, and Yilmaz found no difference between arms, F(2,94) = 1.241, p = .294.

In both trials, the apparent benefit depended on the assessment instrument, illustrating how the choice of endpoint can determine the conclusion.

He ran a pre-registered factorial experiment with 336 medical novices reading chest CT scans for lung nodules, varying whether AI was present during training and diagnosis. Participants trained with AI and then tested without it scored 71% against 68%, a difference of 0.03, p < 0.05, d = 0.43. The confidence interval’s lower bound rounded to zero at two decimals.14 The effect was small, and the study remains a preprint.

The acquisition evidence discussed here comes from high school mathematics, neurosurgical simulation and chest CT reading. These studies offer useful findings, although their applicability to sustained clinical training remains uncertain.

What institutions can do while the evidence is built

Two programs have introduced practices intended to preserve independent performance, although published competency outcomes remain unavailable in the accounts discussed here. I recommend piloting both because there are plausible reasons to expect them to help, provided their effects are measured.

Abernethy and colleagues at Johns Hopkins ran a six-month AI scribe pilot in internal medicine. Of 48 residents offered the tool, 31 used it, generating nearly a thousand notes. The researchers published seven practices mapped to ACGME competencies, beginning with a requirement that learners demonstrate their ability to write clear, accurate notes independently before using a scribe. Their proposed benchmark of six months and 50 encounters draws on decades of collective experience, a basis they explicitly distinguish from empirical data. The program instituted a review of trainees’ self-generated notes before permitting scribe use.15 The published account does not establish its effect on competence, although assessing independent performance before introducing assistance is consistent with the founding review’s finding about baseline competence.

OHSU’s graduate medical education policy requires trainees to disclose AI use verbally, using wording such as “the differential I am presenting was generated by AI.”16 The requirement can be implemented at no cost and needs no vendor or accreditor. It allows a supervisor to recognize, during the encounter, when a trainee has delegated reasoning to AI. Its weakness is that disclosure can diminish within a month if trainees believe it makes them look lazy and attendings fail to ask. The low cost and plausible mechanism justify a pilot, although evidence of an effect on competence remains limited.

Both practices should be piloted with an acquisition endpoint. Institutions that adopt them without measuring independent performance will remain unable to determine whether they prevent never-skilling.

The case for AI fluency

My recommendation for AI fluency extends beyond what the evidence currently establishes and therefore depends on the reasoning that supports it.

Although AI fluency has limited evidence, I regard it as the most important available intervention because it can be taught and used without waiting for changes elsewhere in the system. Guardrailed design depends on a vendor, access gating on a program director, and selective withholding on procurement decisions. A clinician can apply AI fluency in a hospital with no AI governance function, and teaching it requires a place in the curriculum rather than a procurement cycle. The founding review’s finding about baseline performance strengthens this rationale: effective collaboration requires clinicians to recognize what they contribute, making fluency a prerequisite for the other remedies to work.

The evidence remains limited partly because evaluations rarely examine the outcomes that matter. Woods reviewed 27 AI training programs for healthcare workers against the Kirkpatrick-Barr hierarchy. Nineteen measured reactions, 21 assessed attitudes or knowledge, two reached behavior change, and none reached level four, which concerns organizational change and patient benefit.17 Thousands of clinicians have received training, while evidence about its effects on those outcomes remains limited.

Qazi’s study challenges the case for fluency because 20 hours of training did not prevent 44 physicians from following inaccurate AI. A single-session accuracy endpoint, however, cannot capture the full range of changes training might produce. Clerc gave 76 middle-school students a two-hour workshop on how language models function and fail, then compared them two days later with 40 students who had not attended. After an underspecified prompt, 59.2% of trained students asked a follow-up question, compared with 27.9% of untrained students, d = 0.80, while task scores rose by about one point, p = .040.18 This quasi-experimental preprint studied schoolchildren in one session with AI present, so it fails the acquisition standard applied elsewhere in this paper. Its relevance is limited to the difference between its behavioral and performance outcomes. The behavioral effect was several times the size of the score effect, meaning an evaluation confined to scores would have described a substantial change in how students used the tool as marginal.

The selective-prediction study offers the closest available clinical comparison discussed here. Clinicians with previous experience using a decision support tool lost 9 percentage points when exposed to inaccurate AI and recovered all of them once the model began abstaining. Those without previous experience lost 12 points and recovered 3. Previous exposure determined whether they recovered once the model abstained, despite failing to prevent the initial harm. Experience was recorded through a single self-reported yes-or-no question, the subgroups were unbalanced at 78 and 181 participants, and the analysis was exploratory. These limitations make the result a basis for further investigation.9
Anyone promoting AI fluency, myself included, should also consider Zhang’s findings. In a study of 288 K-12 teachers in Taiwan, correlations between self-rated AI ability and objective test results ranged from 0.07 to 0.24. The weakest relationship concerned the dimension closest to appraising an output.19 Applying this result to physicians requires an assumption, but there is no reason to expect better calibration among clinicians who have never been trained in the skill. A program that certifies fluency through a questionnaire measures confidence, leaving the purchasing organization without evidence of competence.

What trainees can do now

Institutional change will take time, while trainees are already using these systems extensively. In a national survey of 286 Canadian medical students, 96.5% used a language model, 90.2% had encountered inaccurate output, and 11% had received formal instruction.20 The following recommendations reflect that immediate need, with the basis for each made explicit.

Write your differential before you open the tool. Evidence that this preserves reasoning remains limited, and the adjacent study found no accuracy advantage.7 I recommend it because recording the differential before consultation allows you to identify which thoughts were your own. The proposed benefit follows from how the exercise works and requires experimental evaluation.

Verify one claim back to its source, every session. This recommendation has the strongest support of those listed here. Schoolchildren who received two hours of instruction questioned a weak prompt at roughly twice the rate of untrained students, including on tasks the instruction had not rehearsed.18 With nine in ten of your peers already having encountered an incorrect answer, checking claims is an immediate practical need.
Ask the model to argue against its own conclusion. Verification requires a standard against which to assess an answer, and treating the output as a provisional claim helps establish that habit.3

Do not trust your own sense of how good you are at this. Self-rated AI ability has little relationship with tested ability.19 Confidence is particularly unreliable when assessing an output, precisely where errors in judgment matter most.

Ong’s unannounced break from AI offers the strongest test described here, but programs will need to organize it. Asking trainees to administer the test themselves during a busy service requires them to choose a slower way of completing a task under time pressure. Almost nobody will volunteer, and those who do need the exercise least. Scheduling the break therefore requires funding for someone to cover the additional time.

What to measure

Ong identified unaided performance, tested when the learner is not expecting it, as the relevant assessment. Bastani used this design with an exam, and he used it with a diagnosis. Removing assistance is necessary to establish whether the learner acquired the competency being demonstrated.

The consequences extend to licensure, which assumes that competence can transfer across settings. Credentials awarded before ambient AI became available, or through assessments that did not explicitly control its use, offer limited insight into this question. The professional requirements discussed here do not establish a consistent approach to demonstrating unaided reasoning across licensing and certification bodies in the United States and Europe. Existing credentials may therefore leave uncertainty about whether demonstrated performance depends on AI assistance.

Professional bodies have begun to require independent capability without specifying how it should be measured. Germany’s Bundesärztekammer stated in January 2025 that undergraduate, specialty and continuing education should continue teaching sufficient AI-independent knowledge and skills. A month later, the European Society of Radiology recommended regular re-certification “with or without AI.”21,22 Neither specifies how unaided performance should be assessed.

Radiology already collects data from which a related instrument could be developed. The ACR-SIIM parameter that defines a qualified end-user through unaided competence also requires acceptance, rejection and overrides of AI outputs to be logged in standardized form.6 The profession has therefore specified a competence requirement and a source of performance data, although the parameter does not connect them.

Using those logs would require addressing their scope and interpretation. The parameter applies to clinical practices rather than residency programs, and an override rate alone cannot show whether a trainee is exercising sound judgment. A falling rate could mean that a resident has correctly learned when to trust the model, which would be a desirable outcome. Pairing the log with case-level ground truth about whether each override was warranted would allow calculation of an appropriate-reliance rate. That measure would assess disagreement in relation to the cases that required it. It could provide a continuous measure of whether trainees are still thinking, using data imaging departments already generate. Appropriate reliance should become a standard.

Accreditors should take the first step because programs depend on health systems to fund new requirements, while vendors need a reason to build the necessary tools. If the ACGME or its European equivalents made appropriate reliance a reportable milestone, that requirement would give both groups a reason to act.

The concern about never-skilling is credible, and the proposed frameworks provide a useful basis for investigation. Establishing whether the remedies work will require research that follows trainees through an AI-rich rotation and tests their independent performance after removing the AI. Until that evidence is available, the recommendations presented here, including my own, remain arguments awaiting direct evaluation.

Declaration of interests

I am employed by GE HealthCare as Global Head of AI Advocacy, and I lead HelloAI, an AI fluency program. GE HealthCare develops and sells clinical AI products, including imaging AI of the kind governed by the ACR-SIIM practice parameter discussed here, and provides customer education on their use. Readers should weigh both against the argument I make for AI fluency in particular, which is the section where my professional interest and my stated position align most closely. No funding was received for this work and no colleague or business function at GE HealthCare reviewed or approved it.

References

  1. Budzyń K, Romańczyk M, Kitala D, Kołodziej P, Bugajski M, Adami HO, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. Lancet Gastroenterol Hepatol 2025;10(10):896-903. doi:10.1016/S2468-1253(25)00133-5 ↩︎
  2. Burk-Rafel J. Proceedings and abstracts of the 2024 Artificial Intelligence and Medical Education Macy Conference, November 18, 2024. Atlanta: Josiah Macy Jr. Foundation, 2024. Cited as the source for “never-skilling” in reference 3, where the surname appears as “Rafel JB”. The published proceedings of the same conference, Acad Med 2025;100(9S Suppl 1):S4-S14, doi:10.1097/ACM.0000000000006099, describe the concern but do not use the term. ↩︎
  3.  Abdulnour RE, Gin B, Boscardin CK. Educational strategies for clinical supervision of artificial intelligence use. N Engl J Med 2025;393(8):786-797. doi:10.1056/NEJMra2503232 ↩︎
  4. Ong AY, Merle DA, Pollreisz A, Wagner SK, Sevgi M, Keane PA, Huemer R, Oehling J, Jäger M, Huemer J. Flight rules for clinical AI: lessons from aviation for human-AI collaboration in medicine. npj Digit Med 2026;9(1). doi:10.1038/s41746-026-02410-1 ↩︎
  5. Ke Y, Jin L, Ong JCL, Thirunavukarasu AJ, Car J, Cheung CY, Tham YC, Ting DSW, Ong MEH, Compton S, Narayan A, Keane PA, Wong TY, Bates DW, Tan P, Liu N. AI-induced never-skilling in medical education. Nat Med 2026;32(6):1997-2006. doi:10.1038/s41591-026-04438-y ↩︎
  6.  American College of Radiology, Society for Imaging Informatics in Medicine. ACR-SIIM Practice Parameter for Imaging Artificial Intelligence. Approved at the ACR 2026 annual meeting, Washington DC, 5 May 2026. ↩︎
  7. Everett SS, Bunning BJ, Jain P, Lopez I, Agarwal A, Desai M, Gallo R, Goh E, Kadiyala VB, Kanjee Z, Koshy JM, Olson A, Rodman A, Schulman K, Strong E, Chen JH, Horvitz E. From tool to teammate in a randomized controlled trial of clinician-AI collaborative workflows for diagnosis. npj Digit Med 2026;9(1). doi:10.1038/s41746-026-02545-1. NCT06911645 ↩︎
  8.  Buçinca Z, Malaya MB, Gajos KZ. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proc ACM Hum-Comput Interact 2021;5(CSCW1):Article 188. doi:10.1145/3449287 ↩︎
  9. Jabbour S, Fouhey D, Banovic N, Shepard SD, Kazerooni E, Sjoding MW, Wiens J. Selective prediction reduces the negative effects of automation bias overall but increases false negatives. arXiv:2508.07617v2. Preprint. ↩︎
  10. Qazi IA, Ali A, Khawaja AU, Akhtar MJ, Sheikh AZ, Alizai MH. Automation bias in large language model-assisted diagnostic reasoning among physicians trained in AI literacy: a randomized clinical trial. NEJM AI 2026. doi:10.1056/AIoa2501001. NCT06963957 ↩︎
  11. Bastani H, Bastani O, Sungu A, Ge H, Kabakcı Ö, Mariman R. Generative AI without guardrails can harm learning: evidence from high school mathematics. Proc Natl Acad Sci USA 2025;122(26):e2422633122. doi:10.1073/pnas.2422633122 ↩︎
  12. .  Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of artificial intelligence tutoring vs expert instruction on learning simulated surgical skills among medical students: a randomized clinical trial. JAMA Netw Open 2022;5(2):e2149008. ↩︎
  13. Yilmaz R, Bakhaidar M, Alsayegh A, Abou Hamdan N, Fazlollahi AM, Tee T, Langleben I, Winkler-Schwartz A, Laroche D, Santaguida C, Del Maestro RF. Real-time multifaceted artificial intelligence vs in-person instruction in teaching surgical technical skills: a randomized controlled trial. Sci Rep 2024;14:15130. ↩︎
  14. He VF, Li S, Puranam P, Lin F. Predictive AI can support human learning while preserving error diversity. arXiv:2502.16411v5. Preprint. ↩︎
  15.  Abernethy J, Shah A, Chen B, Reynolds S, Wright SM, O’Rourke P. Integrating AI scribes into medical education: guardrails for preserving clinical reasoning. J Gen Intern Med 2026;41(9):2598-2602. doi:10.1007/s11606-025-10149-w ↩︎
  16. Oregon Health & Science University School of Medicine. Policy GME 48: Artificial Intelligence (AI) Use in Graduate Medical Education. Version 2, effective 12 February 2026. ↩︎
  17. Woods L, Lyons K, Van Der Vegt A, Olsen Q, Huang W, Khor JS, Xu N, Sullivan C. Assessing the effectiveness of artificial intelligence education and training for healthcare workers: a systematic review. BMC Med Educ 2026;26:549. doi:10.1186/s12909-026-08969-3 ↩︎
  18. Clerc O, Abdelghani R, Desvaux C, Poisson E, Oudeyer PY, Sauzéon H. Teaching students to question the machine: an AI literacy intervention improves students’ regulation of LLM use in a science task. arXiv:2604.01955, 2026. Preprint. ↩︎
  19. Zhang S, Xiao R, Botelho AF, Liao G, Chiu TKF, Stamper J, Koedinger KR. How to assess AI literacy: misalignment between self-reported and objective-based measures. In: Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference. New York: ACM, 2026. doi:10.1145/3785022.3785088 ↩︎
  20. Barr AA, Rozman RC, Liu K, et al. Medical students’ use of large language models: a national survey. Int J Med Inform 2026;220:106589. doi:10.1016/j.ijmedinf.2026.106589 ↩︎
  21. Bundesärztekammer. Stellungnahme “Künstliche Intelligenz in der Medizin”. Vorstandsbeschluss 14 January 2025. Dtsch Arztebl 2025;122(4):A-238. doi:10.3238/arztebl.2025.Stellungnahme_KI_Medizin  ↩︎
  22. Kotter E, Akinci D’Antonoli T, Cuocolo R, Hierath M, Huisman M, Klontzas ME, Martí-Bonmatí L, May MS, Neri E, Nikolaou K, Pinto dos Santos D, Radzina M, Shelmerdine SC, Bellemo A, on behalf of the European Society of Radiology. Guiding AI in radiology: ESR’s recommendations for effective implementation of the European AI Act. Insights Imaging 2025;16:33. doi:10.1186/s13244-025-01905-x ↩︎

Share