Artificial Intelligence as a Tool for Feedback in Internal Medicine Residency Training

Feedback represents a fundamental form of interaction between educators and residents, with performance feedback being crucial for promoting effective learning. Artificial intelligence (AI) is increasingly being utilized in medical education, with applications ranging from assessing clinical skills to developing educational objectives and performance evaluation. This study aimed to describe and compare differences in resident perceptions of educator-generated, AI-enhanced feedback on the performance evaluations.

Available feedback from faculty during an internal medicine (IM) rotation was selected and sent to residents. Prior to distribution, the feedback was assigned to one of two groups: Educator or AI-enhanced, using ChatGPT4. For the AI-enhanced group, feedback was revised via ChatGPT 4 with the following prompt: "Rewrite the following trainee feedback to emphasize a growth mindset. Frame areas for improvement as opportunities for learning and development. Provide specific examples and actionable suggestions to help the trainee improve their performance. The tone should be encouraging and supportive. Avoid, such as "generalizations," "negative language," "judgmental language," "personal criticism," and "offense. " 

Feedback was sent to residents, who were blinded to its source, and they then completed an online survey on the fairness, constructiveness, and supportiveness of the feedback. The survey instrument was not formally validated. 

Of the 56 residents, 50 completed the survey, with 54% receiving educator feedback and 46% receiving AI-enhanced feedback; nearly half of the residents received feedback from their Nephrology rotation. Participants were distributed across PGY-1 (46%), PGY-2 (24%), and PGY-3 (30%) levels. Overall, residents rated feedback positively: fairness (M=4.2, SD=0.7), supportiveness (M=4.1, SD=1.0), constructiveness (M=3.8, SD=0.9), and perceived impact on future performance (M=3.7, SD=1.0). Across all four dimensions, AI feedback was rated numerically higher; however, none were statistically significant (perceived future performance p=0.176, constructiveness p=0.106, fairness p=0.356, supportiveness p=0.631). Absolute differences were small (a few-tenths of a Likert point) (Figure 1A). Subgroup analysis of Nephrology rotation participants revealed higher constructiveness ratings for AI feedback (4.1 vs. 3.3, p = 0.044, exploratory, unadjusted for multiplicity), with trends toward a greater perceived impact on future performance (4.3 vs. 3.2, p = 0.118) (Figure 1B). Thematic analysis revealed three domains: improvement plans, perceptions of helpful feedback, and suggestions for refinement. AI feedback was noted for its structure and actionability, while educator feedback was valued for its personalization. AI feedback prompted fewer unclear improvement directions (4.3% vs. 22.2%) and greater satisfaction (30.4% vs. 18.5%), though some participants perceived AI feedback as impersonal. 

AI-enhanced feedback in IM residency was perceived as well-structured, detailed, and actionable, particularly supporting clear improvement plans. While educator feedback was valued for its personal and individualized nature, AI feedback may serve as a complementary tool to enhance the clarity and constructiveness of trainee evaluations. The small sample, ordinal outcomes, and non-validated instrument limit precision and preclude definitive inference.