[The following notes were generated by Andrea Belovich, PhD.]

The opening seminar for the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?” was co-presented on January 7th, 2021 by Dr. Mark L. Jordan, Professor of Urology and Residency Program Director at the University of California, Irvine and Dr. Justin La, Chief Resident of Urology at the University of California, Irvine. In this presentation, entitled “USMLE Transition to Pass/Fail: Implications for Resident Candidate Assessment and Selection,” Dr.’s Jordan and La provided an overview of how the USMLE Step 1 exam’s transition to pass/fail may impact residency applications from both the Program Director and Resident perspectives.

Dr. Jordan began with an overview of the current role of the USMLE Step 1 as a screening tool for residency program applications and the impetus behind its transition to a pass/fail assessment. He emphasized that the initial purpose of the USMLE was to determine the minimal competency of physicians to enter practice, rather to gauge an applicant’s qualifications for a residency program. Despite this, the Step 1, Step 2 CK, and Step 3 components of the USMLE exam have all been assessed using a 3-digit numerical score, while the Step 2 CS exam is assessed on a pass/fail basis. In 2008, the Committee to Evaluate USMLE Program (CEUP) did consider converting the Step 1 exam to pass/fail, though, at the time, the 3-digit scoring system was upheld. However, several points of discussion emerged regarding the benefits and drawbacks of using a numerical scoring system, which Dr. Jordan then addressed.

The predominant argument in support of a 3-digit USMLE Step 1 score has been that the exam is standardized, and therefore offers a “level playing field” for all test-takers. This serves as a reliable source of “hard data” which residency program directors can use to screen and filter residency applications. In contrast, other parameters such as grades, class rank, letters of recommendation, etc. often lack uniformity between schools and are difficult to compare. Dr. Jordan also discussed the disadvantages of utilizing a numerical score for the USMLE Step 1, and placed emphasis on the developing consensus that holistic evaluation of resident candidates is no longer best-served by utilizing a high-stakes exam. Dr. Jordan also explained that the focus on numerical scores of the USMLE often results in “USMLE mania,” where achieving high scores becomes the predominant goal of preclinical medical education. This phenomenon contributes to increased student anxiety as well as the development of parallel curriculum, or “teaching to the test.” Furthermore, as Dr. Jordan addressed in detail towards the end of the seminar, the 3-digit score introduces an inherent bias against diverse applicants during recruitment and does not promote equitable distribution of residency positions.

Ultimately, the Invitational Conference on USMLE Scoring (InCUS) convened in March 2019 by the AAMC, AMA, ECFMG, FSMB and NBME concluded that the USMLE Step 1 no longer served its stakeholders and recommended a transition of the USMLE Step 1 to pass/fail after January 2022.

During the next segment of the presentation, Dr. La presented the perspective of current residents in context of the USMLE transition to pass/fail. While no data has yet been published on the impact of resident input in the application evaluation process, Dr. La hypothesized that current resident input contributes greatly to building a successful and collegial team and environment that facilitates learning and sound patient care. Regardless of COVID-19, the majority of residency applicants to a urology residency program reported interactions with current residents as the most important component of their interview day [1]. Current residents also are able to assess the “fitness” of applicants for residency as well as their overall “fit” amongst the cohort by using non-USMLE components of their application [2]. As mixed data exists regarding the ability of the USMLE Step 1 to predict resident success, Dr. La presented scoping literature reviews demonstrating that other selection criteria are heterogeneously studied, with publications differing even in definition and assessment of resident success [3, 4]. In light of these data, Dr. La concluded that there is no single factor predictive of success in residency.

During the final section of the seminar, Dr. Jordan discussed the challenges the USMLE Step 1 pass/fail transition poses for program directors and the implications for applicants, along with a wide range of studies to demonstrate the advantages and disadvantages of other evaluation metrics. With the loss of an objective evaluation tool for screening and evaluating residency applications, program directors face an increased burden in reviewing applications, raising some concern that applicants could be disadvantaged, as individual applications may receive a less in-depth review. Applicants are then left with uncertainty about how to distinguish themselves from their peers when applying for competitive specialties, particularly if they are apply from newer, international, or less well-known medical schools. There is also uncertainty over whether greater emphasis will now be given to the numerical score of the Step 2 CK exam.

Finally, the effects of the COVID-19 pandemic are also anticipated to be felt during the transition of the USMLE Step 1 to pass/fail, as many in-person tools for assessment of residency candidates (such as visiting rotations and in-person interviews) have not been feasible. Evaluation of qualities and factors such as professionalism, accountability, social responsibility, team performance, peer interactions, and technical skills cannot be effectively assessed by an electronic application. Prior to COVID-19, visiting/away clinical rotations were heavily relied upon to assess applicants, as they allow the most important determinants of future success to be observed. The COVID-19 pandemic therefore introduces greater uncertainty in terms of how candidates are not only assessed for residency, but also how they are able to gain consideration for interviews.

Dr. Jordan also polled the live audience to gauge which tools they would now primarily rely upon for resident selection, what positive changes they anticipated could result from Step 1 transitioning to pass/fail, and what greatest challenges they faced for residency selection during the pandemic. He then concluded the presentation with a summary of potential solutions to the challenges he had outlined during the seminar:

• Standardized Candidate Assessment Tools such as clinical skills assessments based on specialty, nontraditional assessments (e.g., Jefferson Empathy Scale, Grit Scale), predictors of self-control, wellness and conscientiousness, etc.
• Standardized MSPE, rotation evaluations, grades, letters of recommendation, and transcripts
• Increased medical school transparency on applicant strengths, professionalism, performance
• Program director-driven development of mission-based holistic criteria for application review
• ERAS modification to permit extraction of information for holistic review
• Early application cycle to demonstrate program interest

References:

1. Kenigsberg AP, Khouri Jr. RK, Kuprasertkul A, Wong D, Ganesan V, and Lemack GE. Urology Residency Applications in the COVID-19 Era. Urology. 2020
2. Nemani VM, Park C, and Nawabi DH. What makes a “great resident”: the resident perspective. Curr Rev Musculoskelet Med. 2014.
3. Yang A, Gilani C, Saadat S, Murphy L, Toohey S, and Boysen-Osborn M. Which Applicant Factors Predict Success in Emergency Medicine Training Programs? A Scoping Review. AEM Educ Train. 2020.
4. Bowe SN, Laury AM, and Gray ST. Associations between Otolaryngology Applicant Characteristics and Future Performance in Residency or Practice: A Systematic Review. 2017.