[The following notes were generated by Andrea Belovich, PhD.]

The second installment of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 14th, 2021 by Dr. Douglas Gould, Professor and Chair of the Department of Foundational Medical Studies at the Oakland University William Beaumont (OUWB) School of Medicine. In his webinar, entitled “Pass/Fail Step 1: Implications for a Foundational Sciences Department” Dr. Gould provided an overview of how the transition of the USMLE Step 1 scoring to pass/fail (P/F) may impact biomedical/foundational sciences departments across medical schools. Throughout his presentation, he incorporated audience polling questions, enabling participants to interact in real-time.

Dr. Gould began by introducing the changes in USMLE Step 1 scoring and a poll question gauging the audience’s familiarity with the topic. Based on the response, Dr. Gould provided an abridged review of the USMLE Step exam sequence and the recommendations made by the AAMC, AMA, ECFMG, FSMB, and NBME’s Invitational Conference on USMLE Scoring (InCUS) conference in March, 2019. As recommended by the InCUS, the transition of USMLE Step 1 to P/F scoring will be implemented beginning in January, 2022, with no numeric score reported. The content of USMLE Step 1 will remain unchanged, as it will continue to focus on foundational sciences including Anatomy, Biochemistry, Microbiology, Pharmacology, and Physiology. USMLE Step 2 CS will remain P/F, although it may continue to be suspended due to COVID-19, as it involves in-person interaction with Standardized Patients, and USMLE Step 2 CK and 3 will remain numerically scored.

Dr. Gould provided an overview of the anticipated impacts of the transition of the USMLE Step 1 to P/F scoring. He addressed how the loss of the numeric score eliminates a valuable tool for residency program directors to screen and filter applicants and suggested the possibility that the transition may disadvantage students from newer or less-well known medical schools. This could possibly amplify disparities for underrepresented minorities, women, and international students if they lack equitable access to the subjective components of residency applications. Later in the webinar, he would mention the likelihood for increased pressure on students to attend more prestigious medical schools and apply to greater number of residencies, both of which would contribute to increased student debt.

Next, he focused on what residency program directors may use to replace the Step 1 numeric scores in screening applications and how preclinical curricula may be generally impacted. Citing the 2018 Program Director survey results from the National Resident Matching Program, Dr. Gould identified the following top-seven criteria used to rank residency applications (in order of decreasing importance): Step 1 score, Letters of Reference, MSPE, Step 2 (CK) score, Personal Statement, Grades in required clerkships, and Failed USMLE attempts. Other factors emphasized included AOA membership (#14), ‘other’ life experience (#16), Step 2 (CS) score (#17), and finally, Involvement in Research (#26) [1]. Dr. Gould then mapped the results from a poll question administered during the previous week’s webinar asking webinar participants which factors they themselves would consider most likely to be relied upon to screen residency applications in the absence of a Step 1 numeric score. The audience results indicated the top three factors to be 1) Step 2 (CK) score, 2) Grades in required clerkships, and 3) Letters of Reference. Dr. Gould suggested that these IAMSE webinar audience response results underscore the variation and unpredictability of which factors may become most important going forward, emphasizing the need for the flexibility of medical schools as they adjust to the Step 1 P/F change. With these results in mind, Dr. Gould discussed the potential for preclinical, foundational sciences curricula to transition to P/F as well, and may experience reorientation towards preparing students for Step 2 CK, as this exam will still be assessed with a numeric score. In short, Dr. Gould anticipates further de-emphasis on basic science education going forward.

Importantly, an expose of the process of holistic review of the residency application dossier revealed opportunities for biomedical/foundational sciences to promote strong applications by providing students with research activities. Dr. Gould then presented the mission statement of his home institution, OUWB, and showcased their Department of Foundational Medical Studies (FMS), which encompasses approximately 40 faculty members, including those in basic science and medical education, medical humanities, community/public health, traditional (bench) researchers and medical librarians. As part of the department’s approach to research activities, he discussed their longitudinal, 4-year required research program (Embark). Students are paired with mentors and receive training across multiple domains of the research design and implementation process, including navigating IRB requirements, presentation skills, abstract writing, and paper publication. Dr. Gould noted the possibility of using the outcomes of student research activities as quantifiable metrics for residency application, which will favor the development of research programs.

Finally, Dr. Gould described the COVID-19 pandemic as a significant catalyst for the changes medical education has already been experiencing recently. This includes an increase in medical school and residency applications, which have not been accompanied by an increase in federally-funded residency programs. While not necessarily a novel trend, this does contribute to application inflation and makes holistic review more challenging. Other significant changes include the transition of USMLE Step 2 CS to a virtual format, along with a vast majority of medical education. COVID-19 has also forced a more rapid acceleration in the digitization of teaching materials across the literature. He highlighted factors that contributed to his FMS department’s adaptation to the COVID-19 pandemic, crediting the department’s collaborative nature as well as the presence of faculty already devoted to medical education in mounting a rapid and effective response to the unforeseen changes. A robust faculty development program also enabled a quick transition to online learning and asynchronous education, while promoting evaluation of effectiveness.

While the number of traditionally-structured foundational sciences departments has been decreasing over the last two decades, Dr. Gould suggested this may be due to department name changes and combination of disciplines into single departments. In fact, according to the AAMC, overall numbers of basic/foundational science departments have increased 33% over the last five years, and the number of foundational science faculty has increased from 14,047 in 1998 to 18601 in 2018, with 21% of medical school faculty holding a PhD as of 2018 [2]. As continued changes to medical education are inevitable, Dr. Gould strongly promoted the increase and development of dedicated foundational science medical education faculty, who are invested in maintaining and innovating effective education in the face of unforeseen events, such as the COVID-19 pandemic and the transition of the USMLE Step 1 to P/F.

References:

1. National Resident Matching Program. Program Director Survey Results. 2018.
2. Association of American Medical Colleges. 2018 US Medical School Faculty Report. https://www.aamc.org/data-reports/faculty-institutions/interactive-data/2018-us-medical-school-faculty

[The following notes were generated by Andrea Belovich, PhD.]

The opening seminar for the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?” was co-presented on January 7th, 2021 by Dr. Mark L. Jordan, Professor of Urology and Residency Program Director at the University of California, Irvine and Dr. Justin La, Chief Resident of Urology at the University of California, Irvine. In this presentation, entitled “USMLE Transition to Pass/Fail: Implications for Resident Candidate Assessment and Selection,” Dr.’s Jordan and La provided an overview of how the USMLE Step 1 exam’s transition to pass/fail may impact residency applications from both the Program Director and Resident perspectives.

Dr. Jordan began with an overview of the current role of the USMLE Step 1 as a screening tool for residency program applications and the impetus behind its transition to a pass/fail assessment. He emphasized that the initial purpose of the USMLE was to determine the minimal competency of physicians to enter practice, rather to gauge an applicant’s qualifications for a residency program. Despite this, the Step 1, Step 2 CK, and Step 3 components of the USMLE exam have all been assessed using a 3-digit numerical score, while the Step 2 CS exam is assessed on a pass/fail basis. In 2008, the Committee to Evaluate USMLE Program (CEUP) did consider converting the Step 1 exam to pass/fail, though, at the time, the 3-digit scoring system was upheld. However, several points of discussion emerged regarding the benefits and drawbacks of using a numerical scoring system, which Dr. Jordan then addressed.

The predominant argument in support of a 3-digit USMLE Step 1 score has been that the exam is standardized, and therefore offers a “level playing field” for all test-takers. This serves as a reliable source of “hard data” which residency program directors can use to screen and filter residency applications. In contrast, other parameters such as grades, class rank, letters of recommendation, etc. often lack uniformity between schools and are difficult to compare. Dr. Jordan also discussed the disadvantages of utilizing a numerical score for the USMLE Step 1, and placed emphasis on the developing consensus that holistic evaluation of resident candidates is no longer best-served by utilizing a high-stakes exam. Dr. Jordan also explained that the focus on numerical scores of the USMLE often results in “USMLE mania,” where achieving high scores becomes the predominant goal of preclinical medical education. This phenomenon contributes to increased student anxiety as well as the development of parallel curriculum, or “teaching to the test.” Furthermore, as Dr. Jordan addressed in detail towards the end of the seminar, the 3-digit score introduces an inherent bias against diverse applicants during recruitment and does not promote equitable distribution of residency positions.

Ultimately, the Invitational Conference on USMLE Scoring (InCUS) convened in March 2019 by the AAMC, AMA, ECFMG, FSMB and NBME concluded that the USMLE Step 1 no longer served its stakeholders and recommended a transition of the USMLE Step 1 to pass/fail after January 2022.

During the next segment of the presentation, Dr. La presented the perspective of current residents in context of the USMLE transition to pass/fail. While no data has yet been published on the impact of resident input in the application evaluation process, Dr. La hypothesized that current resident input contributes greatly to building a successful and collegial team and environment that facilitates learning and sound patient care. Regardless of COVID-19, the majority of residency applicants to a urology residency program reported interactions with current residents as the most important component of their interview day [1]. Current residents also are able to assess the “fitness” of applicants for residency as well as their overall “fit” amongst the cohort by using non-USMLE components of their application [2]. As mixed data exists regarding the ability of the USMLE Step 1 to predict resident success, Dr. La presented scoping literature reviews demonstrating that other selection criteria are heterogeneously studied, with publications differing even in definition and assessment of resident success [3, 4]. In light of these data, Dr. La concluded that there is no single factor predictive of success in residency.

During the final section of the seminar, Dr. Jordan discussed the challenges the USMLE Step 1 pass/fail transition poses for program directors and the implications for applicants, along with a wide range of studies to demonstrate the advantages and disadvantages of other evaluation metrics. With the loss of an objective evaluation tool for screening and evaluating residency applications, program directors face an increased burden in reviewing applications, raising some concern that applicants could be disadvantaged, as individual applications may receive a less in-depth review. Applicants are then left with uncertainty about how to distinguish themselves from their peers when applying for competitive specialties, particularly if they are apply from newer, international, or less well-known medical schools. There is also uncertainty over whether greater emphasis will now be given to the numerical score of the Step 2 CK exam.

Finally, the effects of the COVID-19 pandemic are also anticipated to be felt during the transition of the USMLE Step 1 to pass/fail, as many in-person tools for assessment of residency candidates (such as visiting rotations and in-person interviews) have not been feasible. Evaluation of qualities and factors such as professionalism, accountability, social responsibility, team performance, peer interactions, and technical skills cannot be effectively assessed by an electronic application. Prior to COVID-19, visiting/away clinical rotations were heavily relied upon to assess applicants, as they allow the most important determinants of future success to be observed. The COVID-19 pandemic therefore introduces greater uncertainty in terms of how candidates are not only assessed for residency, but also how they are able to gain consideration for interviews.

Dr. Jordan also polled the live audience to gauge which tools they would now primarily rely upon for resident selection, what positive changes they anticipated could result from Step 1 transitioning to pass/fail, and what greatest challenges they faced for residency selection during the pandemic. He then concluded the presentation with a summary of potential solutions to the challenges he had outlined during the seminar:

• Standardized Candidate Assessment Tools such as clinical skills assessments based on specialty, nontraditional assessments (e.g., Jefferson Empathy Scale, Grit Scale), predictors of self-control, wellness and conscientiousness, etc.
• Standardized MSPE, rotation evaluations, grades, letters of recommendation, and transcripts
• Increased medical school transparency on applicant strengths, professionalism, performance
• Program director-driven development of mission-based holistic criteria for application review
• ERAS modification to permit extraction of information for holistic review
• Early application cycle to demonstrate program interest

References:

1. Kenigsberg AP, Khouri Jr. RK, Kuprasertkul A, Wong D, Ganesan V, and Lemack GE. Urology Residency Applications in the COVID-19 Era. Urology. 2020
2. Nemani VM, Park C, and Nawabi DH. What makes a “great resident”: the resident perspective. Curr Rev Musculoskelet Med. 2014.
3. Yang A, Gilani C, Saadat S, Murphy L, Toohey S, and Boysen-Osborn M. Which Applicant Factors Predict Success in Emergency Medicine Training Programs? A Scoping Review. AEM Educ Train. 2020.
4. Bowe SN, Laury AM, and Gray ST. Associations between Otolaryngology Applicant Characteristics and Future Performance in Residency or Practice: A Systematic Review. 2017.