[The following notes were generated by Andrea Belovich, PhD.]

The fourth session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 28^th, 2021 by Drs. Bruce Morgenstern and Brenda Roman. Dr. Morgenstern is the Vice Dean for Academic and Clinical Affairs at the Roseman University College of Medicine, and the immediate past President of the Alliance for Clinical Affairs. Dr. Roman is the Associate Dean of Medical Education and a Professor of Psychiatry at Wright State University Boonshoft School of Medicine, and the current President of the Alliance for Clinical Education. In this webinar, “Step 1 Going Pass-Fail: Are We Just Kicking the Can Down the Road?,” the speakers discussed issues with the current residency application process that will remain to be solved after the USMLE Step 1 is scored pass/fail (P/F). The audience also engaged in a rich discussion about how Undergraduate Medical Education (UME) can help address these issues.

Dr. Morgenstern began the webinar with an overview of the original purpose of the USMLE Step series exams by quoting the USMLE’s mission statement, “To provide to licensing authorities meaningful information from assessments of physician characteristics including medical knowledge, skills, values, and attitudes that are important to the provision of safe and effective patient care” [1]. He contrasted this with the Step exams’ off-label use to distinguish residency applicants from one another and questioned how well exams consisting of multiple-choice questions can assess abstract qualities such as values and skills. Given these limitations and the recent discontinuation of the Step 2 Clinical Skills (CS) exam, Dr. Morgenstern raised the question of how the USMLE will continue to move forward in advancing its mission statement.

Dr. Morgenstern then pointed out that medical education, in general, faces a similar challenge. The ultimate goal of medical education is, of course, to produce “good” physicians. However, since the field lacks an operational definition of what a “good” physician is, medical education struggles to articulate how and when a candidate is determined to be one. Competency-based medicine is a trend evolving to meet this challenge, although Dr. Morgenstern added the caveat that the term “competency” itself may not be a sufficient descriptor due to the word stigma of “bare minimum” often associated with the term. Beyond clinical knowledge, the areas commonly used to measure competence are also difficult to assess, including clinical skills, professional identity development, professionalism, values and attitudes, health systems science-related curricular content, and growth mindset. As an example, while Dr. Morgenstern believes the ability to communicate in a culturally humble way is paramount, assessing this skill is far more difficult, especially when the focus on USMLE Step 1 has created a parallel curriculum for medical schools.

The driving factor behind this focus on USMLE Step 1 is the residency “match frenzy,” which has encouraged program directors find ways to screen and rank applicants. Citing a collection of Bryan Carmody’s research [2], Dr. Morgenstern acknowledged that Step 1 and Step 2 CK scores predict passage of standardized specialty boards, but in a more global context, Step 1 scores do not correlate with other measures of overall success in residency. Despite the data, perception is reality, so the practice of relying upon Step 1 scores to develop interviewee lists for residency slots has been generally accepted by both students and program directors, causing significant stress for students [3, 4, 5].

In addition to the overwhelming number of residency applications per residency slot, a significant contributor to the reliance on ranking tools is the lack of staffing available to residency program directors in support of the conductance of holistic application review. As program directors experience pressure to select applicants likely to pass specialty boards, they prefer methodologies to meaningfully rank applicants. Subjective metrics, such as Medical Student Performance Evaluation (MSPE) letters, tend to lack consistency regarding clinical grading and undergo “modifier inflation” in the description of medical students. Even the term “excellent” is no longer the highest superlative, since medical students as a group are generally very intelligent and capable individuals (reminiscent of the Lake Wobegon effect). Taken together, this leads to a general lack of trust between UME and GME regarding the MSPE.

Dr. Morgenstern concluded his portion of the webinar by positing that the current residency application process itself may not be fair. Even ERAS contributes greatly to the overwhelming number of applicants, and the National Residency Match Program algorithm is proprietary, so it is not possible to verify whether/how it favors applicants or programs, not to mention diversity, equity, and inclusion issues. Finally, he emphasized that the underlying issues with the current residency match process will remain after the USMLE Step 1 transitions to P/F, so, going forward, how can program directors screen applicants and conduct holistic reviews? Even after consideration for the Lake Wobegon effect, 50% of all applicants will perpetually be “below average.” To protect these students, Dr. Morgenstern emphasized that it is incumbent upon UME to develop tools that are more intentional in their design and purpose to assess “good” residents compared to the minimum competency licensure exams previously coopted for the match process.

Dr. Roman then continued the webinar by expanding upon problems that will remain inherent to the residency match process despite the USMLE Step 1 P/F transition. In particular, student stress is not addressed by this change, rather it will probably shift from Step 1 to Step 2 CK, as program directors are likely to focus more on Step 2 CK (a stronger predictor of clinical performance than Step 1 [6]). This is likely to impact the clerkship years and may detract from patient care and clinical learning, especially in light of the discontinuation of Step 2 CS. Medical schools and programs are still evaluated by match rate, and the job performance of Student Affairs Deans are tied to successful matches. Students with well-explained gaps in their education may be negatively viewed by program directors for requiring more time to complete medical school. Students and programs will still try to game the system. Despite these issues, almost all students are successful in matching to a residency program.

A brief overview of the history of the match process from its inception in 1920 shows that these underlying themes have plagued the transition from UME to GME for an entire century. Dr. Roman then asked the audience for their input on a series of open-ended and polling questions to help identify solutions to these long-standing challenges.

When asked, “What characteristics of a future ‘good’ clinician can the pre-clerkship educators identify about their students?” audience answers included “professionalism, teamwork, emotional intelligence, diligence, maturity, hard-working, and curious.” These responses underscored the collective observation that the system has become over-reliant on measuring knowledge by numbers.

Following up on this question, Dr. Roman asked, “What can the pre-clerkship educators document in a narrative format about students that would be helpful for consideration by program directors?” Audience answers included, “Professionalism, ability to think critically, curiosity, grit, and determination,” which described resident characteristics seen as desirable by program directors.

Dr. Roman then asked the audience whether educators should “agree to a common/standardized vocabulary in describing medical students in evaluations and letters of recommendation?” and “Should other measures be used in the application process regarding fit to the medical specialty, like ‘dexterity assessments” for skills-based specialties or communication assessments for ‘people-centered specialties’?” The audience responded in an overwhelmingly positive manner to both questions.

Finally, Dr. Roman polled the audience with the following question: “Since data regarding ‘predicting success in residency’ is not robust, a radical solution would be for students to identify a specialty of choice, geographic preference and a few key attributes about what they would like in a program (academic medical center versus community-based program, research opportunities) and do away with interviews…?” The audience was more evenly split in their response to this question.

As the webinar drew to a conclusion, Dr. Roman reminded the audience that medical schools can improve the residency application process and trust between UME and GME by developing better ways to identify and addressing professionalism issues. This may include involving basic science faculty who can detect concerning patterns, and greater distinction between professionalism concerns that can be remediated versus a long-standing pattern of behavior and documenting these.

Finally, Dr. Roman left with the audience with a provocative proposal intended to spark thought and discussion. With a disclaimer that the idea in its current form was not intended for serious consideration, Dr. Roman suggested the following:

1. Programs should define optimal fit for residents in their programs
2. Applicants define their characteristics
3. The match algorithm does not use rank lists, but rather one that prioritizes compatibility between applicant and program

Dr. Roman also suggested that medicine should also ask that schools and hospitals stop being ranked by media publications, as this creates a false reality and may drive students to applying to so many residencies. Ultimately, it is important for UME and GME to work together more to avoid simply moving the stress from Step 1 to Step 2 CK, and to involve clinicians and basic medical science faculty in helping program directors assess applicants.

References:

1. USMLE Mission Statement. United States Medical Licensing Examination. https://www.usmle.org/about/
2. https://thesherriffofsodium.com/2019/03/05/the-mythology-of-usmle-step-1-scores-and-board-certification
3. Beck Dallaghan. Medical School Resourcing of USMLE Step 1 Preparation: Questioning the Validity of Step 1. Med Sci Educ. (2019) 29:1141-1145
4. Bryan Carmody. On Residency Selection and the Quantitative Fallacy. J Grad Med Educ. (2019) 11 (4): 420-421.
5. Bryan Carmody. Medical Student Attitudes toward USMLE Step 1 and Health Systems Science – A Multi-Institutional Survey. Teaching and Learning in Medicine. December 8, 2020.
6. Akshita Sharma, Daniel P. Schauer, Matthew Kelleher, Benjamin Kinnear, Dana Sall, and Eric Warm. USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency. J Grad Med Educ. (2019) 11(4): 412-419.

[The following notes were generated by Andrea Belovich, PhD.]

The third session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 21st, 2021 by Dr. Jonathan Amiel, Interim Co-Vice Dean for Education, Senior Associate Dean for Curricular Affairs, and Associate Professor of Psychiatry at the Columbia University Vagelos College of Physicians and Surgeons. In this webinar, “USMLE Step 1 P/F: A UME Curriculum Dean’s Perspective,” Dr. Amiel discussed how the transition of the USMLE Step 1 to pass/fail (P/F) is anticipated to impact undergraduate medical education (UME) curriculum.

Dr. Amiel began with an overview of the USMLE Step exam licensure program, outlining the recommendations of the 2019 Invitational Conference on USMLE Scoring (InCUS) that culminated in the decision to change the USMLE Step 1 from a numerical scoring system to P/F after January 2022. He then discussed the role of a Curriculum Dean in UME, and how, from this perspective, the Step 1 P/F shift would likely impact UME curricula. Quoting Sir Isaac Newton, “And to every action there is always an equal and opposite or contrary reaction,” this portion of the webinar was framed as an exploration of both the changes that the Step 1 P/F shift is likely to introduce to UME, and the potential implications of UME’s responses to those changes.

He explained that a curriculum dean’s role is to coordinate between several departments to create a coherent and cohesive educational program. He listed major factors that could be impacted as UME adjusts to the Step 1 P/F transition: 1) Educational program mission, objectives, values, and ethics, 2) Accreditation, 3) Logistics, 4) Curriculum architecture and engineering, 5) Faculty development and 6) Resource management.

In terms of the mission, objective, values, and ethics of educational programs and accreditation, Dr. Amiel expressed optimism that pre-clerkship faculty may be able to take a more holistic approach to the knowledge and learning skills students acquire in pre-clerkship curricula. Rather than “teaching to the test,” institutions may utilize Step 1’s shift to P/F as an opportunity to work with the USMLE program and the NBME to approach Step 1 from a competency-based perspective. Dr. Amiel emphasized shifting the exam’s focus towards assessing what students need to know to be competent in clinical settings. Dr. Amiel stated that he did not anticipate a major impact in the accreditation arena, with the caveat that the NBME may decide to increase the passing threshold for Step 1. If implemented, this increased threshold could introduce a higher fail rate that may affect accreditation.

Whether or not the Step 1 passing threshold is raised, curriculum deans must consider how the P/F shift will affect the logistics of preparing students for standardized exams and how student support services may be impacted. Learners who have difficulty with standardized tests will still require support for licensure exams, and institutions need to ensure that they are still providing this support. This includes greater support for the USMLE Step 2 CK exam, which may now grow in importance for residency application ranking, as it will still be scored numerically. Additionally, a thoughtful approach to revising assessment methods is needed. For example, NBME subject exams are often currently used to prepare students for USMLE, but Dr. Amiel urged participants to consider other assessment methods to help prepare students for passing the Step 1 exam, including utilizing more open-ended questions and applied-knowledge assessments.

Next, Dr. Amiel discussed how the architecture of UME curricula may be impacted by the Step 1 P/F shift. He predicted that the most significant architectural impact would be on dedicated study time for the Step exams. Schools may need to dedicate more time to Step 2 CK preparation, which will affect the senior curriculum. While it may be tempting to reduce the dedicated study time for Step 1 to create space for Step 2 CK preparation, students who experience difficulty with standardized testing may be put at greater risk of failing Step 1. Elective rotations may also be reduced, but these are important for residency interviews and gaining clinical experience. Ultimately, these issues may result in individualized study time/tracks for students.

With the potential changes to assessment methods, Dr. Amiel also anticipated that faculty development programs will need to provide new training to faculty. He emphasized this as an opportunity to reframe “assessment of learning” as “assessment for learning,” and to help faculty broaden their thinking about how to test medical knowledge early in the curriculum. He then discussed the potential impact of the Step 1 P/F shift on resources and resource management. As institutions have learned during the COVID-19 pandemic era, resources need to be used wisely, and the process of making changes to curricula (especially in terms of architecture, dedicated study time, and timing of exams) is resource-intensive. Parallel curricula for different graduating classes may be required, which can result in confusing messaging for students. Changes in testing may also require more resources dedicated to testing support.

To summarize, Dr. Amiel outlined UMSE’s likely reactions to some of the changes that anticipated to result from the USMLE Step 1 shift to P/F. Firstly, students may be required to take Step 2 CK earlier in their curriculum than they do now in order to obtain scores needed for residency applications. This would then require UME curricula to prepare students for the Step 2 CK earlier as well. Dedicated study time for Step 1 may be decreased, but new dedicated time would likely be introduced for Step 2 CK preparation. Schools that administer the Step 1 exam later in students’ training (like Columbia University) will need to consider shifting both the Step 1 and Step 2 CK exams earlier. Specialties may also begin to develop and administer their own exams to sort and rank applicants. If this occurs, curriculum design would be further impacted in order to help students be prepared and competitive for their desired specialties.

Dr. Amiel suggested that shifting the USMLE Step 1 to P/F will be positive for medical education, but stakeholders must take care to mitigate any negative impacts that could result from the reactions to this shift. He stressed the importance of basing initiatives in education theory and best-practices to assess medical knowledge, promoting competency-based medical education. He also mentioned working with the NBME and USMLE program to establish optimal curricular times dedicated to the Step 1 exam as part of the gateway into the clinical phase of UME curricula. He suggested that students should be able to demonstrate their competency in a basic body of knowledge and some clinical reasoning skills before moving to clinic, and that, in its current form, the USMLE Step 1 itself may currently not be sufficient for this purpose. This raises the question of whether the USMLE Step 1 is even the optimal test for the transition from pre-clerkship training to the clinic. However, Dr. Amiel did stress that all parts of an assessment program should be designed to help further learner’s progress and learning, and should include experience with high-stakes assessments.

Finally, Dr. Amiel cautioned that any new methods of programmatic assessment should be mindful of holistic review and how generating data about learner performance may impact residency applications. In particular, institutions must be careful to employ anti-racist pedagogy and assessment practices to avoid increasing inequities and disadvantaging students who may not have had as many enriched educational experiences as others. Dr. Amiel concluded with the recognition that the USMLE Step 1 P/F shift will result in an abundance of scholarly activity opportunities.

[The following notes were generated by Andrea Belovich, PhD.]

The second installment of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 14th, 2021 by Dr. Douglas Gould, Professor and Chair of the Department of Foundational Medical Studies at the Oakland University William Beaumont (OUWB) School of Medicine. In his webinar, entitled “Pass/Fail Step 1: Implications for a Foundational Sciences Department” Dr. Gould provided an overview of how the transition of the USMLE Step 1 scoring to pass/fail (P/F) may impact biomedical/foundational sciences departments across medical schools. Throughout his presentation, he incorporated audience polling questions, enabling participants to interact in real-time.

Dr. Gould began by introducing the changes in USMLE Step 1 scoring and a poll question gauging the audience’s familiarity with the topic. Based on the response, Dr. Gould provided an abridged review of the USMLE Step exam sequence and the recommendations made by the AAMC, AMA, ECFMG, FSMB, and NBME’s Invitational Conference on USMLE Scoring (InCUS) conference in March, 2019. As recommended by the InCUS, the transition of USMLE Step 1 to P/F scoring will be implemented beginning in January, 2022, with no numeric score reported. The content of USMLE Step 1 will remain unchanged, as it will continue to focus on foundational sciences including Anatomy, Biochemistry, Microbiology, Pharmacology, and Physiology. USMLE Step 2 CS will remain P/F, although it may continue to be suspended due to COVID-19, as it involves in-person interaction with Standardized Patients, and USMLE Step 2 CK and 3 will remain numerically scored.

Dr. Gould provided an overview of the anticipated impacts of the transition of the USMLE Step 1 to P/F scoring. He addressed how the loss of the numeric score eliminates a valuable tool for residency program directors to screen and filter applicants and suggested the possibility that the transition may disadvantage students from newer or less-well known medical schools. This could possibly amplify disparities for underrepresented minorities, women, and international students if they lack equitable access to the subjective components of residency applications. Later in the webinar, he would mention the likelihood for increased pressure on students to attend more prestigious medical schools and apply to greater number of residencies, both of which would contribute to increased student debt.

Next, he focused on what residency program directors may use to replace the Step 1 numeric scores in screening applications and how preclinical curricula may be generally impacted. Citing the 2018 Program Director survey results from the National Resident Matching Program, Dr. Gould identified the following top-seven criteria used to rank residency applications (in order of decreasing importance): Step 1 score, Letters of Reference, MSPE, Step 2 (CK) score, Personal Statement, Grades in required clerkships, and Failed USMLE attempts. Other factors emphasized included AOA membership (#14), ‘other’ life experience (#16), Step 2 (CS) score (#17), and finally, Involvement in Research (#26) [1]. Dr. Gould then mapped the results from a poll question administered during the previous week’s webinar asking webinar participants which factors they themselves would consider most likely to be relied upon to screen residency applications in the absence of a Step 1 numeric score. The audience results indicated the top three factors to be 1) Step 2 (CK) score, 2) Grades in required clerkships, and 3) Letters of Reference. Dr. Gould suggested that these IAMSE webinar audience response results underscore the variation and unpredictability of which factors may become most important going forward, emphasizing the need for the flexibility of medical schools as they adjust to the Step 1 P/F change. With these results in mind, Dr. Gould discussed the potential for preclinical, foundational sciences curricula to transition to P/F as well, and may experience reorientation towards preparing students for Step 2 CK, as this exam will still be assessed with a numeric score. In short, Dr. Gould anticipates further de-emphasis on basic science education going forward.

Importantly, an expose of the process of holistic review of the residency application dossier revealed opportunities for biomedical/foundational sciences to promote strong applications by providing students with research activities. Dr. Gould then presented the mission statement of his home institution, OUWB, and showcased their Department of Foundational Medical Studies (FMS), which encompasses approximately 40 faculty members, including those in basic science and medical education, medical humanities, community/public health, traditional (bench) researchers and medical librarians. As part of the department’s approach to research activities, he discussed their longitudinal, 4-year required research program (Embark). Students are paired with mentors and receive training across multiple domains of the research design and implementation process, including navigating IRB requirements, presentation skills, abstract writing, and paper publication. Dr. Gould noted the possibility of using the outcomes of student research activities as quantifiable metrics for residency application, which will favor the development of research programs.

Finally, Dr. Gould described the COVID-19 pandemic as a significant catalyst for the changes medical education has already been experiencing recently. This includes an increase in medical school and residency applications, which have not been accompanied by an increase in federally-funded residency programs. While not necessarily a novel trend, this does contribute to application inflation and makes holistic review more challenging. Other significant changes include the transition of USMLE Step 2 CS to a virtual format, along with a vast majority of medical education. COVID-19 has also forced a more rapid acceleration in the digitization of teaching materials across the literature. He highlighted factors that contributed to his FMS department’s adaptation to the COVID-19 pandemic, crediting the department’s collaborative nature as well as the presence of faculty already devoted to medical education in mounting a rapid and effective response to the unforeseen changes. A robust faculty development program also enabled a quick transition to online learning and asynchronous education, while promoting evaluation of effectiveness.

While the number of traditionally-structured foundational sciences departments has been decreasing over the last two decades, Dr. Gould suggested this may be due to department name changes and combination of disciplines into single departments. In fact, according to the AAMC, overall numbers of basic/foundational science departments have increased 33% over the last five years, and the number of foundational science faculty has increased from 14,047 in 1998 to 18601 in 2018, with 21% of medical school faculty holding a PhD as of 2018 [2]. As continued changes to medical education are inevitable, Dr. Gould strongly promoted the increase and development of dedicated foundational science medical education faculty, who are invested in maintaining and innovating effective education in the face of unforeseen events, such as the COVID-19 pandemic and the transition of the USMLE Step 1 to P/F.

References:

1. National Resident Matching Program. Program Director Survey Results. 2018.
2. Association of American Medical Colleges. 2018 US Medical School Faculty Report. https://www.aamc.org/data-reports/faculty-institutions/interactive-data/2018-us-medical-school-faculty