[The following notes were generated by Andrea Belovich, PhD.]

The fifth session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on February 4th, 2021 by Drs. John (Jack) Boulet and Robert Cain. Prior to retiring in 2020, Dr. Boulet served as Vice President of Research and Data Resources for the Foundation for Advancement of International Medical Education and Research (FAIMER) and the Educational Commission for Foreign Medical Graduates (ECFMG). Dr. Cain is the President & CEO of the American Association of Colleges of Osteopathic Medicine (AACOM). In this webinar, “USMLE Step 1: Osteopathic and International Perspectives,” the speakers discussed how the USMLE Step 1 shift to pass/fail (P/F) is likely to impact International and Osteopathic medical training programs.

Dr. Boulet began by discussing recent changes to the USMLE Step exam series and how international medical graduates (IMGs) may be impacted. Nearly one-fourth of active physicians and residents in the United States are IMGs, and future IMGs applying to U.S. residency programs may be impacted by the changes to the USMLE Step Exam Series beyond the residency application and ranking process. In order for IMGs to be admitted to residency or fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME), they must first receive ECFMG Certification [1].

Dr. Boulet described the current ECFMG Certification requirements, which include examination in both medical science and clinical skills. Whereas medical science knowledge is assessed by the USMLE Step 1/Step 2 Clinical Knowledge (CK) exams, clinical skills have been assessed using the USMLE Step 2 Clinical Skills (CS) exam before its suspension in early 2020 and recent elimination in January 2021. According to Dr. Boulet, the ECFMG now allows applicants to demonstrate clinical skills through alternative avenues, including having an license to practice medicine in another country or having passed a standardized clinical skills exam for medical licensure. Other avenues include graduating from a medical school that 1) is accredited by an agency recognized by the World Federation for Medical Education, 2) participates in the U.S. Federal Student Loan Program, or 3) issues degree a jointly with U.S. medical school accredited by the Liaison Committee on Medical Education. Finally, without the communication requirements assessed by the USMLE Step 2 CS, IMGs must also pass the Occupational English Test in medicine to receive ECFMG Certification.

Dr. Boulet next discussed how the reporting of USMLE Step 1 scores as P/F is likely to affect IMGs during the residency application process. On average, IMGs have a ~60% match rate with U.S. residencies compared to a >90% for medical graduates trained in the United States. IMGs therefore tend to submit a great number of applications (~150), increasing competition for interviews. Dr. Boulet shared that since IMGs can no longer depend upon a strong numerical USMLE Step 1 scores to stand out from their peers, they must now reconsider how to distinguish themselves during the transition to Graduate Medical Education (GME).

Dr. Boulet suggested that the likely use of the numerically scored Step 2 CK exam to screen residency applications may disadvantage certain IMGs, depending on whether their school’s curriculum that provides early clinical experience or can be altered to do so. Schools may differ in this ability, depending on whether the school uses a 4-year curriculum (such as the Caribbean schools) or a 6-7 year curriculum. Schools with a small minority of graduates who apply to U.S. residencies may also not wish to restructure their curriculum, which could disadvantage their graduates seeking U.S. residency training. Given these concerns, IMGs’ motivation to secure U.S. residency programs may decrease.

In terms of the educational impact of the USMLE Step 1 P/F transition, Dr. Boulet anticipated that students may be less likely to neglecting preclinical coursework in favor of Step 1 preparation. However, the decreased pressure may reduce the perceived value of the basic sciences, which may result in reduced motivation for students to perform as well in the basic sciences. Overall, the lack of pressure to score highly on Step 1 may allow time for a more balanced education, which is expected to positively affect wellness and reduce burnout.

Nevertheless, the question remains in regard to how IMGs will distinguish themselves during the Undergraduate Medical Education (UME) to GME transition. In addition to the increased reliance on Step 2 CK numeric scores, U.S. residency program directors may rely on the international reputation of medical schools, including whether they are ECFMG accredited. Dr. Boulet concluded by emphasizing the need to reevaluate the residency selection process, and highlighted efforts by institutions such as the Coalition for Physician Accountability, the ECFMG, and the National Board of Osteopathic Medical Examiners (NBOME) to make the UME to GME transition more efficient. These efforts include exploring assessment innovations to allow ranking across multiple dimensions needed for success in residency, creating structured holistic reviews of credential portfolios, and developing more holistic metrics such as a “Resident Readiness Index.”

Dr. Cain continued the webinar by sharing the overall organizational impact of the USMLE Step 1 transition to P/F on the osteopathic medicine community, despite the community having its own independent licensing body, the NBOME. Dr. Cain emphasized that osteopathic students and graduates are becoming an increasingly large part of the physician workforce, with 25% of medical students graduating with a Doctor of Osteopathy (DO) degree. Approximately 20% of medical schools offer the DO degree, an 20% of those schools are located in rural and underserved regions. Altogether, this underscores the need for DO graduates to have access to high quality residency training, which is becoming more competitive due to increased volume of applications, including those from IMG and MD graduates. Dr. Cain also expressed concerns that residency programs who have previously accepted DO students may be less likely to do so as a result of the USMLE Step 1 P/F transition, which may result in a loss of DO students in leadership and other prominent positions.

The impact of many residency programs requiring DO students to take the USMLE Step 1 to be considered is associated with significant costs. For students, this cost is felt financially, personally, and relationally, while schools feel the impact of the USMLE Step 1 requirement on their preclinical curricula. These costs raise an ethical question regarding the fairness of requiring students to take two exams, particularly when considering the additional opportunity cost: during the time students must prepare to this extra exam, they must forgo other extracurricular or leadership opportunities.

Dr. Cain then discussed whether it is therefore a benefit or a detriment to require DO students to take the USMLE Step 1 exam, particularly after the transition to P/F occurs. The requirement serves to benefit DO students when a numerical score is provided, as it allows students (particularly high-performing individuals) to distinguish themselves. However, does that benefit change when the Step 1 exam is P/F, if both the COMLEX Level 1 and the USMLE Step 1 test the same material and the COMLEX is also acceptable for medical licensure?

To explore this question, Dr. Cain discussed the advantages and disadvantages of associated with the USMLE Step 1 scoring system transition to P/F. Advantages include decreased stress and cost for students, and schools will experience less pressure to “teach to the test.” Furthermore, the reduced pressure to score highly on an assessment designed for allopathic medical education aligns with and supports the development of the osteopathic professional identity. However, disadvantages include concerns regarding reduced transparency surrounding the residency application screening process due to the loss of the numeric Step 1 score and whether program directors will rely upon school reputation or their own familiarity with the individual DO programs. Finally, a greater proportion of DO students pursue clinical clerkship training in non-hospital based settings than MD students, such as ambulatory and physical rehabilitation training, which disadvantages them from scoring as highly on the USMLE Step 2 CK exam.

Dr. Cain next explained the process for DO licensure, underscoring that DO students may achieve licensure either through the NBOME’s COMLEX exams or through the National Board of Medical Examiners’ (NBME) USMLE exams. While DO students are required to pass the COMLEX exams to graduate and most commonly use this pathway to achieve licensure, they are not required to take both COMLEX and USMLE exams. For DO students, this raises the concern that the USMLE exams have been misused in regards to their primary purpose to inform licensure when used as a tool for residency placement, particularly when there are differences in performance compared to allopathic students.

Dr. Cain also provided a more in-depth look at the purpose of the NBOME’s COMLEX exam series, and explained importance to osteopathic medical education. In addition to the biomedical, clinical, and health systems sciences integral to allopathic medical education, a significant component of the osteopathic medical curriculum includes over 200 hours of biomechanical science, complexity science, and osteopathic philosophy. All of these aspects need to be assessed in order to determine knowledge, skill, and behavior of students as they progress through their training. The COMLEX exams are designed to test the osteopathic-specific components at each stage of development, in addition to the biomedical, clinical, and health systems sciences. In all, the COMLEX is not designed as an add-on exam, but is rather embedded throughout the four-year experience.

To conclude the webinar, Dr. Cain emphasized that work is currently being done by the Coalition for Physician Accountability to review the transition from UME to GME, including the development of a more holistic approach to residency recruitment and interviewing. To summarize the general sentiment of the osteopathic community, Dr. Cain stated that the change of the USMLE Step 1 scoring to P/F could have a negative impact on DO students, when there really should be no impact. This highlights a great need for continuing education around the use of the COMLEX exam and understanding the tools for determining residency readiness.

References:

1. About ECFMG Certification. Educational Commission for Foreign Medical Graduates. https://www.ecfmg.org/certification/

[The following notes were generated by Andrea Belovich, PhD.]

The fourth session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 28^th, 2021 by Drs. Bruce Morgenstern and Brenda Roman. Dr. Morgenstern is the Vice Dean for Academic and Clinical Affairs at the Roseman University College of Medicine, and the immediate past President of the Alliance for Clinical Affairs. Dr. Roman is the Associate Dean of Medical Education and a Professor of Psychiatry at Wright State University Boonshoft School of Medicine, and the current President of the Alliance for Clinical Education. In this webinar, “Step 1 Going Pass-Fail: Are We Just Kicking the Can Down the Road?,” the speakers discussed issues with the current residency application process that will remain to be solved after the USMLE Step 1 is scored pass/fail (P/F). The audience also engaged in a rich discussion about how Undergraduate Medical Education (UME) can help address these issues.

Dr. Morgenstern began the webinar with an overview of the original purpose of the USMLE Step series exams by quoting the USMLE’s mission statement, “To provide to licensing authorities meaningful information from assessments of physician characteristics including medical knowledge, skills, values, and attitudes that are important to the provision of safe and effective patient care” [1]. He contrasted this with the Step exams’ off-label use to distinguish residency applicants from one another and questioned how well exams consisting of multiple-choice questions can assess abstract qualities such as values and skills. Given these limitations and the recent discontinuation of the Step 2 Clinical Skills (CS) exam, Dr. Morgenstern raised the question of how the USMLE will continue to move forward in advancing its mission statement.

Dr. Morgenstern then pointed out that medical education, in general, faces a similar challenge. The ultimate goal of medical education is, of course, to produce “good” physicians. However, since the field lacks an operational definition of what a “good” physician is, medical education struggles to articulate how and when a candidate is determined to be one. Competency-based medicine is a trend evolving to meet this challenge, although Dr. Morgenstern added the caveat that the term “competency” itself may not be a sufficient descriptor due to the word stigma of “bare minimum” often associated with the term. Beyond clinical knowledge, the areas commonly used to measure competence are also difficult to assess, including clinical skills, professional identity development, professionalism, values and attitudes, health systems science-related curricular content, and growth mindset. As an example, while Dr. Morgenstern believes the ability to communicate in a culturally humble way is paramount, assessing this skill is far more difficult, especially when the focus on USMLE Step 1 has created a parallel curriculum for medical schools.

The driving factor behind this focus on USMLE Step 1 is the residency “match frenzy,” which has encouraged program directors find ways to screen and rank applicants. Citing a collection of Bryan Carmody’s research [2], Dr. Morgenstern acknowledged that Step 1 and Step 2 CK scores predict passage of standardized specialty boards, but in a more global context, Step 1 scores do not correlate with other measures of overall success in residency. Despite the data, perception is reality, so the practice of relying upon Step 1 scores to develop interviewee lists for residency slots has been generally accepted by both students and program directors, causing significant stress for students [3, 4, 5].

In addition to the overwhelming number of residency applications per residency slot, a significant contributor to the reliance on ranking tools is the lack of staffing available to residency program directors in support of the conductance of holistic application review. As program directors experience pressure to select applicants likely to pass specialty boards, they prefer methodologies to meaningfully rank applicants. Subjective metrics, such as Medical Student Performance Evaluation (MSPE) letters, tend to lack consistency regarding clinical grading and undergo “modifier inflation” in the description of medical students. Even the term “excellent” is no longer the highest superlative, since medical students as a group are generally very intelligent and capable individuals (reminiscent of the Lake Wobegon effect). Taken together, this leads to a general lack of trust between UME and GME regarding the MSPE.

Dr. Morgenstern concluded his portion of the webinar by positing that the current residency application process itself may not be fair. Even ERAS contributes greatly to the overwhelming number of applicants, and the National Residency Match Program algorithm is proprietary, so it is not possible to verify whether/how it favors applicants or programs, not to mention diversity, equity, and inclusion issues. Finally, he emphasized that the underlying issues with the current residency match process will remain after the USMLE Step 1 transitions to P/F, so, going forward, how can program directors screen applicants and conduct holistic reviews? Even after consideration for the Lake Wobegon effect, 50% of all applicants will perpetually be “below average.” To protect these students, Dr. Morgenstern emphasized that it is incumbent upon UME to develop tools that are more intentional in their design and purpose to assess “good” residents compared to the minimum competency licensure exams previously coopted for the match process.

Dr. Roman then continued the webinar by expanding upon problems that will remain inherent to the residency match process despite the USMLE Step 1 P/F transition. In particular, student stress is not addressed by this change, rather it will probably shift from Step 1 to Step 2 CK, as program directors are likely to focus more on Step 2 CK (a stronger predictor of clinical performance than Step 1 [6]). This is likely to impact the clerkship years and may detract from patient care and clinical learning, especially in light of the discontinuation of Step 2 CS. Medical schools and programs are still evaluated by match rate, and the job performance of Student Affairs Deans are tied to successful matches. Students with well-explained gaps in their education may be negatively viewed by program directors for requiring more time to complete medical school. Students and programs will still try to game the system. Despite these issues, almost all students are successful in matching to a residency program.

A brief overview of the history of the match process from its inception in 1920 shows that these underlying themes have plagued the transition from UME to GME for an entire century. Dr. Roman then asked the audience for their input on a series of open-ended and polling questions to help identify solutions to these long-standing challenges.

When asked, “What characteristics of a future ‘good’ clinician can the pre-clerkship educators identify about their students?” audience answers included “professionalism, teamwork, emotional intelligence, diligence, maturity, hard-working, and curious.” These responses underscored the collective observation that the system has become over-reliant on measuring knowledge by numbers.

Following up on this question, Dr. Roman asked, “What can the pre-clerkship educators document in a narrative format about students that would be helpful for consideration by program directors?” Audience answers included, “Professionalism, ability to think critically, curiosity, grit, and determination,” which described resident characteristics seen as desirable by program directors.

Dr. Roman then asked the audience whether educators should “agree to a common/standardized vocabulary in describing medical students in evaluations and letters of recommendation?” and “Should other measures be used in the application process regarding fit to the medical specialty, like ‘dexterity assessments” for skills-based specialties or communication assessments for ‘people-centered specialties’?” The audience responded in an overwhelmingly positive manner to both questions.

Finally, Dr. Roman polled the audience with the following question: “Since data regarding ‘predicting success in residency’ is not robust, a radical solution would be for students to identify a specialty of choice, geographic preference and a few key attributes about what they would like in a program (academic medical center versus community-based program, research opportunities) and do away with interviews…?” The audience was more evenly split in their response to this question.

As the webinar drew to a conclusion, Dr. Roman reminded the audience that medical schools can improve the residency application process and trust between UME and GME by developing better ways to identify and addressing professionalism issues. This may include involving basic science faculty who can detect concerning patterns, and greater distinction between professionalism concerns that can be remediated versus a long-standing pattern of behavior and documenting these.

Finally, Dr. Roman left with the audience with a provocative proposal intended to spark thought and discussion. With a disclaimer that the idea in its current form was not intended for serious consideration, Dr. Roman suggested the following:

1. Programs should define optimal fit for residents in their programs
2. Applicants define their characteristics
3. The match algorithm does not use rank lists, but rather one that prioritizes compatibility between applicant and program

Dr. Roman also suggested that medicine should also ask that schools and hospitals stop being ranked by media publications, as this creates a false reality and may drive students to applying to so many residencies. Ultimately, it is important for UME and GME to work together more to avoid simply moving the stress from Step 1 to Step 2 CK, and to involve clinicians and basic medical science faculty in helping program directors assess applicants.

References:

1. USMLE Mission Statement. United States Medical Licensing Examination. https://www.usmle.org/about/
2. https://thesherriffofsodium.com/2019/03/05/the-mythology-of-usmle-step-1-scores-and-board-certification
3. Beck Dallaghan. Medical School Resourcing of USMLE Step 1 Preparation: Questioning the Validity of Step 1. Med Sci Educ. (2019) 29:1141-1145
4. Bryan Carmody. On Residency Selection and the Quantitative Fallacy. J Grad Med Educ. (2019) 11 (4): 420-421.
5. Bryan Carmody. Medical Student Attitudes toward USMLE Step 1 and Health Systems Science – A Multi-Institutional Survey. Teaching and Learning in Medicine. December 8, 2020.
6. Akshita Sharma, Daniel P. Schauer, Matthew Kelleher, Benjamin Kinnear, Dana Sall, and Eric Warm. USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency. J Grad Med Educ. (2019) 11(4): 412-419.

[The following notes were generated by Andrea Belovich, PhD.]

The third session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on January 21st, 2021 by Dr. Jonathan Amiel, Interim Co-Vice Dean for Education, Senior Associate Dean for Curricular Affairs, and Associate Professor of Psychiatry at the Columbia University Vagelos College of Physicians and Surgeons. In this webinar, “USMLE Step 1 P/F: A UME Curriculum Dean’s Perspective,” Dr. Amiel discussed how the transition of the USMLE Step 1 to pass/fail (P/F) is anticipated to impact undergraduate medical education (UME) curriculum.

Dr. Amiel began with an overview of the USMLE Step exam licensure program, outlining the recommendations of the 2019 Invitational Conference on USMLE Scoring (InCUS) that culminated in the decision to change the USMLE Step 1 from a numerical scoring system to P/F after January 2022. He then discussed the role of a Curriculum Dean in UME, and how, from this perspective, the Step 1 P/F shift would likely impact UME curricula. Quoting Sir Isaac Newton, “And to every action there is always an equal and opposite or contrary reaction,” this portion of the webinar was framed as an exploration of both the changes that the Step 1 P/F shift is likely to introduce to UME, and the potential implications of UME’s responses to those changes.

He explained that a curriculum dean’s role is to coordinate between several departments to create a coherent and cohesive educational program. He listed major factors that could be impacted as UME adjusts to the Step 1 P/F transition: 1) Educational program mission, objectives, values, and ethics, 2) Accreditation, 3) Logistics, 4) Curriculum architecture and engineering, 5) Faculty development and 6) Resource management.

In terms of the mission, objective, values, and ethics of educational programs and accreditation, Dr. Amiel expressed optimism that pre-clerkship faculty may be able to take a more holistic approach to the knowledge and learning skills students acquire in pre-clerkship curricula. Rather than “teaching to the test,” institutions may utilize Step 1’s shift to P/F as an opportunity to work with the USMLE program and the NBME to approach Step 1 from a competency-based perspective. Dr. Amiel emphasized shifting the exam’s focus towards assessing what students need to know to be competent in clinical settings. Dr. Amiel stated that he did not anticipate a major impact in the accreditation arena, with the caveat that the NBME may decide to increase the passing threshold for Step 1. If implemented, this increased threshold could introduce a higher fail rate that may affect accreditation.

Whether or not the Step 1 passing threshold is raised, curriculum deans must consider how the P/F shift will affect the logistics of preparing students for standardized exams and how student support services may be impacted. Learners who have difficulty with standardized tests will still require support for licensure exams, and institutions need to ensure that they are still providing this support. This includes greater support for the USMLE Step 2 CK exam, which may now grow in importance for residency application ranking, as it will still be scored numerically. Additionally, a thoughtful approach to revising assessment methods is needed. For example, NBME subject exams are often currently used to prepare students for USMLE, but Dr. Amiel urged participants to consider other assessment methods to help prepare students for passing the Step 1 exam, including utilizing more open-ended questions and applied-knowledge assessments.

Next, Dr. Amiel discussed how the architecture of UME curricula may be impacted by the Step 1 P/F shift. He predicted that the most significant architectural impact would be on dedicated study time for the Step exams. Schools may need to dedicate more time to Step 2 CK preparation, which will affect the senior curriculum. While it may be tempting to reduce the dedicated study time for Step 1 to create space for Step 2 CK preparation, students who experience difficulty with standardized testing may be put at greater risk of failing Step 1. Elective rotations may also be reduced, but these are important for residency interviews and gaining clinical experience. Ultimately, these issues may result in individualized study time/tracks for students.

With the potential changes to assessment methods, Dr. Amiel also anticipated that faculty development programs will need to provide new training to faculty. He emphasized this as an opportunity to reframe “assessment of learning” as “assessment for learning,” and to help faculty broaden their thinking about how to test medical knowledge early in the curriculum. He then discussed the potential impact of the Step 1 P/F shift on resources and resource management. As institutions have learned during the COVID-19 pandemic era, resources need to be used wisely, and the process of making changes to curricula (especially in terms of architecture, dedicated study time, and timing of exams) is resource-intensive. Parallel curricula for different graduating classes may be required, which can result in confusing messaging for students. Changes in testing may also require more resources dedicated to testing support.

To summarize, Dr. Amiel outlined UMSE’s likely reactions to some of the changes that anticipated to result from the USMLE Step 1 shift to P/F. Firstly, students may be required to take Step 2 CK earlier in their curriculum than they do now in order to obtain scores needed for residency applications. This would then require UME curricula to prepare students for the Step 2 CK earlier as well. Dedicated study time for Step 1 may be decreased, but new dedicated time would likely be introduced for Step 2 CK preparation. Schools that administer the Step 1 exam later in students’ training (like Columbia University) will need to consider shifting both the Step 1 and Step 2 CK exams earlier. Specialties may also begin to develop and administer their own exams to sort and rank applicants. If this occurs, curriculum design would be further impacted in order to help students be prepared and competitive for their desired specialties.

Dr. Amiel suggested that shifting the USMLE Step 1 to P/F will be positive for medical education, but stakeholders must take care to mitigate any negative impacts that could result from the reactions to this shift. He stressed the importance of basing initiatives in education theory and best-practices to assess medical knowledge, promoting competency-based medical education. He also mentioned working with the NBME and USMLE program to establish optimal curricular times dedicated to the Step 1 exam as part of the gateway into the clinical phase of UME curricula. He suggested that students should be able to demonstrate their competency in a basic body of knowledge and some clinical reasoning skills before moving to clinic, and that, in its current form, the USMLE Step 1 itself may currently not be sufficient for this purpose. This raises the question of whether the USMLE Step 1 is even the optimal test for the transition from pre-clerkship training to the clinic. However, Dr. Amiel did stress that all parts of an assessment program should be designed to help further learner’s progress and learning, and should include experience with high-stakes assessments.

Finally, Dr. Amiel cautioned that any new methods of programmatic assessment should be mindful of holistic review and how generating data about learner performance may impact residency applications. In particular, institutions must be careful to employ anti-racist pedagogy and assessment practices to avoid increasing inequities and disadvantaging students who may not have had as many enriched educational experiences as others. Dr. Amiel concluded with the recognition that the USMLE Step 1 P/F shift will result in an abundance of scholarly activity opportunities.