[The following notes were generated by Andrea Belovich, PhD.]

The fifth session of the Winter 2021 IAMSE Web Seminar Series, “USMLE Step-1 is Going to Pass/Fail, Now what do we do?”, was presented on February 4th, 2021 by Drs. John (Jack) Boulet and Robert Cain. Prior to retiring in 2020, Dr. Boulet served as Vice President of Research and Data Resources for the Foundation for Advancement of International Medical Education and Research (FAIMER) and the Educational Commission for Foreign Medical Graduates (ECFMG). Dr. Cain is the President & CEO of the American Association of Colleges of Osteopathic Medicine (AACOM). In this webinar, “USMLE Step 1: Osteopathic and International Perspectives,” the speakers discussed how the USMLE Step 1 shift to pass/fail (P/F) is likely to impact International and Osteopathic medical training programs.

Dr. Boulet began by discussing recent changes to the USMLE Step exam series and how international medical graduates (IMGs) may be impacted. Nearly one-fourth of active physicians and residents in the United States are IMGs, and future IMGs applying to U.S. residency programs may be impacted by the changes to the USMLE Step Exam Series beyond the residency application and ranking process. In order for IMGs to be admitted to residency or fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME), they must first receive ECFMG Certification [1].

Dr. Boulet described the current ECFMG Certification requirements, which include examination in both medical science and clinical skills. Whereas medical science knowledge is assessed by the USMLE Step 1/Step 2 Clinical Knowledge (CK) exams, clinical skills have been assessed using the USMLE Step 2 Clinical Skills (CS) exam before its suspension in early 2020 and recent elimination in January 2021. According to Dr. Boulet, the ECFMG now allows applicants to demonstrate clinical skills through alternative avenues, including having an license to practice medicine in another country or having passed a standardized clinical skills exam for medical licensure. Other avenues include graduating from a medical school that 1) is accredited by an agency recognized by the World Federation for Medical Education, 2) participates in the U.S. Federal Student Loan Program, or 3) issues degree a jointly with U.S. medical school accredited by the Liaison Committee on Medical Education. Finally, without the communication requirements assessed by the USMLE Step 2 CS, IMGs must also pass the Occupational English Test in medicine to receive ECFMG Certification.

Dr. Boulet next discussed how the reporting of USMLE Step 1 scores as P/F is likely to affect IMGs during the residency application process. On average, IMGs have a ~60% match rate with U.S. residencies compared to a >90% for medical graduates trained in the United States. IMGs therefore tend to submit a great number of applications (~150), increasing competition for interviews. Dr. Boulet shared that since IMGs can no longer depend upon a strong numerical USMLE Step 1 scores to stand out from their peers, they must now reconsider how to distinguish themselves during the transition to Graduate Medical Education (GME).

Dr. Boulet suggested that the likely use of the numerically scored Step 2 CK exam to screen residency applications may disadvantage certain IMGs, depending on whether their school’s curriculum that provides early clinical experience or can be altered to do so. Schools may differ in this ability, depending on whether the school uses a 4-year curriculum (such as the Caribbean schools) or a 6-7 year curriculum. Schools with a small minority of graduates who apply to U.S. residencies may also not wish to restructure their curriculum, which could disadvantage their graduates seeking U.S. residency training. Given these concerns, IMGs’ motivation to secure U.S. residency programs may decrease.

In terms of the educational impact of the USMLE Step 1 P/F transition, Dr. Boulet anticipated that students may be less likely to neglecting preclinical coursework in favor of Step 1 preparation. However, the decreased pressure may reduce the perceived value of the basic sciences, which may result in reduced motivation for students to perform as well in the basic sciences. Overall, the lack of pressure to score highly on Step 1 may allow time for a more balanced education, which is expected to positively affect wellness and reduce burnout.

Nevertheless, the question remains in regard to how IMGs will distinguish themselves during the Undergraduate Medical Education (UME) to GME transition. In addition to the increased reliance on Step 2 CK numeric scores, U.S. residency program directors may rely on the international reputation of medical schools, including whether they are ECFMG accredited. Dr. Boulet concluded by emphasizing the need to reevaluate the residency selection process, and highlighted efforts by institutions such as the Coalition for Physician Accountability, the ECFMG, and the National Board of Osteopathic Medical Examiners (NBOME) to make the UME to GME transition more efficient. These efforts include exploring assessment innovations to allow ranking across multiple dimensions needed for success in residency, creating structured holistic reviews of credential portfolios, and developing more holistic metrics such as a “Resident Readiness Index.”

Dr. Cain continued the webinar by sharing the overall organizational impact of the USMLE Step 1 transition to P/F on the osteopathic medicine community, despite the community having its own independent licensing body, the NBOME. Dr. Cain emphasized that osteopathic students and graduates are becoming an increasingly large part of the physician workforce, with 25% of medical students graduating with a Doctor of Osteopathy (DO) degree. Approximately 20% of medical schools offer the DO degree, an 20% of those schools are located in rural and underserved regions. Altogether, this underscores the need for DO graduates to have access to high quality residency training, which is becoming more competitive due to increased volume of applications, including those from IMG and MD graduates. Dr. Cain also expressed concerns that residency programs who have previously accepted DO students may be less likely to do so as a result of the USMLE Step 1 P/F transition, which may result in a loss of DO students in leadership and other prominent positions.

The impact of many residency programs requiring DO students to take the USMLE Step 1 to be considered is associated with significant costs. For students, this cost is felt financially, personally, and relationally, while schools feel the impact of the USMLE Step 1 requirement on their preclinical curricula. These costs raise an ethical question regarding the fairness of requiring students to take two exams, particularly when considering the additional opportunity cost: during the time students must prepare to this extra exam, they must forgo other extracurricular or leadership opportunities.

Dr. Cain then discussed whether it is therefore a benefit or a detriment to require DO students to take the USMLE Step 1 exam, particularly after the transition to P/F occurs. The requirement serves to benefit DO students when a numerical score is provided, as it allows students (particularly high-performing individuals) to distinguish themselves. However, does that benefit change when the Step 1 exam is P/F, if both the COMLEX Level 1 and the USMLE Step 1 test the same material and the COMLEX is also acceptable for medical licensure?

To explore this question, Dr. Cain discussed the advantages and disadvantages of associated with the USMLE Step 1 scoring system transition to P/F. Advantages include decreased stress and cost for students, and schools will experience less pressure to “teach to the test.” Furthermore, the reduced pressure to score highly on an assessment designed for allopathic medical education aligns with and supports the development of the osteopathic professional identity. However, disadvantages include concerns regarding reduced transparency surrounding the residency application screening process due to the loss of the numeric Step 1 score and whether program directors will rely upon school reputation or their own familiarity with the individual DO programs. Finally, a greater proportion of DO students pursue clinical clerkship training in non-hospital based settings than MD students, such as ambulatory and physical rehabilitation training, which disadvantages them from scoring as highly on the USMLE Step 2 CK exam.

Dr. Cain next explained the process for DO licensure, underscoring that DO students may achieve licensure either through the NBOME’s COMLEX exams or through the National Board of Medical Examiners’ (NBME) USMLE exams. While DO students are required to pass the COMLEX exams to graduate and most commonly use this pathway to achieve licensure, they are not required to take both COMLEX and USMLE exams. For DO students, this raises the concern that the USMLE exams have been misused in regards to their primary purpose to inform licensure when used as a tool for residency placement, particularly when there are differences in performance compared to allopathic students.

Dr. Cain also provided a more in-depth look at the purpose of the NBOME’s COMLEX exam series, and explained importance to osteopathic medical education. In addition to the biomedical, clinical, and health systems sciences integral to allopathic medical education, a significant component of the osteopathic medical curriculum includes over 200 hours of biomechanical science, complexity science, and osteopathic philosophy. All of these aspects need to be assessed in order to determine knowledge, skill, and behavior of students as they progress through their training. The COMLEX exams are designed to test the osteopathic-specific components at each stage of development, in addition to the biomedical, clinical, and health systems sciences. In all, the COMLEX is not designed as an add-on exam, but is rather embedded throughout the four-year experience.

To conclude the webinar, Dr. Cain emphasized that work is currently being done by the Coalition for Physician Accountability to review the transition from UME to GME, including the development of a more holistic approach to residency recruitment and interviewing. To summarize the general sentiment of the osteopathic community, Dr. Cain stated that the change of the USMLE Step 1 scoring to P/F could have a negative impact on DO students, when there really should be no impact. This highlights a great need for continuing education around the use of the COMLEX exam and understanding the tools for determining residency readiness.

References:

1. About ECFMG Certification. Educational Commission for Foreign Medical Graduates. https://www.ecfmg.org/certification/