[The following notes were generated by Mark Slivkoff, PhD]

IAMSE Webinar Series, Spring 2019

Speakers: Michelle Daniel, MD, MHPE, FACEP
Assistant Dean for Curriculum and Associate Professor of Emergency Medicine and Learning Health Sciences
University of Michigan School of Medicine, Ann Arbor, MI USA

Title: Moving USMLE Step 1 After Core Clerkships: Rationale, Challenges, and Early Outcomes
Series: The Role of Basic Science in 21st Century Medical Education

The focus this week was once again on USMLE Step 1. Dr. Michelle Daniel of the University of Michigan School of Medicine presented this final seminar of our series which focused on the placement of Step 1 in the curriculum, a topic that has generated debate recently.

Currently, the ideal placement for Step 1 is unknown. Its current placement, usually at the end of the second year, is mainly based on tradition rather than an alignment of learning goals or outcomes. In her presentation, Dr. Daniel explained the rationale for, breaking with tradition from and altering the timing of Step 1. She also addressed early outcomes associated with the change and discussed potential challenges—and how to overcome them.

She started with a poll in which she asked the audience to choose the stage of change that describes your institution concerning the timing of Step 1:
• Pre-contemplative stage (Students take Step 1 following the preclinical curricula before entering clinical clerkships and currently, there are no thoughts of changing)
• Contemplative stage (Students take Step 1 following the preclinical curricula, but the institution is currently considering moving Step 1 after clerkships)
• Determined stage (Students take Step 1 following the preclinical curricula, but plans are currently underway to move Step 1 after clerkships)
• Action stage (Students currently take Step 1 after the core clerkships)
• N/A (Students do not take Step 1)

The results were similar to what Dr. Daniel has seen nationally. Over half (55%) of schools are in the pre-contemplative stage, about a third (32%) are in the contemplative stage, 1% are in the determined stage, while 8% are in the action stage.

As is the case with a majority of medical schools, Step 1 is given after years 1 and 2, the basic science years. Students memorize and purge the information during the examination, then promptly forget it. Schools who desire to move Step 1 usually argue that moving the timing of Step 1 after the core clinical year helps promote better retention of foundational science knowledge and basic and clinical science integration. Most schools who alter the timing of Step 1 do so as just one part of the process of determining the means by which science is delivered across the curriculum.

Anecdotally—Dr. Daniel continued—students report that when they study for Step 1 after core clerkships, the basic science information “feels stickier” since they are able to recall real patients they have encountered with certain conditions and it becomes easier to remember basic science concepts when they can create linkages to clinical care. The long question stems encountered on the USMLE are therefore easier to understand.

Another reason that schools move Step 1 is to boost their scores. As Dr. Daniel points out (and as last week’s poll also demonstrated), a majority of individuals desire making Step 1 pass/fail but the reality is that the score matters to residency directors. However, Dr. Daniel believes that this reason for change should be lowest on the list. What is critically important is that when schools have changed the timing, their scores have NOT been negatively impacted. Schools, therefore, can feel free to make sweeping changes to their curriculum, and this seems to be the trend since the list of schools who are making the change is increasing.

Dr. Daniel then took a dive into some of the data which she has published with her colleagues concerning the characteristics of schools which have moved Step 1 (Daniel et al. 2017). She then presented outcome data from a follow-up paper (Jurich et al. 2018). Specifically, they were interested in what happened to Step 1 scores and failure rates after moving the examination. Results indicated a slight increase in scores and a reduction in failure rates. Dr. Daniel additionally pointed out a highlight from the failure data in that the number of failures went down from 48 to 6. She also currently has a paper in review which focuses on Step 2 outcomes.

The conclusion of her webinar focused on the challenges (which she has published as well with her colleagues) (Pock et al. 2019). Since they lack a Step 1 study period to consolidate knowledge, students may not seem as prepared to start clerkships. Communication with clinical faculty must emphasize that learners entering the clerkships are “different” – not better or worse than historical students, but different. Curricula that foster the integration of basic and clinical sciences, as well as those that emphasize active learning in the pre-clerkship phase can help smooth this transition.

Another challenge to moving Step 1 is that weaker students may not be identified early and provided with appropriate academic support. Supplementing institutional exams with NBME basic science subject exams, the comprehensive basic science exam or comprehensive basic science self-assessment can help identify such learners before they get to Step 1.

Shelf exam performance weaknesses are another concern. Many institutions who have moved Step 1 after core clerkships report a decline in shelf exam performance, particularly in the early clerkships. The declines in shelf scores, as Dr. Daniel explained, are not surprising when one considers that students have not had as much practice with timed tests and NBME style questions.

Other challenges include students extending the Step 1 study period if there is not a firm “deadline” of when to take the exam, students becoming overly concerned about what residency specialty choices may be open to them until they know their Step 1 scores, and lastly, learners may be concerned about having enough time to take and pass Step 1, Step 2 CK, and 2 CS in rapid succession. Dr. Daniel suggested that all these things can be easily managed, however.

Dr. Daniel concluded her presentation just as she did at the beginning: with a poll. She asked now that you have learned a bit more, what would you advise your institution to do?
• Keep Step 1 after the pre-clerkship phase
• Move Step 1 after core clerkships
• It depends on the needs of curricular re-design, but I would be comfortable advocating either position

Approximately half of the respondents chose the latter, and the remaining responses were split between keeping Step 1 as is or moving it. Dr. Daniel once again emphasized the large drop in failure rates.

Similar to the previous four webinars, this one was also followed by numerous questions, too many to address before the hour concluded.

We wish to thank again all presenters and the audience who made this series such a success!

References:

• Daniel M., A. Fleming, C. O’Conner Grochowski, V. Harnik, S. Klimstra, G. Morrison, A. Pock, M. L. Schwartz, and S. Santen. Why Not Wait? Eight Institutions Share Their Experiences Moving United States Medical Licensing Examination Step 1 After Core Clinical Clerkships. Acad Med 2017;92:1515-1524.
• Jurich, D., M. Daniel; M. Paniagua, A. Fleming, V. Harnik, A. Pock, A. Swan-Sein, M.A. Barone, and S. Santen. Moving the United States Medical Licensing Examination Step 1 After Core Clerkships: An Outcomes Analysis. Acad Med 2019;94:371-377.
• Pock, A. M. Daniel, S.A. Santen, A. Swan-Sein, A. Fleming, and V. Harnik. Challenges Associated With Moving the United States Medical Licensing Examination (USMLE) Step 1 to After the Core Clerkships and How to Approach Them. Acad Med 2019. DOI: 10.1097/ACM.0000000000002651

[The following notes were generated by Mark Slivkoff, PhD]

IAMSE Webinar Series, Spring 2019

Speakers: David Chen
Third Year Medical Student
The University of Washington, Seattle, WA, USA

Kathryn Andolsek, MD MPH
Professor, Community and Family Medicine
Assistant Dean, Premedical Education
Duke University, Durham, NC USA

Title: Stepping Beyond the Step 1 Climate
Series: The Role of Basic Science in 21st Century Medical Education

In this week’s webinar our attention was moved away from specific curricula, as discussed during the previous two webinars, to a more global topic: Step 1 of the United States Medical Licensing Examination (USMLE). The two presenters included David Chen, a third-year medical student at the University of Washington in Seattle, Washington, and Dr. Kathryn Andolsek, a professor and assistant dean at Duke University in Durham, North Carolina. That our list of presenters included a student and experienced medical educator gave the audience a unique opportunity to hear two perspectives on the ongoing debate of the importance placed on the USMLE examinations.

Mr. Chen led off the webinar. When he went into medicine, he explained, he hoped to address health care disparities, but upon entering medical school he was surprised at how little attention was given to systemic barriers to health and social medicine. Even if they were covered in class, he and his classmates were inclined to not pay attention since the material was not relevant to USMLE Step 1 which was seen as the sole take away of the preclinical years.

He took Step 1 last year and expressed that it was “the single worst educational experience of his life”, a needless period of suffering during which self-care was neglected and burn out was real. He spent long hours memorizing the content of questionable relevance and accuracy while having physical and emotional exhaustion. Mr. Chen documented his experiences and reached out to other students and faculty members, and was invited by Academic Medicine to write a commentary on the subject (Chen et al. 2019). They wrote that the “Step 1 Climate”, a termed coined by he and his fellow authors, was a national phenomenon. It has profound effects on the learning environment, student diversity, and student well-being.

The learning environment suffers since the test-preparation materials become the de facto national curriculum, which affects students and their instructors, and the relationship between them. The examination serves as a barrier to student diversity and success of underrepresented minorities, women, and students from low-income families. Furthermore, as Mr. Chen explained, the examination sets up “tiered worthiness” among medical students in that Step 1 scores define the competitiveness for residencies and their prestige. Family medicine, for example, sits at the low end of the scale which may deter students from applying. These factors all lead to ill effects on student well-being by increasing depression, burnout, and in extreme cases, suicide.

Mr. Chen continued his discussion by addressing an article which was a rebuke to theirs (Katsufrakis and Chaudhry 2019). Briefly, he pointed out that despite their claim there have been studies which have linked residency success to Step 1 performance (Prober et al 2016; McGaghie et al. 2011; Gliatto et al. 2016). Mr. Chen also expressed that he is unsure if the authors have a conflict of interest, but he does rightly claim that taking the boards is expensive.

A group of stakeholders who need more attention in this debate is the patients. Board examinations, as is commonly claimed, help ensure quality care for patients. Mr. Chen argues that given the diversity of patients, can a multiple-choice examination truly measure competency?

The last point that Mr. Chen makes is that the time is ripe for change. Over 100 years ago, the Flexner Report was published (in 1910) and about 70 years ago the first multiple-choice NBME exam was given (in 1916). The world is much different today, and “The need for a fundamental redesign of the content of medical training is clear” (Cooke et al 2006). Mr. Chen commented that he is not sure exactly what the barriers to change are, then parted with a final note: it is an ominous sign for patients who interact with the medical establishment if we in medical education are unable to address our problems.

Dr. Andolsek then took over the screen and the webinar. She began by adding some additional details to Mr. Chen’s brief historical account of Step 1. The main role of early examinations was to serve as a common licensing examination such that physicians could practice across state borders. It was therefore initially designed as a pass/fail assessment. Dr. Andolsek reiterated that the examination is expensive, usually with related costs (Step 1 resources) greater than $800. The examination correlates with performance on subsequent examinations (such as board examinations) but not on residency progression, faculty evaluations, or clinical skills. Furthermore, groups including women, historically underrepresented persons, non-traditional students, and those with financial need perform “differently” on the examination, usually less well.

Stakeholders of Step 1, she continued, include medical students, medical licensing boards, medical schools, and program directors. Similar to Mr. Chen’s comments, Dr. Andolsek questioned patients as belonging to the stakeholder group. She mentioned that in her 40 years of practice a patient never asked her how she did on the boards.

Medical students have responded to Step 1 in that they recognize that it is the de facto curriculum. The use it to judge whether a medical school’s curriculum is relevant if it’s “high yield” and linked to Step 1 content. If it’s not, then curricular content has a tendency to get deemed as uninteresting and devalued in their medical education.

The majority stakeholders, Dr. Andolsek continued, should be the medical licensing boards since licensing was the original intent of the examination. Most states have it written into their legislation, however, the actual number of the examination is never mentioned, only if it was passed or failed. Thus, this adds to the argument that the examination should be a pass/fail assessment.  

Many medical schools use the Step 1 to determine if students are ready to enter clinical education, although this was not an intended use for the examination. Some schools also use the scores to determine gaps in their curriculum, and to determine how likely their students will “match.”

The next group which she addressed in more detail are the program directors who use Step 1 scores to identify which medical students will get full review of their application and interviewed. Some program directors use “cut off scores” frequently set at < 220-240, with higher scores for more competitive specialties. Dr. Andolsek added that 90% of residency graduates who passed Step 1 with lower scores (200-227) passed their board examinations in fairly competitive specialties, demonstrating that passing correlates with passing. Dr. Andolsek also made the point that many good residency applicants are passed up by directors since their applications are ignored when certain scores are not attained. The programs are also judged by the ACGME, the residency accrediting body, which has historically used board passing rates as one measure to judge a program’s quality. A positive change is underway, as Dr. Andolsek noted, in that the ACGME has changed the threshold for passing rates.

Program directors need More relevant measures to assess and communicate medical school performance. They need some type of test or mechanism which may include ACGME competencies and milestones, core Entrustable Professionable Activities (EPAs) and any or all tools developed for resident selection that correlates with performance.

Dr. Andolsek concluded with a polling question sent out to schools. The question asked was should the Step 1 be pass/fail? The results?

• Yes: 71%
• Undecided: 16%
• No: 14%

The session was then opened up for questions, which lasted well over 20 minutes indicating the provocativeness of their webinar.  

References:

• Andolsek KM. One Small Step for Step 1. Acad Med. 2019 Mar;94(3):309-313
• Berner ES, Brooks CM, Erdmann JB. IV: Use of the USMLE to Select Residents. Acad Med. 1993;68:753-759
• Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfield BI, Laitman BM. Student Perspectives on the “Step 1 Climate” in Preclinical Medical Med. Acad Med. 2019;94:302–304.
• Cooke M, Irby DM, Sullivan W, Ludmerer KM. American Medical Education 100 Years after the Flexner Report. N Engl J Med. 2006;355(13):1339–44
• Dias RD, Gupta A, Yule SJ. Using Machine Learning to Assess Physician Competence: A Systematic Review. Acad Med. 2019 Mar;94(3):427-439
• Gauer JL,Jackson JB. Relationships of demographic variables to USMLE physician licensing exam scores: a statistical analysis on five years of Adv Med Educ Pract. 2018;10:39-44.

medical student data.

• Gliatto P, Leitman IM, Muller D. Scylla and Charybdis: The MCAT, USMLE, and Degrees of Freedom in Undergraduate Medical Education. Acad Med. 2016;91(11).
• Haider SR. Beyond USMLE Step 1. Acad Med. 2018;93(4).
• Haist SA, et al. The Evolution of the US Medical Licensing Examination (USMLE): Enhancing Assessment of Practice-Related Competencies. JAMA. 2013;310:2245-2246.
• Haist SA, Butler AP, Paniagua MA. Testing and evaluation: the present and future of the assessment of medical professionals. Adv. Physiol. Educ. 2017;41:149-153.
• Katsufrakis PJ, Chaudhry HJ. Improving Residency Selection Requires Close Study and Better Understanding of Stakeholder Needs. Acad Med. 2019;94:305–308.
• Kent JA, Patel V, Varela NA. Gender Disparities in Health Care. Mt Sinai J Med. 2012;79(5):555–9.
• Kim RH Tan TW. Interventions that affect resident performance on the American Board of Surgery In-Training Examination: A systematic review. J Surg Educ. 2015;72:418-29
• Lewis CE, et al. Numerical Versus Pass/Fail Scoring on the USMLE: What do Medical Students and Residents Want and Why? J Grad Med Educ. 2011;3:59-66.
• Marcus-Blank B et al. Predicting Performance of First-Year Residents: Correlations Between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study. Acad Med. 2019 Mar;94(3):378-387 28,
• McGaghie WC et al. Are US Medical Licensing Exam Step 1 & 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions? Acad Med. 2011;86:48-52
• Prober CG, et al. A Plea to Reassess the Role of United States Medical Licensing Examination Step 1 Scores in Residency Selection. Acad Med. 2016; 91:12-15.
• Rubright JD, et al. Examining Demographics Prior Academic Performance and United States Medical Licensing Examination Scores. Acad Med. 2019 Mar;94(3):364-370
• Williams RD, Cooper AL. Reducing Racial Inequities in Health: Using What We Already Know to Take Action. Int. J. Environ. Res. Public Health 2019, 16(4), 606;
• Zuckerman SL et al. Predicting Resident Performance from Preresidency Factors: A Systematic Review & Applicability to Neurosurgical Training. World Neurosurg. 2018;110:475-484

Education.