IAMSE Spring 2019 WAS Session 4 Highlights

[The following notes were generated by Mark Slivkoff, PhD]

IAMSE Webinar Series, Spring 2019

Speakers: David Chen
Third Year Medical Student
The University of Washington, Seattle, WA, USA

Kathryn Andolsek, MD MPH
Professor, Community and Family Medicine
Assistant Dean, Premedical Education
Duke University, Durham, NC USA

Title: Stepping Beyond the Step 1 Climate
Series: The Role of Basic Science in 21st Century Medical Education

In this week’s webinar our attention was moved away from specific curricula, as discussed during the previous two webinars, to a more global topic: Step 1 of the United States Medical Licensing Examination (USMLE). The two presenters included David Chen, a third-year medical student at the University of Washington in Seattle, Washington, and Dr. Kathryn Andolsek, a professor and assistant dean at Duke University in Durham, North Carolina. That our list of presenters included a student and experienced medical educator gave the audience a unique opportunity to hear two perspectives on the ongoing debate of the importance placed on the USMLE examinations.

Mr. Chen led off the webinar. When he went into medicine, he explained, he hoped to address health care disparities, but upon entering medical school he was surprised at how little attention was given to systemic barriers to health and social medicine. Even if they were covered in class, he and his classmates were inclined to not pay attention since the material was not relevant to USMLE Step 1 which was seen as the sole take away of the preclinical years.

He took Step 1 last year and expressed that it was “the single worst educational experience of his life”, a needless period of suffering during which self-care was neglected and burn out was real. He spent long hours memorizing the content of questionable relevance and accuracy while having physical and emotional exhaustion. Mr. Chen documented his experiences and reached out to other students and faculty members, and was invited by Academic Medicine to write a commentary on the subject (Chen et al. 2019). They wrote that the “Step 1 Climate”, a termed coined by he and his fellow authors, was a national phenomenon. It has profound effects on the learning environment, student diversity, and student well-being.

The learning environment suffers since the test-preparation materials become the de facto national curriculum, which affects students and their instructors, and the relationship between them. The examination serves as a barrier to student diversity and success of underrepresented minorities, women, and students from low-income families. Furthermore, as Mr. Chen explained, the examination sets up “tiered worthiness” among medical students in that Step 1 scores define the competitiveness for residencies and their prestige. Family medicine, for example, sits at the low end of the scale which may deter students from applying. These factors all lead to ill effects on student well-being by increasing depression, burnout, and in extreme cases, suicide.

Mr. Chen continued his discussion by addressing an article which was a rebuke to theirs (Katsufrakis and Chaudhry 2019). Briefly, he pointed out that despite their claim there have been studies which have linked residency success to Step 1 performance (Prober et al 2016; McGaghie et al. 2011; Gliatto et al. 2016). Mr. Chen also expressed that he is unsure if the authors have a conflict of interest, but he does rightly claim that taking the boards is expensive.

A group of stakeholders who need more attention in this debate is the patients. Board examinations, as is commonly claimed, help ensure quality care for patients. Mr. Chen argues that given the diversity of patients, can a multiple-choice examination truly measure competency?

The last point that Mr. Chen makes is that the time is ripe for change. Over 100 years ago, the Flexner Report was published (in 1910) and about 70 years ago the first multiple-choice NBME exam was given (in 1916). The world is much different today, and “The need for a fundamental redesign of the content of medical training is clear” (Cooke et al 2006). Mr. Chen commented that he is not sure exactly what the barriers to change are, then parted with a final note: it is an ominous sign for patients who interact with the medical establishment if we in medical education are unable to address our problems.

Dr. Andolsek then took over the screen and the webinar. She began by adding some additional details to Mr. Chen’s brief historical account of Step 1. The main role of early examinations was to serve as a common licensing examination such that physicians could practice across state borders. It was therefore initially designed as a pass/fail assessment. Dr. Andolsek reiterated that the examination is expensive, usually with related costs (Step 1 resources) greater than $800. The examination correlates with performance on subsequent examinations (such as board examinations) but not on residency progression, faculty evaluations, or clinical skills. Furthermore, groups including women, historically underrepresented persons, non-traditional students, and those with financial need perform “differently” on the examination, usually less well.

Stakeholders of Step 1, she continued, include medical students, medical licensing boards, medical schools, and program directors. Similar to Mr. Chen’s comments, Dr. Andolsek questioned patients as belonging to the stakeholder group. She mentioned that in her 40 years of practice a patient never asked her how she did on the boards.

Medical students have responded to Step 1 in that they recognize that it is the de facto curriculum. The use it to judge whether a medical school’s curriculum is relevant if it’s “high yield” and linked to Step 1 content. If it’s not, then curricular content has a tendency to get deemed as uninteresting and devalued in their medical education.

The majority stakeholders, Dr. Andolsek continued, should be the medical licensing boards since licensing was the original intent of the examination. Most states have it written into their legislation, however, the actual number of the examination is never mentioned, only if it was passed or failed. Thus, this adds to the argument that the examination should be a pass/fail assessment.  

Many medical schools use the Step 1 to determine if students are ready to enter clinical education, although this was not an intended use for the examination. Some schools also use the scores to determine gaps in their curriculum, and to determine how likely their students will “match.”

The next group which she addressed in more detail are the program directors who use Step 1 scores to identify which medical students will get full review of their application and interviewed. Some program directors use “cut off scores” frequently set at < 220-240, with higher scores for more competitive specialties. Dr. Andolsek added that 90% of residency graduates who passed Step 1 with lower scores (200-227) passed their board examinations in fairly competitive specialties, demonstrating that passing correlates with passing. Dr. Andolsek also made the point that many good residency applicants are passed up by directors since their applications are ignored when certain scores are not attained. The programs are also judged by the ACGME, the residency accrediting body, which has historically used board passing rates as one measure to judge a program’s quality. A positive change is underway, as Dr. Andolsek noted, in that the ACGME has changed the threshold for passing rates.

Program directors need More relevant measures to assess and communicate medical school performance. They need some type of test or mechanism which may include ACGME competencies and milestones, core Entrustable Professionable Activities (EPAs) and any or all tools developed for resident selection that correlates with performance.

Dr. Andolsek concluded with a polling question sent out to schools. The question asked was should the Step 1 be pass/fail? The results?

• Yes: 71%
• Undecided: 16%
• No: 14%

The session was then opened up for questions, which lasted well over 20 minutes indicating the provocativeness of their webinar.  

References:

• Andolsek KM. One Small Step for Step 1. Acad Med. 2019 Mar;94(3):309-313
• Berner ES, Brooks CM, Erdmann JB. IV: Use of the USMLE to Select Residents. Acad Med. 1993;68:753-759
• Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfield BI, Laitman BM. Student Perspectives on the “Step 1 Climate” in Preclinical Medical Med. Acad Med. 2019;94:302–304.
• Cooke M, Irby DM, Sullivan W, Ludmerer KM. American Medical Education 100 Years after the Flexner Report. N Engl J Med. 2006;355(13):1339–44
• Dias RD, Gupta A, Yule SJ. Using Machine Learning to Assess Physician Competence: A Systematic Review. Acad Med. 2019 Mar;94(3):427-439
• Gauer JL,Jackson JB. Relationships of demographic variables to USMLE physician licensing exam scores: a statistical analysis on five years of Adv Med Educ Pract. 2018;10:39-44.

medical student data.

• Gliatto P, Leitman IM, Muller D. Scylla and Charybdis: The MCAT, USMLE, and Degrees of Freedom in Undergraduate Medical Education. Acad Med. 2016;91(11).
• Haider SR. Beyond USMLE Step 1. Acad Med. 2018;93(4).
• Haist SA, et al. The Evolution of the US Medical Licensing Examination (USMLE): Enhancing Assessment of Practice-Related Competencies. JAMA. 2013;310:2245-2246.
• Haist SA, Butler AP, Paniagua MA. Testing and evaluation: the present and future of the assessment of medical professionals. Adv. Physiol. Educ. 2017;41:149-153.
• Katsufrakis PJ, Chaudhry HJ. Improving Residency Selection Requires Close Study and Better Understanding of Stakeholder Needs. Acad Med. 2019;94:305–308.
• Kent JA, Patel V, Varela NA. Gender Disparities in Health Care. Mt Sinai J Med. 2012;79(5):555–9.
• Kim RH Tan TW. Interventions that affect resident performance on the American Board of Surgery In-Training Examination: A systematic review. J Surg Educ. 2015;72:418-29
• Lewis CE, et al. Numerical Versus Pass/Fail Scoring on the USMLE: What do Medical Students and Residents Want and Why? J Grad Med Educ. 2011;3:59-66.
• Marcus-Blank B et al. Predicting Performance of First-Year Residents: Correlations Between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study. Acad Med. 2019 Mar;94(3):378-387 28,
• McGaghie WC et al. Are US Medical Licensing Exam Step 1 & 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions? Acad Med. 2011;86:48-52
• Prober CG, et al. A Plea to Reassess the Role of United States Medical Licensing Examination Step 1 Scores in Residency Selection. Acad Med. 2016; 91:12-15.
• Rubright JD, et al. Examining Demographics Prior Academic Performance and United States Medical Licensing Examination Scores. Acad Med. 2019 Mar;94(3):364-370
• Williams RD, Cooper AL. Reducing Racial Inequities in Health: Using What We Already Know to Take Action. Int. J. Environ. Res. Public Health 2019, 16(4), 606;
• Zuckerman SL et al. Predicting Resident Performance from Preresidency Factors: A Systematic Review & Applicability to Neurosurgical Training. World Neurosurg. 2018;110:475-484

Education.