[The following notes were generated by Michele Haight, PhD.]

Arianne Teherani PhD, Professor of Medicine and Education Scientist, Director for Program Evaluation and Education Continuous Quality Improvement,  Karen Hauer, MD, PhD, Clinical Professor, Associate Dean for Assessment,  University of California San Francisco School of Medicine

Achieving Equity in Assessment for Clinical Learners

Equity in assessment  occurs when all learners have fair and impartial opportunities to learn, be coached and receive feedback, be assessed and graded, be advanced and graduated, and be selected for subsequent training and job opportunities. (Lucey et al. Academic Med 2020)

Equity in clinical education is a relatively unexplored area. There is not a singular definition of equity. Equity is built upon the following principles:

• Acknowledgment that differences are not deficits
• Supporting the success of all learners
• Implementation of multiple types of assessments
• Holding all learners to the same high standards
• Moral, legal, political, economic and social dimensions
• What constitutes an equitable outcome?

Conceptual framework for thinking about equity

• Closing achievement gaps (key outcomes we desire to achieve)
• Equal treatment (adequacy and opportunity)
• Fairness (personal or social status are not obstacles to achieving educational potential) and Inclusion (ensuring a standard of education for all)
• Justice

What does it mean to have equity in assessment?

Core ideas of two prevailing equity in assessment  frameworks (Gibbs and Lucey) have been organized into one framework that is focused on macro and micro level solutions. Both levels of solution need to be addressed to propel change.

• Macro level of solution (procedures/policies, top down mandate)
• Micro level of solution (occurs in micro settings/one on one interaction)

MACRO

• Intrinsic Equity (assessment practices are just and fair for all groups and minimize bias)
• Instrumental Equity (results are shared and used to create opportunity; interpretation of results is just and fair)
• Definition of Achievement (takes into account how achievement is defined; what does the institution and the teacher consider achievement; what are the knowledge and skills that are equated with achievement? )

MICRO

• Contextual Equity (learning environment factors, teachers’ beliefs, fairness in the learning environment, bias-free learning environment)
• Multifactorial (learner identity, self-esteem, motivation, expectations for success)

Key Evidence about Equity and Clinical Performance Assessment

• Differential attainment addresses how educational outcomes diverge for different demographic groups which are assessed in the same way
• Studies show differences in clinical performance assessments favor non-UIM (underrepresented in medicine) learners.
• Amplification Cascade shows how a small difference in assessed performance leads to larger differences in grades and selection for awards.
• A UCSF study of its own learners showed evidence of an amplification cascade for UIM learners and demonstrated that academic outcomes differences favor non UIM learners throughout their medical education experience.
• UCSF conducted a Root Cause Analysis to assess what happened, why it happened and how it can be prevented. Results indicated the following:

1. Factors that caused the differences were personal, interpersonal, cultural and structural.

2. Personal/Interpersonal factors that included stereotyping, microaggressions, unconscious biases, etc. disproportionately impacted UIM learners.

3. Cultural/Structural factors embedded in institutional policies (e.g., normative vs. criterion-based grading) and present in the clinical learning environment (e.g., rapidly changing team assignments) disproportionately impacted UIM learners.
4. These impacts resulted in fewer honors grades and awards for UIM learners and affected UIM residency specialty choices and the potential for academic career choices.

To address equity in clinical performance, there needs to be an integrated, systems approach to change which focuses on the following:

• redesign of medical school curricula
• redesign of learning and assessment methods
• faculty development and training programs to identify and avoid bias
• use of frequent observations and feedback as a best practice for assessment
• use of narrative assessments over time
• avoidance of comparing  learners

Countermeasures and Solutions implemented by UCSF

• Advocacy and Health Equities curriculum in Internal Medicine clerkship
• Altered clerkship assessment practices (e.g., grading committees)
• Re-envisioned honors societies’ membership processes
• Faculty training to recognize microaggressions and bias, especially in the learning environment
• Reinforce Educational CQI habit of mind; establish and implement Equity Evaluation Plan.

The clinical learning environment is complex and raises questions about equity.

Concerns about bias found UIM bias affects all aspects of assessment. Non UIM learners were rated slightly higher than UIM.

• Bias is not merely an individual problem; it is a systemic problem.
• Many biases are harmful.
• It is not possible to remove all bias.
• Bring awareness to biases so we can make a choice to avoid acting with harmful discrimination.

We need to design systems that allow all students to learn and become best. This requires a cultural change in medicine.

Factors to consider in determining a clerkship grade:

• Did team members observe the student contributing to patient care?
• Did the learning environment allow learners to perform at their best?
• Do learners and team members share a common understanding of expectations?
• What structural factors in learners’ educational experiences before and during medical school might lead to differences in grades?
• What process does clerkship director use to determine final grade?

There are three components of equity in assessment:

• Intrinsic equity : the way one designs and conducts assessment practices.
• Contextual equity: addresses the environment in which learning and assessments occur.
• Instrumental equity: how use assessment information such as  grades, to determine selection for next opportunity or honors and awards.

Intrinsic Equity:

Older Model of Assessment Practices
Purpose: Classify and rank learners; some students learn more and better. There is  little feedback; promotes fixed mindset
Learner ability: Some will learn more and be better
Feedback:  infrequent, correct/incorrect
Progress: Time-based
Context:  De-contextualized
Mindset: Fixed

Current Model of Mastery Assessment Practices
Purpose: Promote and shape learning (all learners have capacity to learn and succeed)
Learner ability: All learners have the capacity to learn
Feedback:  Frequent, immediate and actionable
Progress: Based on progress map/competency-driven
Context: Real world/interact with patients
Mindset: Growth

We need to challenge faculty to think differently to overcome old assumptions and adopt a growth mindset.

Characteristics of a Growth Mindset in Assessment

• Recognizes and rewards growth and improvement.
• Rethinks the cultural affinity for excellence; the view of excellence in medicine is that a few learners and a few physicians are truly the top and we can identify those. This is a faculty myth based on working with learners for a few days or a week.
• Reframes meritocracy. Not all learners start at same point or have the same experiences. In the learning environment, with attention all learners can be highly successful.
• Establishes a non-competitive learning environment; changes the learner focus from keeping achievement higher than peers to individual focus on how to be better tomorrow than today.

UCSF Interventions:

• Faculty Diversity, Equity and Inclusion Champion Training
• Website for Assessment Guidelines and Equity Checklist for Assessment  https://meded.ucsf.edu/faculty-educators/equity-assessment-guidelines-and-checklist Faculty are encouraged to continue to reflect upon their own biases, and observe learners to better understand all the ways they are contributing to patient care.
• Changes to clerkship assessments: Establish grading committees, eliminate core clerkship honors grades in third year so learners have a year to learn, grow and receive feedback.

Contextual Equity: (Learning and Assessment Environment)

• Environment offers opportunities for all students participate and learn.
• Faculty preparation including allyship against stereotype threat and microaggressions.

Factors influencing performance of UIM learners:

• Stereotype threat: phenomenon in which performance is impaired due to fear of confirming negative stereotypes.
• Microaggressions (can emanate from patients, team members, observations and images)
• Microaggressions can appear as:
  • Microassaults
  • Microinsults
  • Microinvalidations
  • Environmental Images

Bystander Intervention Guide

• Pre brief : anticipate that microaggressions and will occur; engage learners in how they would like a response.
• Recognize and analyze microaggression.
• Consider type of response, e.g., bear witness, intervene with patient, pause, exit room.
• Respond in real time.
• Check in with learner immediately.
• Follow up (based on learner preference).

Instrumental Uses of Assessments: Use of assessments for purposes for which they are not designed. For example, clerkship exams are intended to assess and ensure minimum knowledge competency at the end of the clerkship, but are used to determine clerkship grades, AOA selection or residency selection.

UCSF changed its selection practice for AOA away from metrics to  a holistic process in which learners submit an application and describe how they feel they meet the criteria for AOA, which center on serving the sick.  UCSF no longer participates in AOA due to a conflict of values.

Changing assessments in medical schools raises questions about residency selection. Many national organizations are working to improve the residency application process to ensure it promotes learning across the continuum rather than maintaining a focus on a high stakes selection process.

Recommendations for achieving equity in assessments:

• Examine your own data.
• Design your assessment to fit your purpose.
• Continuously improve assessment.
• Think of assessment as a system.
• Consider the intrinsic, contextual and instrumental aspects of equity in assessment.

[The following notes were generated by Michele Haight, PhD.]

Karen Eley Sanders, MS, EdD, Vice Provost for College Access, Charlotte Baker DrPH, MPH, CPH, Richard Vari, PhD, Senior Dean for Academic Affairs, Joanne Greenawald MD, Director of Problem-Based Learning, Emily M. Holt Forest, MA, Director Counseling and Academic Enrichment Services, Virginia Tech Carilion School of Medicine

Strategies to Recognize and Address Implicit or Explicit Bias in Small Group Teaching

Dr. Eley Sanders presented a video of a small group learning session in which an administrator and medical student actors portrayed a small group learning facilitator and group participants respectively. The video showed group introductions for a problem-based learning  opening session. Dr. Eley Sanders encouraged the webinar participants to consciously reflect upon their impressions, thoughts and feelings during the video presentation.

Upon completion of the video, Dr. Eley Sanders conducted a debriefing session in which she initiated a discussion on “microaggressions.”  She described a system of bias. She explained how stereotypes can lead to prejudice. Acting upon prejudice results in discrimination. Microaggressions are an extension of discrimination, prejudice and bias.

• Bias is an inclination for or against something.
• Stereotypes are generalizations about a person which do not permit for individual differences or variations.
• Prejudice is an opinion, pre-judgment or attitude about a group or its members.
• Discrimination is behavior that treats people unequally because of their group memberships. This often begins with negative stereotypes and prejudices.
• D. Wing Sue classifies microaggressions as “ brief, everyday exchanges that send denigrating messages to certain individuals because of their group membership.”

Dr. Eley Sanders described eight different categories of microaggressions:

• Alien in One’s Own Land
• Second Class Citizen
• Ascription of Intelligence
• Pathologizing Values/Communications Styles
• Color Blindness
• Assumption of Criminal Status
• Denial of Racism/Sexism/Homophobia/Transphobia
• Myth of Meritocracy

Microaggressions can have short term and long terms impacts such as the following:

• Feeling of “onlyness” and loneliness
• Physical avoidance
• Disengaging
• Physiological stress
• Anxiety and/or Depression
• Sleep Difficulties
• Inability to Focus
• Dropping Out
• Patients who experience microaggressions are less likely to follow through on treatment plans, and less likely to return to a medical professional who shows bias.

Dr. Eley Sanders noted that the persons committing microaggressions might be otherwise well-intentioned and unaware of the impact of their words, but this does not minimize the impact of their words.

Dr. Charlotte Baker guided the group to view the video again, pausing to analyze each example of a microaggression. She provided alternative comments and strategies to avoid inappropriate comments. She noted that some people might not be offended by certain comments, but she cautioned that this might not always be the case.

Dr. Baker reviewed other situations that commonly occur in medical education and can lend themselves to microaggressions.

Drs. Eley Sanders and Baker concluded the session by responding to questions posed by the webinar participants.

1. How do you recommend a bystander respond to a microaggression that has been observed in the classroom? Simply say “Ouch” or “Wow.” These responses provide the opportunity for discussion. Give everyone the opportunity to learn from the incident.
2. Is it appropriate to acknowledge that you might have difficulty pronouncing someone’s name, but you will make every effort to learn how to pronounce it? Yes, acknowledge this and you might work with the student to perfect your pronunciation.
3. Participants noted that the examples in the video were more “macro” than “micro” aggressions. Eley Sanders stated that this is a correct observation and clarified that microaggressions can be further broken down to micro-assaults, micro-insults and micro-invalidations.
4. Baker noted that it is important to establish relationships with students, so we need to be constantly aware of “how” we ask questions.
5. What are your recommendations for navigating the student/faculty power differential in conversations? Create a “safe space” in the learning environment for learners to have these types of discussions. Acknowledge and apologize to the learner for any missteps and apologize in the group (if this is where the misstep occurred) in order to role model the behavior we seek from our learners.
6. Should students be called out for committing microaggressions? Absolutely yes. We have to help them learn appropriate behaviors and engagement.
7. Baker and Eley Sanders recommended reading the following article: “My Name is Not Interpreter,” Montenegro, R. JAMA, 2020, 323(17):1700-1701. https://doi.org/10.1001/jama.2020.2976

[The following notes were generated by Michele Haight, PhD.]

Amy Caruso Brown MD, MSc, MSCS, HEC-C , Associate Professor of Bioethics and Humanities and Associate Professor of Pediatrics, SUNY Upstate Medical University, Syracuse, New York, USA  brownamy@upstate.edu

Creating Diverse, Equitable and Inclusive Content in Medical Education

Diversity, equity, inclusion and bias in health sciences education is a compendium of many different perspectives.

Addressing issues related to bias requires a shared definition for commonly used terms. These terms can be positive, negative or both.

• Prejudice: Preconceived opinion that is not based on reason or experience.
• Stereotype: Association of a member of a group with a consistent set of traits.
• Bias: Inclination or prejudice for or against something
• Implicit or Unconscious Bias: Relatively unconscious and relatively automatic features of prejudiced judgment and social behavior; contrasted with explicitly endorsed beliefs. Implicit to Explicit represents a spectrum, not a dichotomy.

Bias in health sciences education can be placed into four categories:

• Who?  Who is admitted to medical school? Who is hired to teach?
• What?  What content is presented to learners?
• How #1? How are learners taught, mentored and evaluated?
• How #2? How do learners see health professionals treat patients and families?

There are four categories of bias within the area of content:

• Scientifically inaccurate information.
  • Promotion of race as a biological as opposed to a social construct, racial essentialism.

• Overemphasis of certain topics; Neglect of other topics
  • Unequal distribution of time in pharmacology dedicated to men’s and women’s reproductive issues.
  • Disproportionate representation of white skin and white able-bodied images.

• Promotion of bias, shame and stigma towards people belonging to certain groups or with certain medical conditions.
  • E.g., fat-shaming, poverty=laziness, all patients who speak Spanish are undocumented workers.

• Irrelevant to medicine and promotes bias and discrimination
  • Humor and microaggressions

What doe bias look like on multiple-choice exams?

• Diseases associated with specific races
• Male and female stereotypes
• Instantaneous judgments of patients with certain characteristics
• Epidemiology without context
• Use of race as a proxy for genetics
• Absence of white privilege
• Race as meaningful for only non-white identities
• Race is represented negatively and associated with disparities without mention of community strengths

Bias in content and the learning environment is costly:

• Lowers faculty productivity and student/faculty retention
• Reduces quality of care

Our biases are mitigated by our:

• Positionality
• Discomfort with discussing racial issues
• Lack of real-world data
• Lack of time and effort for closer examination

Current curricular and assessment paradigms in health sciences education make it difficult to identify biases.

Dr. Caruso Brown has developed a checklist instrument for identifying content biases; this tool is publicly available at https://tinyurl.com/UpstateBiasCheklist

Guiding Questions for examining content include the following:

1. Why is the content at risk for bias?
2. How might it impact learners?
3. What is the goal/learning objective for this part of the content? Why is the content there in the first place?
4. Should it be changed?
5. How should it be changed?

There are four recommended methods for changing content:

• Remove content
• Replace content with images that challenge stereotypes
• Add content
• Apologize (last resort)

Mapping the attributes of hypothetical patients used in the curriculum via the spreadsheet provided in the checklist tool provides an opportunity for a systematic review of potential biases across the curriculum. This type of review also helps to identify the pervasiveness of biases across the curriculum and provides the opportunity to enhance equity within the curriculum.

Addressing Bias in Content Development

1. Offer faculty and students a session unpacking race as a social construct before they start learning from case studies or real patients.
2. Use tables to track representation over entire courses and curricula. Regularly reassess for over or under representation. (Equity mapping)
3. Practice counter-stereotypic imaging.
4. Incorporate discussion of structural and social determinants of health into cases including a discussion of a patient’s relative vulnerability or privilege based on the patient’s racial classification.
5. When race and stereotype may align within the case, faculty should talk explicitly about that and promote stereotype replacement, individuating and perspective-taking.
6. Consider introducing the patient’s race in a paragraph with additional information about the patient’s family and social and cultural background, rather than in the first sentence.

Pros and Cons for Including Race in Case Development:

Pros

• People notice race
• Racial and ethnic identities and experiences matter and can impact care
• Learners often make assumptions about race if it is not mentioned

Cons

• It is easy to fall back on race as proxy for genetics
• Faculty are not trained to view racism as a risk factor
• Faculty do not want to write about rare combinations

[The following notes were generated by Andrea Belovich, PhD.]

In response to the enthusiastic reception of the IAMSE Winter 2021 WAS series, “The USMLE Step 1 is Going to Pass/Fail, Now what do we do?,” an additional webinar was presented by Dr. Elise Lovell, Clinical Professor of the University of Illinois at Chicago and Co-Chair of the Coalition for Physician Accountability UME-GME Review Committee (UGRC) on February 25^th, 2021. During this sixth and final webinar, made available free-of-charge to registrants, Dr. Lovell provided a cutting-edge update of the UGRC’s work to review the transition from Undergraduate Medical Education (UME) to Graduate Medical Education (GME).

Dr. Lovell began by introducing the composition and mission of the Coalition for Physician Accountability. The Coalition is a collaboration between the national organizations responsible for the education, assessment, accreditation, certification, and licensure processes ranging from UME to practicing physicians. (A full list of member organizations is available on the Coalitions’ website [1].) As a collective, the Coalition’s mission is “to advance health care and promote professional accountability by improving the quality, efficiency, and continuity of the education, training and assessment of physicians” [2].

To provide the audience with context for the development of the Coalition’s UGRC, Dr. Lovell shared the four recommendations resulting from the Invitational Conference on USMLE Scoring (InCUS) in March 2019:

• Consider score reporting changes, e.g., pass/fail; composite score; categorical/tiered scoring for USMLE
• Research on how USMLE is (or is not) related to performance in residency and/or practice
• Continue work to address group (racial/demographic) differences in USMLE
• Undertake comprehensive overview of UME-GME transition system

Dr. Lovell emphasized that while the first recommendation regarding the USMLE Step 1 P/F scoring has received significant attention from the medical education community, the InCUS recognized that the USMLE Step 1 numerical score was merely one factor of many that contributes to the broader problems surrounding the UME-to-GME transition. After briefly discussing the progress being made towards meeting the first three recommendations, Dr. Lovell focused on the fourth InCUS recommendation: the review and overhaul of the current UME-GME transition system. This review process was anticipated to involve eight major areas of focus: 1) Improving the application process, 2) Reducing the number of applications submitted by applicants, 3) Improving transparency of the UME-GME transition process, 4) Identifying potential adjustments to the Match, 5) Improving Program Directors’ ability to more holistically review applicants, 6) Improving the trust/transparency of medical school-based assessments, 7) Reviewing the role of standardized testing in UME-GME transition, and 8) Developing assessments for other important measures beyond knowledge.

As the InCUS developed the recommendation to review the UME-to-GME transition, the Coalition for Physician Accountability was identified as the organization best suited to addressing the task. In September of 2019, the Coalition formed a Planning Committee (PC) to identify the construct and membership of the UME-GME Review Committee (UGRC), develop a process for selecting UGRC membership/leadership, and determine key questions, deliverables, milestones, timeline, and budget for the UGRC.

As a result of the PC’s efforts, membership of the UGRC is comprised of representatives from each of the Coalition member organizations, as well as medical students, resident physicians, members of the public, and support staff. Along with Dr. Lovell, Dr. George Mejicano was selected to co-chair the UGRC. Areas of focus for the UGRC were identified to include fit of applicants for residency programs based on abilities and interests, increase of trust between medical schools and residency programs, and mitigation of reliance on licensing exams during the application process. In addition, the UGRC’s focus encompasses ensuring fairness for all types of applicants (allopathic, osteopathic, and international medical graduates) while considering cost and the potential for systemic bias and diversity issues throughout the transition from UME to GME. The broader scope of the UGRC was also determined to include consideration of specialty-specific competencies, including when and how specialty-specific preparatory training for residency should occur, as well as where responsibility should rest in resolving any gaps in residency readiness.

Once established, the UGRC created four work groups to address the charges from the Coalition’s Planning Committee, and added their own overall considerations of diversity, equity, inclusion, fairness, student wellbeing, and the public good. The work groups were also asked to proactively anticipate outcome assessment of their recommendations and to plan for research for continual modification to support the formation of an iterative review process of the UME-GME transition going forward.

Work Group A’s focus is on ensuring residency readiness by identifying competencies students should be learning during UME and improving the advising process as students choose residencies for application. Work Group B’s charge is to improve the mechanics of the application/election process from the UME perspective, which includes information sharing by medical schools with residency programs, residency application content, and application mechanics (e.g., away rotations, etc.). Work Group C’s task is to improve the mechanics of the application/election process from the GME perspective, including information sharing by residency programs with medical schools, interview processes, and the match process with an emphasis on considering financial and opportunity costs to students. Work Group D is concerned with post-match optimization and improving the “soft transition” into residency during the fourth year of medical school. In particular, this group is addressing how to ensure that junior residents are prepared to meet specialty-specific competencies and how responsibility for any readiness gaps should be shared in the interest of patient safety and students’ professional development. This process is anticipated to involve information sharing/feedback from the residency program back to UME in order to more successfully educate and prepare future students.

Dr. Lovell then provided an overview of the UGRC’s process and work-to-date. Following up from a well-planned and intentional virtual brainstorming session held January 14^th and 15^th, 2021, the UGRC’s work groups have achieved/established the following: 1) Communication framework including bi-directional Stakeholder engagement; 2) Central Repository for reference literature; 3) Work Group level-setting (defined present status of UME-GME transition and data resources); 4) Root cause analysis of current problems; 5) Consideration of “what’s good/worth keeping” from current state; 6) Envisioned success, including measurement of outcomes; 7) Solicited Stakeholder visions of the transition’s future Ideal State.

Dr. Lovell then invited audience members to follow UGRC meeting updates on physicianaccountability.org, and provided an expected timeline for the delivery of the URGC’s recommendations, which are currently being built. These draft recommendations will be reviewed by the UGRC in March 2021, and made available for public review in April 2021. Final UGRC recommendations are expected to be delivered to the Coalition for Physician Accountability in May 2021.

The URGC is currently seeking UME-to-GME innovation input from stakeholders, which will be accepted through March 12^th, 2021 [3].

References:

1. Coalition for Physician Accountability. https://physicianaccountability.org/members/
2. Coalition for Physician Accountability. https://physicianaccountability.org/about/
3. “UME-to-GME Review Committee (UGRC) Seeks Input on “Innovations” being Used in UME-to-GME Transition.” Coalition for Physician Accountability. https://physicianaccountability.org/wp-content/uploads/2021/02/UGRC-Innovations-Invitation-1.pdf