IAMSE Spring 2021 Session 3 Highlights

[The following notes were generated by Michele Haight, PhD.]

Arianne Teherani PhD, Professor of Medicine and Education Scientist, Director for Program Evaluation and Education Continuous Quality Improvement,  Karen Hauer, MD, PhD, Clinical Professor, Associate Dean for Assessment,  University of California San Francisco School of Medicine

Achieving Equity in Assessment for Clinical Learners

Equity in assessment  occurs when all learners have fair and impartial opportunities to learn, be coached and receive feedback, be assessed and graded, be advanced and graduated, and be selected for subsequent training and job opportunities. (Lucey et al. Academic Med 2020)

Equity in clinical education is a relatively unexplored area. There is not a singular definition of equity. Equity is built upon the following principles:

• Acknowledgment that differences are not deficits
• Supporting the success of all learners
• Implementation of multiple types of assessments
• Holding all learners to the same high standards
• Moral, legal, political, economic and social dimensions
• What constitutes an equitable outcome?

Conceptual framework for thinking about equity

• Closing achievement gaps (key outcomes we desire to achieve)
• Equal treatment (adequacy and opportunity)
• Fairness (personal or social status are not obstacles to achieving educational potential) and Inclusion (ensuring a standard of education for all)
• Justice

What does it mean to have equity in assessment?

Core ideas of two prevailing equity in assessment  frameworks (Gibbs and Lucey) have been organized into one framework that is focused on macro and micro level solutions. Both levels of solution need to be addressed to propel change.

• Macro level of solution (procedures/policies, top down mandate)
• Micro level of solution (occurs in micro settings/one on one interaction)

MACRO

• Intrinsic Equity (assessment practices are just and fair for all groups and minimize bias)
• Instrumental Equity (results are shared and used to create opportunity; interpretation of results is just and fair)
• Definition of Achievement (takes into account how achievement is defined; what does the institution and the teacher consider achievement; what are the knowledge and skills that are equated with achievement? )

MICRO

• Contextual Equity (learning environment factors, teachers’ beliefs, fairness in the learning environment, bias-free learning environment)
• Multifactorial (learner identity, self-esteem, motivation, expectations for success)

Key Evidence about Equity and Clinical Performance Assessment

• Differential attainment addresses how educational outcomes diverge for different demographic groups which are assessed in the same way
• Studies show differences in clinical performance assessments favor non-UIM (underrepresented in medicine) learners.
• Amplification Cascade shows how a small difference in assessed performance leads to larger differences in grades and selection for awards.
• A UCSF study of its own learners showed evidence of an amplification cascade for UIM learners and demonstrated that academic outcomes differences favor non UIM learners throughout their medical education experience.
• UCSF conducted a Root Cause Analysis to assess what happened, why it happened and how it can be prevented. Results indicated the following:

1. Factors that caused the differences were personal, interpersonal, cultural and structural.

2. Personal/Interpersonal factors that included stereotyping, microaggressions, unconscious biases, etc. disproportionately impacted UIM learners.

3. Cultural/Structural factors embedded in institutional policies (e.g., normative vs. criterion-based grading) and present in the clinical learning environment (e.g., rapidly changing team assignments) disproportionately impacted UIM learners.
4. These impacts resulted in fewer honors grades and awards for UIM learners and affected UIM residency specialty choices and the potential for academic career choices.

To address equity in clinical performance, there needs to be an integrated, systems approach to change which focuses on the following:

• redesign of medical school curricula
• redesign of learning and assessment methods
• faculty development and training programs to identify and avoid bias
• use of frequent observations and feedback as a best practice for assessment
• use of narrative assessments over time
• avoidance of comparing  learners

Countermeasures and Solutions implemented by UCSF

• Advocacy and Health Equities curriculum in Internal Medicine clerkship
• Altered clerkship assessment practices (e.g., grading committees)
• Re-envisioned honors societies’ membership processes
• Faculty training to recognize microaggressions and bias, especially in the learning environment
• Reinforce Educational CQI habit of mind; establish and implement Equity Evaluation Plan.

The clinical learning environment is complex and raises questions about equity.

Concerns about bias found UIM bias affects all aspects of assessment. Non UIM learners were rated slightly higher than UIM.

• Bias is not merely an individual problem; it is a systemic problem.
• Many biases are harmful.
• It is not possible to remove all bias.
• Bring awareness to biases so we can make a choice to avoid acting with harmful discrimination.

We need to design systems that allow all students to learn and become best. This requires a cultural change in medicine.

Factors to consider in determining a clerkship grade:

• Did team members observe the student contributing to patient care?
• Did the learning environment allow learners to perform at their best?
• Do learners and team members share a common understanding of expectations?
• What structural factors in learners’ educational experiences before and during medical school might lead to differences in grades?
• What process does clerkship director use to determine final grade?

There are three components of equity in assessment:

• Intrinsic equity : the way one designs and conducts assessment practices.
• Contextual equity: addresses the environment in which learning and assessments occur.
• Instrumental equity: how use assessment information such as  grades, to determine selection for next opportunity or honors and awards.

Intrinsic Equity:

Older Model of Assessment Practices
Purpose: Classify and rank learners; some students learn more and better. There is  little feedback; promotes fixed mindset
Learner ability: Some will learn more and be better
Feedback:  infrequent, correct/incorrect
Progress: Time-based
Context:  De-contextualized
Mindset: Fixed

Current Model of Mastery Assessment Practices
Purpose: Promote and shape learning (all learners have capacity to learn and succeed)
Learner ability: All learners have the capacity to learn
Feedback:  Frequent, immediate and actionable
Progress: Based on progress map/competency-driven
Context: Real world/interact with patients
Mindset: Growth

We need to challenge faculty to think differently to overcome old assumptions and adopt a growth mindset.

Characteristics of a Growth Mindset in Assessment

• Recognizes and rewards growth and improvement.
• Rethinks the cultural affinity for excellence; the view of excellence in medicine is that a few learners and a few physicians are truly the top and we can identify those. This is a faculty myth based on working with learners for a few days or a week.
• Reframes meritocracy. Not all learners start at same point or have the same experiences. In the learning environment, with attention all learners can be highly successful.
• Establishes a non-competitive learning environment; changes the learner focus from keeping achievement higher than peers to individual focus on how to be better tomorrow than today.

UCSF Interventions:

• Faculty Diversity, Equity and Inclusion Champion Training
• Website for Assessment Guidelines and Equity Checklist for Assessment  https://meded.ucsf.edu/faculty-educators/equity-assessment-guidelines-and-checklist Faculty are encouraged to continue to reflect upon their own biases, and observe learners to better understand all the ways they are contributing to patient care.
• Changes to clerkship assessments: Establish grading committees, eliminate core clerkship honors grades in third year so learners have a year to learn, grow and receive feedback.

Contextual Equity: (Learning and Assessment Environment)

• Environment offers opportunities for all students participate and learn.
• Faculty preparation including allyship against stereotype threat and microaggressions.

Factors influencing performance of UIM learners:

• Stereotype threat: phenomenon in which performance is impaired due to fear of confirming negative stereotypes.
• Microaggressions (can emanate from patients, team members, observations and images)
• Microaggressions can appear as:
  • Microassaults
  • Microinsults
  • Microinvalidations
  • Environmental Images

Bystander Intervention Guide

• Pre brief : anticipate that microaggressions and will occur; engage learners in how they would like a response.
• Recognize and analyze microaggression.
• Consider type of response, e.g., bear witness, intervene with patient, pause, exit room.
• Respond in real time.
• Check in with learner immediately.
• Follow up (based on learner preference).

Instrumental Uses of Assessments: Use of assessments for purposes for which they are not designed. For example, clerkship exams are intended to assess and ensure minimum knowledge competency at the end of the clerkship, but are used to determine clerkship grades, AOA selection or residency selection.

UCSF changed its selection practice for AOA away from metrics to  a holistic process in which learners submit an application and describe how they feel they meet the criteria for AOA, which center on serving the sick.  UCSF no longer participates in AOA due to a conflict of values.

Changing assessments in medical schools raises questions about residency selection. Many national organizations are working to improve the residency application process to ensure it promotes learning across the continuum rather than maintaining a focus on a high stakes selection process.

Recommendations for achieving equity in assessments:

• Examine your own data.
• Design your assessment to fit your purpose.
• Continuously improve assessment.
• Think of assessment as a system.
• Consider the intrinsic, contextual and instrumental aspects of equity in assessment.