The Standardized Patient Assessment Examination: Integration with the Basic Sciences Curriculum

INTRODUCTION

In August of 1998, Wake Forest University School of Medicine (WFUSM) initiated a new curriculum: Prescription for Excellence: A Physician’s Pathway to Lifelong Learning. This curriculum incorporated the best aspects of the traditional lecture-based and problem-based learning tracks that existed since the creation of the problem-based curriculum in 1987. The goals of the new curriculum are to foster:

• the development of professional attitudes and behaviors
• core biomedical science knowledge
• clinical skills
• problem solving/clinical reasoning skills
• communication skills
• self-directed learning/lifelong skills
• information management skills

The development of the new curriculum occurred over a three year time period. Each of the components of the two existing curricula tracks were scrutinized for the ability to assist students in achieving the above stated goals. The IPA, or Individualized Process Assessment, utilized in the Parallel Curriculum, was one such component. Its development into the SPA (Standardized Patient Assessment) exam, an integral part of the Prescription for Excellence curriculum, is the focus of this report.

History of the IPA
The Parallel Curriculum (PC), a problem-based learning, educational program during the first two years of medical school, started at WFUSM in 1987.¹ One essential component of the curriculum was a performance-based assessment entitled the Individualized Process Assessment (IPA). The IPA was a two-part exercise designed to assess a student’s physical examination and interviewing skills, clinical reasoning, basic science knowledge acquisition, and oral presentation skills.

The IPA consisted of two parts: Part I and Part II, which were conducted at two different times during one week examination periods. Part I was a Standardized Patient (SP) encounter in which the student had 1 hour and 15 minutes to conduct a thorough history and physical examination. The student was observed by a clinical faculty member through a one-way mirror using a standardized checklist. Oral feedback was presented to the student after completion of the exercise. Following the completion of Part I, the student had two hours to write up the case, develop a problem list with mechanistic hypotheses, and construct a preliminary list of learning issues. The student was then given two days to do library research, revise their hypotheses, log his/her clinical reasoning process, and prepare a basic science presentation. In Part II of the IPA, the student met with two faculty members, a clinical and basic science faculty team, for approximately 2 hours. In this time, the student gave a brief oral case presentation, discussed his/her clinical reasoning process, presented basic science learning issues, and reviewed a self-assessment.

PC students completed five IPA exercises in the first two years of medical school-three in the first year (October, January, and April) and two in the second year (December and May). These exercises were deemed successful for the 24 students of the student body each year who matriculated through the Parallel Curriculum, but the question was asked “what about the rest of the class??

Should We Do It?
?What about the rest of the class?? was a guiding question when restructuring the curriculum. The curriculum committee encouraged the use of IPA-like examinations for assessment and program evaluation, but recognized that expanding the exercise to the entire medical school class would be a logistical challenge. Ultimately, the assessments were seen to provide information regarding the achievement of all seven goals for the Prescription for Excellence curriculum. It was also noted that these exercises would provide useful assessment data for other courses in the curriculum such as the Foundations of Clinical Medicine (FCM) course, the Basic and Clinical Sciences Problem-based learning course, and the basic science core courses.

Creation of the SPA
The Standardized Patient Assessment (SPA) resulted from the decision to incorporate performance-based assessments for all students into the new curriculum. The SPA is similar in structure to the IPA examinations, but with a different administrative schedule. Part I remains a 1 hour and 15 minute history and physical exam of a SP encounter with a clinical faculty observer through a one-way mirror, and Part II a written and 2 hour oral presentation of clinical and basic science reasoning to a basic science/clinical faculty member team. To accommodate a full class and fit into the new curriculum, the exam structure was redesigned to build upon each experience for the student and end with a complete Part I and Part II by the middle of the second year. Considerations about facilities and staff, student preparation, faculty recruitment, and SP recruitment and training all guided the design process.

Facilities and Staff
The first question was whether there were sufficient facilities and staff to conduct the exam for the full class of 108 students. WFUSM has ten fully equipped examination rooms and six small group tutorial rooms. Each room has one-way viewing windows, wall-mounted video cameras, microphones, and whiteboards. A SPA staff was assembled which included a volunteer medical director, a full-time administrative director who doubles as the SP Coordinator, and six support-staff team members.
The Medical Director oversees all activities pertaining to the SPA. The director works with the SPA coordinator to orient new faculty and students, to develop and/or revise student documentation forms, and collaborates with content expert faculty to develop the clinical cases.

During the SPA itself, the Medical Director is available on-site during the exam to supervise, fill in for last minute faculty vacancies, and to participate in grading and evaluation of students. After the SPA exam, the Medical Director reviews videotapes of students with substandard performance on the examination and makes recommendations for remediation.

Student Preparation
Student preparation was the second challenge. Students are prepared for these examinations in a variety of ways. First is the Foundations of Clinical Medicine (FCM) course, which is a weekly course with two components ? Physical Examination (PE) and Doctor/Patient Relationship (DPR) teaching. In 24 weeks, all students are taught the components of conducting a complete adult physical examination. In that same 24-week period, students are also receiving training in the Doctor/Patient Relationship on alternate weeks. Five to six students meet with two faculty facilitators to learn the intricacies of both the interpersonal relationship between doctor and patient, as well as how to conduct a thorough adult patient interview.

In order to prepare students specifically for each SPA examination, all information is posted on the Intranet several months prior to the examination period and videotapes of outstanding previous performances are on file in the library. One month before the exam, an orientation session is offered by the medical and administrative directors to explain the process and answer questions.

Faculty Recruitment
With 108 medical students, faculty recruitment was the biggest concern when determining whether or not the SPA could be accomplished. With increasing demands on clinical and research time, could we actually get enough faculty to volunteer two hours or more of their time to this exercise?

Over the summer, a memo was sent to all faculty regarding educational opportunities available for the upcoming year. Three months prior to the examination a recruitment letter was sent to all full-time, part-time, and emeritus faculty. One month before the examination, a recruitment letter was sent to all house officers. Throughout the three months leading up to the exam, memos were sent to the Department chairs updating them on their department’s participation. Faculty are asked to give a “block” of time, either a morning or afternoon if possible, yet any volunteered time is accepted. The block of time reduces the actual number of different faculty required for the exercise, safeguards against “no shows” for the second slot, and reduces the number of faculty who need training in this process. This exercise has the full support of the Dean of the medical school who, himself, participates in the SPA.

Standardized Patient Recruitment & Training
WFUSM has a 100 person strong, on-going SP program that actively recruits from the nine local colleges and universities, the North Carolina School of the Arts (a fine arts and performance art school), the nine major hospitals, and the medical school.
A three-day examination process requires anywhere from 16-28 SP’s depending upon scheduling. Existing SPs train, in groups, for two to three hours with the trainer and often a clinical faculty member. New SPs spend four to five hours on touring the facility, checklist training, question practice, and physical exam preparation.

Cost
A final consideration when designing the SPA was cost. The actual number of SP examination encounters dropped from five to three in the MS I and MS II years. However, the cost of using SP’s increased from approximately $600 per exam ($3,000 per two years) to nearly $2,800 per exam ($8,400 per two years). A full-time standardized patient trainer was hired to support this and other curricular processes.

Scoring and Grading
The evaluation process for the SPA consists of the following grades ? Honors, Pass, Low Pass, Fail. Descriptive criteria for each grade are reviewed with faculty participants and are also available to the students. Table 1 indicates the cumulative SPA grade distribution for December 1999.

RESULTS

The First SPA (Formative Part I) – November
PC students, because of the small size of the group (24 students), had covered all 12 sessions of physical examination by the first SP exercise in November. The students in the new curriculum had covered only five sessions of physical examination by the first examination week.

Therefore, it was decided that this initial assessment (SPA Part I) would be a “practice exercise” focused primarily on the provision of formative feedback, and counting for only five percent of the FCM course grade.

Students completed a one hour and 15 minute interview as well as a physical examination covering the parts of the exam they had reviewed in class. Upon completion of the SP encounter, students were required to submit a write-up of the patient within 24 hours.

MS II’s were used as “graders” for this exercise. This was done for several reasons: 1) having second-year student graders would lower the anxiety of first-year students; 2) this avoided any unintentional comparison on the part of faculty of the skills of the former PC students and the new curriculum students; 3) it was a maturing experience for the MS II’s who received the benefit of “being on the other side of the one-way mirror.? The second-year students wrestled with grades and feedback, which increased their understanding of faculty graders in other activities. The MS II’s attended a “faculty” orientation and were trained to use the standardized checklist and provide appropriate feedback. Faculty supervision of the second-year students during the exam was provided by the medical director.

The Second SPA (Part I and Formative Part II) ? February
A complete Part I was administered including a one hour and 15 minute comprehensive adult interview and physical examination with a clinical faculty member observing through a one way mirror. Each faculty member graded the student based upon a standardized, case-specific checklist, which included elements that were ? Critical, ? Important, or Optional. After the student completed the history and physical examination, the faculty observer joined the student in the exam room to elicit feedback from the SP. The faculty member then reviewed the checklist with the student, provided feedback, and assigned a grade for the exercise. The grade constituted 25% of the first year FCM grade. The total time commitment for faculty was two hours.

Following the completion of the Part I exercise, students had four hours to write-up the case, develop a problem list with mechanistic hypotheses, and a brief list of learning issues. The write-up was submitted electronically and graded by the faculty member who observed the student performing the history and physical exam. Once this information was submitted, students were given laboratory data on the patient and were expected to answer an essay question related to this case within 24 hours. A protocol answer was created by faculty from the Department of Biochemistry and the discussion question was graded by faculty with content expertise in this subject matter. This experience provided practice in Part II of the SPA examination as well as formative feedback on students’ clinical reasoning and independent learning skills. The grade constituted 5% of the final grade for the Cellular and Subcellular Processes course.

The Third SPA (Part II) ? May
The third SPA in May is designed to measure the student’s clinical reasoning process. In this exercise, a correct diagnosis is less important than the student’s ability to collect and analyze data in a systematic manner, be able to explain underlying mechanisms, and support his/her hypotheses.

Three individual but related neurological cases were written to correspond with the teachings from the previous eight-week Systems Pathophysiology block of neuroscience and musculoskeletal systems. Each case was electronically released by computer to a previously assigned group of students 48 hours prior to the oral presentation. The students had two hours from the case release time to electronically submit a problem list, mechanistic hypotheses, and a preliminary list of learning issues. The remaining 46 hours were spent with students researching the case, revising initial hypotheses, keeping a well-documented clinical log of the reasoning process, ordering lab tests, and preparing a basic science presentation on three learning issues.

On the oral examination day, the student met with a 2 member, clinical/basic science faculty team in the following format:

• 20 minutes: Evaluator preparation time
• 5 minute: Oral presentation of the case
• 30 minutes: Student discusses the patient’s major problems through problem identification and hypothesis formulation
• 30 minutes: Basic science learning issues
• 30 minutes: Self-assessment process and grading

The clinical reasoning assessment constitutes 10% of the Basic and Clinical Science Problems small group grade from the first year and the basic science presentation counts for 10% of the eight-week Systems Pathophysiology block grade.

The Fourth SPA (Part I and Part II) ? December
The fourth and final SPA takes place in December; three months prior to students beginning the clinical ward rotations. The students are expected to “put it all together,? by performing a complete Part I and Part II in one week. Again, cases were scripted to correspond with the Systems Pathophysiology phase teachings from the previous 12-week block. This exam was seen as the culmination of two years of physical examination, doctor-patient relationship, and basic science reasoning skill development. The Part I grade counted for 25% of the second year FCM grade. Part II, the clinical reasoning process, counted for 10% of the Basic and Clinical Science Problems small group grade and the basic science discussion of Part II for 15% of the Systems Pathophysiology grade (see Table 2).

Integration of the SPA with the Basic Science Curriculum
Another goal of the SPA was to integrate clinical cases with the basic science curriculum. For example in the February SPA, a patient with diabetic ketoacidosis was used for the Part I examination (history and physical examination). The students were then required to answer essay questions on fatty acid metabolism and ketogenesis related to this case. In Part II of the SPA the student makes a basic science presentation relevant to the clinical case to a clinical/basic science faculty team. In previous years, the grade for this exercise was combined with the clinical reasoning process grade. However, starting in December 1999, to emphasize the importance of the basic sciences, the basic science presentation became a separate grade for Part II of the SPA. Having the basic science component count as a substantial part of the grade for Part II of the SPA emphasizes to the student the importance of learning basic concepts of pathophysiology, anatomy, pharmacology, etc. and relating these issues back to the clinical case.

The relationship of student performance in the basic science core courses and the May 2000 and December 1999 SPA basic science discussion was examined. Correlational analyses of the overall core course percent scores (calculated without any SPA components) and the SPA basic science discussion grades revealed modest to weak relationships.

Pearson correlation coefficients for the May 2000 SPA grades and the first year core course percent scores ranged from .266 to .394 (n=107) as shown in Table 3. All were statistically significant (p The relationships of the December 1999 basic science SPA grades and all the core basic science courses were somewhat weaker, r ranging from .139 to .359 (n=103) as shown in Table 4. The case used was related to a topic that students had completed 4 weeks prior to SPA.

The strengths of SPA and core course relationships were somewhat greater overall for the May 2000 than the December 1999 SPA. This may reflect differences in the educational level or abilities of the students, in the nature of the case, or it may indicate that basic science knowledge development and basic science knowledge assessment in SPA are becoming more integrated as the curriculum becomes more refined.

Use of Computers
The implementation of the new curriculum brought new technology to WFUSM. Each student, upon matriculation, is provided with a laptop computer with network access, Internet access, and on line curriculum delivery through the Intranet. The Office of Academic Computing was created to service the needs of the computerized curriculum and provide technical support to students and faculty.

Curricular Feedback
The SPA also serves as a vehicle for curricular feedback and the first year brought forth a number of curricular issues that needed to be addressed. Time constraints limited the students from completing a patient-centered interview.

Therefore, changes were made in the FCM course to incorporate the need for patient-centered interviewing coupled with efficient and effective history taking methods. The SPA also highlighted redundancies or missing components of the interview/examination within the teaching process.

SPA performance feeds into the small group case course that occurs weekly. Students who perform poorly on Part II, the clinical reasoning process and mechanistic thinking, are referred to their small group leader for additional attention during small group. Not only does this reinforce SPA concepts, but it strengthens a student’s performance within the small groups.

Finally, SPA tapes and performances have been evaluated for the Promotions Committee when decisions about a student’s behavior, communication skills, or clinical reasoning are in question.

DISCUSSION

The goals of the SPA experience are to develop:

• Competence in the technical skills of performing a physical examination.
• Problem solving and clinical reasoning skills.
• Skills in interviewing and communications.
• Professional attitudes and behaviors such as the ability to admit mistakes and lack of knowledge.
• Presentation skills and self-assessment.
• Self-directed learning skills.

Students have reported increased confidence in their ability to conduct histories and physical examinations and have shown progress in clinical reasoning and presentation skills. Faculty, after observing students, report increased confidence in the skills these students will bring with them to the third and fourth year clinical rotations. Faculty also expressed the belief that this type of examination process is producing a more well-rounded, prepared medical student.

A concern with the increased number of students is the subject of cheating. How could we control intentional or inadvertent speaking about the exam in the halls with 108 students? To control for this issue we sacrificed some standardization by utilizing male and female SP’s for the same case, and in other exercises by changing the case presentations or laboratory values to lead students toward different diagnostic conclusions. Students also operated under the WFUSM Honor Code and Policy during the examination.

The recruitment of faculty for this process was another major concern when designing the SPA. The first year proved to be a challenge in terms of faculty participation. Faculty were recruited up to the time of the SPA exam and some faculty simply did not show for the assigned time slot(s) or left early. The SPA Medical Director and Associate Dean for Medical Education were “on call” to fill in as were residents who could not block out scheduled time, but agreed to be called at the last minute.

Fear of the unknown has been reported as a reason for faculty not participating in these types of activities. SPA exams and SP’s were not a part of medical school 20 years ago. Increased exposure to SP’s and alternate assessment methods has increased faculty participation at WFUSM. In addition, a plan for departmental participation based upon the number of faculty members in each department was developed. This has helped SPA recruitment efforts for the 1999-2000 academic year in faculty participation and medical education overall.

There is a continuing call for structured observation of students’ clinical skills by faculty. A recent summary from the LCME database² sources documented the infrequent use of structured observation of students’ clinical skills. There was a positive trend of an increase from 19.1 to 48% in the number of schools using standardized patients in comprehensive fourth-year clinical assessments: however, there was no mention of how many schools use faculty observers of first-and second-year students’ interviewing and physical diagnosis skills.

The underlying justification for performance-based assessment of clinical skills using multiple station clinical encounters with standardized patients (OSCE – Objective Structured Clinical Examination) is the difficulty in getting faculty to systematically observe students in structured clinical settings. ³ In studies of the validation of standardized patient ratings, faculty observations and ratings are used as the criterion standard. ⁴ Some authors have gone so far as to say that faculty rating is the gold standard for the validation of standardized patient ratings. ⁵ Even though there are legitimate concerns regarding the inter-rater reliability of faculty ratings of students’ clinical skills, ⁶ the use of structured encounters and standardized patients reduces some of the sources of reliability error.

Students were asked to complete an evaluation of the May 2000 SPA experience. The overall response rate was 86%. Students overwhelmingly agreed with statements that the SPA allowed adequate demonstration of their clinical reasoning ability (83.5% agreed or strongly agreed) and adequate demonstration of their relevant basic science knowledge (84.4% agreed or strongly agreed). Almost 71% of the students also agreed or strongly agreed that the skills and attributes upon which they were evaluated were appropriate.

CONCLUSIONS

More formalized research needs to be conducted on the actual benefits of this type of examination over smaller more focused OSCE exams. However, the National Board of Medical Examiners (NBME) plans to implement an SP examination as part of the USMLE (United States Medical Licensing Examination) step 2,. This will be a required element of the step 2 licensing process. Therefore, in addition to being an integral part of the Prescription for Excellence curriculum, the SPA also serves to better prepare WFUSM students for the Step 2 examination.

REFERENCES

1. Philp, J.R., and Camp, M.G. The Problem-Based Curriculum at Bowman Gray School of Medicine. Academic Medicine 65:363-364, 1990
2. Kassebaum, D.G. and Eaglen, R.H. Shortcomings in the Evaluation of Students’ Clinical Skills and Behaviors in Medical School. Academic Medicine 74:842-849, 1999
3. Barrows, H.S., Williams, R.G. and Moy, R.H. A Comprehensive Performance-Based Assessment of Fourth-Year Students’ Clinical Skills. Journal of Medical Education. 62:805-809, 1987.
4. Colliver, J.A. and Swartz M.H. Assessing Clinical Performance with Standardized Patients. Journal of the American Medical Association 278:790-791, 1997.
5. Colliver, J.A. Validation of Standardized-Patient Assessment: A Meaning for Clinical Competence. Academic Medicine 70:1062-1064, 1995
6. Kalet, A., Earp, J.A. and Kowlowitz, V. How Well Do Faculty Evaluate the Interviewing Skills of Medical Students? Journal of General Internal Medicine. 7:499-505, 1992

LIST OF ABBREVIATIONS
IPA: Individualized Process Assessment
SPA: Standardized Patient Assessment
PC: Parallel Curriculum
SP: Standardized Patient
FCM: Foundations of Clinical Medicine
WFUSM: Wake Forest University School of Medicine
MS I: 1st year Medical Student
MSII: 2nd year Medical Student
H&P: History and Physical Examination
DOS: Disk Operating System
VB: Script Visual Basic Script
ODBC: Open Database Connectivity
DSN: Data Source Name
URL: Uniform Resource Locator
CGI: Common Gateway Interface
GI: Gastroenterology
HTML: Hypertext Markup Language
MRI: Magnetic Resonance Imaging
CT: Computed Tomography
OSCE: Objective Structured Clinical Examination
NMBE: National Board of Medical Examiners
USMLE: United States Medical Licensing Examination

NOTE: Please refer to BSE Vol 10 No 1&2 Complete PDF File for Appendix