Formative and Summative Assessment of the Problem-Based Learning Tutorial Session Using a Criterion-Referenced System

INTRODUCTION

Great strides in curricular reform have been introduced into many medical schools since the presentation of the SPICES model by Harden.^1,2 This model promotes a Student centered, Problem-based, Integrated, Community-oriented curriculum with Elective modules and a Systematic approach to learning, hence the acronym SPICES. The School of Medicine Tec de Monterrey has developed a competency-based, integrated, spiral curriculum in which PBL is the predominant teaching-learning strategy.

Because of the logarithmic growth of medical information, medical students cannot reasonably be expected to master it all. Medical educators struggle with curricular overload while striving to foster application of students’ knowledge and facilitate their independent and critical thinking skills. Meanwhile, medical students have to concentrate on the relevance of basic science to medicine, how to identify and solve clinical problems, and develop the behavior of lifelong learning. PBL is a pedagogic approach designed to achieve these diverse educator/student goals.³

Published studies have reported that PBL has four main objectives: 1) to apply a base of knowledge, 2) to develop clinical reasoning and judgment and decision making skills, 3) to foster self-directed learning and 4) to promote collaborative work.^{3, 4, 5, 6, 7, 8, 9, 10} Tutorial sessions play the major role in the attainment of these objectives. Guided by the tutor (teacher) and through collaboration among them, students establish the learning objectives for each problem and then commit to independent study. During small group discussion they then apply their self-acquired knowledge to the patient’s problem and use their clinical reasoning and decision-making skills to solve it and attain the learning objectives.

A model for a problem-based small group process includes:¹¹ 1) presentation of the learning scenario (problem), 2) definition of unfamiliar language or concepts, 3) brainstorm (comments) by students about issues that come to mind regarding the scenario, 4) identification of key areas for potential learning and organization of these within a logical, conceptual framework, 6) development of a learning plan with specific questions that ensures that all group members understand and subscribe to the learning plan, 7) self-study (independent study) centered around the pre-established learning plan, 8) discussion of the problem centered around the learning plan and 9) evaluation of the learning experience of the particular session.

Not only are the learning method and the intent of its outcomes important, but also the evaluation of their achievement. Commonly, assessment of PBL tutorials focuses on the process only; the way students go through the process of the strategy and acquire self-study and thinking skills. One of the important principles of assessment is to match the assessment method to the learning mode, developmental level, subject matter and program outcomes.¹² Thus, assessment of tutorials should also include the knowledge that is being progressively attained. PBL assessment then, must consider student achievement of the objectives sought by the course and those promoted by PBL. Such assessment must be an integral part of the teaching-learning process; it should be continuous and not just take part at the end of it, and it should be both, summative and formative. Formative assessment is a part of the developmental or ongoing teaching-learning process. It includes delivery of feedback to the student, with the aim of improving teaching, learning and the curriculum. Summative assessment occurs at the end of a term or course and is used primarily to provide information about how much the student has learned and how well the course was taught.13

Because both the course objectives and those promoted by PBL are fostered and achieved during the tutorial sessions where there is an opportunity of ongoing assessment, a need of an assessment tool that focuses on the continuous attainment of both, the course and PBL objectives was identified. This was particularly important at the School of Medicine Tec de Monterrey where new faculty members were participating in PBL courses. Assessment standardization of the tutorial sessions was needed so that teachers knew exactly what to expect from students and vice versa.

The primary objective of this report was to present a criterion-based system that combines specific course objectives with those inherent to PBL. This system described herein is flexible enough to be adapted and used in any type of basic or clinical science PBL course.

MATERIALS AND METHODS

Achieving a valid formative and summative assessment requires identification of criteria for each PBL objective. The four main PBL objectives were described as “rubrics”. In this report pre-established objectives for the Nutrition and Metabolism PBL core basic science course were considered. Specific criteria for each of the four objectives-rubrics were defined by the author and integrated with the course objectives. A fifth rubric with criteria was added, professional behavior, being an outcome emphasized in all courses of the curriculum. A numeric scale, ranging from one (not developed) to six (very well-developed) was used for each criterion as well as a summative scale to integrate criteria within each rubric (Tables 1, 2 and 4). Three final checklists were developed: 1) an electronic checklist to assess daily student achievement, performed by the tutor (Table 1), 2) a self-assessment checklist (Tables 2 and 3) and 3) a peer-assessment checklist (Table 4).

Student assessment performed by the tutor (faculty member)
Tutorial sessions of seven to nine students ran three times a week for two hours, every other day, for a total of 11 tutorials per monthly rotation in the four-month course. The scale for each criterion ranged from one (not developed) to six (very well-developed). Every criterion was not assessed daily; criteria that were considered for each tutorial session depended on the objectives to be covered in that particular session, which were defined during the previous session, and they also depended on the PBL step the group of students was working on. In each tutorial session the tutor of the course assigned each student one grade that ranged from one to six, for each of the five rubrics. The tutor added the total score for each rubric, giving a maximum of 30 points per tutorial session (six points per rubric). At the end of the month, the tutor summed the score obtained by each student for every tutorial. Tutor assessment of students had both, formative and summative value. A detailed description of the assessment system, including the three checklists, was provided to the students the first day of the course.

As set by departmental guidelines, daily assessment by the tutor had a summative value of 30% of the final monthly grade; 10% corresponded to assignments and 60% to the written monthly exam. An electronic format used to facilitate the tutor’s work is shown in Table 5. Once daily grades were entered, formulas calculated monthly grades. The tutor also used the daily assessment to give continuous feedback to students. At the end of the monthly rotation, the tutor assessed performance of the roles taken by each student during the tutorials (ie. leader, secretary and participant). Role assessment served a formative purpose to improve student performance in future tutorials, but had no summative value.

Self-assessment
Self-assessment took place at the end of each monthly rotation, using the rubrics and criteria shown in Tables 2 and 3. Self-assessment was a formative way to get students to reflect on their abilities, performance and attitudes, but had no summative value.

Peer-assessment
Peer-assessment took place at the end of each monthly rotation, using the rubrics and criteria shown in Table 4 (peer-assessment format). Each student handed over this written peer-assessment format to each one of his peers. Peer-assessment also took place in an open session, at the end of the rotation, in which each student gave oral feedback to every other student and to the teacher. Peer-assessment had no summative value; it fostered reflection by students on how their classmates assessed their performance.

RESULTS

This paper describes a criterion-referenced system using Tables 1-4. It was first used one and a half years ago for the Nutrition and Metabolism course. With the intent to standardize the assessment system at the school, as stated by the Assessment Committee, it has been incorporated into each of the four PBL core courses of the Basic Medical Science Department, and has been adapted by the course directors for the Gynecology and Obstetrics core PBL clinical course to increase objectivity of tutorial assessment. Although no quantifiable data to assess the checklists have been obtained, through oral feedback received, it has been perceived that teachers’ and students’ opinions have been positive. This checklist system has established objective standards for both teachers and students. Teachers know what to expect and students know the performance standard.

It has especially helped new faculty who have just begun teaching in a PBL course. Rubrics and criteria let them know exactly what to expect from students during a tutorial. Before its use, many teachers awarded all students with the maximum number of points that each tutorial session was worth. Teachers now assigned students a daily grade, which according to oral feedback from students, has been more fair. Feedback to students has been enhanced since the use of this checklist system; individual and group feedback has been fostered.

During individual meetings with teachers, they reported an increase in student participation. Presumably, this was because students knew what was expected from them. Moreover, teachers indicated that students’ participation was more directed towards knowledge sharing and decision making skills. Additionally, it helped standardize PBL tutorial assessment across the basic medical science department and has since been adapted for one clinical rotation PBL course.

By assessing knowledge base, critical thinking and decision making skills, self-study, collaborative work and professional attitudes, through this criterion-based system we observed four main groups of students: 1) those who studied much and strived to make an effort, but who had underdeveloped critical thinking and decision making skills, 2) students who did not study much but had developed critical thinking and decision-making abilities, 3) those very few with personal attitudes and or collaboration problems and 4) those students that studied much, had developed critical thinking and decision making skills and had good attitudes.

DISCUSSION AND CONCLUSIONS

Critical thinking and a scientific approach to problem-solving are imperative not only for research scientists, but also for physicians.³ Life long skills such as self-directed learning and collaboration with other health care professionals are also necessary attributes of practicing physicians. These skills are incorporated into medical student’s training. PBL provides a pedagogical environment that is conducive to developing these skills. However, an objective-based assessment system must be utilized to ensure their attainment.

Although PBL is commonly used in many medical schools, too much emphasis is placed on the process itself. Little attention has been given to the outcomes of PBL. Assessment of PBL courses and/or curricula is usually incongruent with the sought objectives. PBL assessment relies on evaluation principles similar to other teaching-learning modalities. In particular, student assessment should test the individual’s ability to fulfill pre-established learning objectives.⁸ A PBL assessment tool was defined at the School of Medicine Tec de Monterrey that encompasses both course objectives and those fostered by the PBL teaching-learning strategy. This assessment system consists of three criterion-based checklists: one for tutor assessment of students, another for self-assessment and the third for peer-assessment. Five rubrics were defined for these checklists: knowledge application, critical thinking and decision-making skills, self-directed study, collaborative work and professional attitudes. Criteria for each rubric were also specified.

The value of self and peer assessment has been reported by Friedman.¹³ Results from Friedman’s study indicate that assessment plays an important role in further developing multiple dimensions of the medical profession. Self-assessment evidence and peer-assessment are legitimate as long as assessment standards are being met. The assessment tool presented in this study includes students’ self and peer assessment. The criteria defined for each rubric are the standards students are expected to complete.

Frequently, PBL assessment is used for formative purposes. Those opposed to the summative value of PBL assessment state that evaluation of cognitive and behavioral skills during small group work is subjective. Tutor assessment of students, described herein (table 1) is used for both, summative and formative evaluation. We believe that the teacher’s summative assessment should always include a formative component. If both forms of assessment are based on defined criteria, subjectivity can be somewhat reduced and variability of the evaluations across instructors can also be reduced. Using focus groups and a questionnaire administered to students in a PBL curriculum, Willis¹⁴ reported that his students supported summative assessment of PBL groups. In addition, students in the Willis study felt that summative assessment of PBL should also measure behavior (attitude) that contributes to motivation of the group process and cognitive skills relating to the content of the group discussion.

At first glance, the criterion-referenced system presented may seem cumbersome or too time consuming. However, one must consider its advantages. Our criterion-referenced system has set objective standards for student performance. Second, it has helped to guide teachers and students during the PBL process. Third, it has fostered feedback to students, and fourth, it has helped identify students with critical thinking and decision-making deficiencies, allowing teachers to focus on individual students.

Our long-term intent is to use these checklists as our “gold standard” for assessing tutorials of all PBL courses in the curriculum. With the five rubrics, criterion within each can be adapted according to the type of course and the course objectives.

The learning strategy used in a course must keep coherence with the content area and with the evaluation system. Hence, PBL assessment should focus not only on the process itself, but also on the outcomes: it should incorporate both, course objectives and those fostered by PBL. All three domains of assessment within the classroom, as well as both types of objectives are included in the criterion-based assessment tool presented (i.e. tutor assessment of students, self-assessment and peer-assessment). Self and peer review are learning experiences, in and of themselves. They promote the development of reflective skills that are required for self-directed, life-long learning of prospective physicians. The checklists described herein, are tools to assess attainment of acceptable standards of performance during PBL tutorials.

REFERENCES

1. Harden R.M., Sowden S. & Dunn W.R. Some educational strategies in curriculum development: the SPICES model. ASME Medical Education Booklet No. 18. Medical Education 1984; 18: 284-297
2. Harden R.M. and, Davis M.H. The continuum of problem-based learning,. Medical Teacher. 1998; 20 (4): 317-322.
3. Ludvigsson, J. A curriculum should meet future demands., Medical Teacher. 1999; 21, (2), pp.: 127-129).
4. Barrows, H.S. A taxonomy of problem-based learning methods. Medical Education. 1986; 20: 481-486.
5. Norman, G.R. and Schmidt, H.G. The psychological basis of problem-based learning: a review of the evidence., Academic Medicine. 1992; 73 (10), pp.: 1068-1071.
6. Albanese, M.A and Mitchell, S. Problem-based learning: A review of literature on its outcomes and implementation issues,. Academic Medicine. 1993; 68:, pp. 52-81.
7. Des Marchais, J.E. and Vu, N.V. Developing and evaluating the student assessment system in the preclinical problem-based curriculum at Sherbrooke,. Academic Medicine. 1996; 71:, pp. 274-283.
8. Davis M.H and Harden R.M. AMEE Medical Education Guide No. 15: Problem-based learning-a practical guide. Medical Teacher. 1999; 21 (2:), pp. 130-140.
9. Dahle, L.O., Brynhildsen, J., Behrbohm M., Rundquist, I. and Hammar, M. Pros and cons of vertical integration between clinical medicine and basic science within a problem-based undergraduate medical curriculum: examples and experiences from Linkoping, Sweden., Medical Teacher. 2002; 24 (3:), pp. 280-285.
10. Groves, M., Scott I. and Alexander H. (2002) Assessing clinical reasoning: a method to monitor its development in a PBL curriculum., Medical Teacher. 2002; 24 (5:), pp. 507-515.
11. Baptiste, S. Problem Based Learning: a self-directed journey. Thorofare, NJ, USA: SLACK incorporated; 2002.
12. Friedman Ben-David, M. The role of assessment in expanding professional horizons., Medical Teacher. 2000; 22 (5:), pp. 472-477
13. Wojtczak, A. Glossary of medical education terms: Part 1., Medical Teacher. 2002; 24:, pp. 216-219.
14. Willis, S.C., Jones, A., Bundy, C., Budett, K., Whitehouse C.R. and O’Neill P.A. Small-group work and assessment in a PBL curriculum: a qualitative and quantitative evaluation of student perceptions of the process of working in small groups and its assessment., Medical Teacher. 2002; 24 (5:), pp. 495-501.