The Satisfaction Levels of Students on Academic Support and Facilities, Educational Activities and Tutor Performance in a PBL Program

INTRODUCTION

Since the early eighties, growing awareness of the discrepancy between the undergraduate medical education and Turkey’s basic health needs started a review process regarding the undergraduate education of physicians. A Turkish Parliament report in 1991, demonstrated that only a slight proportion of medical practitioners were capable of coping with the health demands of the country.¹ The main problems of medical education declared by the Turkish Medical Association were the lack of integration between basic and clinical sciences, incompetence of physicians in basic professional skills, the lack of awareness of community health problems and difficulties in the transfer of acquired knowledge and skills to professional life.² These reports and International developments in the field of medical education led Dokuz Eylul University School of Medicine (DEUSM) to seek solutions to the problems of undergraduate medical education. Based on our literature review, preparatory visits to PBL implementing medical schools and considering PBL principles such as a student centered approach, facilitation of integration of basic and clinical sciences, learning to learn and transfer of knowledge and skills to professional life, a PBL curriculum was adopted. It was implemented in the 1997-1998 academic year.³

The duration of our undergraduate medical education is six years and PBL is the principal educational strategy in the first three years of the undergraduate program. Task-based learning was adopted as an educational strategy for clerkships in the 2000-2001 academic year.^4,5 The first three years of undergraduate education are structured on a modular basis. Through PBL sessions, which are the main foci of the modules, the curriculum focuses on knowledge, comprehension, application, analysis, and evaluation in the cognitive domain. PBL provides a learning environment in which competence is fostered not primarily by teaching to impart knowledge but through encouraging an inquisitive style of learning. The cognitive effects of PBL on student learning are increased retention of knowledge, enhanced integration of basic science concepts in clinical problems, development of self-directed learning skills, and enhanced intrinsic interest in the subject matter.⁶ The new curriculum has all of the aforementioned advantages. The lectures, usually limited to one hour per day, provide some clues to increase students’ motivation and curiosity and support our students during periods of independent learning. Basic clinical skills are acquired in clinical skills laboratories and basic science practicals are implemented in laboratories. During these educational activities trainers adopt a coaching role and facilitate student learning until they reach competency level. Community-based educational activities begin in the first year and continue until the end of the sixth year. Field studies help students gain knowledge about the health organization and problems of the country, and help them learn how to evaluate humans in biological, psychological, and social contexts.⁷ Starting with the first year, special study modules provide students with an opportunity to gain in depth knowledge and skill in a field that is of interest to them. Continuous efforts are made for the horizontal and vertical integration of the curriculum.

In DEUSM, module-end, semester-end, phase-end written and practical assessments are used to evaluate the students’ knowledge and skill levels. The passing grade is 70 out of 100 points. Different assessment methods such as multiple choice questions, mini scenarios, essay questions, and, Objective Structured Clinical Examinations (OSCEs) are used to determine knowledge, understanding, application, analysis, synthesis, and evaluation levels within the cognitive domain. Tutors use a “Tutorial Student Assessment Form” to evaluate the students’ performance in PBL sessions.

The relationship between the administrative structures of the medical school and teaching hospitals, the other responsibilities of teachers and administrators, and the intricacies of the curriculum as a system of interrelated components means that any real change has wide repercussions.⁸ An ongoing evaluation is essential to determine if the new system is working to produce a better product.⁹ Curriculum evaluation includes gathering information about the merits of the educational program and program monitoring helps to determine whether corrective measures are indicated.^10,11 In DEUSM, several methods such as the evaluation of students based on their performances in PBL sessions and the other educational activities by tutors and trainers, achievement scores of students in exams, tutors?, trainers?, and students’ oral and written feedbacks and educational research studies are being used for the evaluation of the educational program. In addition to existing quantitative and qualitative studies of the program evaluation, the following research questions were developed:

• What are the satisfaction levels of PBL students concerning the academic support and facilities of DEUSM, their educational activities, and tutor performance?
• Does the level of satisfaction vary from one year to the other?

The main objective of the present study was to gather information from students in order to improve the educational program. This study is an example of the first level of Kirkpatrick’s program evaluation model.¹² The results of this study were presented and discussed in the meetings of the educational committees and the Undergraduate Medical Education Committee to improve the quality of the educational process in DEUSM.

MATERIALS AND METHODS

At the end of the 1999, 2001, and 2003 academic years, this survey was repeated to monitor the changes in students’ satisfaction levels and opinions. Only the preclinical students participated in the survey. They were asked to fill in a self-administered questionnaire in 20 minutes. The participating students were informed about the purpose of the study and their oral consent was obtained. The main limitation of the study is that since there were no third year PBL students in 1999, the questionnaire was given only to first and second year students. In 2001 and 2003, the PBL students of the three preclinical years were included in the study. The students’ response rates on their satisfaction levels for the years 1999, 2001 and 2003 respectively were as follows; 90.3% (196), 93.3% (403), 94.2% (374).

The questionnaire consisted of open-ended questions regarding the students’ opinions and expectations, and a three-dimensional five point scale on the satisfaction levels of students (Appendix 1). The dimensions of the scale were academic support and facilities of DEUSM, educational activities, and the tutor performance.

The students rated the performance of their tutor during the last tutoring period. The items used in the evaluation of the tutors were based on a description of a tutor’s tasks.¹³ Different rating scales were reviewed for developing the rating scale.^14-17

The student satisfaction levels on academic support and facilities provided by DEUSM, educational activities, and tutor performance were determined as dependent variables and academic years as independent variables.

The inclusion of open-ended questions aimed to encourage students to express their opinions and feelings relevant to the previously mentioned dimensions of the scale. In order to enhance confidentiality, the students were asked to leave the questionnaire anonymous.

At the end of the 1997-1998 academic year, a pilot study was applied to first year students and reliability studies were performed. The Cronbach alpha coefficient of the scale on educational activities was 0.85. In 1999, items on academic facilities and tutor performance were added to the scale and reliability studies were repeated following their application. The reliability analysis of the items of the scale were as follows; academic facilities : 0.80, educational activities : 0.84, tutor performance : 0.95. The coefficients of all dimensions being greater or equal to 0.80, the scale was considered as reliable. The findings of the year 1999 were used in the Master of Health Profession Education Program thesis of the first author in Maastricht University, Netherlands.

SPSS program and One Way Anova test were used for data and statistical analysis.

RESULTS

The average satisfaction points of the three-dimensional scale are presented in Tables 1, 2 and 3. The satisfaction levels for academic support and facilities of DEUSM are shown in Table 1. The satisfaction scores for communication with teachers were higher than the scores of other items for all three years (4.19?0.9, 3.80?1.1, 3.90?1.0). The scores on social activities were under the midpoint (=3.0) of the scale. Compared to 1999, the scores attributed to physical conditions of the library, learning resources, and computer facilities increased in the year 2003. Except for the scores attributed to the physical conditions of the library and computer facilities, the scores showed a certain decrease in 2001 compared to 1999 and an increase was observed in 2003.

The satisfaction levels for educational activities are shown in Table 2. The points attributed to PBL sessions, professional skills program, basic science practicals, and self-study activities were higher than the overall assessment point of educational activities. The scores attributed to lectures follow these educational activities.

Tutor performance scores given by the students are shown in Table 3. The item on the provision of a comfortable, trustworthy group environment was highly rated in all years. The items on analysis and synthesis of knowledge and motivation of tutors were also highly rated. Stimulating an extensive reporting of the information collected during self-study period was rated less than other items. In chronological order, the overall assessment of tutor performance, on a five point scale, was 4.32?0.8, 3.87?1.1 and 3.99?1.0 points.

The highest satisfaction scores were observed in 1999 and followed by a moderate decrease in 2001 and a moderate increase in 2003.

DISCUSSION

In the present study, among the items on support and facilities provided by DEUSM, the highest score was attributed to communication with tutors. This was attributed to the underlining tutors’ openness to bilateral communication and their adoption of roles facilitating learning.

In line with the transition to the PBL program, efforts to improve the physical conditions of the library and learning resources have been intensified and they are still actively pursued. The relative increase in scores attributed to these items in year 2003, compared to those of 1999, is thought to parallel the attempt to improve both. In answers given to open-ended questions, the intensity of students’ expectations on the development of library’s learning resources is noteworthy.

In Olmesdahl’s study, second and third-year medical students were asked to rank their ten most distressing needs and concerns. Content overload was identified as the most serious concern followed by learning problems and time management.¹⁸ In Miller’s study, perceptions of first-year medical students and strains imposed on them were presented. In addition to academic workload, another major problem faced by these students was how to manage leisure activities and social relationships.¹⁹ In the present study, almost 1/3 of students complained about the inadequacy of time for social activities and insufficiency of sociocultural activities. The scores attributed to social activities were under the midpoint of the scale for all years. The efforts of the Sociocultural Activities Committee are believed to have contributed to an increase in the scores in 2003. The students’ demands and expectations concerning the augmentation of social activities are still present.

Considering students’ satisfaction points on educational activities; PBL sessions, professional skills, and basic science practicals have higher satisfaction points than the overall assessment point of educational activities. PBL has advantages such as providing a learning experience that is much more enjoyable than sitting through hours of lectures.²⁰ It promotes the activation and elaboration of prior knowledge and enables a cognitive process called epistemic curiosity.²¹ Students play an active role and use an iterative process of seeking new information. The advantages of PBL may have supported students’ positive perceptions. Professional skills and basic science practicals are implemented with competency-based approach and trainers take on a coaching role working with their students until they reach competency level. The trainers’ supportive role and physical conditions of the laboratories such as anatomical models, mannequins may have contributed to students’ high satisfaction levels.

In the scale evaluating students’ perception of their tutors’ performance, the item on “the provision of a comfortable trustworthy group environment” was highly rated. A tutor should establish a climate of openness that allows students to say what they believe or know, without fear of censorship or being put down. Learning can only occur when ideas can be freely expressed.²² In the present study, high satisfaction points were attributed to the feedback opportunity provided by the tutor. This study gives the impression that the tutors provided a positive learning climate and an opportunity for self-reflection and evaluation of group process. In a PBL session, tutors’ metacognitive questions assist the organization, generalization, and evaluation of knowledge. The items in the scale on asking questions towards analysis and synthesis of knowledge, and the motivation of tutor were given subsequent high points. It is also noteworthy that the lowest score was attributed to the item “stimulating an extensive reporting on information collected during self-study.? When this finding was discussed at tutor meetings, the explanations of tutors generally converged on the same point. At this stage, particularly when a topic was outside of their field of expertise, tutors had difficulty asking appropriate questions and generating an in depth discussion of a particular topic. The provision of more detailed subject specific information for tutors was proposed as a solution to this problem.

Although variations were observed in items concerning tutor performance, all related items were above the midpoint of the scale. This was seen as a positive sign in favor of the tutors’ performance. In DEUSM, faculty members are required to take “Basic Training Skills” and “PBL” courses before they take tutoring role. PBL course participants are given the chance of observing at least two PBL sessions and sharing their experiences with course trainers.²³ This training process facilitates tutors’ adoption and actualization of the PBL philosophy. In another study carried out in DEUSM, it was shown that tutors had positive thoughts on the efficacy of PBL.²⁴ The school’s efforts to adapt tutors to the system and tutors’ adaptation may have contributed to the students’ satisfaction points on tutor performance.

In general, the students’ ratings showed high satisfaction levels regarding the educational activities, educational facilities of DEUSM and tutors. These findings were consistent with the feedback given by the students throughout the year. Besides the cognitive effects, positive learning environment, and more opportunities for student-faculty interaction, several other factors may have contributed to the high satisfaction levels. During the curriculum revision process, the management style was changed from a rigid, mechanistic and hierarchical structure to a more flexible, organic and participative structure. The faculty members were motivated by their participation in the process. Almost all of them were involved in the change process as an educational committee member, tutor, trainer, evaluator, lecturer, or case writer. Continuous educational activities such as Basic Training Skills Course, PBL Course and weekly tutor meetings were carried out to facilitate the faculty members’ adaptation to the new curriculum. The PBL curriculum and positive organizational climate of the inaugural years may have contributed to the high satisfaction levels of the students. A slight decrease was observed in the satisfaction levels of students three years after the implementation of the new program. In 2003, an increase in nearly all the parameters was viewed as a positive finding. In comparison with 1999 and 2003, the low ratings in 2001 were interpreted as an aberration. This may reflect the variations in institutional work rhythm and motivation. Factors like weariness of the academic staff and administrative changes may have led to the reduction of initial high energy and motivation levels. These factors may have caused the slight decrease in the satisfaction scores of students in 2001. A subsequent increase in administrative support, meetings of the educational committees on a regular basis, in-service training activities, and the development of standards and algorithms on the functioning of committees enhanced the functioning of the educational system and motivation of the academic staff. These may be the contributory factors to the positive change in the satisfaction points in 2003.

CONCLUSIONS

Studies based on opinions and satisfaction levels of students may have a considerable role in monitoring, identifying positive and problematic areas and implementing necessary revisions of an educational program. In the present study, the satisfaction levels of preclinical years’ students in the years 1999, 2001 and 2003 were evaluated. The findings of the present study are continuously used as part of program evaluation studies. The present study may constitute an example for PBL-implementing schools in the integration of student opinions into program evaluation studies.

ACKNOWLEDGEMENTS

The present study was sponsored by the Dokuz Eylul University Scientific Research Projects Centre.

REFERENCES

1. Turkish Parliament Research Commission; Turkish Parliament. Report on medical education in Turkey: Point of view of the student. Ankara, 1991.
2. Kılıc, B. and Sayek, I. Turkish Medical Association. Report on Undergraduate Medical Education. Ankara, 1997.
3. Elci, O.C. and Aksakoglu, G. Dokuz Eylul joins the family: active medical education. Medical Education. 1998; 32: 222-223.
4. Ozkan, H., Degirmenci, B., Musal, B., Itil, O., Akpinar, H., Akalin, E., Ozkan, S. and Alici, E. Task based learning in Dokuz Eylul University Medical School, Turkey. Medical Teacher. 2004; 26(3): 279-280.
5. Ozkan, H., Degirmenci, B., Musal, B., Itil, O., Akalin, E., Kılınc, O., Ozkan, S. and Alici, E. Task based learning program for clinical years of medical education. Education for Health. 2006; 19(1): 32-42.
6. Musal, B., Aksakoglu, G. and Ucku, R. Community-based education program of Dokuz Eylul School of Medicine. Education for Health. 2003a; 16(2): 218-221.
7. Craig, P. and Bandaranayake, R. Experiences with a method for obtaining feedback on a medical curriculum undergoing change. Medical Education. 1993; 27: 15-21.
8. Grant, J. and Gales, R. Changing medical education. Medical Education. 1989; 2: 252-257.
9. Coles, C.R. and Grant, J.G. Curriculum evaluation in medical and health-care education. Medical Education. 1985; 19: 405-442.
10. Bandaranayake, R. Implementing change in medical education in developing countries. Medical Teacher. 1989; 11: 39-45.
11. Dolmans, D.H.J.M. and Schmidt, H.G. The advantages of problem-based curricula. Techniques in medical education: problem-based learning. Postgraduate Medical Journal. 1996; 72: 535-538.
12. Kirkpatrick, DL. Evaluating Training Programs. 2nd ed. San Francisco: Berret-Koehler Publishers, Inc; 1998.
13. Barrows, H.S. and Tamblyn, R.M. Problem-Based Learning: An approach to Medical Education. New York: Springer Publishing; 1980.
14. Dolmans, D.H.J.M., Wolfhagen, H.A.P. and van der Vleuten, C.P.M. Long-term stability of tutor performance. Academic Medicine. 1996; 71: 1344-1346.
15. Kalaian, H.A. and Mullan, P.B. Exploratory Factor Analysis of Students’ Ratings of a Problem-based Learning Curriculum. Academic Medicine. 1996; 71: 390-392.
16. Robins, L.S., Gruppen, L.D., Alexander, G.L. and Fantone, J.C. A predictive model of student satisfaction with the medical school learning environment. Academic Medicine. 1997; 72: 134-139.
17. Litzelman, D.K., Stratos, G.A., Marriot, D.J. and Skeff KM. Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Academic Medicine. 1998; 73: 688-695.
18. Olmesdahl, P.J. The establishment of student needs: an important internal factor affecting course outcome. Medical Teacher. 1999; 21(2): 174-179.
19. Miller, P.M.C.C. The first year at medical school: some findings and student perceptions. Medical Education. 1994; 28: 5-7.
20. Albanese, M.A., Mitchell, S. Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine. 1993; 68(1): 52-76.
21. Schmidt, H.G. Foundations of problem-based learning: Some explanatory notes. Medical Education. 1993; 27: 422-432.
22. Barrows, H.S. A specific, problem-based, self-directed learning method designed to teach medical problem-solving skills, and enhance knowledge retention and recall. In: HG Schmidt, ML De Volder, editors. Tutorials in problem-based learning: A new direction in teaching the health professions. Van Gorcum Ltd; 1984. p. 16-32.
23. Musal, B., Abacioglu, H., Dicle, O., Akalin, E., Sarioglu, S. and Esen, A. Faculty development programs in Dokuz Eylul School of Medicine: In the process of curriculum change from traditional to PBL. Medical Education Online [serial on the Internet]. 2002; 7: 2. Available from: http://www.med-ed-online.org/pdf/res00030.pdf
24. Musal, B., Taskiran, C. and Kelson, A. Opinions of tutors and students about effectiveness of PBL in Dokuz Eylul University School of Medicine. Medical Education Online [serial on the Internet]. 2003; 8:16. Available from: http://www.med-ed-online.org/pdf/f0000073.pdf

Download Appendix – http://www.iamse.org/jiamse/volume16-1/16-1_appendix.pdf