Does Human Simulator-Aided Learning Improve Long-Term Retention of Autonomic Pharmacology Concepts and Facts by Year II Medical Students?

INTRODUCTION

Interest in using human simulators to enhance student learning of concepts and principles in biomedical sciences has existed for decades.¹ Applications of patient simulations in small group teaching and learning environments have been increasingly applied to various student populations in medicine, nursing and allied health sciences during the past decade.^2-6 The use of computer-controlled human patient simulators is receiving increasing support as an educational tool in the training and evaluation of the clinical skills of medical students, residents and physicians.⁷ Medical students appear to value simulation-based learning highly.⁸ Students especially value the opportunity to apply their theoretical knowledge in a safe and realistic setting and to develop a systematic approach to a clinical problem.⁸ Simulators are an important part in the training of medical personnel to help improve patient safety.⁹

Simulation can be integrated into problem-based curricula by simply “bringing to life” existing case material.¹⁰ The use of simulators in preclinical courses, such as pharmacology and physiology, has been shown to help link theory and practice in medical education.^11,12 A simulator-aided learning environment appears intuitively to be superior to paper-based or video-assisted learning environment due to its applicability to real life situations. However, limited research has been published on the actual value of simulation as an educational and/or evaluation tool for pre-clinical medical students, particular in pharmacology.⁷ Few studies have addressed the efficacy of patient simulators in long-term retention of new knowledge. Morgan et al.¹³ compared the effectiveness of simulator- and video-based, faculty-assisted learning by final year medical students at the University of Toronto participating in a short-term (6 week) anesthesia rotation. Comparison of the final examination scores of students trained by either method showed no significant differences in short-term knowledge retention between the two groups. However, the authors could not predict the comparative effects of the two teaching methods on the long-term retention (> 6 weeks) of knowledge.

The purpose of our study was to investigate the effects of a high fidelity human patient simulator (Laerdal SimMan®) on long-term knowledge retention by second year medical students enrolled in the Medical Pharmacology course. Our hypothesis was that second year medical students participating in a trained facilitator-guided clinical case-based small group discussion assisted by a human patient simulator would demonstrate increased knowledge retention when compared to students participating in a facilitator-guided traditional paper-based small group discussion. We assessed retained knowledge at four time points over eight months.

MATERIALS AND METHODS

Study Groups: All medical students enrolled in the Medical Pharmacology course, including those participating in this study, attended a series of autonomic pharmacology lectures one week before participating in the related small group session. Faculty prepared syllabi, slide sets and lectures, as well as standard reference texts were available to all students. For the purpose of this study, twenty-six second year medical students were randomly assigned to either a computerized human patient simulator group (Group S: 5M/8F) or a control paper group (Group P: 7M/5F). One student assigned to Group P was absent on the day of the small group session, and was, therefore, excluded from this study. Informed consent was obtained from all participating students in accordance with an IRB approved protocol.¹⁴

Facilitated Small Group Learning: For the past decade at the University of Cincinnati, College of Medicine pharmacology faculty have delivered a series of facilitated small group sessions (ten over eight months) in paper-based clinical case scenario format to promote pharmacology knowledge retention in second year medical students. This format encourages students to apply newly acquired knowledge in clinically relevant contexts. A trained facilitator, either a pharmacology faculty member or an advanced graduate student (MD/PhD program), guides 12-14 medical students through analysis and solutions to cases during each 90-minute session.

For the purposes of this pilot study, sessions were facilitated by two senior graduate students with equivalent levels of experience with and knowledge of this subject matter. The graduate students were directly trained by the Tutorial Director (RK) and the Simulation Center Director (GH) and certified as facilitators for both the paper-based and simulator-based small sessions.

Study Design: Clinical case scenarios (3-4 per session) and related self-study questions for each small group session were posted electronically on an Intranet course system Blackboard) one week prior to the session. Students were asked to independently prepare their answers to the case-based questions prior to the facilitated discussion session with their peers. Immediately before the start of their respective facilitated discussion sessions, all study participants were surveyed (see Student Survey, Appendix 1) with respect to preferences in learning style and the use of learning aids. All students completed a graded 5-question quiz on core pharmacological facts and principles related to the autonomic pharmacology lectures and small group cases at the start of the session. Group P discussed the case materials using a traditional case presentation format guided by a trained facilitator. Group S discussed the same case materials aided by a human patient simulator (SimMan®) pre-programmed to simulate a patient’s clinical cardiopulmonary symptoms and the physiological responses (e.g., changes in heart rate, blood pressure, respiratory rate, blood oxygen saturation, etc.) to a variety of autonomic pharmacology interventions in each of the clinical case scenarios discussed at the session. No student had prior exposure to SimMan®.

Subsequently, all study participants were tested using graphical, tabular, and narrative Step I USMLE format questions on key autonomic pharmacology principles and concepts and on important drug facts discussed at the small group session through a series of four cumulative course examinations administered over an 8-month period. These course examinations were held 1 week (exam 1), 5 weeks (exam 2), 22 weeks (exam 3), and 33 weeks (exam 4), respectively, following the autonomic pharmacology small group session. The number of questions that related to autonomic pharmacology decreased with each exam due to the cumulative nature of the exams. More examination items (n =14) were included on exam 1 administered most proximate to the lecture and small group session than on each of the subsequent 3 examinations; 5, 4, and 4 items, respectively. The same 27 test items, each used one time, were used to compare performance between the two study groups.

Test Item Selection: Examination questions on autonomic pharmacology (and all other course topics) were selected from a secure item bank. These test items were written by course directors and lecturers and subsequently selected based on previous validation for use in examinations by a senior faculty committee, none of whom were associated with the study design or conduct. Each exam was designed to include questions that varied in degree of difficulty based on previous students’ performance on these items.

Data Analysis: All data collection was performed in a blinded manner to maintain participant confidentiality and segregation of participant identity from individual participant scores. All quantitative data were first subjected to a test for normal distribution. Quantitative data determined to be normally distributed, were analyzed by unpaired one-tailed Student’s t-test. Otherwise, the Wilcoxon Rank Sums Test was applied to data sets that were not normally distributed. The criterion of p<0 was established for identification of significant differences. Statistical analyses were performed using SAS version 9.0 (SAS Institute, Cary, North Carolina, USA

RESULTS

Surveys regarding learning styles and study-aid preferences administered before the start of the small group session revealed that students in Group S and Group P had similar “Learning Style” preferences; 67% of Group S and 58% of Group P preferred to “Learn on Their Own”. There were no significant differences between the groups with respect to preference for learning in a small group context or for the use of “Learning Aids”. There were also no differences between the groups with respect to age (Group P: 22-28 yrs; Group S: 21-27 yrs; 18-22 yrs = 8%; 23-25 yrs = 64%; 26-29 yrs = 28%) or gender (total: 12M/13F; Group P: 7M/5F; Group S: 5M/8F). We calculated an 83% chance of detecting a difference between the mean scores of the groups using a one-tailed t-test at a 0.05 level of significance, based on metrics of our data set.

The scores on the 5-item quizzes administered at the start of the sessions revealed no difference in performance between the Groups S and P (Table 1). Comparison of the Group S and P scores on autonomic pharmacology test items in three of four post-session course examinations also failed to reveal a statistically significant difference between the groups (Table 1). The only outcome difference we detected was student performance on the four autonomic pharmacology test items on exam 4; Group P performed better than Group S (p = 0.04). Analysis of the 27 autonomic pharmacology test items from all four examinations yielded no significant differences between the two groups. We also used the Wilcoxon Rank Sums Test to compare the mean scores of students in Groups S and P for the pre-session quiz and on two course exams (2 and 4) because these scores were not normally distributed. This analysis yielded similar results; i.e., no statistically significant differences between Group S and Group P for the pre-session quiz or for exam 2, but on exam 4, Group P outperformed Group S (p= 0.04). An unpaired Student’s t-test showed no significant differences in performance by gender, making it unlikely that the unequal distribution of men and women in Groups S and P affected the study outcome.

DISCUSSION

Our hypothesis, that students exposed to human patient simulator-aided case presentations (Group S) would retain significantly more knowledge of autonomic pharmacology drug facts and principles over an eight-month period than students who received the traditional paper-based training (Group P), was not supported. The results of our study did not reveal improvement in long-term retention of newly acquired autonomic pharmacology knowledge by second year medical students trained using a human patient simulator (SimMan®). Groups S and P performed similarly on all 27 autonomic pharmacology test items included in four examinations administered over an eight month period, and on each set of autonomic pharmacology test items in individual examinations; except for exam 4, in which Group P performed better. We failed to identify a benefit for patient simulator use in acquiring and retaining new knowledge of autonomic pharmacology over the course of 8 months. Morgan et al.¹³ previously reported no significant differences in short-term (6 week) knowledge retention between two groups of medical students completing an anesthesia rotation after receiving simulator- or video-based training. Our study suggests that simulator-aided training is neither better nor worse as an educational tool than more traditional facilitated small group discussion.

Several limitations of this study constrain the interpretation and application of the results. Because group size was small, extrapolation of our results to a larger study population cannot be reliably made. In addition, any potential benefit of the human patient simulator may have been limited by the fact that the students in Group S had only a single exposure to simulator-aided training. Feingold et al. suggested that repeated exposure to realistic clinical simulations over a long period of time with real-time feedback and correction of errors may be necessary to observe significant benefits with this method.¹⁵ Our ability to detect a true difference in new knowledge retention between the groups at over an eight month assessment period may have been limited by the relatively small number of test items included on exams 2-4. Another limitation of this study is that paper examinations may not be the best method for detecting enhancement of skill or knowledge gained by using the simulator as they are contextually very different from the simulator sessions. A particular advantage of simulation-based training is its ability to incorporate elements that more closely resemble real-life patient encounters, thereby enhancing students’ confidence in making decisions in these settings.¹⁶ It is possible that follow-on assessments conducted in a more realistic setting, such as those that incorporate a standardized or simulated patient, may be better able to detect improvements in knowledge retention facilitated by simulator-aided training.¹⁷ Consistent with this, Gilbart et al.¹⁶ noted significant improvement in trauma management skills by students exposed to a computer-based trauma simulator versus students who received no additional training. This skill improvement was only evident when performance was evaluated using a clinically based performance assessment. Improvement was not observed on a paper-based examination. This suggests that traditional paper-based assessment instruments may not be able to discern transfer of knowledge retention that is promoted by simulator-aided training. It is possible that a different assessment modality for evaluating students’ retention of newly acquired autonomic pharmacology principles and facts knowledge, such as a case scenario using a standardized patient or patient simulator, would demonstrate a benefit of simulator-aided training not detected using traditional paper-based examinations. However, it must also be acknowledged that differences in individual motivation and learning style preferences, and the use of pharmacology texts, course syllabus, and electronic sources, as well as participation in informal peer study groups are variables beyond the control of the study conditions that potentially confound the study outcome and its interpretation.

Future studies investigating the effects of human patient simulator-aided training on the acquisition and retention of medical knowledge may benefit from incorporating multiple simulator training sessions into the study design. In addition, the use of assessment modalities that closely mimic real-life clinical settings may facilitate the detection of any benefits obtained using the simulator as an educational tool for long-term retention of new knowledge.

ACKNOWLEDGEMENTS

The authors wish to thank Steven Lisco, M.D., Department of Anesthesiology, University of Cincinnati, and Stephan Pelikan, Ph.D., Department of Mathematical Sciences, University of Cincinnati, for their time and support.

REFERENCES

1. Gordon, M.S. Cardiology patient simulator. Development of an animated manikin to teach cardiovascular disease. American Journal of Cardiology. 1974; 34:350-354.
2. Euliano, T.Y. Small group teaching: clinical correlation with a human patient simulator. Advances in Physiology Education. 2001; 25:36-43.
3. Goodrow, M.S., Rosen, K.R., and Wood, J. Using cardiovascular and pulmonary simulation to teach undergraduate medical students: Cases from two schools. Seminars in Cardiothoracic and Vascular Anesthesia. 2005; 9:275-289.
4. Gordon, J.A., Brown, D.F., and Armstrong, E.G. Can a simulated critical care encounter accelerate basic science learning among preclinical medical students? A pilot study. Simulation in Healthcare. 2006; 1:13-17.
5. Kabanza, F., Bisson. G., Charneau, A., and Jang, T.S. Implementing tutoring strategies into patient simulator for clinical reasoning learning. Artificial Intelligence in Medicine. 2006; 38:79-96.
6. Sinz, E. Simulation-based education for cardiac, thoracic, and vascular anesthesiology. Seminars in Cardiothoracic and Vascular Anesthesia. 2005; 9:291-307.
7. Morgan, P.J., and Cleave-Hogg, D. Simulation technology in training students, residents and faculty (Review). Current Opinion in Anaesthesiology. 2005; 18: 199-203.
8. Weller, J.M. Simulation in undergraduate medical education: bridging the gap between theory and practice. Medical Education. 2004; 38: 32-38.
9. Moorthy, K., Vincent, C., and Darzi, A. Simulation based training (Editorial). British Medical Journal. 2005; 330: 493-494.
10. Gordon, J.A., Oriol, N.E., and Cooper, J.B. Bringing good teaching cases “to life”: a simulator-based medical education service. Academic Medicine. 2004; 79: 23-27.
11. Winston, I., and Szarek, J.L. Problem-based learning using a human patient simulator. Medical Education. 2005; 39: 526-7.
12. Mueller, M.P., Christ, T., Dobrev, D., Nitsche, I., Stehr, S.N., Raven, K., and Koch, T. Teaching antiarrhythmic therapy and ECG in simulator-based interdisciplinary undergraduate medical education. British Journal of Anaesthesia. 2005; 95: 300-304.
13. Morgan, P.J., Cleave-Hogg, D., McIlroy, J., and Devitt, J.H. Simulation technology: a comparison of experiential and visual learning for undergraduate medical students. Anesthesiology. 2002; 96: 10-16.
14. University of Cincinnati IRB #05-08-10-04-E: Effects of simulator-aided learning on retention of principles of autonomic pharmacology by year II medical students.
15. Feingold, E.F., Calaluce, M., and Kallen, M.A. Computerized patient model and simulated clinical experiences: evaluation with baccalaureate nursing students. Journal of Nursing Education. 2004; 43: 156-163.
16. Gilbart, M.K., Hutchison, C.R., Cusimano, M.D., and Regehr, G.A. Computer-based trauma simulator for teaching trauma management skills. The American Journal of Surgery. 2000; 179: 223-228.
17. Hanson, G. Refocusing the skills laboratory. Nurse Educator. 1993; 18: 10-12.