Creation of an Online Curriculum to Introduce and Supplement the Learning of Pharmacology in a Problem-based Learning/Lecture Hybrid Curriculum

INTRODUCTION

Utilization of Problem-Based Learning (PBL) during the basic science years has become increasingly prevalent at medical schools both in the United States and internationally, as advances in cognitive and learning sciences theories continue to influence medical education.^1-3 While there are many potential benefits to having medical information presented in an integrated fashion in the context of case studies, this curricular format presents particular challenges to the teaching of many longitudinal disciplines, including pharmacology.^4-8 If the curriculum is also organ-systems based, this presents additional complications, as subjects such as pharmacology can be difficult to integrate within a single organ system, and require a great deal of interdisciplinary coordination.⁵

Many PBL curricula allot a discrete period of time for the teaching of pharmacology, expecting that students will learn pharmacology largely on their own over the course of the curriculum.⁹ If the teaching of pharmacology does not begin early in the curriculum, students may fail to pay attention to faculty-intended self-directed learning goals and may later find that they have fallen behind.¹⁰ Additionally, students are often confused as to how much time and effort they should devote to studying pharmacology; it is easy to ignore this field altogether, with the competing demands of other disciplines that are more explicitly covered in the curriculum.¹⁰ Discrete periods of pharmacology didactics also lend themselves to increased student “cramming,” or massed learning, immediately before a curriculum exam or before medical board exams, when distributed learning (i.e., learning pharmacology in smaller units along the way) would lead to improved retention of such material.^11-13 These difficulties are somewhat paradoxical, because pharmacology naturally synthesizes fundamental principles from the foundations in biochemistry, physiology, and pathology in order to provide the basis for therapeutic intervention.⁸ However, online interventions can help increase distributed learning habits; Magid and Schindler found that weekly open-book, computer-based quizzes increased student performance on final examinations and resulted in positive student feedback in general pathology, another longitudinal discipline at our institution.¹⁴

Intervention Goals and Objectives

Within the Western Reserve2 Curriculum of Case Western Reserve University School of Medicine, the basic science curriculum spans 18 months and is organized into six “blocks” (Figure 1). Each block is approximately 12 weeks long, with the exception of the first block.¹⁵ The curriculum is systems-based, with PBL-style cases accounting for approximately half of student contact time. The remaining six hours of student contact within the systems-based block is utilized for lectures and other smaller group sessions that are led by content specialists. The Western Reserve2 Curriculum was introduced two years prior to the pharmacology learning intervention described herein.

Within this systems-based teaching structure, fundamental principles of pharmacology are officially presented within the “Homeostasis” curricular block at the end of the first year. Although therapeutic agents are encountered within the PBL-style cases and briefly mentioned in lecture from the start of the curriculum, we were concerned that students failed to recognize the opportunities for learning pharmacology prior to the official didactic sessions scheduled near the end of their first year. Though they were presented with therapeutic agents in PBL-style cases throughout the curriculum, it was repeatedly observed by faculty facilitators that students often did not devote as much time to the pharmacology learning objectives during discussion groups as the other basic science disciplines. Many students provided feedback that they felt they had minimal exposure to pharmacology prior to the “Homeostasis” block, and that they were unhappy or uncomfortable with their pharmacology knowledge prior to studying for their medical board examinations. The following comment from an end-of-block survey was typical: “I feel, as do most of my peers, that I am completely inept when it comes to pharmacology.” Students stated that they did not feel confident that they had the means to study pharmacology on their own, and expressed trepidation in doing so: “Pharm is hard for some of us to understand, and reading everything out of a book to learn for that subject just does not work very well,” and “Please teach pharmacology to the students.” Consistent with international medical students, students at our institution requested additional educational support regarding solving pharmacology problems and increased clinical integration.¹⁶

To facilitate learning of pharmacology earlier in the first year, we chose to create a supplement to the curriculum that could augment pharmacology learning without requiring any additional lecture or PBL time. Our objectives were as follows:

Objectives

•Supplement the curriculum with online, self-paced modules that align with topics students are currently studying throughout the systems-based blocks
•Increase student familiarity with fundamental principles of pharmacology and therapeutic agents
•Increase the amount of time students spend on learning pharmacology (increase student engagement)
•Improve student self-efficacy so that students feel they have the tools to learn pharmacology

The main outcomes we hoped to achieve with our supplemental curriculum were as follows:

Outcomes

•Increase student pharmacology knowledge
•Increase student time spent working on pharmacology
•Increase student self-efficacy in learning pharmacology

In summary, we wished to increase formal learning opportunities in pharmacology, while creating a flexible method for students to access the material, working at their own pace and at times that were convenient for them. We hoped that they would gain additional pharmacology knowledge, spend more time engaged with the material, and feel more confident in their ability to learn the subject matter.

Building an Improved Pharmacology Curriculum

Many steps were taken to create the new, supplemental pharmacology curriculum. A focus group comprised of second-year students was conducted to obtain insights into what students felt were strengths and weaknesses of the pharmacology curriculum, and what they thought could be done to improve it. Students expressed the desire for more explicit pharmacology didactics earlier in the curriculum, as well as a more integrated approach; students reported that they did not know what they should be learning with regard to pharmacology, or useful methods for learning about therapeutic agents. With the new changes to the curriculum, didactic time had been decreased considerably, including time dedicated to pharmacology. As with any new curriculum, optimal means of presenting the information within the constraints of decreased contact time were in the process of being identified and implemented. Faculty had also expressed frustration with the reduction in contact hours, and student satisfaction with regard to pharmacology teaching had significantly decreased, when it had previously received outstanding marks in the pre-Western Reserve2 curriculum.

Given this impetus for augmenting the pharmacology curriculum from both students and faculty, we conducted a review of the literature to determine how others had solved this problem in their curricula. We found that other medical schools struggled with similar difficulties, i.e., students were treating pharmacology learning objectives within PBL cases too superficially.¹⁰ However, an online curriculum that was focused on use of herbs and dietary supplements showed significant knowledge improvements and retention at the long-term follow up assessment, six to ten months later.¹⁷ Results from another online pharmacology curriculum indicated that students (nurse prescribers) reported substantial increases in their perceived knowledge and satisfaction with pharmacology, further suggesting that an online supplemental curriculum might be efficacious at our institution.¹⁸ The University of Melbourne found a combination of multiple-choice questions (MCQs) and PBL-style questions with case vignettes to be useful methods for assessing a PBL curriculum.¹⁹ Students at our institution already had essay examinations as a part of the existing curriculum, including pharmacology questions, so we did not focus on augmenting such assessment strategies. However, we did include short MCQs in the weekly self-assessment quizzes in our supplemental curriculum, to provide the students an avenue for assessing their learning.

Weighing the different options, we decided to use an online supplement, as this could be completed by students at their convenience, in any location. Previous research has shown that students will use digital content to learn basic science topics.^20-21 This supplemental material is not an effective substitution for the core curriculum, as students who used online lectures as a replacement for live lecture attendance in pharmacology performed more poorly on summative assessments.²¹ However, students in another study were found to have a positive subjective response to online content, and the majority felt that online lectures were at least as effective as live class lectures.²⁰ At our institution, although online lectures from the previous iteration of the curriculum are available to students, the ongoing live lectures are not available online. Student feedback indicated that these lectures, while useful, did not appropriately address their introductory pharmacology learning needs. Hence, the decision was made to create supplemental online modules. All students were required to purchase a laptop computer upon matriculation, so access to online modules was not a concern. In addition, this approach did not require additional didactic time, a commodity that is decidedly difficult to obtain within a curriculum of limited student contact time.

At other institutions, pharmacology modules appear to have been created professionally or with significant computer programming support.^18,23 The University of Cincinnati College of Medicine chose to create their instructional technology and online modules for their basic science curriculum in-house, in order to maintain the integrity of the finished product, but their annual costs were $96,000, on average.²⁴ While we eventually intend to seek funding, with the goal of broadening our modules’ appeal and usefulness in order to share them with other schools, we felt that we could initiate this task creatively, largely taking advantage of existing resources instead of requesting additional support. We were able to create our supplemental curriculum with one medical student’s summer support and “volunteer” time from faculty, making it unnecessary to involve outside computer professionals to create the modules, thereby retaining more control over the final product. In this manner, students, faculty, and staff were able to work collaboratively without the further complications of working with outside vendors.

Deciding upon the computer method of delivery required significant investigation. Pachyderm, a flash-based modular program with pre-fabricated templates that could be used for creation of modules with limited computer expertise, was initially considered. However, since we wished to assess the efficacy of the intervention, this program did not prove suitable, as it did not include the ability to capture quiz and survey data.

Since students expressed a desire to have pharmacology presented on a more longitudinal basis, and doing so had the potential to increase learning and retention by providing distributed as opposed to massed learning, we decided to create nearly weekly modules that would complement the existing didactics and case studies presented during the formally scheduled curriculum. In this way, students’ pharmacology learning could be integrated with their other learning and potentially placed within a more meaningful context. In addition, frequent modules would increase student awareness of pharmacology learning opportunities much earlier in the curriculum, further facilitating distributed learning rather than massed learning right before examinations, as suggested by the literature.¹⁴

Getting Stakeholder Support

We worked with our medical school’s Academic Computing office to develop a plan for inclusion of the online pharmacology supplement within the existing infrastructure of the current curriculum online portal. Utilization of this electronic framework permitted us to track student participation and assessment outcomes in a de-identified manner.

The supplemental pharmacology curriculum also required support from additional medical school committees that monitor curriculum changes and student evaluations. Multiple meetings and presentations to these committees were necessary to obtain support. In addition, changes were made by these bodies, such as the wording, length, and frequency of our program evaluation questionnaires, due to concern for student survey response burden. We also met with various members of the administration, including the vice dean for medical education, the curricular affairs dean, and the director of program evaluation, to obtain their support. Ultimately, approval to offer the supplemental online pharmacology curriculum to students on an optional basis was attained. Since this supplemental pharmacology curriculum was not required, a limited number of students chose to utilize the resource initially, making the interpretation of results somewhat more complex.

Overview of Supplemental Pharmacology Curriculum

Given the feedback we received from students and faculty, we wished to provide a longitudinal supplement to the existing curriculum. In this manner, we could introduce pharmacology principles much earlier, and more formally highlight pharmacology-related learning objectives. Some of these pharmacology learning opportunities were already present in the existing PBL-style cases in the curriculum, but we anticipated that the supplement would increase awareness of these opportunities. As stated above, fundamental principles of pharmacology are formally presented during the “Homeostasis” block that occurs at the end of the first year of the curriculum. As a means of achieving remote contact with the students, the web-based pharmacology supplemental curriculum was initiated near the start of the students’ first year, during the “Human Blueprint” block that includes the disciplines of molecular biology, genetics, development, reproduction, endocrine systems, and cancer biology. Though student participation was optional, students were informed of the supplement via an in-person introductory session, flyers distributed during PBL-style groups, and existing institutional online student portals.

The general format of the web-based supplement consisted of short “modules,” which were comprised of: 1) a brief topic introduction and didactic information, pointing out applications to the existing curriculum, when possible; 2) a short reading assignment/resource suggestions (generally one to three pages of reading); 3) a short (three question) quiz used for student self-assessment and to solidify learning; and 4) brief explanations of the basis for the correct answers and the problems with the incorrect answers. Each module was intended to take no more than 20-30 minutes to complete, including the brief textbook or literature-based reading assignment and quiz. The formats of each module differed slightly, including more traditional didactic presentations, case studies, and even a “choose your own adventure”-type format, where students chose which therapeutic option to administer to the “patient,” with explanations of the rationale for each treatment. Students had access to new modules on a weekly basis, mirroring the existing weeks of the curriculum. Students were encouraged to keep up with the weekly modules, as they were more relevant to the existing curriculum when done concurrently, but all modules were permanently available to students after they were released. The modules were written by faculty members, with input from medical students. A unique feature of these modules included a mascot, affectionately named “PharmFriend,” who highlighted key points and take-home messages for students. PharmFriend’s comments were largely written by a medical student. We felt that this student voice was important, to optimally engage students and convey the material at a level that was accessible to most beginning medical students.

To assess the effectiveness of the supplement, a number of means of curricular assessment were instituted:

1. Pre- and post-tests for each curricular block. To assess increases in pharmacology knowledge, a pre-test was included at the beginning of each curricular block, prior to the release of any supplemental modules. Similarly, each series of modules associated with a systems-based curricular block was concluded with a post-test. These post-tests contained the same questions from the pre-test, providing a means to measure an increase in knowledge, as well as additional related questions designed to assess the transfer and application of knowledge. All students were encouraged to take the pre- and post-tests, regardless of whether or not they completed the online supplement.
2. NBME-style examinations. A formative but mandatory NBME-style exam at the end of each curricular block, consisting of retired NBME questions selected by faculty members, was an existing part of the curriculum. Pharmacology questions included within these NBME-style exams were also used to assess increases in pharmacology knowledge.
3. End-of-block essay examinations. All students were required by the existing curriculum to take an integrated essay exam covering material from that curricular block. Scores for pharmacology-related sub-questions were separated from non-pharmacology essay sub-questions, and compared.

Additional goals of the online supplement included increasing student engagement with the material, measured by a self-reported increase of time spent studying and discussing pharmacology. Another goal of these modules was to increase student self-efficacy in pharmacology; the goal was for students to feel more confident about their ability to learn the material and approach pharmacology problems on their own. Student satisfaction with pharmacology as a discipline was also an important goal, as we expected that the supplement would help students learn and feel as if they had more support at an earlier point in the curriculum. These goals were assessed by optional surveys at the beginning and end of curricular blocks, usually presented in the context of existing curricular assessment surveys. The collaboration with our Academic Computing office provided an avenue to track the number of times each student accessed the online supplemental modules, as well as the number of completed self-assessment quizzes and pre- and post-tests. We used this information as an additional proxy for engagement with pharmacology, as increased “hits” to the modules and quiz completions would likely correlate with increased interest and/or effort spent learning pharmacology.

Design of Pharmacology Supplements Aligned with Curricular Blocks

“The Human Blueprint” is the second block of the Western Reserve2 Curriculum, but is the first block to deal extensively with molecular and organ-based medical sciences, including molecular biology, genetics, development, reproduction, endocrine systems, and cancer biology. It was during this block that the pharmacology supplement was initiated. The PBL-style cases early in “The Human Blueprint” block largely focus on developmental/chromosomal defects and genetics, in which drugs and therapeutic agents do not play a large role in treatment. Thus, while every effort was made to present materials in the pharmacology supplement that were relevant to students’ weekly PBL-style cases, it was not always possible to do so directly. During lectures, however, students were presented the topics of cellular receptors and tyrosine kinase inhibitors. Thus, we drew from lecture topics as an additional source of linkage with the existing curriculum.

The pharmacology supplement for “The Human Blueprint” block consisted of eight modules, with the following topics:

•What is Pharmacology?
•Drug Delivery
•Drug-Receptor Interactions
•Receptor Regulation
•Chronic Myelogenous Leukemia & Tyrosine Kinase Inhibitors
•Introduction to Endocrine and Reproductive Drugs
•Pharmacogenetics: Focus on Erlotinib
•Breast Cancer: Focus on HER-2 Inhibitors

We provided a brief introduction in the first module of the supplement (“What is Pharmacology?”) to the discipline of Pharmacology, as well as an introduction to the online supplement itself. In the second module, “Drug Delivery,” the topics of bioavailability, routes of drug administration, steady-state, and half-life were introduced, as these were viewed to be basic pharmacology concepts that would be useful for the students to learn as they began considering individual drugs. The third module, “Drug-Receptor Interactions,” encompassed the subjects of receptors, agonists/antagonists (competitive and non-competitive), drug-receptor interactions, and graded log dose-response curves (including concepts of potency and efficacy). During the first two weeks of the existing curriculum, intracellular signaling pathways and receptors were presented, providing a foundation for the discussion of drug-receptor interactions. The fourth module, “Receptor Regulation,” discussed ligands and receptors, including androgen ablation therapy (relevant to topics of reproductive and endocrine systems more formally covered during the curriculum).

The fifth module, “Chronic Myelogenous Leukemia & Tyrosine Kinase Inhibitors,” involved a departure from the other modules; a patient scenario from a PBL-style case that had been presented during the primary curricular block was presented and expanded within the pharmacology module. The original author of the PBL-style case, a hematologist/oncologist, collaborated with the pharmacology supplement authors to select the actual therapeutic treatments that would be considered as options with such a patient. The physician was also very helpful in providing clinical explanations as to why a particular option was the first-line treatment and what patient characteristics might lead one to consider various alternative treatments. Students were presented with an extension of the PBL-style case, after the patient had failed her initial drug therapy, and asked which treatment they would select next. Students could select one of three different treatment modalities for the “patient,” with explanations for the role of each treatment in the management of chronic myelogenous leukemia. Students were also encouraged to view the explanations for the other treatments they did not select as their first choice in treating the “patient.”

In the sixth module, “Introduction to Endocrine and Reproductive Drugs,” students were provided guidelines regarding how to approach and assimilate information for individual drugs as they progressed through the curriculum. Lists of clinically significant drugs were provided to students, with tips on how to think about therapeutic agents and what were the most important features they should understand. The actual details pertaining to each drug were not provided to students, because we felt that there would be educational value in having students create their own drug “flashcards” or lists to suit their needs. In the seventh module, “Pharmacogenetics: Focus on Erlotinib,” patient variability in responding to drugs, including concepts of pharmacogenetics, was discussed. This module used a patient from one of the PBL-style cases occurring concurrently in the curriculum to further reinforce the material. In the eighth and last module for the curricular block, “Breast Cancer: Focus on HER-2 Inhibitors,” alignment with one of the PBL-style cases from the week was achieved as the topic of monoclonal antibodies and cellular receptors in the treatment of breast cancer was covered and the module expanded what the students had learned from the original case.

During the third curricular block, “Food to Fuel,” which focuses on the topics of biochemistry, metabolism, nutrition, and the gastrointestinal system, the online pharmacology supplements focused on various pharmacokinetic topics. The following nine modules were designed to align with the existing curriculum where possible:

•Vitamins
•Drug Absorption
•Drug Distribution I
•Drug Distribution II
•Drug Absorption: Clinical Case Example
•Drug Distribution: Clinical Case Example
•Introduction to Drug Metabolism
•Drug Metabolism: Focus on Pharmacogenetics
•Introduction to Gastrointestinal System Drugs

Early in the “Food to Fuel” curricular block, students are exposed to concepts in biochemistry and nutrition. The topic of “Vitamins” was therefore chosen as an introductory topic for the pharmacology supplement. Subsequent modules took advantage of the concomitant formal teaching and learning of the gastrointestinal system that takes place during this curricular block and introduced the concepts of drug absorption, distribution and metabolism. Introductions and case-based examples were provided for each of these topics.

Formal teaching of pharmacological principles occurs during the final curricular block of the first year. This “Homeostasis” block includes a focus on cell physiology, cardiovascular physiology, renal physiology, pulmonary physiology, and principles of pharmacology. No online supplements were provided during this block because formalized contact time relating to pharmacology occurs during this curricular block, reinforcing the content from the online modules.

RESULTS: IF YOU BUILT IT, WILL THEY COME?

Since this fledgling supplement to the curriculum was optional, we had significant concerns that few students would peruse the modules. However, for the pharmacology supplement modules that took place during “The Human Blueprint” curricular block, 84% of students (139 students) visited the supplement at least once, and 20% (33 students) completed at least 86% of the available weekly self-assessment quizzes. Significant effort was made to inform the students about the modules, including an in-person presentation that featured students from the previous year who explained that this supplement was a result of their feedback for additional educational support in pharmacology. Student word of mouth was also important; several students reported to the authors that they had heard that Pharmacology was a difficult subject to learn on your own before it was “officially” presented within the “Homeostasis” curricular block, and that they would benefit from some additional support.

Some students completed the online pharmacology supplement modules paced with the required system-based block curriculum as the modules were intended to be approached. Others who had not kept up at this pace completed the modules immediately prior to the start of the “Homeostasis” block, the curricular block in which formal presentation of principles of pharmacology occurs. Anecdotally, students who completed the modules indicated that they did not take more than the expected 20-30 minutes to complete, that the modules helped them to become familiar with pharmacology prior to more formally encountering the material during the “Homeostasis” curricular block, that completion of the modules helped them to feel more comfortable in learning the dense pharmacology formally presented during the “Homeostasis” block, and that they were able to help teach their friends who had not completed the modules and who felt “very confused” and “lost.”

Although a complete discussion of the results of our investigation is beyond the scope of this report, preliminary results suggest that students who regularly used the online pharmacology supplement made significant gains in pharmacology knowledge. There is also an indication that first-year medical students, who had access to the supplement, were more satisfied with their pharmacology learning and knowledge than their counterparts from the previous year, who did not have access to the supplement.

Student perceptions of the curriculum are surveyed at the end of each block. The survey at the end of the “Homeostasis” block for the Class of 2011 (for whom the online pharmacology supplement had not yet been available) revealed a great deal of dissatisfaction with the integration of pharmacology concepts in the curriculum. In contrast to the other four topics surveyed, the mean rating for pharmacology was below the midpoint on a six-point scale (Figure 2). For the Class of 2012, the ratings of pharmacology concept integration were markedly higher than the previous year, although still the lowest of the five concepts surveyed. The pharmacology rating increase of nearly a full point on a six-point scale represents a medium effect size (Cohen’s d = 0.65) and was statistically significant: t (304) = 5.72, p < 0.001. This change appears to be specific to pharmacology; there was no general trend for increased ratings for the other four topics, with two increasing slightly (Cohen’s d = 0.24 and 0.09) and two decreasing slightly (Cohen’s d = -0.23 and -0.11), and no differences significant at an alpha level of 0.01 (all ts < 2.1, all ps > 0.04).

DISCUSSION/LESSONS LEARNED

Based on feedback from students, there was a need to supplement the pharmacology teaching in the existing curriculum at CWRU. Faculty members were also interested in increasing the presence of pharmacology early in the curriculum. With virtually no budget beyond “volunteer” faculty time, one student’s summer support, and a willingness of our Academic Computing office to facilitate our use of the existing electronic curriculum infrastructure, we successfully created supplemental online modules that appeared to increase pharmacology learning. Even though the modules were completely voluntary, a reasonable number of students used the modules and appeared to find them useful. Though they represented a self-selected group, students who used the modules regularly showed significant increases in pharmacology knowledge. Preliminary results and student feedback suggest that the students who had access to the supplement were more satisfied with their pharmacology learning than students from the previous year, who completed the curriculum before the supplement was created.

During this project, the authors learned some valuable lessons about creating a new supplemental curriculum. We were surprised to find that a significant number of students would use the supplement on a purely voluntary basis. Communication with the students was key; though we used several different methods to announce the existence of the supplement to the students, there were students who were upset and claimed that they did not know about the existence of the supplement until partway through the last curricular block of the first year. Increased administrative efforts to enhance student awareness may be necessary, given the diverse commitments that entering medical students have in their schedules. Anecdotally, student word of mouth appeared to be one of the best means of advertisement and reinforcement of the utility of completing the modules.

Creating content for the weekly modules was time-consuming. Even though the modules were generally very brief, the content had to be established, de novo, often in collaboration with multiple faculty members. Even formatting the modules with pictures and graphs into the available electronic curriculum system required considerable effort, and the learning curve was initially steep. For the first block of the supplement, modules were not completed until the morning the module was to be released. Now that the modules have been created, much less time is required for maintaining/updating content. In the future, additional modules will be designed to further supplement learning of therapeutic agents that occurs in the second year of the curriculum.

Figure 2. Student Ratings of Integration and Understanding for Curriculum Topics before and after Introduction of Online Pharmacology Supplement. Student ratings for pharmacology after the introduction of the online pharmacology supplement for the class of 2012 were markedly higher than the previous year (Class of 2011, before the introduction of the supplement), while no consistent trend for improvement was evident in the ratings of the other curriculum topics surveyed.

Obtaining organizational buy-in and approval from stakeholders was also more time consuming than anticipated. The effort required for engaging various faculty committees, the pharmacology faculty, and the administration is not to be underestimated. Ultimately, however, reactions have been positive and preliminary results suggest that this supplement increases student learning and satisfaction. We continue to work toward further integrating this longitudinal pharmacology supplement into the existing curriculum and enhancing student learning of fundamental principles of pharmacology and the appropriate utilization of therapeutic agents.

ACKNOWLEDGEMENTS
The authors would like to thank Terry M. Wolpaw, MD, MHPE, Klara K. Papp, PhD, Irene Medvedev, PhD, Wei Wang, MD, MS, and Siu Yan Scott, LSW, for their invaluable assistance in project development and data collection.

REFERENCES

1. Patel V.L., Yoskowitz, N.A., and Arocha, J.F. Towards effective evaluation and reform in medical education: a cognitive and learning sciences perspective. Adv Health Sci Educ. 2009; 14(5):791-812.
2. Antepohl, W., and Herzig, S. Problem-based learning versus lecture-based learning in a course of basic pharmacology: a controlled, randomized study. Med Edu. 1999; 33(2):106-133.
3. Dahle, L.O., Brynhildsen, J., Behrbohm Fallsberg, M., Rundquist, I., and Hammar, M. Pros and cons of vertical integration between clinical medicine and basic sciences within a problem-based undergraduate medical curriculum: examples and experiences from Linköping, Sweden. Med Teach. 2002; 24(3):280-285.
4. Neville, A.J. Problem-based learning and medical education forty years on. Med Prin Pract. 2009; 18(1):1-9.
5. Faingold, C.L., and Dunaway, G.A. Teaching pharmacology within a multidisciplinary organ system-based medical curriculum. N-S Arch Pharmacol. 2002; 366:18-25.
6. Schmidt, H. Integrating the teaching of basic sciences, clinical sciences, and biopsychosocial issues. Acad Med. 1998; 73(9):S24-S31.
7. Pinto Pereira, L.M., Telang, B.V., Butler, K.A., and Joseph, S.M. Preliminary evaluation of a new curriculum—incorporation of problem based learning (PBL) into the traditional format. Med Teach. 1993; 15(4):351-364.
8. Kwan, C.-Y. Problem-based learning and teaching of medical pharmacology. N-S Arch Pharmacol. 2002; 366(1):10-17.
9. Woodman, O.L., Dodds, A.E., Frauman, A.G., and Mosepele, M. Teaching pharmacology to medical students in an integrated problem-based learning curriculum: an Australian perspective. Acta Pharm Sinic. 2004; 25(9):1195-1203.
10. Kwan, C.-Y. Learning of medical pharmacology via innovation: a personal experience at McMaster and Asia. Acta Pharm Sinic. 2004; 25(9):1186-1194.
11. Donovan, J.J., and Radosevich, D.J. A meta-analytic review of the distribution of practice effect: now you see it, now you don’t. J Appl Psychol. 1999; 84:795–805.
12. Bahrick, H.P., and Hall, L.K. The importance of retrieval failures to long-term retention: a metacognitive explanation of the spacing effect. J Mem Lang. 2005; 52:566–577.
13. Cepeda, N.J., Pashler, H., Vul, E., Wixted, J.T., and Rohrer, D. Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychol Bull. 2006; 132:354–380.
14. Magid, M.S., and Schindler, M.K. Weekly open-book open-access computer-based quizzes for formative assessment in a medical school general pathology course. J Int Assoc Med Sci Educ. 2006; 17(1):45-51.
15. Ornt, D.B., Aron, D.C., King, N.B., Clementz, L.M., Frank, S., Wolpaw, T., Wilson-Delfosse, A., Wolpaw, D., Allan, T.M., Carroll, M., Thompson-Shaheen, K., Altose, M.D., and Horwitz, R.I. Population medicine in a curricular revision at Case Western Reserve. Acad Med. 2008; 83(4):327-331.
16. Shankar, R.P., Dubey, A.K., Palaian, S., Pranaya, M., Saha, A., and Deshpande, V.Y. Favorable student attitudes towards pharmacology in a medical college in western Nepal. J Int Assoc Med Sci Educ. 2005; 15:31-38.
17. Beal, T., Kemper, K.J., Gardiner, P., and Woods, C. Long-term impact of four different strategies for delivering an on-line curriculum about herbs and other dietary supplements. BMC Med Educ. 2006; 6:39.
18. Lymn, J.S., Bath-Hextall, F., and Wharrad, H.J. Pharmacology education for nurse-prescribing students – a lesson in reusable learning objects. BMC Nurs. 2008; 7:2.
19. Azer, S.A., and Frauman, A.G. Seeing the wood for the trees: approaches to teaching and assessing clinical pharmacology and therapeutics in a problem-based learning course. Ann Acad Med Singap. 2008; 37:204-9.
20. Lovell, K., and Plantegenest, G. Student utilization of digital versions of classroom lectures. J Int Assoc Med Sci Educ. 2009; 19(1):20-25.
21. Fernandes, L., Maley, M., and Cruickshank, C. The impact of online lecture recordings on learning outcomes in pharmacology. J Int Assoc Med Sci Educ. 2008; 18(2):62-70.
22. McNulty, J.A., Espiritu, B., and Halsey, M. Medical student use of computers correlates with personality. J Int Assoc Med Sci Educ. 2002; 12(1):9-13.
23. Kemper, K.J., Gardiner, P., and Woods, C. Changes in use of herbs and dietary supplements (HDS) among clinicians enrolled in an online curriculum. BMC Complem Altern M. 2007; 7:21.
24. Marsh, K.R. Developing computer-aided instruction within a medical college. J Int Assoc Med Sci Educ. 2008; 18(1S):17-28.