Analysis on the Effects of Block Testing in the Medical Preclinical Curriculum

INTRODUCTION

Preclinical medical curricula, especially in schools using a traditional discipline-based curricular structure, are often characterized by frequent single course examinations which may lead students to memorize that course material for short term retention. One result of this type of “study and forget” cycling is difficulties when the students are facing USMLE Part 1 examinations, because they must relearn much of the material and have no overall understanding of how the subject material from the different courses correlates. Block testing was initiated to encourage a more integrated learning model (see Figure 1).

In this innovative testing model, students are allowed a long period of time (4 or 5 weeks) that is examination-free to independently spend the time in learning and correlating the subject material for all the courses in the study block. At the end of this period, Block week occurs–4 days of free time for study followed by one day (Friday) with 6 hours of a comprehensive examination with all subjects taught in that Block represented proportionally on the examination (for a description of the testing times during the Block Examination, see Materials and Methods). Since each course is represented by questions proportional to the time spent in that course, the problem of having a comprehensive examination which only tests a subset of knowledge is avoided.¹ In this manuscript we discuss how the implementation of the Block Examination process at two medical schools has affected USMLE Part 1 scores, students, and faculty.

MATERIALS AND METHODS

Block Examination Committee
At the University of Louisville, the Block Examination Committee for the 6 Block tests consists of all second year course directors and two clinical representatives. The Chair of this committee, who oversees the Block Exam initiative, is appointed by the Dean of Curriculum at the Medical School. At the Medical University of South Carolina the Year 1 and Year 2 committees, composed of course directors, are responsible for the preparation of 6 Block tests in each preclinical year.

Block Examination question development

Each participating course (usually there are 3-4 courses participating in each Block test) submits 3-4 questions per lecture hour to the Block Exam Committee for the examination, proportional to their lecture time in that Block. The questions are submitted a minimum of two to three weeks prior to the test. The questions are carefully read by the Committee, and any questions that do not conform to the NBME format are discarded and the question writers are asked to submit new questions or to rewrite the discarded question.² In addition, the text of the questions is optimized to reflect the nature of the question in the least number of words possible. Questions with extended matching and clinical scenario questions are encouraged by the Committee from the courses.^3,4 The questions are also reviewed for both basic science and clinical accuracy by the Committee. Any revised questions are returned to the writers of the questions to verify that the changes are acceptable and resubmitted for final approval. The Chair, then requests the

course directors to examine possibilities for merging questions across discipline lines but testing similar subject matter. This can be done, for instance, by having a clinical scenario or description to which specific questions from different disciplines can be directed. Our Block Examinations usually have approximately⁵ such merged questions.

Assembly of the Block Examination
At the University of Louisville, once the question set is approved, the questions are entered into the LXR Test Version 6 system (www.lxr.com) and randomized into 6 sets of no more than 50 questions per set, distributing the questions with any attached illustrations, tables, and graphs. The assembled examination is proofread carefully by the Chair of the Block Exam Committee to make sure cross-referencing questions are not in the same section, detect any duplicate questions, and ensure that the sections are relatively equally balanced. The examination then is copied and assembled into six booklets with a cover page that states the examination number, section of the exam, and the number of questions and pages represented in this section of the examination. Answer sheets are also imprinted with the section number. A similar methodology is used in year 1 block tests at the Medical University of South Carolina; however, only two booklets are printed, one for a morning session and one for an afternoon session. Each session is 3 hours long, and the maximum number of questions per session is 150. An identical protocol for administration of the tests is followed in year 2, but the Year 2 coordinator does the assembly and randomization of test items.

Examination Day
At the University of Louisville, the Block Examination Committee Chairman distributes to the class of 144 students the schedule for the examination day (see Table 1). That day starts with instructions to the class, and then after each testing section (there are 6 total with a maximum of 50 questions in each) the students turn in their answer sheets, which are duplicated and graded. During the 15 minute break time, the Chairman and associates remove all the previous section booklets from the student seats and replace them with next section booklets and answer sheets. At the end of the Block Examination, the six answer sheets for each student are copied, collated and made available to the students. Then, in a locked room, the entire Block test is displayed by sections on the wall, along with several copies of the key and a designation of the course that provided each question. At that time, students may grade their tests, examine the questions, and discuss the examination with

others in the room. They may also draft challenges to specific questions on sheets mounted on the wall by each section.

At the Medical University of South Carolina the morning and afternoon booklets are distributed to the class of 146 students and collected at the end of each session (see Table 1). The booklets contain a comment sheet where the students are encouraged to express concerns about any item. All comments are carefully evaluated before the test is graded. After the test is graded, once all students have completed the exam (including any who postponed the test for reasons considered valid by the Office of Student Affairs and Academics), and course grades have been returned, a proctored review session is scheduled for students to review the exam and their answers. During the exam review session, coordinated by the Office of Student Affairs and Academics, the students may review their individual exam with the answer key. Students are also permitted to discuss the exam during the session with their classmates and may write challenges to questions. When a student has completed reviewing the examination materials, all materials, including written challenges to the questions, are returned to the session proctor.

Challenges to questions
At the University of Louisville, a committee of 8 students is elected by the class to examine challenges and verify the challenges using books, notes and other sources. The Chair of this committee then submits the approved challenges to the Chair of the Block Examination Committee by Monday evening following test week, who then distributes them to the appropriate course directors. Each course decides which challenges are acceptable and modifies the scoring key to accommodate any changes. The examination is rescored and final scores are separated for each course and distributed to the participating course directors, who release them to the students, if possible, by Friday following the Block Examination. No total Block test grade is used in grading. At the Medical University of South Carolina, the written challenges are forwarded to the respective course directors by the Office of Student Affairs and Academics; students meet with the course director and follow up about their questions and challenges; the course directors then decide which challenges are acceptable and modify the scoring key to accommodate any changes.

Satisfaction surveys
At the University of Louisville, the Office of Medical Education administers a survey to all students at the end of the Fall and Spring semesters. Included in this survey are questions regarding the Block Examination process and the student opinions about this type of testing methodology. The survey is done using a Likert type scale (1=strong disagree to 5=strongly agree).

At the Medical University of South Carolina, students complete end of the year surveys at the end of the first and second year to provide feedback about the overall curriculum and particular curricular goals. Two items on the surveys directly assess students’ perceptions of the block testing through a Likert-type scale (as above): “The comprehensive exams helped me integrate learning issues

from different classes” and “I like the comprehensive exams rather than individual course exams.?

Statistical analysis
Analysis of USMLE Part 1 scores for students at both universities prior and post implementation of Block Examinations was done with SPSS, Version 14.0.

RESULTS

Student and faculty satisfaction
Results from the satisfaction survey for the students at the University of Louisville are shown in Table 2. Survey results indicate that students in the first class exposed to this testing were apprehensive and in some cases negatively disposed toward this new testing model. However, attitude became more positive over time. Students have also begun to view Block Examinations as a positive preparation for the USMLE Part 1 examination. The administration and implementation of the Block tests has been optimized with time and experience. The course faculty have been generally supportive (based on willingness to participate extensively in the examination process) of this testing method for several reasons: the Block tests are sequestered and this allows faculty to simply modify and upgrade questions year to year, rather than keep producing new and consequently often over-detailed questions; also, the Block Examination Committee assembles and runs the examinations so that duty


is obviated from departmental faculty. A byproduct of this process is the constant improvement of questions by the Block Examination Committee and the faculty to more closely approximate the USMLE Step 1 tests; in addition, the challenge process involves only receiving documented challenges from the student challenge committee, rather than taking time to discuss the same challenges over and over with individual students; finally, any concern about this testing method has been alleviated because student grade averages in individual courses generally have not suffered with this testing, though taking an examination where the questions are integrated among the different disciplines represented, represents a different test taking paradigm than they have experienced previously in single course/subject testing.

Since the introduction of the Block testing at the Medical University of South Carolina, results from students on the end of year 1 and end of year 2 curriculum effectiveness surveys have been consistently positive (Table 3). The quality of the tests has consistently improved, in part, because of faculty development efforts to improve course director and teaching faculty test writing skills and also because the tests are not released except under proctored circumstances and the honor code is upheld. Thus, there is opportunity to continually improve the quality of the items without having to complete a new test every year. The faculty has responded well to the system and the advantages of group review of test items and consideration of post-test comments have become obvious, as reflected by the fact that successful challenges to questions are very rare.

Effect on USMLE Part 1 Scores
This testing method (above) differs radically from traditional stand-alone, course-based examinations. By providing students with 4-5 weeks free of examinations and then several days free of all academic responsibilities for study, it was hoped that the students would be able to correlate the material in all of the courses being taught and gain deeper, more permanent understanding of underlying principles. This, in time, could translate into better performance on the USMLE Part I examination, which tests overall understanding and often includes questions that require thinking across discipline lines. The results at the University of Louisville, School of Medicine and at the Medical University of South Carolina are shown in Figures 2a and 2b and Table 4. Statistical analysis measuring only the USMLE 1 scores independently from any other factors for the University of Louisville group shows that for the years 1996-2002 (n= 946, prior to Block testing) the mean Part 1 score was 209.85 with a SD of 22.16 and SEM of 0.720 while for the years following initiation of Block tests (n=412, 2003-2005) the mean Part 1 score was 215.04 with SD=22.01 and SEM of 1.084. By T-test, df=1356, t= -3.976 and p 0.001. These data suggest the Block tests are shown to have made a statistically significant contribution to the board scores of students at the University of Louisville. Data from MUSC also indicate a similar positive significant influence on USMLE Step 1 scores following the adoption of Block tests. For the years 1996-2000 (n=695), the mean Part 1 score was 205.77 with a SD of 21.18 and SEM of 0.80, while for the years following the implementation of Block tests, 2001-2005 (n=671), the mean Part 1 score was 215.41 with a SD of 22.86 and SEM of 0.88. By T-test, df=1364, t=8.085 and p 0.001. Though these results are encouraging, it must be noted that at both schools there is a multitude of board preparation efforts and curriculum implementations which may also contribute partially to this success, though many of those efforts bracket the entire time examined, pre- and post-Block Examinations. In addition, an examination of entering grade point and MCAT scores revealed that the classes at both universities have been relatively homogenous with a slight increase in MCAT scores in the 2005 entering class at the University of Louisville (Table 5). Figures 2a and 2b also reveal the overall shift of individual scores to higher levels at both schools following initiation of Block testing. This finding suggests that the Block Examination method may have a positive influence on retention and assimilation of basic science content as tested on the Part 1 exam.

DISCUSSION

Both medical schools represented in this study have documented positive and sustainable increases in USMLE Part I results following initiation of Block testing, although there may be other factors (i.e. other Board preparation efforts such as question groups, shelf tests, etc) which may also partially influence these results. The MCAT and GPA scores have been relatively homogenous and don’t appear to play a significant part in this increase.^5-7 Nevertheless, increased student satisfaction and the general acceptance and

participation by the teaching faculty for this method of testing have allowed Block testing to become an integral component of the evaluation system at both of these medical schools. This cooperation among the course directors in implementing Block testing has also led to improvement of the tests over the last couple of years with better organization and increasingly better questions. Moreover, the review of the Block test by the Block Examination Committee leads to linking of questions across course lines, as well as a better appreciation of other course curricula by the course directors, leading at times to synchronization of lecture presentations.

Another result of Block testing has become apparent over time. At the University of Louisville, we allow students ample review time to examine their answers against posted keys and actual reproductions of the examination that identify the department testing each question. This permits the students to calculate an approximate score in each subject, pending challenge; to challenge questions; and, most importantly, the opportunity to discuss the test with classmates. These discussions provide students with an additional learning opportunity where students explain to other students why certain answers are correct while others are not. On average, about a third of the class participates in this discussion, which we believe reinforces retention of the preclinical material among participants.

The challenge process is well received by both students and faculty. At the University of Louisville, having a student committee examine the challenges and verify the valid ones with references to notes, books and sometimes even published papers, makes the challenge process much less adversarial and more efficient with between 60 and 70% of all committee-approved challenges being accepted by the Departments. Moreover, since course faculty are not directly involved in this process, other than to consult with the course director on accepting a challenge to their question, individual students no longer take up faculty time with challenges.

At the Medical University of South Carolina, the post-test review process is not as elaborate but also gives the students an opportunity to learn from their errors. In addition, there has been a concerted effort to reduce the time between the test and grade posting to allow the students a quick transition from a test debate into learning the new topics that are presented as soon as the test is over.

Block testing also demonstrates to the students the physical and mental rigors of a whole day examination, which is similar to USMLE Part 1 testing. It also provides practical evidence that studying methods need to change from short-term memorization to a more integrated long-term understanding. Finally, the change from a single subject examination where the students know the origin of the next

question, to an examination where questions from different disciplines alternate, requires an adjustment to test taking skills. However, based on evidence from student satisfaction surveys at both medical schools as well as a survey of course grades (below), the students appear to adapt to this test taking from both physical and mental aspects quite quickly (usually by the second Block Examination) (see Table 2). Since the course average scores for the class in individual courses, usually by the second Block Examination, reach identical levels as in the years prior to Block tests, we feel the students become comfortable with integrated testing and that this comfort zone extends ultimately to the USMLE Part 1 examination.8 As an example, in Medical Microbiology and Immunology, the course average fluctuated between 81.2 and 83.1 in a few years prior to Block testing. In the last three years, post Block testing, the course average score has been: 82.9, 81.8, and 81.8.

This paper shows the effect of implementing Block testing at two different medical schools. The Medical University of South Carolina initiated Block testing in both preclinical years simultaneously in 2001. At the University of Louisville based on the apparent success at MUSC, but for logistical, internal reasons, Block testing in 2003 could be started only in the second preclinical year, which contains Medical Microbiology and Immunology, Pathology, Medical Genetics, Introduction to Clinical Medicine, and Clinical Neurosciences. Consequently, it is interesting that the implementation of Block testing only in the second year at the University of Louisville yielded results that were quite similar to those from MUSC, where Block testing has been run for both preclinical years. This may indicate that experience with this kind of testing even for one school year prepares the students sufficiently for the physical and mental rigors of the USMLE Part 1 examination and helps them achieve better scores. It is also possible that this type of testing changes the way students study and retain information, integrating second year subjects and being better prepared for the material presented on the USMLE Part 1 examination.

The progress of Block testing at both schools will be monitored with the expectation that results will remain as reported. While a few years of experience have allowed both schools to run the Block Examinations seamlessly, there are still many interesting avenues to explore. For instance, it will be very interesting to determine if learning increases with the implementation of Block testing. This would need to be examined both at the clinical expertise level and also in national examinations beyond Step 1. We need to find a compatible mechanism for offering subject Shelf Tests within the Block testing system. We are also continually striving to incorporate integrated questions which include more than a single department questioning a single clinical scenario. It will be important to determine how students respond to these types of questions.² The University of Louisville has also now instituted Block testing in the second semester of the first pre-clinical year and we are interested to know how this class will perform on the USMLE Part 1 examination after 9 Block Examinations.

In sum, initiation of Block testing has so far been an extremely positive experience for both medical schools and their student populations. This system should be applicable to many medical, dental, osteopathic, and veterinary schools which need to prepare their students for qualifying examinations.

REFERENCES

1. Markert, R.J. When to use a comprehensive final examination. Journal of Medical Education. 1981; 6: 198-199.
2. Case, S.M. and Swanson, D.B. Constructing written questions for the basic and clinical sciences (3rd edition/revised).National Board of Medical Examiners. 2002.
3. Case, S.M. and Downing, S.M. Extended matching items: a practical alternative to free response questions. Teaching and Learning in Medicine. 1993; 5:107-115.
4. Bhakta, B., Tennant, A., Horton, M., Lawton, G. and Andrich, D. Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education. BioMedCentral Medical Education. 2005; 5:9 (http://www.biomedcentral.com/1472-6920/5/9).
5. Jones, R.F. and Thomae-Forgues, M. Validity of the MCAT in predicting performance in the first two years of medical school. Journal of Medical Education. 1984; 59: 455-464.
6. Vancouver, J.B., Reinhart, M.A., Solomon, D.J. and Haff, J.J. Testing for validity and bias in the use of GPA and MCAT in the selection of medical students. Academic Medicine. 1990; 65: 694-697.
7. Veloski, J.J., Callahan, C.A., Xu, G., Hojat, M. and Nash, D.B. Prediction of students’ performances on licensing examinations using age, race, sex, undergraduate GPAs, and MCAT scores. Academic Medicine. 2000; 75(supp): S28-30.
8. Ripkey, D.R., Swanson, D.B. and Case, S.C. School-to-school differences in Step 1 performance as a function of curriculum type and use of Step 1 in promotion/graduation requirements. Academic Medicine. 1998;73: S16-18.