At Northern U School of Medicine the faculty has been “asked” to write a higher percentage of exam questions that follow the National Board style where a short clinical vignette is followed by a question with 5 to 10 optional independent answers to chose from. Each question is to have a single correct answer. The answer set is not to have “None of the Above”, “All of the above”, combinations of two or more of other answer options or other forms of “K-type” variations. Many of the basic science faculty who teach in the “pre-clinical” curriculum feel very uncomfortable about how to write the vignettes and many faculty members, clinicians and basic scientists alike, find it hard to come up with good answer tests for many of the questions they write. Because of heavy demands on their time, the administration (the Curriculum Office) agreed to facilitate the development of a Question Bank with the long term goal of collecting a large assortment of validated questions for faculty to pick from rather than having to write new questions from year to year. To begin to establish this Question Bank it was decided that students of the current year would not be able to see exams except in the testing times.
In the past students could either get copies of the exam questions after the exam was finished or at least be able to view the exam again at some point after the testing time. While allowance to review the exam was designed to allow the students one additional learning opportunity, what was happening was that students handed down exams from one academic year to the next class of students. This practice therefore required the faculty to write new questions every year for all exams.
If a Question Bank was going to be developed, the student practice of “passing down” questions would have to be stopped. The faculty decided the quickest way to develop the Question Bank was to make it so that there would be no opportunity after the scheduled testing period for students to again see the questions. So this year they changed the policy and no courses are to allow any student to see exam questions at any time other than during the test session.
The students are outraged! They spoke strongly that they felt this significantly limited a wonderful learning opportunity. They, by way of the classes’ officers, sent an e-mail to the Dean asking that he intervene and override the faculty’s decision. What should he do? What is your opinion of the faculty’s need and the students’ needs in this situation?
Student’s Response
Medical school has redefined the word “busy” for me. I find that I work harder and longer now than I ever did during 4 years of engineering study or 10 years as a chemical engineer at a major oil and gas corporation. I understand that medical school is exhausting for faculty as well as students, so I can sympathize with professors who are concerned that the practice of passing down old exams will place extra demands on their time. However, the mounting demands on my time as a student and the pressure to pass National Board exams make it imperative that my study time be as focused and efficient as possible. Being able to review my tests in detail allows me to direct my effort to the right areas and correct any misconceptions I might have had.
Whether you like the term “High Yield” or not, I have come to learn that the phrase has definite application in medical education. Information comes at us by the truck-load with only an evening to digest it, understand it, commit relevant parts to memory, and then prepare for the next day’s load. I’m a bit embarrassed to admit that many days go by when I don’t even have time to read the textbooks – I may have 4 or 5 power point presentations (each with 60 or so slides) to review from daily lectures, which only leaves time for looking at pictures and tables in the text or for reviewing a board exam preparation book. Given these constraints, when I take a medical school exam I need to know whether or not I have mastered the topics being taught.
In academic circles I hear the words ‘summative’ and ‘formative’ used quite often to describe exams. While I know these two words have good qualifying purpose, they get away from the layman’s term that describes the real purpose of medical school exams: feedback. If an exam score is all that is provided and that score is anything less than 100%, then it is obvious that concepts are going unmastered, but which ones? Perhaps I am aware that there is a specific fact I do not know, but there is also the possibility that I believe I understand when I really do not.
When I was working as an engineer we had a common practice when a more seasoned engineer was teaching a younger colleague: after the teaching was done, the mentor would then erase what was drawn on the board and hand the chalk to the mentee and say “Now, explain to me everything we just talked about.” This was the test, and the feedback was immediate! If the concepts were not learned, then time was wasted and even worse, mistakes could be made. Medical school, or for that matter any academic subject, is no different. Honest and open feedback is the only road to improvement, and the teaching is incomplete without it.
In the case of Northern U School of Medicine, the Dean should meet with the faculty and student representatives to discuss the situation. He/she should encourage that group to come up with an innovative way to allow both the educational needs of the students and the long-term objectives of a Question Bank be achieved. This actually could become a win-win situation rather than the disaster that has developed to this point.
Faculty Member’s Response
The issue is complex as described. There are enough problems to insure all parties have an equal opportunity to be incorrect in their handling of the issue. There is also ample room for each group to improve the outcome for the benefit of all concerned.
There are a limited number of ways to write a question and once they are used, we will by necessity start repeating old test questions. The best way to prevent this is – don’t return the tests. This also prevents students from trying to learn only what will be on the test instead of learning how to be a successful physician.
Problem for the faculty member:
Students have a right to know what is expected from them in a course or in their clinical rotations. A quick review of CurrMIT clearly indicates a practical problem for the student. Objectives written by faculty across the country are: 1) unclear, 2) do not tell the student what and how their knowledge will be measured, 3) frequently fail to describe the depth of knowledge expected, and 4) often omit material that will most likely be included on the exam (and certainly on STEP exams). If these objectives are all students have to direct their study, their desire to see the exam (and old exams given previously by the professor) is obvious. This is the ONLY way the student can determine what is expected. The objectives do little to help the students and in many cases reinforce the student’s belief that medical education is a confrontational (us vs. them) learning environment.
Summary:
We are moving in this direction by requiring students and residents to demonstrated competency in required areas. Once again, this requirement MUST be presented to the student clearly, using well written objectives before the course or clerkship starts. Failure to do so is akin to professional misconduct by the faculty and the school. Telling students what is expected from them in a course of clerkship is not only okay – it is ethically required. It is also ethical to require each student to demonstrate their ability to perform in real life situations. This goes far beyond getting a test question correct. It demands the students demonstrate their ability to provide the quality of care needed by society. Requiring students to critically think rather the memorize information is a great starting point. Requiring students to demonstrate the application of critical thinking skills in patient simulators and supervised clinical settings is even better. When this approach is accepted as a primary teaching methodology in medical schools, the issue of returning exams will become a moot point – and we will all be raised to a new level in medical education.
Dean’s Response
The faculty and the students each have a valid point. Another aspect of this case is that institutional change always creates angst; students hand down more than exam questions, they hand down experiences and any change in that experience is most often met with skepticism. However, in this case, the faculty has it right if they are to offer the students the highest quality testing experience that they can create.
An effort should be made to get the students to understand that it is the in their best interest to be well prepared for licensing exams, as these exams have a significant impact on their future. Writing questions that are of the same quality as the USMLE questions takes much time and effort. The best questions are reviewed, edited and tested before they enter a permanent question bank. The process of writing good questions and the various rules that apply to their construction should be reviewed with students and the commitment to the students to create high quality tests that will better prepare them for their future should be affirmed.
In addition, the students should be assured that materials to help them prepare for the exams will be made widely available through a learning resource center. There are a variety of review materials for USMLE-type questions and a good array of these should be made available to students. Ideally, a site license for a computer-based test bank for the use of all students should also be purchased and installed for use in such a center.
Respondents
- 1. Student Respondent – Mr. Jason P. Cooper, MSII, Texas Tech University Health Sciences Center, Lubbock TX
2.Faculty Respondent – Dr. Herb Janssen, Ph.D., Professor, Department of Physiology, Texas Tech University Health Sciences Center, Lubbock TX
3.Dean Respondent – Dr. Dani McBeth, Ph.D. Associate Dean for Student Affairs, Associate Professor of Microbiology & Immunology, The Sophie Davis School of Biomedical Education, The City College of New York, New York NY