Generating Multiple-Choice Questions from an Existing Short-Answer Radiology Examination

INTRODUCTION

Multiple-choice exams are used frequently in medical education. They are reliable and allow for rapid marking and immediate feedback for students.1 However, creating and writing effective multiple-choice questions is a time-intensive process that involves 3 steps: writing clear objectives, defining levels of learning and following basic test writing rules. The three basic levels of learning are knowledge, combined comprehension and application, and problem solving.²

The National Board of Medical Examiners and the Medical Council of Canada have both published guides to aid in constructing effective test questions.^3,4 These guidelines demonstrate how to generate clear questions and appropriate answers. At the University of Ottawa, medical students are taught basic radiology as a component of a broad Foundations course. Short-answer radiology questions have traditionally been incorporated into a practical Foundations exam, in which examinees are tested on anatomy and microbiology as well. Examinees rotate through 12 radiology stations. The anatomy and histology components of the exam are tested in the multiple-choice format. Thus, marking the radiology component of the exam has been more difficult than marking the other components of the exam. This article describes the process of converting the previously existing short-answer questions to multiple-choice questions.

METHODS

After reviewing the existing short-answer questions, the exam writer formulated an educational objective for each question. The objective helped define the level of learning of each question. Several of the previously existing questions examined the knowledge level of learning, and some of these questions were omitted. After determining which level of learning was to be examined the exam writer formulated the question, ensuring that the question was related to the educational objective. The exam writer also ensured that the stem was a clear, direct question. Then, the exam writer generated -distracters by incorporating common learner errors into the options. An example of this process is found in Appendix. The exam was then reviewed and approved by members of the Faculty of Medicine.

DISCUSSION

Writing effective multiple-choice examinations is time-intensive, and it can be a difficult process. Several steps are involved in writing just one effective multiple-choice question. Writing clear objectives and determining the level of learning examined by individual questions are important steps that should be done prior to writing multiple-choice questions. Objectives should focus on important, relevant topics rather than on trivia and should be measurable. Vague statements, which use such words as understand and learn, result in unclear, immeasurable objectives.²

Multiple-choice questions can be used to examine higher levels of learning.² Adding clinical stems is one way to examine application of knowledge rather than simple recall of isolated trivia.⁴ When testing radiology, higher level questions require examinees to identify structures or findings correctly, interpret the image, and then use that information to answer questions correctly. Questions that require examinees to identify structures are examining at the knowledge learning level rather than at the application of knowledge learning level. However, when a learning objective states that recall or recognition is the task required of the examinee, questions that examine the knowledge level of learning are appropriate.²

Basic rules should be followed when one writes multiple-choice exams. Questions should be clear, and if possible, students should be able to answer the question without looking at the options.⁴ All information necessary to answer the question should be provided2 and options, including the distracters should be homogenous.⁴ For example, if the question asks which structures are involved in inflammatory arthritis, all answers should be anatomical structures. Generally, negatively worded questions that include not or except should be avoided.²

A criticism of multiple-choice exams is that of a cueing effect which can aid students in identifying the right answer.⁵ Scores have been shown to be higher on multiple-choice exams than on open-ended exams, but this effect seems to diminish with a higher level of expertise on the part of the examinee.⁵ Questions related to identification of a diagnosis have demonstrated a more prominent cueing effect than questions related to management of patients and interpretation of data.⁶

This article demonstrates the process of converting short answer questions to multiple-choice questions. Multiple-choice questions are a reliable form of assessment that allow for rapid marking.^1,2 Following established and well- published guidelines can aid in the development of effective multiple-choice questions that examine high levels of learning.⁴

REFERENCES

1. Fowell, S., Bligh, J.G. Recent developments in assessing medical students. Postgrad. Med. J. 1998; 74(867):18-24.
2. Collins, J. Education techniques for lifelong learning: writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radiographics. 2006;26(2):543-551.
3. Case, S.M., Swanson, D.B. Constructing written test questions for the basic and clinical sciences. http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf. [Accessed: March 26, 2010]
4. Claire, T. The Medical Council of Canada guidelines for the development of multiple-choice questions. http://www.mcc.ca/pdf/MCQ_Guidelines_e.pdf.pdf . [Accessed: March 26, 2010]
5. Schuwirth, L.W., Van der Vleuten, C.P., Donkers, H.H. A closer look at cueing effects in multiple-choice questions. Med. Educ. 1996;30(1): 44-49.
6. Swanson, D., Norcini, J., Grosso, L. Assessment of clinical competence: written and computer based simulations. Assessment and Evaluation in Higher Education. 1987;12(3): 220-246.

Appendix

The original short answer question:

Using the radiographs answer the following questions:

• What is the arrow pointing to?
• In what pattern(s) of arthritis is this seen?
• What is the typical target site?

The multiple choice question generated:

A 40 year old woman presents to her family doctor with increasing fatigue over the last several months. She has pain, swelling and stiffness in her hands and feet. The following radiographs of her feet are obtained: