- Title: Constructing Written Test Questions For the Basic and Clinical Sciences
- Authors: Case and Swanson
- Access the original paper here
- Listen to a deep-dive podcast:
Paper summary
This document, “Constructing Written Test Questions For the Basic and Clinical Sciences,” is a guide for creating effective multiple-choice questions for medical examinations. It emphasizes assessing application of knowledge rather than rote memorization, advocating for clinically relevant scenarios and avoiding flawed question structures that benefit test-wise students. The manual details various question formats, including one-best-answer and extended-matching item types, providing templates and examples. It also addresses issues like item flaws, standard setting, and the use of clinical vignettes to enhance question quality and validity. Finally, it includes examples of effective test questions across various medical disciplines and learning styles.
What are the key implications for teachers in the classroom?
Based on the sources, here are some key implications for teachers in the classroom:
Teachers should design test questions that align with the level of practice expected of the examinee, particularly for management-focused questions.
Assessment is a critical component of instruction. Teachers should use assessments to communicate what they view as important, motivate students to study, identify areas of deficiency, determine grades, and identify weaknesses in the curriculum.
Tests should align with educational goals. Teachers should develop tests that assess skills in interpreting data and making decisions, not just recall of isolated facts. The method used for assessment does not determine the component of competence measured by the test.
Teachers should sample topics and skills when creating assessments, ensuring the amount of attention given to a topic reflects its relative importance. The sample of test questions should represent the instructional goals.
Teachers should use a variety of testing methods because no one method can assess all the skills of interest.
The psychometric characteristics of assessments are important for high-stakes tests. For tests used for promotion or graduation decisions, the results must be reproducible and accurate. For low-stakes tests, the primary focus should be on directing student learning.
Teachers should avoid using true/false questions because they often lead to ambiguity and assess recall of isolated facts. One-best-answer questions are better for assessing application of knowledge, integration, synthesis, and judgement.
Teachers should avoid negative A-type questions such as “Which of the following is NOT correct?”.
Teachers should be aware of technical item flaws related to testwiseness (e.g., grammatical or logical cues, long correct answers) and irrelevant difficulty (e.g., long, complicated, or double options, vague frequency terms). Such flaws make it easier for some students to answer correctly based on test-taking skills alone.
Each test item should focus on an important concept, typically a common or potentially catastrophic clinical problem, rather than trivial facts.
Each item should assess application of knowledge, not recall of isolated facts. Clinical vignettes provide a good basis for questions.
The stem of an item should pose a clear question that can be answered without looking at the options.
All distractors should be homogeneous, plausible, grammatically consistent, logically compatible, and of the same length as the correct answer.
Teachers should avoid using “none of the above” as an option, which essentially turns the item into a true/false item.
Teachers should use experimental and clinical vignettes to test application of knowledge in the basic sciences.
Teachers should focus on key concepts and principles that are essential for students to understand without references.
Teachers should avoid items that only require recall of isolated facts or esoteric topics that are not essential.
Teachers should use clinical vignettes to pose medical decisions in patient care situations for the clinical sciences.
Teachers should structure questions to assess application of knowledge rather than recall of isolated facts by structuring them in a more realistic manner.
Teachers should use patient vignettes to start item stems and include relevant components in a specified order: Age, Gender, Site of Care, Presenting Complaint, Duration, Patient History, Physical Findings, Results of Diagnostic Studies, and Initial Treatment.
Teachers should provide reference materials in test questions when, in real life, one would be likely to refer to a reference source.
Teachers should avoid questions that include “red herrings” or that try to trick students. The focus should be on important findings.
Teachers should be aware that “vague frequency terms” are not consistently interpreted, even by experts.
Teachers can use case clusters, which are multiple-choice questions associated with the same patient presentation. In this case, teachers should avoid “cueing” and “hinging”.
Teachers can use extended-matching (R-type) items. For these, teachers should write the theme, lead-in, and options first, and the item stems last.
Teachers can use Pick N items, which instruct examinees to select a specific number of responses.
Teachers should interpret item analysis results to evaluate the quality of an item and the accuracy of the answer key.
Teachers need to make judgements when setting standards, and involving multiple judges will reduce bias.
Teachers should consider using frequent short quizzes rather than infrequent tests.
Teachers can use take-home and open-book exams to stimulate broad and deep learning.
Teachers should consider cumulative tests to encourage inter-relationships among topics.
Teachers should use integrative, cross-course tests to encourage students to see relationships among disciplines and topics.
Quote
A primary purpose of testing is to communicate what you view as important. Tests are a powerful motivator, and students will learn what they believe you value
