- Title: Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review
- Authors: Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X.
- Access the original paper here
- Listen to a deep-dive podcast:
Paper summary
This literature review from the Review of Educational Research examines the creation, analysis, and use of distractors in multiple-choice tests. The authors synthesize existing research on distractor development strategies, focusing on identifying common misconceptions and creating similar-but-incorrect options. They explore various psychometric methods for evaluating distractor quality, including classical test theory and item response theory. The optimal number and placement of distractors are also discussed, along with recommendations for best practices. Finally, the authors propose innovative distractor development methods using automatic item generation and key features to address challenges in creating large numbers of high-quality items.
What are the key implications for teachers in the classroom?
Based on the provided document, here are some key implications for teachers in the classroom regarding the development, analysis, and use of multiple-choice tests, particularly focusing on the role of distractors:
- Distractor Creation: Teachers should focus on creating plausible distractors that reflect common student misconceptions or errors in thinking. This can be achieved by reviewing students’ responses to open-ended questions, using verbal reports, or considering typical student errors.
- Similarity: Distractors should be similar in content and structure to the correct option. This includes similarities in length, complexity, formatting, and grammar. For numeric options, this could mean using factors from the correct answer to create incorrect options.
- Analysis of Distractors:
- Teachers should analyze the frequency with which students select each distractor. Distractors chosen by less than 5% of students might be nonfunctioning and need revision.
- Teachers should utilize trace line plots to identify distractors that do not differentiate between low and high-achieving students.
- The choice mean of the correct option should be higher than the choice mean of any distractor. If a distractor has a higher choice mean than the correct option, it needs to be reviewed.
- Teachers can use the point-biserial correlation to evaluate the effectiveness of distractors. A value greater than -0.05 indicates a distractor is not discriminating adequately.
- Diagnostic Information: Distractors can provide valuable diagnostic information about student understanding and misconceptions. By analyzing which distractors students choose, teachers can identify areas where students need additional instruction.
- Item Writing Guidelines: Teachers should adhere to item-writing guidelines for creating multiple-choice questions. Key guidelines include using plausible distractors, placing them in a logical order, keeping them independent and homogeneous, and avoiding clues to the correct answer.
- Number of Distractors: There is evidence that using two distractors and one correct option is often optimal. This can reduce test development time, reduce student test-taking time, and improve item quality.
- Ordering of Distractors: The position of the correct answer and distractors can affect the difficulty of a multiple-choice item. While some researchers suggest placing distractors in a logical or numerical order, others recommend randomizing the order to avoid bias.
- Automated Item Generation (AIG): Teachers may use automated processes to generate test items, as this process can help with efficiency. In order to generate multiple choice items, teachers can use key features to generate plausible distractors. Teachers can also create distractors based on content similarity by searching databases for semantically related concepts.
In summary, teachers should see distractors not just as incorrect options, but as a valuable tool to understand student thinking, improve test quality, and enhance instruction. Teachers should therefore invest time in careful development and analysis of distractors when designing multiple choice tests.
Quote
Multiple-choice testing is considered one of the most effective and enduring forms of educational assessment that remains in practice today
