- Title: Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research
- Authors: Michael C. Rodriguez
- Access the original paper here
- Listen to a deep-dive podcast:
Paper summary
This meta-analysis synthesizes 80 years of research on multiple-choice item writing, focusing on the optimal number of response options. The study uses meta-analytic techniques to examine how altering the number of options impacts item difficulty, discrimination, test reliability, and validity. Results indicate that three options are generally optimal, maximizing test efficiency and potentially improving content coverage. However, the method of distractor removal significantly impacts reliability, with random removal proving detrimental. The findings suggest a revision of existing item-writing guidelines to prioritize three well-constructed options over a larger number of less effective distractors.
What are the key implications for teachers in the classroom?
Based on a meta-analysis of 80 years of research, the optimal number of options for multiple-choice (MC) items is three. This finding has several implications for teachers in the classroom:
- Efficiency in test development: Teachers can save time by creating only two plausible distractors instead of three or four, without compromising the quality of the test.
- Improved content coverage: With three-option items, teachers can administer more items in the same testing time, potentially improving content coverage. The inclusion of more high-quality items can also improve test score reliability, which can provide additional validity-related evidence.
- Item Analysis: Teachers should be aware that item analysis data is tentative and sample specific and should be used carefully. From an IRT perspective, the elimination of distractors makes items easier, which is irrelevant. In addition, in a standard setting context, item difficulty plays a significant role in popular methods. In a classroom context, grading standards may need adjustment if tests become easier due to the use of fewer options.
- Test Validity: Validity evidence should be gathered to ensure the quality of test item development, and item-writing guidelines could serve as a checklist. Test developers should provide a rationale for the number of options used in MC items.
- Reduced test-taking time: Students may complete three-option tests more quickly than tests with four or five options, which can be beneficial in timed testing situations. However, on highly speeded tests, lower-ability students may resort to blind guessing, which may have a detrimental effect on validity.
- Focus on distractor quality: Instead of focusing on creating more distractors, teachers should focus on creating high-quality, plausible distractors. The quality of the distractors, not the number of them, is what guards against awarding undeserved credit.
- Option deletion methods: When reducing the number of options, randomly deleting distractors can be more detrimental to reliability than deleting ineffective distractors. Teachers should consider which distractors to eliminate thoughtfully.
- Use of 3-option items: Based on the meta-analysis, teachers should use three options for multiple choice questions (one correct option and two plausible distractors), because using more options does not significantly improve item and test score statistics and often results in implausible distractors.
In summary, using three-option multiple-choice items can help teachers create more efficient, reliable, and valid assessments, as long as they focus on creating high-quality items and thoughtfully consider how to eliminate options when needed.
Quote
Three options are optimal for MC items in most settings
