- Title: Madiha Sajjad, Samina Iltaf, Rehan Ahmed Khan
- Authors:
- Access the original paper here
- Listen to a deep-dive podcast:
Paper summary
This 2020 article from the Pakistan Journal of Medical Sciences examines the quality of multiple-choice questions (MCQs) used in a medical school exam. The researchers analyzed distractor efficiency, identifying non-functional distractors (those chosen by less than 5% of students) as indicators of poorly written questions. They found a high frequency of item-writing flaws, such as low cognitive level questions and illogical distractors, impacting distractor efficiency and overall exam quality. The study’s findings highlight the need for improved MCQ writing to enhance assessment accuracy and student performance. The authors suggest that addressing these flaws can improve the reliability and validity of medical exams.
What are the key implications for teachers in the classroom?
The sources discuss several implications for teachers regarding the use of multiple-choice questions (MCQs) in the classroom, primarily focusing on how to improve the quality of these questions by analyzing distractor efficiency. Here are some key points for teachers to consider:
- Understanding Distractor Efficiency: Teachers should be aware of the concept of distractor efficiency, which refers to how well the incorrect options (distractors) in an MCQ function. A distractor is considered functional if it is chosen by more than 5% of the examinees, and non-functional if chosen by less than 5%. Analyzing distractor efficiency helps identify flaws in the construction of MCQs.
- Analyzing Item Flaws: Teachers should analyze MCQs for item-writing flaws, especially in items with low to moderate distractor efficiency. This analysis can help them understand why some distractors are not being chosen by students. Flaws can be categorized as:
- Within option flaws: These include distractors that are not homogenous in length or content, and repeated distractors.
- Alignment flaws: These occur between options and the stem or lead-in of the question, such as linguistic cues, logic cues, and irrelevant distractors.
- Other flaws: These include low cognitive level items and unnecessarily complicated stems.
- Focus on Cognitive Level: Teachers should ensure that MCQs are written at an appropriate cognitive level for their students. Questions should not focus on trivial facts but rather on more complex concepts and applications. Low cognitive level items, which rely on recall, may be too easy and cause students to ignore the distractors.
- Creating Plausible Distractors: Teachers should focus on creating plausible distractors that are likely to be chosen by students who have not fully grasped the concept being tested. It’s important to recognize that some questions inherently have fewer plausible options. The quality of distractors is more crucial than the number of distractors.
- Avoiding Cues: Teachers need to avoid cues within the questions that lead students to the correct answer without them needing to understand the content. These can be linguistic or logical cues found within the stem or distractors.
- Clarity in Stem and Lead-in: The stem (background information) and lead-in (the question itself) of an MCQ should be clear and unambiguous. Vague or confusing terms in the stem or lead-in can negatively impact student performance.
- The Impact of Language: Teachers should be aware that language can affect how students understand and respond to MCQs, particularly in settings where the medium of instruction is not the students’ primary language. The effect of language on the validity of test results needs to be considered.
- Importance of Item Analysis: Post hoc item analysis is a useful tool that teachers can use to assess the quality of their MCQs. This analysis can provide valuable insights into the reliability and validity of test items.
- Training and Development: Faculty development programs that train teachers in standard item-writing guidelines can significantly improve the quality of MCQs.
By paying attention to these key factors, teachers can develop more effective MCQs that accurately assess student learning and reduce the impact of “test-wiseness”. Improving the quality of distractors also enhances the discriminatory ability of the questions. Ultimately, this leads to better evaluations and a better understanding of students’ knowledge.
Quote
Analyzing the low to medium DE items for item writing flaws, provides valuable information about item writing errors which negatively impact the distractor efficiency
