- Title: Diagnostic Questions: Is There Value in Just One?
- Authors: Dylan Wiliam
- Access the original paper here
- Listen to a deep-dive podcast:
Paper summary
This paper contrasts two approaches to data-driven instruction: a “data-push” model, where teachers receive periodic test results, and a “decision-pull” model, exemplified by the DIMS project. The DIMS project focuses on using single, high-quality questions to inform immediate instructional decisions, emphasizing the interpretability of both correct and incorrect answers. The authors argue that while a single question has lower reliability than a longer test, its utility in real-time classroom adjustments outweighs this limitation, especially when considering the class as a whole. The paper details characteristics of effective “decision-pull” questions and discusses how to select questions that provide good coverage of the learning domain. Ultimately, the authors advocate for integrating assessment seamlessly with instruction
What are the key implications for teachers in the classroom?
- Teachers can use single, high-quality questions to make quick decisions about what students understand and to adjust their teaching to meet students’ learning needs. This is called a “decision-pull” approach to data-driven instruction, and it contrasts with the “data-push” approach where teachers are given large amounts of data that they may not have time to use or may not find helpful.
- Questions used in a decision-pull approach should be designed so that teachers can easily and quickly collect information from every student in the class. Multiple-choice questions are often used for this purpose, and questions with multiple correct answers are especially helpful because they provide more information about student understanding. Teachers can also use “mini-whiteboards” or “exit tickets” to collect information from students.
- In addition to being easy to collect, good decision-pull questions should also have incorrect answers that help teachers understand what students do not understand and provide ideas about what to do about it. This means that the incorrect answers should be based on common student misconceptions or errors. It is also important that the correct answers be interpretable, meaning that students who select the correct answer are doing so because they understand the concept.
- While the reliability of a single question is lower than the reliability of a test with multiple questions, a single question can still provide enough information to guide instruction in real-time, especially if the question is well-chosen. Teachers can increase the efficiency of their assessments by choosing questions that cover the most important concepts in the domain being assessed. They can also use questions that are likely to elicit common student misconceptions.
The sources also highlight the importance of teachers being able to use assessment information to make instructional decisions. Teachers who collect information about student achievement but are unable to do anything with that information get worse results than teachers who do not collect such information at all. For assessment to be effective, it needs to be integrated with instruction. The decision-pull approach to assessment, where teachers use single questions to make real-time instructional decisions, is one way to achieve this integration.
Quote
While the accuracy of a decision based on a single item may be far from perfect, it is better than a decision based on no data at all.
