More tips from Kate Jones
Video transcript
okay can we have your fifth and final tip please yes this is about effective multiple choice question design i’m a big fan as you are multiple choice questions but sadly there’s a lot of really badly designed questions out there and i worry about early careers teachers who i need to be trained on how to write an effective multiple choice question and i worry because they may go online and look for questions which which i do but if an experienced teacher’s uploaded a quiz you may assume that that’s effective and it might not be and probably the biggest mistakes that i see um including an obviously incorrect response and we can just write that off immediately and then the other one is what i refer to as the bradley walsh effect which is great on the chase where they have comedy options but not good with teaching and learning so that’s entertainment and i love the chase and i love when they do have a funny option and um but when we do that in school when teachers are trying to be hip and funny we’re diluting the level of challenge and the effectiveness of retrieval practice because i see it all the time which of these was henry viii’s wife lady gaga you know which character from romeo juliet justin bieber honestly that they’re everywhere and i’ve probably done them myself as well so that’s the first thing as well have a very clear question especially with the language that you’re using and your correct answer and the plausible distractors the distractors should be able to tell us something else and i know you’ve done a lot about this in maths i’ve seen some great examples in primary maths where the three options we had the correct answer and it was an addition question and the other option would be the answer if they subtracted or if they multiplied so then if they selected that option you know exactly where they’ve gone wrong and i can design similar in history i have read a book about it’s a book just about multiple choice questions by patty shank and and she says actually two plausible distractors is enough the more plausible distractors we add the harder it can be for us as teachers so we’ve we’ve got that aspect to it now this is interesting as well because this is something that dylan william uh he edited a book of mine previously and he’s not a fan of but i am and researchers so andrew c butler suggested adding i don’t know yet as an option and the reason why dylan william and i do understand um his concerns is that that’s like i’m douglas as well um that’s an opt-out way it’s going oh i don’t know i don’t know the reason why i like it is because it helps students with send it makes even more low stakes i have a conversation with students that say i would rather you say i don’t know than just guess because if you’ve guessed and you guessed it correctly i don’t know that although i will say there can be recall involved in power elimination and guesswork because a student might see a question and not know it but they know it’s not that one and i know it can’t be that one so it has to be this one but that’s still retrieval practice because they’ve recalled information so what i say with the i don’t know is for students to be really honest and if they select it then i know they don’t know they they know we know there’s a gap in knowledge i’ll go over it to avoid the i don’t know i don’t know i will perhaps not include it on some questions that are easier or i will say to students you can only use it three times now doug lamov and i have wrote a blog about this we were chatting and douglamov made a really good point he said well students will know something even if they don’t know that answer so why don’t you add the option i know that and then they get to write something so i had a question um what does the word conscription mean and my i had the correct answer two plausible distractors i know that and it was really interesting because some students said i don’t know what conscription means but i know lots of men signed up to fight because of propaganda some students did know what conscription meant and still wanted to add i know that so there’s just so many ways that we could be really flexible with multiple choice questions great with mini whiteboards as well great for elaboration why did you select the answer why do you think that one is incorrect as well so there’s all these different things that we can do the primary school that i work with they have a column at the side for optional working out and if they’re students struggling they’re encouraged to do that and then again if they’ve gone wrong we can see where they’ve gone wrong as well so multiple choice question design um all those things i’ve said about the plausible distractors um making sure the level of challenge that there’s opportunities for retrieval success but there’s also some difficult questions and that they are accessible for everyone that language isn’t a barrier either amazing okay three three quick points i could talk to you all day about multiple choice uh questions okay but like just three points on this the first is um we made a decision when we were coming up with diagnosticquestions.com the website just to have and three distractors and one right answer just because it made data collection much easier and you could compare you could say if only 20 of kids got this one right then 60 got that one right we could start to say things about question difficulty and so on but i’ve also seen uh with dylan and also some of your examples that um having either more than one correct answer or sometimes no correct answers and often in humanity subjects one answer is kind of more correct than another you could argue and so on i assume you’re kind of keen for varieties of kind of question types with multiple choice questions well this is something actually that i’m really against and this is again andrew c butler wrote about multiple choice questions and avoid all of the above or none of the above and there’s a few reasons for that um so the first one if the correct well students can think that we’re trying to trick them catch them out and it’s we don’t want them to think that if the answer is all of the above but the student selects an answer above how do we mark that because if we say it’s incorrect it’s not incorrect it’s just not the answer that i’m looking for but then if we say oh that is correct they may later remember it being correct and assume the other options are incorrect so especially for younger students that’s really confusing but none of the above is even worse like what is the point in that because we’ve just exposed students to incorrect answers you could answer that correctly without knowing the answer so i i use the example which battle took place in 1066 i list battles but the one the right answer is none of the above a shooter could get that correct but not know what battle took place in 1066 or what if they do the frustration of not being able to tell you that right answer and another problem with having two correct answers to select is later on where the student what if a student gets one right and one wrong and then they forget which one they got incorrect or correct and there’s this potential for confusion as well so i have especially younger students as well primary key stage three i do think that we keep it simple um and consistent don’t have a word and then a sentence and in terms of the layout as well so there’s there’s so much research and blogs out there on multiple choice question design um that i do think it’s really interesting and another tip would be to look as much as you can at the quizzes that are out there and and i always find great questions or i found questions i think well that’s good but i could actually make that a little bit better as part of my research for retrieval practice 2 i answered so many multiple choice quizzes set by teachers on things i knew nothing about and was able to score really highly because of bad question design that’s another good point could somebody with no knowledge of this still get it correct well it’s a bad question design then that’s fascinating uh just two more points on this kate um i find i mean i’ve written god knows how many maths i i never want to write another diagnostic question again in my life i’m i’m over them now and but the hardest thing i find and you really only know this i think when you test the question out because you can write a question you think oh this is a great question but then you ask the kids and do a bit of dig and you think well actually i’ve missed something the thing i found the hardest is writing questions that kids can’t get right while still holding a misconception if that makes sense so i’ll give you an obvious example from math and i wonder if this is true in in your subject so you could ask a question this will be a bad example but you could say um which of these numbers is a multiple of 10 for example and you have three distractors but if none of those distractors are factors which is a big misconception kids have they muddle up which is a factor which is a multiple a kid could get that question right while still not being 100 sure what the difference is between the two so on the face of it it looks like a good question but because you haven’t identified the kind of key misconceptions kids may have actually it may not be that reliable an indicator of of actual understanding is that something that that you find in in your subjects or all the subjects that you’ve encountered this potential problem with with diagnostic questions i do think it’s really challenging i honestly think to design an effective multiple choice quiz is not easy and i’ll give you an example my subject where i asked what year was conscription introduced and hardly anyone got that correct and then when i went through that they went oh i forgot what conscription was they actually knew the year it was introduced they and then that’s why the next question next time i asked was which of the following refers to conscription so that was sort of given me wrong data really because they did know the year but it did give me useful information because it showed me they weren’t confident and familiar with that term so the other thing as well is is trying to rephrase um so your question that you gave there you could use different examples to try and illustrate that couldn’t you so you could have different numbers and digits that are essentially asking that same question but and then there should be a pattern that that emerges there if you had within a then that would give you a really good insight there’s also something else actually i want to say about multiple choice questions coming back to the bjorks and retrieval induced forgetting so basically there is information that we teach that is either background contextual or like that story about the tower of london that i gave previously that’s non-essential non-essential information should not be on a multiple choice quiz only the target memories and i’ve done this before where i’ve had filler questions and i’ve put in questions that i thought oh okay i’ll just add this in an example was my class i had 16 questions wanted to round it up to 20 what was the name of the historian we watched in the video then i stopped myself nope they don’t absolutely need to know dan snow’s name but if i did include that in the quiz they may assume that oh i’ve got to learn his name because miss is quizzing me on it and to say oh well it’s harmless it actually that’s not true either because of the retrieval induced forgetting the things that we focus on it will boost their retrieval strength and the things that we neglect obviously the retrieval strength will be low so that could be coming at the risk of something else that is essential to create well decreasing that retrieval strength and boosting the retrieval strength of something that is not essential so another tip i have for teachers is go back through previous quizzes and ask yourself do your students really need to know this if it’s not essential or desirable remove the question and i i’ve since i had this revelation about retrieval induced forgetting i deleted so many quiz questions i had a quiz question about the year henry viii was born and they were shoehorning this random fact into essays was actually no this is the date you need to know the year he became king the year he was born don’t worry about that i’m not going to put that in my quiz so always keep it on i mean i think math is probably different that it’s easier but with something like with humanities and with english literature i see that a lot as well about the plot and the characters i think is this essential so always focus on the essential because of the retrieval and juice forgetting that’s fascinating and the the last thing i’ll say about these because as i say i really could bang on about these all day is i come 100 agree that writing a good diagnostic question a multiple choice question is really tricky but i also think it’s a really useful thing for staff to do for colleagues to do particularly less experienced colleagues to consider where kids might go wrong what a good distractor might be and so on so an exercise i often do with them with with maths departments and i’m wondering whether this is something that transfers across is i’ll say okay i want you to think of a topic that you’re teaching next week or whatever and you work in pairs for this and it doesn’t matter if it’s different topics and on one side of a piece of paper you write your diagnostic question but on the other side you put your four answers a b c d and then you swap pieces of paper but crucially the person your partner can only see your question they can’t see your four answers and their challenge is to think what four answers would they write down to to that question and the other person does it for theirs and then they swap back and compare and that’s always fascinating because they never come up with the same four answers and then you can have a really good discussion about okay well oh you included c was that because you’ve seen kids do this in the past and particularly if you’ve got a less experienced member of staff with a more experienced they can have a really good pedagogical discussion about that and then you end up with kind of a really strong question that takes elements from both is that something that transfers across or something that you’ve you’ve tried yourself oh yeah we absolutely should do that and it is a shame that that never happens and one of the things i really dislike being head of department was all the admin stuff i had to do that was taken away from teaching and learning time in a department meeting that is how i would have loved to have spent department meetings and didn’t always happen like that um it is great that question banks are available um but again as i said i check them i might edit them but i write my own as well so teachers especially early in their career i do think look what questions are out there but yes think about your students and something that has cropped up in your class or that something has been said it was really interesting i spoke to some history teachers in year 7 when we do the battle of hastings the word soldier has been spelt in so many different ways with the j with the g with solids it’s just and it’s really interesting that was like a common thing that we were experiencing so we we could tackle that as a question which one is the correct spelling and use the examples that we have seen but that might not be the same in a different school where perhaps they they they can spell soldier correctly so um i think that sounds i love your idea of doing that and i definitely think question design should be part of cpd and it isn’t and actually even just taking quizzes and sharing them and asking for feedback on them as well and one of the advice is that i’ve given the book pieces of advice is about with cold calling is what’s helped me with that is watching videos online and going into classrooms of seeing other teachers do it and then asking other colleagues to come into my classroom and give me feedback about how i’m using that strategy so the exact same thing i i went years ago in the history department we didn’t share quizzes i would create and use a quiz and my colleague wasn’t and we absolutely should be because we should be asking the same essential questions with that class in this class but it was disjointed so yes all for that i think that would work in any subject and i definitely think it’s something that this is again leaders should try and make the time for this because it’s just so important and it’s not something questioning people think oh you just get better with experience yes you do but to just rely on that that’s quite a slow learning curve we can speed that up by working together reading engaging with research and making opportunities like you’ve just said