Kate Jones

This episode of the Tips for Teachers podcast is proudly supported by Arc Maths
You can download an mp3 of the podcast here.

Kate Jones’ tips:

1. Sometimes it is better to review than retrieve (03:33)
2. Make use of the Encoding specificity principle (19:46)
3. Take a low effort, high impact approach to task and question design (34:28)
4. How to make the best use of technology for retrieval practice (48:34)
5. How to design good multiple choice questions (1:06:21)

Links and resources

• The paper Kate refers to in Tip 1 is here
• Kate’s books are:
  • Retrieval Practice
  • Retrieval Practice: Resource Guide
  • Retrieval Practice 2
  • Retrieval Practice: Primary
  • Wiliam & Leahy’s Five Formative Assessment Strategies in Action
  • Love to Teach

Subscribe to the podcast

• Subscribe on Apple Podcasts
• Subscribe on Spotify
• Subscribe on Google Podcasts
• Subscribe on Stitcher

Watch the videos of Kate’s tips

Podcast transcript

Craig Barton 0:00
Hello, my name is Craig Barton and welcome to the tips for teachers podcast. The show that helps you supercharge your teaching one idea at a time. Each episode I invite our guests from the wonderful world of education to share five tips for teachers to try both inside or maybe even outside of the classroom. With each tip, the challenge is always to ask yourself, what would I have to do or change to make this work for me, my situation and my students, experimentation and frustration may follow, but hopefully something good will come out of it. Now remember to check out our website tips for teachers.co.uk, where you’ll find all the podcasts as well as links resources and audio transcriptions from each episode. But back to the mat. You’ll also find all of the tips from the podcast as single bite size videos, including some of my own tips that I’ve learned over the years. These videos can be used to spark discussion between colleagues at a department meeting, a twilight insect, and so on. Now, just before we dive into today’s fantastic episode, a quick word of thanks to our lovely sponsors. Because this episode of the tips for teachers podcast is once again proudly supported by the lovely people at our mouths are arc Max is a fantastic app designed to help your students remember all the maths content at key stages three and four is built around research into how memory works. Specifically blocks work on the power of retrieval practice and the Spacing Effect ensuring students don’t just practice what they’ve just studied, that are regularly exposed to content they’ve encountered days, weeks and months before. And that is particularly relevant to today’s episode, where we dive deep into retrieval. So if you want to find out more just search arc Max, I mentioned my name. Remember that arc with the C knots. Okay. So back to today’s show. Let’s get learning with today’s guests. They’re fantastic. Kate Jones. Spoiler alert, here are Kate’s five tips. Tip one, sometimes it’s better to review than retrieve. Tip two make use of the encoding specificity principle. Tip three take a low effort high impact approach to task and question design tip for how to make the best use of technology for retrieval practice. And tip number five, how to design good multiple choice questions. Now if you look at the episode description on your podcast player or visit the episode page on tips for teachers dot code at UK, you’ll see I’ve timestamps each of the tips so you can jump straight to anyone you want to listen to first, or if you want to revisit this episode. And just a final word. My audio is a little bit weird on this recording. I won’t go into the long details, but hopefully hopefully won’t put you off too much. Enjoy the show. It’s an absolute cracker.

Well, it gives me great pleasure to welcome Kate Jones to the tips for teachers podcast. Hello, Kate, how are you?

Kate Jones 2:59
Hello. And thank you for asking me because I’m already a big fan of these tips for teachers.

Craig Barton 3:05
Oh, that’s what we like to hear I’m pages to say that that’s fantastic stuff. Okay, can you tell the listener a little bit about yourself ideally in a sentence

Kate Jones 3:12
history teacher for over a decade that I’m about to join evidence based education as teaching and learning associate. And I’ve also written some books.

Craig Barton 3:28
You’ve written a lot of books, and we’ll probably get onto those. Yeah, that’s fantastic. All right. Well, let’s dive straight in. Kate, what’s your first tip for us today?

Kate Jones 3:35
Well, my tips along the theme of retrieval practice, something I’m interested in as I you. My first tip is about knowing when not to do retrieval practice. So when to do review instead. So that might sound surprising to a few people that I’m saying. There’s actually some points where we shouldn’t do retrieval practice.

Craig Barton 4:01
Wow, you’ve hooked me straight in there. Right. So tell tell me about this, then.

Kate Jones 4:05
Okay. Well, I think I’ll need to give a little bit of context. And something that you and I have in common is that we are super fans of professors Robert and Elizabeth Bjork. So I’m going to just draw on their work previously, well, actually, in 1914, Edward Thorndyke, published the law of use and issues. And this was accepted in the, in the community of academic field and memory. And what Thorndyke said was that if you have information in long term memory, but if you don’t use it, you lose it. And I’m paraphrasing that. No, that’s not a direct quote. So the idea is he was saying that it decays, and it disappears. And actually, that seems to make sense and we can understand why it was accepted. And most people believe that term memory works, however, and this is where the blocks come along. So they actually challenged Thorne dikes law of tissues with the new theory of tissues. And this was in the 90s. But I can’t believe that it was in the 90s. And I didn’t know about it when I was trying to train it to be a teacher today. 1010. So the new theory of tissues for anyone not familiar with it, it doesn’t say that what Thorndyke says it doesn’t say that memories decay, it actually has two measures of memory, with storage strength and retrieval strength. So the York’s argue that if it has transferred to long term memory, that it will stay there, it just becomes difficult to access. So storage strength refers to how well embedded entrenched information is in long term memory, no storage strength can’t decline. Or rather, the only way that it would decline would be with physical and quite extreme damage to the brain. So in a classroom context, because we’re not dealing with that, in the classroom contact, we can be quite confident that storage strength doesn’t decline, it can increase. So if it’s transferred to long term memory, it will be there. However, and this is where people go, why do I forget things? Well, that’s because of the travel strength and retrieval strength can and does fluctuate, it can decrease, and it can increase. So I see this all the time when I I’ve been observing lessons recently. And a teacher asked a question, and the students hands shot up in the air. And they answered it correctly, so quickly. And I thought, well, their retrieval strength is high. Because that’s what retrieval strength is, when it’s high that you can recall information quickly, correctly and confidently. But when retrieval strength is low, we’ve all experienced this, students have experienced this. It’s that frustrating feeling. I know this, and it takes a long time. And it’s quite slow and difficult to recall. Or perhaps you don’t recall it in that moment. And this is why I think it’s essential for teachers to know because we could be doing a retrieval task with students, and they don’t recall the information. And then an assumption is made that it wasn’t encoded, it didn’t transfer to long term memory was actually it’s probably there. It’s just that the retrieval strength is low. But don’t panic, because we can boost and increase the retrieval strength. And the way that we do that is either through review and a refresher or through more of retrieval practice. So the reason that I’m saying to teachers, retrieval practice perhaps isn’t always the best option is because there will be points in the academic year, or in terms of your teaching, where retrieval strength will likely be very low. So if we ask students to complete a retrieval practice task, when retrieval strength is low, they won’t do very well. And that’s not good for them because they won’t have retrieval success. It can be really demoralising, knock their confidence, it probably won’t be that insightful for the teacher. And the teacher could misunderstand that data and think I need to reteach it was actually, if they’d have done a refresher over the overview, instead of a retrieval task that would have boosted their retrieval strength. And then perhaps the following week, they could do a retrieval task, and they would do better. So it’s really about understanding, retrieval, strength and storage strength. And by me saying don’t do retrieval practice, do retrieval practice. It’s a it’s a great strategy. But there are certain points where it’s not the best strategy to use. So Does that all make sense?

Craig Barton 9:09
As brilliant case. So the first thing to say that is that’s one of the best summaries I’ve heard of the retrieval, storage strength difference and the history from from the Beatles perspective was was fascinating. Three things I want to dig into as a result of that if if that’s okay, so the first is the obvious question, and that is, how do we know how do we know when it’s a good time to do retrieval practice versus a bad time? What are some of the strategies that we that we can employ to know whether it’s a good time to do retrieval practice?

Kate Jones 9:37
Well, that’s a great question. And we can actually know when the time is out. We should use retrieval practice and we shouldn’t. So I’m going to drop a few names here. Robert Bjork, Dylan William and I co authored an article, and we wrote it in the August for teachers going back to school in September. And that’s a classic example. Students. I’ve had a long break rate of six weeks, and we’ll assume they haven’t been studying like they would in school. So we can assume retrieval strength is low. So if you’ve got a year 11 class or year 13 class or class it from, that you’re carrying on with, it could be tempting to start the year, what can you remember from last academic year? And that won’t turn out? Well, probably because of the retrieval strength being low. So in terms of the academic calendar, we can look at that, we can look at points where there’s been a break in learning. So after the Easter holidays, possibly, although that does depend. If you’ve taught something right on the edge, and they’ve had two weeks, then possibly you could do a retrieval practice task. But if they don’t do as well, as anticipated, it could because retrieval strength is low. Another point to think about is when the content was last revisited, or taught. So is you know who else was don’t need to know about this Ofsted inspectors? I don’t know if they do but the reason I say this is because lots of primary school teachers have told me that Ofsted inspectors are asking students questions about previously taught material. But actually, they really need to be aware of retrieval strength here, because if they were to ask a year six student a question about something on the year four curriculum, the year six student might just panic be blank, and the retrieval strength could be low. The best way for an Ofsted and I know this tips for teachers, not tips for Ofsted. But this can be helpful for teachers to say in an inspection, the best thing for an inspector to do would be able to talk to the classroom teacher. And if for example, the class in year six are studying plants. And they studied that in year four, then the Ofsted inspector can ask them questions about the curriculum content from year four, because the teacher should have revisited that quiz that built on that, and the retrieval strength will be high. So it’s so important for this in terms of understanding learning, and especially making judgments as well. So the things that we need to consider are when there’s been breaks and gaps with learning. But also, if you haven’t revisited something in a very long time, and I’m talking here for months, or it could even be years, then retrieval strength could likely be very low. But there are lots of variables with that, how well that information was taught how thorough for example, because you could ask a student a multiplication question that they’ve done years ago, but that will be really strongly embedded and entrenched. Meaning that the retrieval strength is still high and accessible. But I often see there’s a version of my retrieval grid, my retrieval grid I created in history, but I think it works best in maths, you know, where it’s last lesson last week, last term, and so on. And sometimes there’s a last year question. And my only concern about that last year question is, have you revisited that since a year ago, because if not be very mindful of retrieval strength could be low.

Craig Barton 13:24
This is fascinating this case. And my second question for this and this may be a really silly question. So please forgive me, what’s the difference between a retrieval kind of opportunity or task versus a review task?

Kate Jones 13:37
Yes, I was. So silly question. Honestly, you ask all the great questions. And I’ve felt like that before when I reached out to people like Robert Bjork said, sorry, but that’s a good question. And I’ll give you an example of a view or refresher. So we’re coming back to school after a long break. Why don’t you review a knowledge organiser? Get it up on the board? Have a question and answer discussion, talk about it in pairs, or do a refresher task, which could be comprehension based reading the textbook answering questions. The main difference is that there is support there. And that they have the knowledge organiser, the class notes, the textbook, they can talk about it, they can confer, the teacher can answer and ask questions, in contrast to retrieval, where you would take away that support. Does that answer your question?

Craig Barton 14:35
That does? Yeah, and I think you alluded to this earlier. So it’s a good idea. If something hasn’t been revisited in a long time, do one of these review or refresher tasks, but then maybe a week later or something, then you follow it up with the appropriate retrieval task? Is that right?

Kate Jones 14:48
Yes. And I would tell my students that and I would be really explicit and say, you haven’t done this in a very long time. So I’m not going to quiz you because I know that it will take you a long time. Time. And I had a student, a sixth form student come to me and say, Miss, I tried one of your brain dumps without my idea where I write everything down from memory. And I really struggled. I couldn’t write much. So what were you writing about? And he was starting at the beginning of the course something from a year ago. And that I suppose, where you’re going wrong, you should have read the textbook again, perhaps made some notes and wrote some questions, then waited, and then that will boost your knowledge and your confidence. And he did that. And it worked. Honestly, can you imagine go to a quiz with me fake and I say, I know, it’s just my retrieval strength is low. That’s, that’s how I see the world.

Craig Barton 15:46
is brilliant, this guy? Well, that leads me to just a couple more follow ups on this. If this is alright, we’ve touched upon one there. That’s how much do you think it’s important that the kids are aware of this because I’m of the opinion these days that for many years, I’ve made the mistake of almost kind of keeping the kids in the dark of some of the ideas and strategies I’ve been trying. But this, this feels like something really useful for for kids to be aware of this difference between retrieval, strength, storage strength, and, and not to panic if they think they forgotten something, would that be something you share with the kids?

Kate Jones 16:15
Absolutely essential. I’ve shared it with my students with parents, parents and carers really need to know this. Because even when they know that retrieval practice is more effective than highlighting and underlining. They can still panic when their child forgets and can’t recall information. And that’s where they need to have that deeper understanding of forgetting that actually, it’s there. It’s just not accessible at that moment in time. But don’t worry, we can do something about that. And actually, it’s really powerful when you know that. And it’s really powerful when you know where the gaps are, in your knowledge, because you can do something about that. So all the students that I’ve taught in recent years and their families, retrieval practice is just part of the language of learning. And I do use the term retrieval practice. And I will say, retrieval strength as well. And I try and use it every opportunity in lessons on report and parents evenings. I mean, obviously, I’m the the woman who wrote a book about retrieval practice, but it’s really important that comes from everyone that all teachers and school leaders are all singing from the same retrieval practice hymn sheet. I’m sharing that with students, and I’m working with primaries, and they’re doing a great job of child friendly explanations. And I just think the younger that they can understand this, the better.

Craig Barton 17:44
Final question on this this tip? And this may be a terrible question. And again, Kay, how do we know whether retrieval practice is just really low? Versus that just kids just don’t know it? It’s kind of just not in there at all? Because you could almost feel as a teacher, all right. They didn’t. They didn’t get that they don’t know the answer to that. Or it’s okay. It’s just because we’re retrieval practice is low, versus actually they just don’t have a clue what’s going on? How do we how do we make that distinction?

Kate Jones 18:09
Yeah, again, a great question. And there’s also other things about how do we know if they’re just not trying because retrieval practice is low stakes or no stakes. So there are things that we can do. And I do think first of all, retrieval practice should have some level of accountability. So that if a student isn’t trying isn’t invest in effort, the teacher can have that conversation and say, I’ve noticed this. And now in terms of whether they they just don’t know it, and they never got it, the first time round of retrieval strength is is low. What we can do, and this is where retrieval practice is also brilliant. And I’ll talk about this in another tip as well, about retrieval cues, is that we can increase the retrieval cues and prompts. And if that is still not helping students to reach the correct information, then that is a real clear indicator that it’s it’s worth revisiting, going over in that review and that refresher, and that you’ve recognised this, this gap in knowledge, but if they can get it, so teachers are so good at this without realising they’ll be walking around a class and they’ll see a student struggling and they’ll just give a verbal prompt, or just a little hint to get them to go in and go, Oh, that’s what I meant to be writing about. And then they do that. But still after that verbal prompt, blank, they’re struggling. That’s when we know as a teacher, okay. This is this is something I need to revisit.

Craig Barton 19:45
Fantastic. Okay, Kate, what’s Tip number two, please.

Kate Jones 19:49
Okay, so tip number two is about designing effective retrieval cues. And I’ll talk about something called the encoding specificity principle. Well heard of that.

Craig Barton 20:02
Well, no sounds from sale. Oliver can’t see better jargon. I love it already.

Kate Jones 20:07
Okay, yeah, well, there’s a lot of jargon there’s so well, this principle of Endel Tulving. He is one of the leading experts with human memory. He’s credited for making the distinction between episodic and semantic memory. So semantic memory be knowledge of information and facts and data, and so on. And episodic memories are from episodes in our life. Tulving calls episodic memories, that memory time travel, because it’s so powerful. And it’s connected to an emotion, that you can often remember how you were feeling where you were and who you were with. So Tulving is really well known for that work. But then also, he’s well known for his work with encoding and retrieval, although there is some researchers who disagree with his way, which I’ll touch upon, and then I’ll tell you sort of where I stand on it. But the key idea with this encoding specificity principle is that, in order for retrieval cue to be effective, it should be present at the encoding stage. Now, this is where not everybody agrees, in terms of I think it’s generally agreed that if a cue is present in the encoding stage and retrieval stage, it will help retrieval practice. But it doesn’t necessarily have to be, although it would have to be present in both. So are you with me? So? If I was to

Craig Barton 21:40
think so I’m gonna need some I’m gonna need some practical examples. At some point the following. I think I’ve got my head around the theory so far.

Kate Jones 21:46
Okay. Well, let me also give you this exam, this helps as well. So what we ask students to recall, is a target memory. If I asked you, what did you have for lunch yesterday? You say, Oh, my wife and I went to a restaurant and I saw my friend. Nope, that’s not what I asked Craig, I’m asking What did you have for lunch? The target memory is what you ate. And that can happen in school, where we ask a question. And the answer is irrelevant, because you’re talking about where you went to a restaurant, but it’s not what I’m after. Okay, so what we need to do is, we need to be mindful of that, when we designed retrieval cues that were specifically targeting the target memory. And I will give you a practical example. So I teach history. And I teach the Tudors every year. And I’m really passionate about my subject. And I told the class, that I just been to the Tower of London, and I saw the plaque that says Amberlynn was executed here. And I told him all these stories and anecdotes from my trip. And then when it came to the retrieval task, as you can imagine, they included that in their answers that Miss Jones was at the spot at the Tower of London. No, I don’t want you to recall that information. So it doesn’t mean that we can’t include anecdotes, but we just need to be really clear about what the target memories are. So if I teach him specifically about Henry the Eighth, then he closed the monasteries, I will put up a photograph of the ruins of a monastery. Now I know my students have to recall this later. So this will become their target memory. At this point, I’m not going to go and start telling stories and anecdotes, I’m going to purely focus my explanation on what they need to recall later. So they’re encoding this information that I hope will be their target memory later. Now, later on, some time has passed. And I want to find out what my students can recall about the closure of the monasteries. If I put off that exact same photograph, that are using the encoding stage, that Tulving is saying that that will help and improve their retrieval, because I’ve used the same cue. So there’s a few things there that to sort of unpack that I’ve been very aware of saying, you will need to remember this information. And I will be quizzing you on this, which when I’m talking about Tower of London, I need to make very clear this is just me telling the story. And I’ve used that exact same cue. Now, the reason why people are Alan badly don’t necessarily agree, is because I could still ask my students to recall information without that photograph, and they would. So it’s not that we have to have it present. But with younger students or students with learning difficulties, and we want to help them and boost the success they have, then we can do that by providing the same or similar cues at both stages. So is that example helped a better? Yeah. How’s

Craig Barton 24:57
the housing please open up a couple more questions there. Okay, I’m fascinated by listening Brilliant stuff this. So I’m assuming you’re certainly not saying that we shouldn’t tell stories, right? So we know stories are a good way to get kids into it and so on. So where do you draw the line? Is it you kind of tell the story, but then say, Okay, now this is the bit I need you to remember. How do you How does that work practically?

Kate Jones 25:19
Yeah, you’re absolutely right. And stories can be great that the fact that my students are remembered that is a testament to stories as well. And there’s other purposes of school. Of course, learning is a changing long term memory, but I want my students to be interested in the subject to find out more to go to the Tower of London. So I’m not suggesting that we we don’t tell our stories and things like that. That’s part of us as a teacher and our character. But what we do need to say to students, is when we do a quiz, I’m not going to ask you about this. Or when we do retrieval task, maybe before we begin, you say, Okay, I know, I know what probably happened here. But I don’t want to know about my Tower of London story. Because we if we specially if we tell a story and the class have reacted, whether they’ve laughed, and they’ve been shocked, they’re going to remember that. So we either say at that point, okay, I’m happy to share this with you. But when it comes to the recall, this is this is not relevant. Or just before we did the recall, of a minder, and this is where younger students really struggle. So another example that I had was the brain dump, write down everything, you know, and I did it with year eight, and I did it for year 13. And my older students were really sophisticated answers. And my younger students wrote Pendik was fat. I had a ginger beard. Yes, six waves. Why are you writing this after everything I’ve taught you was a miss, you just said write down everything that we can remember. That’s why they need the guidance, the cues, the pumps and the support. Initially, a primary school teacher told me that they’d it was the topic of plans, they said, What do you remember about year four, when we when you learn about plants, but they just started listing plants in their garden. So they really do need these these cues to be specific and helpful to keep them on track.

Craig Barton 27:20
Got it, got it to two more questions about these cues. If this is okay, Kate. So the first would it be sensible to assume that like a kind of chain of events would be to use the queue during the encoding phase, then use the queue again, the first time we ask kids to retrieve with the ultimate aim that then perhaps the further down the line we can remove the queue? Is that is that the plan?

Kate Jones 27:41
That absolutely it so that we by the time because I say this with primary students, and schools that they should use free recall, but towards the end of the topic, where they’ve got more knowledge, more confidence, they’ve done retrieval tasks, and had success. So they know what they can recall from that experience, what type of things to write and include. But yes, the ultimate goal is obviously not to have any retrieval cues. And that they can answer a free recall question. Because if you think about it, if this if I was doing this with an exam class, they’re not going to have that picture of the monastery on an exam paper in a final assessment. So we don’t just want students to only be able to recall information about monasteries when they see that photo. So this is just that initial retrieval stage. Where we do want students to experience success. Like scaffolding you take that away, totally.

Craig Barton 28:42
Perfect. Probably here is Kate, you’ve opened up every time you speak, I keep thinking of more questions, because you say it’s such good stuff. So I’ve got to squeeze in two more if that’s okay. The first is Do we know anything about what cues work better? Is it is it visual cues tend to be best in this encoding phase or to auditory cues work as well as Is there any insight into that,

Kate Jones 29:00
oh, there’s so much that we need to have. Because there’s, there’s unintentional cues, and we encounter them every day in life, when we smell something or hear something. And that’s something that we sort of need to be mindful of in the class. And that’s why when a class can become crowded, it can provide these unintentional cues. Whereas what the teacher needs to do is provide the intentional cues and think very carefully about this. Now, what I do in terms of my cues is I use a variety where we’ll use images, but I will also perhaps in a different task, use key words as well, because I want my students to be able to use that terminology. So I’d probably use the images first. Students with English as an additional language. The cues are so important and actually for students with sound. The cues are really important See, and that’s something that I’m looking into at the moment is that I think that frustration that EAL students feel with retrieval practice, because it slows that process down because of the translation, the searching for the words. That’s why the prompts do help them. All the students could also have pumps, but the pumps could be a sentence starter, or a structure or a layout. So the prompts that we use will vary with the classes. But there’s also something called the cue overload principle, which if we provide all the principals coming out here today, if we provide too many cues, then it just diminishes the effectiveness of retrieval practice, because obviously, we just given them all the answers and making the retrieval practice not that challenging. But again, this is something where the research won’t have the answers, the teacher will, the teacher will know how many cues to use, and actually the teachers got a lot of power, they can, next time remove or reduce the amount of cues or if a class is struggling in that moment, they can just increase the cues with a verbal prompt with writing something on the board with putting another picture on. So we can really have control and influence of how many cues we use when we use the cues when we take the cues away. There’s quite a lot actually there. I am writing a chapter at the moment about retrieval cues. So that’s why it’s it’s very fresh in my mind.

Craig Barton 31:34
That’s amazing. I promise. This is the last question, are these these queues? Okay, I find this absolutely amazing stuff that I’m really interested in. These are unintentional queues. And I’ve experienced this myself where the kids seem to really understand mathematics when it’s taught in the classroom, when they’re sat next to the same person when we’re in the same room. And then they go into the exam hall. And okay, maybe it’s the stress of the exam, but also, all of a sudden, it’s an unfamiliar environment, they don’t have those kinds of familiar things around and it all goes to pieces. Is this a thing? And and should teachers be aware of these unintentional cues? And is there anything as teachers we can do to kind of break them and stop them having this potentially detrimental effect?

Kate Jones 32:13
Yes. And the really difficult thing about this is that what might be a cue for one person might not necessarily be a cue for another. And this is why it’s very challenging for teachers and men. I know you’ve talked, I don’t think you liked displays Do you?

Unknown Speaker 32:37
Go go down back, I don’t go down that path, it will be kicking off.

Kate Jones 32:41
But that’s again, why would they do need to be really careful about the displays, because when they are in an environment that is different, they haven’t got those cues that they’ve been relying on, which is interesting, because I’ve seen retrieval practice happen. And the students glanced at a display, because the displays have acted as retrieval cues. So I get this moment, I’ve still got a lot more to learn about the unintentional cues. I just think that teachers need to be mindful of this, especially with the learning environments. And also, just the more that we ask questions, and we make observations, and focus on the intentional cues, the things that really matter that will actually support them. But as I’ve said, it’s so difficult. And again, my life now I’m learning about memory. And I do I smell something, and it reminds me of something from years ago. And that’s an unintentional cue that we just have in our life, all the time. And this is something that can actually be really distracting. And we know this as well, with, with little things that I’ve interrupted a class, a teacher told me about how class could hear this, the ice cream ban, everything just went wild. And that’s, you know, the sound is that intent, unintentional cue of of ice cream and what that tastes like. So the only thing that we really can do is be mindful of it. Think about our environment. And then just think carefully, what intentional cues Am I creating that link with the encoding stage? And will directly helps you to to remember the target memory? So that’s all like linked together quite nicely.

Unknown Speaker 34:26
Amazing. Okay, okay. What’s your third tip, please?

Kate Jones 34:30
Oh, my third tip, I need to remember which one, I said I was going, Oh, yes. This is a mantra that I use. And that’s low effort, high impact. This is an approach with retrieval practice, but it can apply to other things. So I’ll just elaborate. First of all, the low effort doesn’t mean no effort. So it’s not about being lazy. Because as teachers, there’s so many things we have to put effort into reports and feedback and planning and So, retrieval practice is going to be sustainable in the sense that teachers are doing it every single day, that it has to be workload friendly. And this is a key message I share with school leaders, this is something that you’re looking for every lesson, then then think about the workload implications of that. And that’s the low effort for the teacher. But the high impact is on student learning. So whatever you do with the retrieval task, in terms of your planning, delivery, and how feedback is provided, it can be low effort for the teacher. But with every retrieval practice task that we design and deliver, it should have a high impact on student learning. Both you and I have done the other way round in the past. So the example that I think of with you, which is high effort from the teacher, and a low impact on learning, was your switch phones. Exactly. It was exactly the same for me and my career. I remember early on, I was creating sorting cards, and I actually had my three sisters helping me one was printing and other was cutting, and other was putting laminated putting them in envelopes. Such a high effort task, I put more effort in the actual task design, instead of the question design. That’s where my effort should have gone. I gave them out in the lesson, the students completed it in about 30 seconds. It wasn’t challenging. It was low impact on learning, but high effort for me. So now I’ve completely flipped that narrative, especially with retrieval practice. And anyone who says that’s not possible with retrieval practice, I can show them how it really is possible.

Craig Barton 36:47
Oh, well, you’ve talked to them. Okay. So a couple of things to say there. Firstly, you’re absolutely right. It’s so frustrating as a teacher, you put a load of effort into designing a task and an activity, and then it just, the impact just doesn’t match anywhere near the effort that you’ve put in. So I’ve certainly tried to go down this path myself and flip it around. So you’ve you’ve done a bit of a hawk there. Can you say that it’s definitely possible? Can you give us some examples of some of these low effort teacher activities that have a high impact on learning?

Kate Jones 37:17
Yes, so I’m going to talk about multiple choice questions and technology. And that’s a great example. There are questions that are already made and available, and can be marked online, instantly, providing immediate feedback to the teacher and the student. Now, the effort that low effort should be the teacher quality, assuring the questions and checking them, and then checking and looking at the results. But that is something that is really easy to implement retrieval practice with mini whiteboards. The effort should be the teacher thinking about the questions that will be asked. But then, obviously, you had a great chat with Adam boxer about mini whiteboards. And that’s it’s immediate responses from everyone in the class. The fact that they robbed the answer off makes it low stakes, the fact that you can skim and scan with multiple choice questions quickly and easily means you could be responsive. And there isn’t that that’s taken me hours to plan. And that’s taken me hours to mark. But it hasn’t done that. But it’s had a high impact, whether it’s check his understanding of retrieval practice. But this is the same for students, I tell them to do retrieval practice, you need a pen and paper. Or sometimes they don’t even need that if they do it verbally. But pen and paper, write down everything you know, from memory. And then they can self assess it, they can check it again alongside the textbook. And Dylan William has said, the best person to mark a test is the person who’s just taken a test. So all of these things, the fact that there’s question banks out there, the fact that the self assessment worked with retrieval practice, there’s templates, all of these are very workload friendly, that support the low effort approach. But if the questions are designed effectively, then they should have a high impact on student learning.

Craig Barton 39:15
That’s fascinating that I want to ask you supplicate that I’ve been meaning to ask you for for a long time. This sounds like a perfect opportunity here. So I love speaking to non mathematicians, because there’s something that happens in maths with regard to retrieval practice, and I’m intrigued whether it also happens in history and so on and so forth. And that is, there are certain topics in maths that it’s quite hard to design retrieval practice activities and tasks for because they require like certain equipment. So a good example is constructions. So you’ve taught students to construct with rulers and compasses and it’s all fine because all less than they’ve been using the ruler and the compass and protractor. But then let’s say next week, you’ve moved on to another topic, but all of a sudden, you want to make sure that they you provide an opportunity for them to return You’ve had to construct something, well, then you’ve got to get out the ruler, get out the compass, get out the protractor. And it’s a real hassle. And another example is a lot of the geometry and graphing work we do in maths may require grids to be drawn axes to be drawn, which have got to be handed out to the kids and so on. So what you often see maths and I’m guilty of this is a lot of the diagnostic questions using mathematics or a lot of the mini whiteboard activities are for the kind of more easily quizzical elements of maths the ones that lend themselves more to the students not needing as much kind of equipment with them. So questions follow up from that, is that something that’s true in like history? For example? Are there certain aspects of history or certain styles of questions that are more that lend themselves better to quizzing in this way? And is there anything we as teachers can do about it? I don’t know if you’ve any thoughts on that?

Kate Jones 40:51
No, I do have have thoughts on that. Well, I would say, and actually, this is moving slightly away. But I’ll come back to your maths example. The learning process by Arthur Melton is described as encoding, storage retrieval. Now, that’s great, that’s really helpful. But the problem that I have with that is that it suggests retrievals, the final step in the learning process, we need to add two more blocks, we need to add application and transfer. So in my subject of history, they’re never going to do multiple choice quiz on an A level exam, they’re going to write an extended answer. And if I rely too heavily on quizzing, and multiple choice, they’ve got a bank of isolated facts, they need to be able to apply them and transfer them where where’s necessary. So whilst I’m not in the same, don’t need physical resources, I, we do have to add every subject, we do have to go beyond the Quizzing. And I’m a big fan of multiple choice questions and quizzing, but it’s limited, it will only take us to a certain point. And I’ve spoke to other subjects that have had that issue. I spoke to her design, and technology teacher, and obviously that there’s a lot of equipment there. One of the things that they did in terms of a retrieval was they provided a comic strip. And they said, Can you from memory, write down this process of how you did that, how you did something, so that there was still so with that, instead of repeating that action, they were still getting them to go through the recall of the step by step. And I know that it’s difficult different to the protractor. And I’m, I always you know, I always use you as an example for maths because I do feel that maths is a really, it all subjects are unique, but lots of the retrieval tasks that I have created, lend themselves really well to humanities and English and even science. And then maths is different. I’ll tell you another way I talk about you a lot in my sessions is I talk about, well, I’ll get to this. Sorry, there’s so much multiple choice questions. I read somewhere that you said mass diagnostic questions, 10 seconds, students should be able to answer them. I don’t know if you remember this. Yeah. And I was I say, Craig Barton is a math expert. So that in his subject, I’m not going to disagree with him. And then I shall a question from my subject, where I say this these longer than 10 seconds. And I purposefully show that to say, we’ve got the maths specialist who knows what works in his subject. And then we’ve got, even though he’s, he’s an expert, and I’ve learned a lot from him, I then have to think, well, what does it look like in my subject? So there’s so much we can learn from each other? And maybe subjects like practical subjects, like design technology, could collaborate with maths in that sense, the same way that I collaborate with English departments. But then also, then we have to as a community, figure out subject community, how do we do this? How do we overcome this? Whether that’s with retrieval practice, or or anything teaching and learning based? So I don’t know if that?

Craig Barton 44:13
Really? It does, it certainly does. It’s really interesting. So I’d never thought to do that, that comic strip idea or, or the equivalence of kind of, alright, describe how you would construct an angle bisector. Or describe how you would measure an angle that’s really smart, I can really see that work. And obviously, alongside that, at some point, they’ve got to have the opportunity to physically do it. But I really, really liked that idea. That sounds really good. The follow up question I had was, often I again, I’m fascinated to know whether this is true in other subjects. One of the key retrieval opportunities that happens in math for certainly for prior knowledge is is in the do now or in the starter. And you can imagine kids come into a room and on the board, there’s five questions or maybe it’s the starter grid that you alluded to before, one from last less than one from last week and so on. And so before. And just going back to what you said there about kind of in history, obviously, you’ve got to practice the whole skill of writing the longer form, kind of essays and arguments and so on. It’s a similar thing in maths where you’ve got to practice the kind of five or six Mark questions that pull in lots of different areas of mathematics and so on. My fear is often that if retrieval only happens in the do now, you kind of get some skewness towards those that do now lends itself better to those kind of short sharp ones, you know, the Quick Fire, retrieve this one mark questions, and so on, and so forth. And I often worry that if if the retrieval kind of gets crammed into the do now, and we don’t provide those other retrieval opportunities to do things, like you say, like do the long form essays or in my case, answer the multi mark more complex exam questions, kids aren’t getting an opportunity to kind of finish, kind of finish the journey. As you say, the retrieval isn’t the end of the kind of story. It’s okay, retrieve all these individual things, but then put it together to do this more complex problem. So that was just a bit of a long ramble just to say that sometimes by concerned with the do now that it lends itself better to a certain type of retrieval. If that makes sense.

Kate Jones 46:07
I think this is a really common problem that some schools are only during retrieval practice, at the start of a lesson with a do now, and I say to them, this is one of the most effective teaching and learning strategies, why would you limit it to just five minutes. And something that I did with curriculum design was fit in lessons probably once every half term that a whole lesson was retrieval, practice, retrieve and reflect? And I recall doing this with a GCSE group. And they had said to me, miss, we finished the content in other subjects, and we’re revising, when are we going to revise in history. And I was outraged, because we had been revising for two years. I didn’t want to cram retrieval at the end, I spaced it out throughout the curriculum. And that was something that wasn’t happening everywhere. But that is sort of very common that it’s five minutes at the start of a lesson to eight minutes. And I understand why it’s a great way to start a lesson. It links in with Rose and shines daily review. But I have not read research that says it has to be at the start of a lesson. If you were to have a lesson and she would need to come in and finish something off. Why can’t you do your retrieval as a transition, I say to teachers about the phrase, like just do it, don’t think more about it like that, with retrieval practice, just make sure you do it. Try not to fixate on, it has to be at this point, it has to be eight to 10 minutes. Sometimes it will be quicker, sometimes something pops up a misconception or a mistake and you think oh, actually, I need to tackle this right now. So we’ve got to have we’ve got to plan retrieval practice in and think carefully about curriculum design, because if you are adding in those extra lessons, then obviously you will finish teaching the content later. But it’s worth it. It’s so worth it. And that class that we were we’d been revising for two years, didn’t feel like we were coming to the end. And we had to feel like reteaching everything again, it was just we’re at a point where we’re just boosting the retrieval strength. So we do need to go go beyond the do now. They’re great for multiple choice. Great, great for cuisine, but it has its limitations.

Unknown Speaker 48:33
Amazing. Okay. Okay, tip number four, please.

Kate Jones 48:36
Okay, my next tip, carrying on with the theme of retrieval practice, is using technology for retrieval practice. So we’ve all had a lot of CPD with technology with online learning. People have different feelings about using technology. But I’m a big advocate for technology in the classroom. With retrieval practice, I have three golden rules with technology. And the first one is should be workload family to support the low effort, high impact. So I’ll give you some examples of online tools that are workload friendly. It also should be user friendly. Now, this is probably the most important use of family for the teacher to the student so that you can use it regularly, it becomes automatic, doesn’t take up that space in working memory. And then the third one is that it must be low stakes. And the reason I say this is because in my previous school, que se to see what online for over a year, and we did online exams, I don’t know how reliable they were at home, but we did them formal exams on Google Forms. And it was a combination of cued recall free recall multiple choice. They had a score, a percentage a grade that went on a report to parents of a key so to me, it didn’t get any more high stakes than that. And I remember when We returned back to the classroom when I did a Google Form quiz and panic setting. Mess. Is this going on at report? Is it getting a grade, I didn’t know you know about retrieval practice. But I then said to my colleagues at a school with using Google Forms for summative assessment, and it works great. Let’s keep it like that. But there’s so many other online quizzes and tools, let’s not use Google Forms for retrieval practice. Let’s have that clear distinction. So that they when they do a Google form, they know very clearly, and they’ve been told this isn’t assessment. But then we can use Kahoot to carousel Quizlet, anything like that, for retrieval practice. And they’re all low stakes in their layout. And they’re the the way they have the images and the music and things like that. So as long as that’s really clear for students as well.

Craig Barton 50:51
This is great. Well, what’s today let’s dive into some of the tech that you’re you’re a fan of k because again, I’m always fascinated whether with my math head on this is something that works well in math or not. So tell me tell me some of the ones that you use and why you like

Kate Jones 51:04
quizzes. So quiz idols ed.com is amazing. And I’m really reluctant to write about online quizzes and tools now because they keep developing and improving at a rapid pace, and quizzes anchor who have actually listened to me and taken on advice. Because it’s grown so much, it was just multiple choice. Now there’s cued recall, you can include an image, so that we said about the encoding cues, you can include images with questions, you can include equations, you can include audio, which is great for some students with learning difficulties, or for languages, the fact that an audio clip can be inserted, there’s also a teleport feature, which I love. So I type in the topic of a quiz that I want to quiz my students on typing the French Revolution, all the public quizzes made by other teachers are available. But if I don’t like some of their questions, then I don’t have to, but I can say, Oh, I’ll take that and you teleport it to your quiz. And even then, once you’ve teleported it, you can edit it and amend it. So that’s just a fantastic feature. And then when you set the quiz, you can personalise it to remove the music, because that’s very annoying and distracting. You can remove the leaderboards, which I do think teachers should do, especially in mixed ability classes, or I’m not sure about in your classes, but it tends to be the same top three on the leaderboard. And it’s just, it shouldn’t really be about that it should be about them and their individual recall. And also remove question timers. And that’s really important for students with sand and Al, but perhaps other students as well. Because the minute that the clock is ticking for some students that makes it high stakes, pressure panic sets in, they form bad habits, such as not reading the question carefully. And we don’t ever want them to rush. I also don’t think it’s fair. And I don’t know if Kahoot still does. So I didn’t use Kahoot for a long time because you couldn’t remove the leaderboard feature. And you could only use the whole quiz not beds, but they have changed. That I didn’t like on Kahoot that you could get more points for answering quickly. Why when you’ve got two students that both got the right answer, should someone be a lot higher than the other students? Because we want them to take their time and get the answer. Correct. And so often when they’re rushing. They go I knew that Miss, I just didn’t read the question. Yeah, I just because the clock is ticking. So the fact that you can personalise all these things on quizzes, you can also share the quiz with your colleagues, you could upload it straight to a Google Classroom, the results, it has a spreadsheet, which is just green and made it so easy. So you can just have a snapshot of the class, or there’s a lot of red for this question. Oh, everyone got this question. Right? tells you your class percentage, which is brilliant, and really, really helpful. This links in with what rose and shine said about the 80% success rate? Well, if you’ve set a class quiz, and it’s 99%, your quizzes probably too easy. But if you’ve then logged in your percentage is 40%. It’s probably too difficult. So there’s all these things with quizzes that I can use to help me I can create an effective quiz quickly. I can set it easily I can share it with colleagues and students. It will mark it for me and provide useful information for me moving forwards. Something else that teachers should do, but they don’t do enough, is repeat quizzes. Don’t just do the quiz and then do it wants to do that quiz again. When you do the quiz again. It reduces the low stakes nature even further because you said we’ve already done this. You could add three or four more questions in From the content that you’ve done in between, and then that is really where students should be seeing the progress. Even if they say, Well, we’ve done this mess. Well, we know how memory works. Or even if you get it right, the second time, your retrieval strength could is still getting a boost. So it’s still a good opportunity. Quizzes. Brilliant.

Craig Barton 55:20
Okay, a few questions. few questions on this. So the first is, where do you tend to use these cases? This does this tend to be more in class with the kids on devices? Or is this homework as well?

Kate Jones 55:31
Yes, you can set it as homework. And but I did work in a school where every student had access to technology. And this is another thing, obviously, that buries for teachers who don’t work in a high tech environment. plickers is, it’s been around a while, but it’s still good. You just one device needed an iPad phone, you scan the class, and Plickers, when they have these paper codes, they cannot cheat. It’s not like a mini whiteboard where they can have a sneaky peek. If someone’s at someone’s answers Plickers. Because the codes are unique in the way they have to hold up the code. It is impossible for them to cheat. They don’t know what their other students have answered. So yeah, and I use that for a long time when I worked in a school without technology. But then when I did work in a school with technology, I thought, Oh, I’m really going to embrace that. And trial. Lots of these. I know you’ve had Adam box around and carousel learn is great. And they have a mini whiteboard mode. So actually, you don’t need technology in class. For that, you can have the questions up on the on the whiteboard. Now it’s on the mini board. So there are ways of doing this. And there’s also features, if you do set it as a homework to try and stop cheating. Like block things or so on. But yeah, I’ve used it mainly in lessons.

Craig Barton 57:03
Fantastic. Just a few more points on there. So we’re gonna go controversial now, Kate. So Dylan, William often makes the point that he he when he uses diagnostic questions, often it’ll just be kids voting with fingers, one for a tooth B, three, C four for D or ABCD. Cards, because he doesn’t like the idea that every answer a child ever gives is recorded somewhere. And and one of his arguments is this raises the stakes of the assessment. And also, it may be kind of causes some kids like it knocks their confidence they they tend to kind of clam up instead of kind of be more open and honest. Well, what’s what’s your view on that as a potential issue with technology for for retrieval,

Kate Jones 57:42
you can use technology for no stakes retrieval practice to make it anonymous. And mentimeter.com. Students don’t put their name in, they answer a question. There’s pros and cons to that. Because as a teacher, you see a snapshot of all the class responses, but you don’t know who the individuals are. But if you are trying to do a no stakes approach, and that’s great, that can be frustrating for the student. They’ve often got Miss. That’s my answer. They want the teacher to know. I didn’t know stakes quiz on quizzes. When my year seventh class, I met them for the first time, probably because we’d been online. And I did not want them to think this was a test or anything like that. So the first quiz I did with them, they had to have a quiz name that was a Harry Potter character. And actually, that was the only time I’ve ever kept the leaderboard up because we it was like Dumbledore hybrid. And it was really fun. And it was anonymous, because I didn’t know who Dumbledore was. And that’s not something I would do regularly. But they’re still going through the act and protest of recall and retrieval practice. Even though I couldn’t see the individuals with their real names, I could still see who score correctly and incorrectly on what questions so it still was useful for me. So we can use technology anonymously. Or we could use something like Padlet and jam board there like digital post it notes as well. But I do think again, it is really difficult the low stakes nature of it because Mentimeter thankfully has a profanity filter. Because if you just say to students, I don’t know who wrote what they could think, Oh, my missus just give us a green light. And actually, that has really been what if a student did just write something they never have, they never have. Once he did write something really, really bad, and I wouldn’t be able to know who wrote that. So it’s just about sort of the pros and cons and the variety. But the more that you do retrieval practice anyway, the more just normalises it as a classroom with The

Craig Barton 1:00:02
Amazing one one more thing on this case. I’m interested in using technology for self Quizzing. Because we know from research, whether it’s the philosophy paper or whatever, that self quizzing is a really good way for for kids to remember. And we’re promoting good study habits and so on. Do you do you find any of these technologies lend them particularly well to kids revising independently at home? And if so, which ones?

Kate Jones 1:00:27
Yeah, so digital flashcards are great for this. And a question that I’m really interested in and I’ve asked Daisy Christodoulou, I’ve asked other people is about should students create their own flashcards or use pre made flashcards? Now obviously, the benefit of them creating them themselves is they’re thinking hard about the questions. And this that’s a good strategy in itself. But as we know, question design is difficult. And if students are going to create flashcards, they need to be taught how they need examples, model and scaffold it all these things was if our sole purpose is retrieval practice, then actually, it could just be better to use ready made flashcards that are specific to an exam board and a topic and so on. And there’s Quizlet, there’s Anki Anki has millions, and in Quizlet, and Anki. Just as to and also quizzes has flashcards. They have the options where you can create your own or select ones that are already made. These are great because actually, if you think about a GCSE how many subjects students have to have, and imagine all those physical copies of flashcards even though I am a bit old school and I prefer if I was revising I would have paper flashcards. But let’s just think practically, a student always has their phone on them was not going to carry around these flashcards packs around with them. But they could be travelling somewhere they could be waiting. And they could just get out their phone and use the flashcards and Anki. Although I don’t know how this works, because it’s the Tech has. When a student answers a question at this point, whether it was correct or not, it records that and it has an algorithm so that it knows next time which questions to focus on. The problem with most online flashcards though, and actually paper flashcards is that students don’t always do the act of retrieval. You have to write it or say it. And that’s a problem with the digital flashcards. And that can be a little bit embarrassing if you you’re not going to say out loud, if you’re on your own, or if you’re on the bus or something. But what ideally students should do is probably read the question, write the answer, then look for the answer and partake of a class. But they don’t they just read the question, think flip it over. And they haven’t done that they haven’t done that physical act of retrieval practice. So flashcards can be brilliant. But they can be bad if they just copy out rereading, if they’re not doing retrieval practice, but digital flashcards, my students have really liked them.

Craig Barton 1:03:21
Fantastic. I’ve just thought of one more question Can I Can you say too much good stuff here. So I’m gonna have to just throw this in, there’s a risk this will open up a whole can of worms, so feel free to swerve this one. But let’s say a student is revising they’ve made themselves as a set of flashcards, digital flashcards, or physical ones for history, geography, English, and so on. Now, what’s the research say here about it, say they’ve got half an hour to revise? Should they do half an hour and spend that half an hour on geography? And then the next day, half an hour in history? And then the next day, half an hour in English? Or should it be half an hour where any call from any subject can come up, and it’s a mixed bag, because I’m really torn on this my own understanding of interleaving is that it’s a good idea to mix things up. But also I understand the idea of kind of thinking deeply and having a coherent kind of chain of thoughts about a subject and, and tying together ideas. So any thoughts on much we mix it up or keep it subject focused?

Kate Jones 1:04:15
I can answer that because I asked John Donne lossky, the exact same question. Let’s go on, he is just amazing professor, anyone who’s not aware strengthen the student toolbox. So I asked him that same question as well, because I have seen students take it to the extreme where they’ve just mashed up all their flashcards. Now, they do need to reshuffle and reorder them. We don’t want them to just learn it. And so by answer by answer, but actually across different subjects than last year was quite clear and said no, I don’t think that’s a good idea. He said of course the idea of interleaving and doing half an hour one night of ones subject, and then all, why not have 15 minutes of geography 15 minutes or history, and then another day have 15 minutes or history, that that type of thing. So there are ways that we can interleave. But yeah, I’m really interested in flashcards, I think we haven’t quite fully got there with really embracing them. Because if we’re asking students to create their own, they need to be creating them from the start of the year. Because otherwise, it’s a huge workload issue just before the exams, making them. And as they are using them online, we need to quality assure and check that they they are correct. But that question, yeah, I put it to the Laski because interleaving in terms of cognitive science, I teach history. And I have a very different approach to interleaving the new in maths, because my curriculum is driven with chronology. So it just would not make sense to teach a lesson about 1930s, Germany 1960s, Germany and then keep going back into so I’m don’t have that same level of expertise. But John Donne lost really confident when he said no, don’t think that’s a good idea. Keep the flashcards and sessions subject and topic based.

Craig Barton 1:06:16
Amazing. I love nothing more than a clear definitive answer, Kate, that’s fantastic. Okay, can we have your fifth and final tip, please?

Kate Jones 1:06:23
Yes, this is about effective multiple choice question design, and a big fan as you are a multiple choice questions. But sadly, there’s a lot of really badly designed questions out there. And I worry about early careers, teachers who need to be trained on how to write an effective multiple choice question. And I will read because he may go online and look for questions which which I do. But I’ve heard experienced teachers uploaded a quiz, you may assume that that’s effective. And it might not be. And probably the biggest mistakes that I see are including an obviously incorrect response. And we can just write that off immediately. And then the other one is what I refer to as the Bradley Walsh effect, which is great on the chase where they have comedy options, but not good with teaching and learning. So that’s entertainment. And I love the chase. And I love when they do have a funny option. And

but when we do that in school, when teachers are trying to be happy and funny, we’re diluting the level of challenge and the effectiveness of retrieval practice. I see all the time. Which of these was heavily dates wife, Lady Gaga, you know which character from Romeo and Juliet, Justin Bieber, honestly, that they’re everywhere, and I’ve probably done them myself, as well. So that’s the first thing as well have a very clear question, especially with the language that you’re using and your correct answer and the plausible distractors. The distractors should be able to tell us something else. And I know you’ve done a lot about this in maths, I see some great examples in primary maths where the three options we had the correct answer, and it was an addition question. And the other option would be the answer if they subtracted or if they multiplied. So then if they selected that option, you know exactly where they’ve gone wrong. And I can design similar in history. I have read a book about a book just about multiple choice questions by Patty shank. And and she says actually, two plausible distractors is enough, the more plausible distractors we add, the harder it can be for us as teachers. So we’ve got that aspect to it. Now, this is interesting as well, because this is something that Dylan William, he edited a book of mine previously, and he’s not a fan of, but I am a researcher. So Andrew C. Butler suggested adding, I don’t know yet as an option. And the reason why Dylan William and I do understand his concerns, is that that’s like and Oblomov as well. It’s an opt out way. I don’t know. I don’t know. The reason why I like it is because it helps students with sand and makes it even more low stakes. I have a conversation with students that say, I would rather you say I don’t know, than just guess, because if you’ve guessed, and you guessed it correctly, I don’t know that. Although what I will say there can be recall involved in parallel elimination and guesswork because a student might see a question and not know it, but they know it’s not that one. And I know it can’t be that one. So it has to be this one. But that’s still retrieval practice because if recalled information, so what I say with the I don’t know is for students to be really honest, and if they select it, then I know they don’t know they Now we know there’s a gap in knowledge. I’ll go over how to avoid the I don’t know, I don’t know. I will perhaps not include it on some questions that are easier. Or I will say to students, you can only use it three times. Now, Doug Lemov. And I have wrote a blog about this. We were chatting. And Doug Lemov made a really good point. He said, Well, students will know something, even if they don’t know that answer. So why don’t you add the option? I know that and then they get to write something. So I had a question. What does the word conscription mean? And my had the correct answer to plausible distractors. I know that. And it was really interesting, because some students, I don’t know what colour scription means. But I know lots of men signed up to fight because of propaganda. Some students didn’t know what conscription meant, and still wanted to add. I know that. So there’s so many ways that we can be really flexible with multiple choice questions great with mini whiteboards as well. Great for elaboration. Why did you select that answer? Why do you think that one is incorrect as well? So there’s all these different things that we can do. The primary school that I work with, they have a column at the side for optional working out. And if the students struggling, they’re encouraged to do that. And then again, if they’ve gone wrong, we can see where they’ve gone wrong as well. So multiple choice, question design. All those things I’ve said about the plausible distractors, making sure the level of challenge that there’s opportunities for retrieval success, but there’s also some difficult questions, and that they are accessible for everyone. That language is a barrier, either.

Craig Barton 1:11:47
Amazing. Okay, three, three quick points. I could talk to you all day about multiple choice questions. But like just three points on this. The first is, we made a decision when we were coming up with diagnostic questions that calm the website, just to have three distractors. And one right answer just because it made data collection, which easier and you could compare, you could say, If only 20% of kids got this one, right, and 60% got that one, right, we could start to say things about questioning difficulty, and so on. But I’ve also seen with Dylan and also some of your examples that having either more than one correct answer, or sometimes no correct answers, and often in humanities subjects, one answers kind of more correct than another, you could argue and so on, I assume you’re kind of keen for varieties of question types with multiple choice questions.

Kate Jones 1:12:36
Wow, this is something actually that I’m really against. This is, again, Andrew C. Butler wrote that multiple choice questions and avoid all of the above or none of the above? And there’s a few reasons for that. So the first one, if the correct answer, while students can think that we’re trying to trick them, catch them out? And it’s, we don’t want them to think that if the answer is all of the above, a student selects an answer above. How do we mark that? Because if we say it’s incorrect, it’s not incorrect. It’s just not the answer that I’m looking for. But then if we say all that is correct, they may later remember it being correct, and assume the other options are incorrect. So especially for younger students, that’s really confusing. But none of the above is even worse. Like, what is the point in that? Because we’ve just expose students to incorrect answers, you could answer that correctly without knowing the answer. So I use the example which battle took place in 1066 or less battles, but the one the right answer is none of the above. A student could get that correct, but not know what battle took place in 1066? Or what if they do? The frustration of not being able to tell you that right answer. And another problem with having to correct answers to select is later on where the student? What if a student gets one right and one wrong? And then they forget which one they got incorrect or correct. And there’s this potential for confusion, as well. So I have especially younger students as well. PrimaryKey stage three, I do think that we keep it simple and consistent, don’t have a word and then a sentence. And in terms of the layout as well. So there’s there’s so much research and blogs out there are multiple choice question, does that mean? That I do think is really interesting. And another tip would be to look as much as you can at the quizzes that are out there? And I always find great questions, or if I’m questions, I think, Well, that’s good, but I could actually make that a little bit better. As part of my research for retrieval practice, too. I answered so many multiple choice quizzes, sat by teachers on things I knew nothing about. and was able to score really highly because of bad question design. That’s another good point. Could somebody with no knowledge of this, still get it correct? Well, it’s a bad question design that.

Craig Barton 1:15:13
That’s fascinating. Just to two more points on this case, I find I mean, I’ve written God knows how many months, I never want to write another diagnostic question again in my life. I’m I’m over them now. But the hardest thing I find are you really only know this, I think when you test the question out, you can write a question think, Oh, this is a great question. But then you ask the kids and do a bit of digging, you think we will actually miss something? The thing I find the hardest is writing questions that kids can’t get right while still holding a misconception, if that makes sense. So I’ll give you an obvious example from math. And I wonder if this is true in in your subject. So you could ask a question, this will be a bad example. But you could say, which of these numbers is a multiple of 10, for example, and you have three distractors. But if none of those distractors are factors, which is a big misconception, kids have the mid the middle up, which is a factor, which is a multiple a kid could get that question, right, while still not being 100% sure what the difference is between the two. So on the face of it, it looks like a good question. But because you haven’t identified the kind of key misconceptions kids may have actually may not be that reliable an indicator of actual understanding, is that something that you find it in your subjects or all the subjects that you’ve encountered this potential problem with, with diagnostic questions?

Kate Jones 1:16:31
I do think it’s really challenging. I honestly think to design an effective multiple choice quiz is not easy. And I’ll give you an example by subject where I asked what year was conscription introduced? And hardly anyone got that? Correct. And then when I went through that, they went, Oh, I forgot what conscription was. They actually knew the year it was introduced. They and then that’s why the next question, next time I asked was, which of the following refers to conscription? So that was sort of given me wrong data, really, because he didn’t know the air. But it did give me useful information, because it showed me they weren’t confident and familiar with that term. So the other thing as well is trying to rephrase your question that you gave there. You could use different examples to try and illustrate that, couldn’t you? So you could have different numbers and digits that are essentially asking that same question. But and then there should be a pattern that that emerges. If you had within a 20 questions, five that were like that, and that would give you a really good insight. There’s also something else actually I want to say about multiple choice questions coming back to the bionics and retrieval induced forgetting. So basically, there is information that we teach that is either background contextual, or like that story about the Tower of London that I gave previously. That’s non essential. non essential information should not be on a multiple choice quiz. Only the target memories, and I’ve done this before where I’ve had filler questions. And I’ve put in questions that I thought okay, I’ll just add this in an example was my class, I had 16 questions, why to round it up to 20? What was the name of the historian we watched in the video? Then I stopped myself. Nope, they don’t absolutely need to know Dan snows name. But if I did include that in the quiz, they may assume that oh, I’ve got to learn his name because misses quizzing me on it. And to say, Oh, well, it’s harmless. It actually that’s not true either. Because of the retrieval induced forgetting, the things that we focus on, it will boost the retrieval strength, and the things that we neglect, obviously, the retrieval strength well below. So that could be coming at the risk of something else that is essential to create wealth, decreasing that retrieval strength and boosting the retrieval strength of something that is not essential. So another tip I have for teachers is go back to previous quizzes, and ask yourself, do students really need to know this? If it’s not essential or desirable? Remove the question. And I’ve since I had this revelation about retrieval induced forgetting, I deleted so many quiz questions. I had a quiz question about the year Henry the Eighth was born and they were shoehorning this random fact into essays was actually no, this is the date you need to know the year he became king. The year he was born. Don’t worry about that. Not going to put that in my quiz. So always keep it on. I mean, I think maths is probably different though. It’s easier but with something like with humanities and with English literature, I see that a lot as well. About the plot The characters is this essential. So always focus on the essential because of trivalent use forgetting,

Craig Barton 1:20:07
as fascinating. And the last thing I’ll say about these, because as I say, I really could bang on about these all day is I come 100% agree that writing a good diagnostic question, a multiple choice question is really tricky. But I also think it’s a really useful thing for staff to do for colleagues to do particularly less experienced colleagues to consider where kids might go wrong, what a good distractor might be, and so on. So an exercise I often do with, with with math departments, and I’m wondering whether this is something that transfers across is I’ll say, Okay, I want you to think of a topic that you’re teaching next week, or wherever, and you work in pairs for this, and it doesn’t matter if it’s different topics. And on one side of the pay piece of paper, you write your diagnostic question, but on the other side, you put your four answers A, B, C, D, and then you swap pieces of paper. But crucially, the person your partner can only see your question they can’t see your four answers. And their challenges think, what four answers would they write down to to that question, and the other person does it for theirs? And then they swap back and compare. And that’s always fascinating, because they never come up with the same four answers. And then you can have a really good discussion about okay, well, are you included? See was that because you’ve seen kids do this in the past, and particularly if you’ve got a less experienced member of staff with a more experienced, they can have a really good pedagogical discussion about that. And then you end up with kind of a really strong question that takes elements from both. Is that something that that transfers across a something that you’ve tried yourself?

Kate Jones 1:21:32
Yeah, we absolutely should do that. And it is a shame that that never happens. And one of the things that I really disliked being head of department was all the admin stuff I had to do that was taken away from teaching and learning time in a department meeting. That is how I would have loved to have spent department meetings, and didn’t always happen like that. It is great that question marks are available. But again, as I said, I check them, I might edit them, but I write my own as well. So it teachers, especially early in their career, I do think look at what questions are out there. But yes, think about your students, and something that has cropped up in your class, or that something has been said, it was really interesting. I spoke to some history teachers in ESL, and when we do the Battle of Hastings, the word soldier has been spelt in so many different ways. With a j with a G with solids, it’s just, and it’s really interesting. That was like a common thing that we were experiencing. So we could tackle that as a question, which one is the correct spelling, and use the examples that we have seen? But that might not be the same in a different school? Where perhaps they they can spell soldiers correctly? So I think that sounds I love your idea of doing that. And I definitely think question design should be part of CPD isn’t and actually even just taking quizzes and sharing them and asking for feedback on them as well. And one of the advice is that I’ve given the both pieces of advice is about with cold calling, is what’s helped me with that is watching videos online and going into classrooms of seeing other teachers do it. And then asking other colleagues to come into my classroom and give me feedback about how I’m using that strategy. So the exact same thing I wouldn’t years ago in the history department, we didn’t share quizzes, I would create news, a quiz. And my colleague wasn’t. And we absolutely should be, because we should be asking the same central questions with that class in this class, but it was disjointed. So yes, all for that. I think that would work in any subject. And I definitely think it’s something that this is again, leaders should try and make the time for this, because it’s just so important. And it’s not something question, and people think, Oh, you just get better with experience? Yes, you do. But to just rely on that, that’s quite a slow learning curve. We can speed that up by working together, reading, engaging with research, making opportunities that you’ve just said.

Craig Barton 1:24:21
That’s fantastic. Okay, these have been five absolutely brilliant tips. So now I’m going to I’m going to hand it over to you what what should I mean, this could take longer than our conversation if you listed all the things that you’ve you’ve written and put together but what would you recommend listeners check out of yours and I’ll put links to these in the show notes,

Kate Jones 1:24:37
as well in terms of what you spoke about today’s retrieval practice, and I’ve authored three books all available, which are kept on retrieval practice. My next book is retrieval practice in a primary context, and that’s available in June. So all of the books though, the pink book, I call it was my first retrieval practice. If you’re just starting off with retrieval. Go there. The blue is just ideas. It’s just a thin resource guide with over 50 retrieval practice tasks and tips. So lots written about retrieval practice in my books.

Craig Barton 1:25:14
Amazing. Well, okay, this has been absolutely fascinating. I wanted to speak to you for ages and this has been Yeah, this has blown my mind has been brilliant. So thank you so much for your time today.

Kate Jones 1:25:22
Thank you all great questions.

Transcribed by https://otter.ai