Research: Is Covert Retrieval an Effective Learning Strategy?

Title: Is Covert Retrieval an Effective Learning Strategy? Is It as Effective as Overt Retrieval? Answers from a Meta‑Analytic Review
Authors: Yu et al
Access the original paper here
Listen to a deep-dive podcast:

Paper summary

This academic meta-analysis investigates the effectiveness of covert retrieval, recalling information mentally, as a learning strategy compared to overt retrieval, which involves physically producing a response. Analyzing data from numerous studies, the research finds that covert retrieval offers a small but significant benefit for learning, especially for complex materials and long-term retention, and when corrective feedback is provided. However, overt retrieval is generally more effective overall than covert retrieval. The findings suggest that while covert retrieval can supplement learning, overt retrieval is a superior method for consolidating memory.

If teachers are to remember one thing from this study, it should be…

While retrieval practice is a powerful learning tool overall, overt retrieval is generally more effective than covert retrieval for enhancing student learning, and educators should prioritize using methods that require students to actively produce responses

Paper Deep Dive

Define any technical terms used in the paper

Here are definitions and explanations for some of the technical terms used in the paper:

Pre-registered protocols: This is a practice where researchers formally document their research plan, including hypotheses, methods, and analysis procedures, before conducting the study or analyzing the data. This is suggested as a way to reduce publication bias and increase transparency in future research.

Retrieval practice / Retrieval practice effect / Testing effect / Test-enhanced learning: These terms refer to the phenomenon where actively retrieving information from memory is one of the most effective strategies for facilitating learning and consolidating long-term memory for studied information. It involves recalling information from memory. This strategy has been consistently demonstrated as a powerful tool for reinforcing long-term learning.

Overt retrieval / Overt retrieval effect: This is a type of retrieval practice where learners recall information from memory and produce overt responses by writing, typing, or speaking aloud the retrieved information. Most previous studies on test-enhanced learning have focused on the effectiveness of overt retrieval. The benefit gained from this type of practice is termed the overt retrieval effect.

Covert retrieval: This involves mentally retrieving information without producing overt responses. It is described as simply bringing information to mind. Covert retrieval is often considered more time-efficient and can be used in situations where overt responses are not feasible.

Meta-analysis: This is a systematic review that integrates data from multiple previous studies. The goal is to synthesize existing results to reach a more reliable conclusion, quantify the magnitude of effects, determine relative efficacy, explore potential moderators and boundary conditions, and examine underlying mechanisms.

Magnitude (of an effect): This refers to the size or strength of a learning enhancement effect, often quantified using statistical measures like Hedges’ g.

Boundary conditions: These are the specific circumstances or factors under which a learning effect is or is not observed, or where its magnitude changes. The meta-analysis aimed to explore these for the covert retrieval effect.

Underlying mechanisms: These are the theoretical explanations or cognitive processes proposed to explain why a particular learning strategy is effective. The paper discusses theories like additional exposure, desirable difficulty, and truncated search as potential mechanisms.

Relative efficacy / relative effectiveness: These terms are used when comparing the learning benefits of two different strategies, specifically overt versus covert retrieval, to see which one is more effective.

Additional exposure theory: This theory proposes that retrieval practice enhances learning by providing extra study opportunity to successfully recalled items. Providing corrective feedback is seen as adding to this re-exposure.

Desirable difficulty theory: This theory posits that learning strategies requiring more mental effort (are “desirably difficult”) may initially impede short-term learning but are ultimately beneficial for long-term learning by forcing deeper cognitive processing. Practicing retrieval is considered desirably difficult because it requires active recall, which is more demanding than passive rereading. This theory can explain why retrieval practice is often more effective after longer retention intervals.

Truncated search theory: This theory suggests that during some forms of retrieval practice (especially covert), learners might put less effort into retrieval and stop their search prematurely before generating a complete answer. Overt retrieval, by requiring a complete response, potentially forces a more exhaustive search.

Recitation: In one of the earliest studies mentioned, this method involved a learner attempting to covertly recall material from a text or word list, looking at the material only when recall failed. While a form of covert retrieval, it differed from the type primarily evaluated in the meta-analysis because the learner controlled the interleaving of recall and review.

Judgments of Learning (JOLs) / Delayed JOLs: This is a mode of covert retrieval where participants covertly retrieve the target answer when probed by a cue (e.g., “apple—??”) and then predict the likelihood of recalling that target later. Making delayed JOLs is assumed to evoke covert retrieval.

Answer monitoring: This is a mode of covert retrieval where individuals judge the accuracy of their own answer or another person’s answer during retrieval practice. In classroom settings, students might monitor the answer provided by a peer who is called upon.

Answer thinking: This is a mode of covert retrieval where participants silently think about the answer to a question or prompt without overtly producing it.

Corrective feedback: This refers to providing the correct answers or information after a retrieval attempt. The meta-analysis investigated whether providing this feedback during the practice phase moderated the effects of covert retrieval.

Control strategy: This refers to the activity that participants in the control condition performed, used as a baseline for comparison against covert retrieval. Common control strategies included rereading the material or engaging in filler/no activity.

Retention interval: This is the amount of time between the end of the learning or practice phase and the final memory assessment (test). The meta-analysis categorized these as short (< 1 day) or long (≥ 1 day) intervals.

Material type: This refers to the different kinds of information studied in the included experiments, such as word lists, word pairs, or education-related materials like texts or lecture content.

Experimental design: This refers to the structure of the study, specifically whether different groups of participants received different conditions (between-subjects design) or whether all participants experienced all conditions (within-subjects design).

Experimental procedure: This relates to how study items were presented, either grouped by condition (pure-list) or mixed together (intermixed-list).

Overt retrieval mode: This refers to the specific way in which overt responses were produced, such as speaking aloud, enacted retrieval (motorically performing the answer), typing, or handwriting.

Hedges’ g (or Cohen’s d): These are statistical measures of effect size used to quantify the magnitude of the difference in final test performance between conditions in the studies. Hedges’ g is a variation of Cohen’s d that applies a correction for potential bias in studies with small sample sizes.

Forest plot: This is a graphical representation used in meta-analyses to show the effect sizes of individual studies and the combined overall effect size.

Heterogeneity: This term indicates that the effect sizes observed across the individual studies included in the meta-analysis vary significantly from one another, suggesting that the effect is not consistent across all conditions or studies.

Publication bias: This is a potential issue where studies with statistically significant or positive results are more likely to be published than those with non-significant or negative results, which can lead to an overestimation of the true effect size in a meta-analysis. The meta-analysis used methods to detect and correct for this bias.

Funnel plot: This is a graph used to visually inspect for publication bias, where the effect sizes of studies are plotted against their standard errors. Asymmetry in the plot can suggest publication bias.

Bayes factors (BF10): A statistical measure used in Bayesian meta-analysis (like RoBMA) to quantify the evidence in favor of one hypothesis (e.g., the existence of an effect or publication bias) relative to another (e.g., the absence of an effect or bias). A BF10 greater than 1 indicates evidence for the hypothesis, with larger values indicating stronger evidence.

What does this paper add to the current field of research?

Based on the sources and our conversation, this paper makes several key contributions to the current field of research on retrieval practice:

Synthesizing Conflicting Findings on Covert Retrieval: Prior research exploring the effectiveness of covert retrieval had yielded conflicting conclusions. Some studies showed that covert retrieval enhances learning compared to control strategies like rereading, while others reported minimal or no enhancement effect. This meta-analysis directly addressed this gap by integrating available data to reach a more reliable conclusion regarding whether covert retrieval boosts learning.
Quantifying the Magnitude of the Covert Retrieval Effect: The study quantified the overall magnitude of the covert retrieval effect through a meta-analysis of data from 2560 participants across 18 studies. It found that covert retrieval enhances learning to a small but significant extent (g = 0.23), although it also noted signals of publication bias suggesting this magnitude might be negligible after correction (g = 0.00).
Quantifying the Relative Effectiveness of Overt and Covert Retrieval: Studies comparing overt and covert retrieval also had conflicting results, making it unclear if covert retrieval was as effective as overt retrieval. This meta-analysis provides a comprehensive integration of these findings, concluding that overt retrieval is generally more effective than covert retrieval. The weighted mean effect size for this difference was g = 0.17, and results from publication bias detection analyses indicated minimal risk of bias for this comparison, supporting the reliability of this finding. This directly contrasts with some previous mini meta-analyses that found no difference.
Identifying Key Moderators: A significant contribution is the exploration of factors that moderate the effectiveness of covert retrieval and the relative effectiveness of overt and covert retrieval. This was an underexplored question considered important for practical guidance, theoretical understanding, and explaining previous inconsistencies. Key moderators identified for the covert retrieval effect include:
- Corrective feedback: Covert retrieval was significantly more effective when corrective feedback was provided (g = 0.59) than when it was not (g = 0.04).
- Control strategy: The effect was substantially larger when compared with filler or no activity (g = 1.13) than when compared with rereading (g = 0.06).
- Retention interval: Covert retrieval was significantly effective for long retention intervals (g = 0.42) but not for short ones (g = -0.07), indicating it consolidates long-term memory.
- Covert retrieval mode: While the overall moderation was not significant, only “answer thinking” (g = 0.23) and “delayed JOLs” (g = 0.31) showed reliable benefits, while “answer monitoring” did not (g = -0.21).
Testing Theoretical Explanations: The paper explicitly tested whether theories accounting for overt retrieval benefits (additional exposure and desirable difficulty) could explain the covert retrieval effect. It found support for both theories, suggesting covert retrieval enhances learning through re-exposure (evidenced by the impact of control strategy and feedback) and effortful cognitive processing (evidenced by the benefit for long-term retention). It also tested theories for the overt vs. covert comparison (truncated search and production effect), finding that the results support the truncated search and desirable difficulty explanations, suggesting overt retrieval’s superiority stems from requiring a more complete search and greater effort due to overt production, while challenging the production explanation’s prediction about list structure.
Providing Practical Implications Based on Integrated Evidence: By synthesizing the evidence, the study provides clearer practical implications for optimizing learning and teaching practices. It clarifies when and how covert retrieval can be useful (e.g., for complex materials, with feedback, for long-term retention, using specific modes like thinking or JOLs) and emphasizes that overt retrieval should generally be prioritized.

In summary, this paper significantly advances the field by providing a much-needed comprehensive meta-analysis that quantifies the effects of covert retrieval and its comparison to overt retrieval, clarifies inconsistencies in previous literature, identifies moderating factors, and provides empirical support for theoretical explanations in this context.

What are the characteristics of the participants in the study?

Based on the sources and our conversation, here is what can be said about the characteristics of the participants in this study:

The meta-analysis integrated data from a total of 2560 participants.
These participants were drawn from 18 different studies.

While the sources provide the total number of participants included in the meta-analysis, they do not offer a detailed breakdown of their specific characteristics (such as age, educational background, or gender distribution) for the combined group of 2560 participants.

However, the sources do note that the broader field of test-enhanced learning, which this study is part of, has been well-established to generalize across a wide range of populations. This typically includes individuals from pre-school students to young and older adults. This suggests that the participants in the included studies likely represented a diverse age range, reflecting the populations commonly studied in research on the testing effect.

What are the key implications for teachers in the classroom?

Based on the sources and our conversation, here are the key implications of this paper’s findings for teachers in the classroom:

Prioritize Overt Retrieval: The meta-analysis found that overt retrieval is overall more effective than covert retrieval. The superiority of overt retrieval showed minimal risk of publication bias, indicating its reliability. Overt retrieval consistently outperformed covert retrieval across different types of materials, experimental designs, and retention intervals. Therefore, learners and educators should prioritize incorporating overt retrieval into learning and teaching situations. Classroom activities like low-stakes class quizzes and written assignments provide students with opportunities to engage in overt retrieval.
Covert Retrieval is a Supplement, Not a Replacement: While covert retrieval can enhance learning to a small but significant extent compared to control strategies like rereading (g = 0.23), it should not replace overt retrieval. Its benefits are modest and its effect size becomes negligible (g = 0.00) when corrected for publication bias. Overt retrieval is a more universally applicable strategy that more effectively accelerates learning across different material types.
When Using Covert Retrieval, Implement it Strategically: If covert retrieval is used, its effectiveness depends on several factors:
- Mode Matters: Only answer thinking (g = 0.23) and making delayed Judgments of Learning (JOLs) (g = 0.31) reliably improved learning outcomes. Answer monitoring (g = -0.21) did not produce detectable enhancement effects. Educators should focus on modes that require learners to actively engage with study materials.
- Corrective Feedback is Crucial: Providing corrective feedback during covert retrieval significantly enhanced its benefits (g = 0.59), whereas covert retrieval without feedback produced little enhancement (g = 0.04). Educators should ensure corrective feedback is provided after retrieval practice, whether covert or overt. This aligns with findings for overt retrieval, where feedback also substantially boosted effectiveness (g = 0.68 with vs. 0.20 without feedback).
- Focus on Long-Term Memory: Covert retrieval is particularly effective for consolidating long-term memory (g = 0.42 for long intervals), but has negligible enhancement effects on immediate test performance (g = -0.07 for short intervals). For goals like long-term knowledge mastery or preparing for cumulative assessments, covert retrieval can be a valuable tool. This pattern is similar to overt retrieval, which is also much stronger for long (g=0.58) than short (g=0.06) intervals.
- May be More Fruitful for Complex Materials: Covert retrieval appeared to produce relatively greater benefits for education-related materials like texts or lecture content (g = 0.39) compared to simpler materials like word lists (g = 0.07).
Encourage Active Thinking from All Students: When asking questions in class, instead of just having one student provide an overt answer while others passively listen or monitor, instructors should ask all students to prepare to answer the question and thoroughly think about the answer. This encourages widespread covert retrieval among the class.
Combine Strategies for Optimal Learning: Educators should aim to create learning environments that leverage both covert and overt retrieval. While prioritizing overt methods due to their greater effectiveness, covert retrieval can serve as a valuable complement, especially when overt retrieval isn’t feasible, when focusing on complex materials and long-term retention, and when implemented with corrective feedback and effortful modes like answer thinking or delayed JOLs.

In summary, teachers should prioritize asking students to overtly retrieve information through methods like quizzes and assignments. When using covert methods, they should ensure students are actively thinking about or judging answers (not just monitoring), receive corrective feedback, and focus on the benefits for long-term retention of complex materials. Covert retrieval can be a useful supplementary tool, particularly when encouraging all students to engage in mental recall.

Why might teachers exercise caution before applying these findings in their classroom?

Based on the sources and our conversation history, teachers might exercise caution before applying the findings of this meta-analysis directly in their classroom for several reasons:

The Overall Effect of Covert Retrieval is Modest and Potentially Negligible After Correcting for Bias. While the meta-analysis found that covert retrieval significantly enhanced learning compared to control strategies overall (g = 0.23), this effect size is considered small. More importantly, the analysis found evidence of significant publication bias for the covert retrieval effect. When corrected for this bias using advanced methods, the magnitude of the covert retrieval effect became negligible (g = 0.00). This suggests that, on average and without considering specific conditions, covert retrieval might not provide a reliable learning advantage over strategies like rereading.
The Effectiveness of Covert Retrieval is Highly Dependent on Specific Conditions (Heterogeneity). The meta-analysis revealed substantial heterogeneity among the effects of covert retrieval, meaning its effectiveness varied widely across studies. Teachers cannot assume it will be effective in just any situation. Its benefits were found to be significantly moderated by:
- The Control Strategy Used for Comparison: The effect was much larger when compared to filler or no activity (g = 1.13) than when compared to rereading (g = 0.06). Since rereading is a common alternative strategy for students, the practical benefit over rereading might be very small or non-existent.
- Provision of Corrective Feedback: The effect was substantially larger when corrective feedback was provided (g = 0.59) than when it was not (g = 0.04). Applying covert retrieval without ensuring timely and accurate feedback might result in little to no learning enhancement.
- Retention Interval: Covert retrieval’s enhancement effect was primarily observed for long-term memory retention (g = 0.42 for long intervals, ≥ 1 day), but there was negligible enhancement for immediate test performance (g = -0.07 for short intervals, < 1 day). If a teacher’s goal is immediate recall or understanding within a single class period, covert retrieval may not be the best strategy.
- Covert Retrieval Mode: Not all forms of covert retrieval were found to be effective. Only answer thinking (g = 0.23) and making delayed Judgments of Learning (JOLs) (g = 0.31) produced detectable enhancement effects, while answer monitoring (g = -0.21) did not. Teachers need to guide students on how to perform covert retrieval actively (mentally retrieving the answer or judging likelihood of recall) rather than just passively monitoring.
Overt Retrieval is Generally More Effective. The meta-analysis consistently found that overt retrieval is more effective than covert retrieval overall (g = 0.17). This finding showed minimal risk of publication bias. While covert retrieval can be a useful supplement when overt retrieval is not feasible, the findings suggest that teachers should prioritize activities that require students to overtly produce responses (writing, typing, speaking aloud) for maximum benefit. Relying solely on covert methods might mean students are missing out on a more powerful learning strategy.
Further Research is Needed on Specific Aspects. The meta-analysis highlights areas requiring more investigation. For instance, there were a limited number of studies on answer monitoring, and more research is needed to understand if or how this common classroom activity can be made more effective. The optimal type and timing of corrective feedback with covert retrieval also need systematic exploration. This suggests the current understanding is not exhaustive, and applying strategies without considering these nuances might not yield desired results.
Translating Findings to the Dynamic Classroom Environment: While the study suggests test-enhanced learning generalizes to classrooms, the practical implementation of specific covert retrieval modes (like encouraging all students to “thoroughly think about the answer” before calling on one) in a busy classroom requires careful consideration to ensure students exert the necessary cognitive effort. Passive listening while one student answers is unlikely to be effective.

In summary, teachers should approach the findings with caution because the overall effect of covert retrieval is small and subject to publication bias, its effectiveness is highly variable and depends on specific conditions like feedback, retention interval, and the exact mode of covert retrieval used, and overt retrieval is generally a more robust strategy. While covert retrieval can be a valuable tool, it needs to be implemented strategically, particularly when focusing on long-term retention of complex materials, providing feedback, and ensuring students engage in effortful thinking rather than passive monitoring.

What is a single quote that summarises the key findings from the paper?

While covert retrieval can enhance learning of educationally relevant materials and consolidate long-term retention, it is generally less effective than overt retrieval