Making it Stick – Implementation of retrieval and interleaving practices

By: Kimberly G. Elder, PharmD, BCPS and Alexis A. Hamelink, PharmD

In the book Make it Stick: The Science of Successful Learning, the authors Brown, Roediger, and McDaniel set the groundwork for inventive practices in the art of making knowledge stick with students beyond the doors of the classroom.¹ As pharmacy educators, our goal is to send students out into practice with the resources and knowledge to succeed. We can teach the material, but it is up to the student to retain the vast amount of information required as a pharmacist. We must ask ourselves if there is anything we can do to improve classroom material retention to serve our students better.

How to make concepts stick:

Make it Stick brings to light evidence-based principles to help with retention, including the art of both retrieval practice and interleaving. Retrieval practice can be done by using low-stakes activities, such as quizzes, which often require active recall. Interleaving is accomplished by spacing out topics through retrieval practice over time. With interleaving, students don’t just study one topic repeatedly until they have it memorized. Instead, they study for a time and then switch to something else. This mimics the way topics are encountered in life.  Both practices are effective at making key concepts stick with students.¹ These concepts were further endorsed in another evidence-based book Small Teaching: Everyday lessons from the science of learning.² These practices were reflected on by the course coordinator of a patient care lab sequence to find a way to incorporate them into the curriculum and assess their perceived impact.

Putting the principles to work:

We implemented these practices in our patient care lab course sequence. We were aware of thousands of review questions available from the therapeutics textbook through the Access Pharmacy® website, a subscription service paid for by our college of pharmacy and health sciences. We wanted to introduce students to the availability of these questions and those associated with other textbooks used throughout the pharmacy curriculum. This was in hopes that students would recognize these as useful study tools for relevant courses in the program that would assist with retrieval practice.

Students completed assignments during weekly patient care lab sessions as part of the typical course structure. In the Fall of 2021, the course coordinator changed the method of accessing these assignments to an adaptive release mechanism within the learning management system (LMS).  Each week, the students logged in to the Access Pharmacy® website, to access and complete four review questions from the therapeutic topic most closely related to the weekly case topic. Then, students answered a perception question that asked how they felt about their performance on the questions to adaptively release the weekly assignment. As the question bank questions could be related to topics other than the weekly assignment topic, this was also an example of interleaving.

We recognized that we could have entered our own review questions directly into the LMS and had students complete them there to gather more data on student performance on the questions themselves.  However, the point of the exercise was for students to physically go to Access Pharmacy® so they would know how to access the questions in the future. 

How did the students react?

During the 2021 Fall quarter, students (n=45) completed five Access Pharmacy® problem sets, and during the 2022 Winter quarter, students (n=39) completed four problem sets. After completing the assigned problem sets, students responded to the perception question (Table 1).

Table 1

How did you feel about your performance on the (p)review questions today?	Fall (Average Responses)	Winter (Average Responses)
Reassured	29.9%	43.3%
Disappointed	33.2%	27.7%
Indifferent	37.0%	32.9%

Students also completed a perception survey of Access Pharmacy® review questions pre-and post-implementation. After the implementation, over 73% of students said that they were very or somewhat likely to utilize Access Pharmacy® questions as a study tool for advanced pharmacy practice rotations, and over 86% responded that they were very or somewhat likely to utilize Access Pharmacy® questions as a study tool for the NAPLEX. Figure 1 shows how often students reported using the Access Pharmacy® questions pre and post-implementation.

Figure 1

Final takeaways:

Since the original implementation of retrieval and interleaving practices in the patient care lab, we have had positive feedback from students regarding this practice.  However, we are still not certain of the full impact on student performance.  Future research could examine how utilizing review questions correlates with exam performance, both in the classroom and at the NAPLEX level. 

We would encourage others to try similar interventions to stimulate retrieval practice and interleaving.  The books referenced, as well as others such as How Learning Works: seven research-based principles for smart teaching³ provide additional evidence-based ideas for teaching. Please reach out to the authors if you’d like to collaborate!

References

1. Brown, Peter C., et al. Make It Stick: the Science of Successful Learning. Cambridge, MA: The Belknap Press of Harvard University Press, 2014.
2. Lang, James M. Small Teaching: Everyday Lessons from the Science of Learning. San Francisco, CA: Jossey-Bass/Wiley, 2016.
3. Ambrose, Susan A., et al. How Learning Works: seven research-based principles for smart teaching. San Francisco, CA: Jossey-Bass, 2010.

---

Author Bio(s):

---

Pulses is a scholarly blog supported by a team of pharmacy education scholars