Intervention Components Paired with Timed Practice
As referenced in the previous post, I finished a meta-analysis focused on intervention studies testing timed math practice, those interested can read the preprint. The OSF page includes all the materials used for the project.
Disclaimer
This post will be focused on information about the additional intervention components that were paired with timed practice! For example,
Can additional components, such as goal setting, reinforcement, immediate feedback lead to greater effects than timed practice in isolation?
How can nuanced details about intervention components lead to different effects?
Goal Setting
It’s difficult to provide definitive advice on the value added of goal setting paired with timed practice. Most studies did not use goal setting in isolation, but rather paired it with other components (which I will discuss below). However, let’s discuss some elements that are percolating in my mind related to goal setting!
Arthur Glenn Dowdy had a problem. While working in his office at Temple, his lunchtime eating habits were making him feel sluggish and impacting his overall feeling of health. He was eating Philly cheese steaks, grabbing a snack from a food truck on campus, or popping on over to the Italian Market to grab a sammie from George’s, a couple slices of pizza from Lorenzo’s, or some pasta from Ralph’s Italian Restaurant. He set a goal to eat healthier at lunchtime. In order to achieve this goal, he developed an actionable plan - he was going to pack a lunch for himself daily, rather than eating out. He opted to plan his lunch as a tin can of sardines and some fruit (would not have been my first choice but we never yuck someone else’s yum). He now has a cabinet full of sardines and a fridge full of fruit that he can quickly grab in the morning to bring to work. Some of us might worry about satiation (getting bored of the same thing) but Art has stuck with this for several years now.
So why the funny anecdote about Art? It highlights several factors we must consider about the use of goal setting as an intervention component.
Who is setting the goal?
An important element to consider is who sets the goal. Is the teacher (or researcher) setting the goal? Is the student setting the goal? Is there a joint effort (teacher supporting student) in setting the goal? Student involvement in the goal setting process may help build an element of ownership over their learning and lead to higher motivation versus a goal being dictated to them.
So what was found in the timed practice literature base? Only two studies attempted to manipulate who set the goal.
Fuchs (1989) identified that high school students who were given a choice of goal (3 options of different performance criteria) outperformed students who were merely assigned a goal. The authors did not an interesting finding, large gains were found from pre-test to the midpoint of the experiment but this growth plateaued from mid-point to post-test. The authors hypothesize that students may have encountered habituation (gotten bored) with the novelty of being able to select their own goal or the goals were inappropriate with their current abilities and the instruction they were receiving.
Gross (2014) evaluated three different conditions with elementary students: (a) researcher set the goal, (b) researcher set the goal with goal lines and graphical feedback, and (c) student set the goal with goal lines and graphical feedback. The question of whether the person setting the goal might influence performance can be evaluated by comparing condition (b) and (c) from above. Counter to their hypothesis, the researchers identified the condition that had researchers setting the goal with graphical feedback (condition b) had greater growth than the condition with students setting their own goal (condition c) – however, this was not a statistically significant difference.
This is an area we need more research. It makes a lot of intuitive sense to me to include the student in the goal setting process - however, I would imagine there would need to be teacher support in how to set appropriately, ambitious goals to work towards. In Gross (2014), they shed some light on the goals student set (and how this compared to the other conditions) and how this impacted the contingent reinforcement paired with goal attainment.
A comparison of goals set by groups indicated that the RO (condition A) and RG (Condition B) groups on average had lower goals than the SG group (Condition C). An examination of the average weekly goal for each group across weeks appears to indicate that the RO and RG goals were similar from week to week, but the SG group had self-selected higher goals for each of the 6 weeks. In addition, the RO and RG groups’ goals increased incrementally from week to week, whereas the SG group’s goals dropped from Week 1 to Week 3 and then appeared to level off from Week 4 to Week 6. It is possible that the students in the SG group were unfamiliar with setting accurate goals and that they needed more instruction for setting goals. Fuchs and colleagues (1991) found similar fluency performa- nce changes across for daily repeated practice of math facts, which they attributed to the use of the same practice procedure. Likewise, the similar goals between the RO and RG groups could be an artifact of the same explicit timing procedures used for daily probes yielding similar daily performance and daily performance goals.
The RO and RG groups had similar levels of goal attainment and the same amount of reward attainment. The SG group had less goal attainment and considerably less reward attainment. Lower rates of goal achievement and reward attainment may explain the difference between the RG and SG groups. The differences in performance between the RG and SG groups and RO group may be best explained by providing goal lines. The goal lines may function as an intermediate reinforcer because the students liked coloring past goal lines or seeing their performance line above their goal line. This echoes previous research, which asserts that the act of reaching performance-approach goals serves as rewarding to individuals (Morisano & Shore, 2010). (Gross et al., 2014, p. 567)
What is the goal?
Because the dependent variables across studies targeted fluent responding, all studies that used goal setting targeted a goal centered around correct items (or digits) solved correctly in a given time frame.
Goal setting though can target other behaviors or performance indicators. This section of writing below is from the preprint I shared above (p. 48).
Carr and Punzo (1993) provide an example of how this decision can play out with students. They examined self-monitoring of accuracy or productivity paired with reinforcement for students with emotional and behavioral disorders engaging in academic tasks across reading, mathematics, and spelling. When students self-monitored and received reinforcement for accuracy, this increased however in some cases productivity decreased. Similarly, when productivity was targeted, this increased however in some cases accuracy decreased – this might align with some individuals concern of timed practicing reinforcing the idea that all that matters is speed. If tied to the instructional hierarchy, it may be more beneficial to target accurate responding during initial acquisition as the goal is to discriminate between correct and incorrect responding. As students acquire new concepts and shift to fluency building, shifting to target fluency (i.e., speed + accuracy) may be more beneficial.
Reinforcement
Reinforcement aims to increase a desired behavior or maintain a desired behavior that is occurring. In many studies, reinforcement was provided immediately following a timed practice trial a student completed (although one single-case study provided reinforcement through behavior specific praise during practice trials). By and large, studies provided reinforcement by allowing students access to preferred activities or access to the treasure box.
What gets tricky for me when reading the articles included in this review is many studies might have had reinforcement present but did not identify it as a component of the intervention. For example, students engage in timed practice and beats their previous score. If the interventionist (researcher or teacher) gives a fist bump and says, “great job today,” and the student felt validated for their effort and it makes them want to continue to improve in their math performance, then this is reinforcement. This fictitious, silly example is something we all can imagine happening in a school-based environment.
Another element that is tricky in reviewing the articles is hypothesizing what might be acting as the reinforcer. For example, let’s consider what Gross (2014) mentioned above.
Student completed a timed practice sheet
Student graphs their new score
Student realized they beat their previous score (potential reinforcer)
Teacher gives a fist bump and says, “great work - you are putting in effort and learning so much” (potential reinforcer)
Student gets access to the treasure box for goal attainment (potential reinforcer)
Is one of these driving the motivational ship? It becomes difficult to unpack.
To me, the key is to consider how to use variables that naturally are present in the environment as a form of reinforcement. Basically, do we want to have to continue to use the treasure box after every probe? Probably not.
Obviously, the treasure box might have a strong reinforcement property for kids so it becomes an easy thing to set up as a reinforcer. However, if you pair this with goal setting and teacher specific praise you can use other variables that occur naturally in the environment to potentially serve as reinforcers. Perhaps, across time, teachers can thin out the frequency in which students get access to the treasure box without seeing dips in motivation because the other two variables can serve as reinforcers.
Gross (2016). I have one more study I found fascinating that I will share here. In this study, the researcher compared three different conditions:
Explicit timing paired with goal setting in isolation.
Explicit timing paired with goal setting and independent contingent reinforcement. Students received reinforcement based on their own performance.
Explicit timing paired with goal setting and group contingent reinforcement. In this condition, three students performance were randomly selected (the students who were selected were kept anonymous to students in the class). If the three students beat their goal, the entire class received reinforcement.
Group-contingent reinforcement has been used in other contexts, specifically within classwide interventions targeting behavior. In this study, independent contingent reinforcement yielded larger gains than group-contingent reinforcement or explicit timing in isolation. Similar gains were found for explicit timing and group-contingent reinforcement.
Immediate Feedback
Another component used was immediate feedback on correct or incorrect responding at the item level. Timed practice is geared around fluency building, which is focused on math items students have high accuracy with. Thus, it might seem counterintuitive to provide item level feedback on correct or incorrect responding. However, immediate feedback was shown to be effective.
Hastings (2010). We have all heard the saying, “A done dissertation is a good dissertation.” My dissertation was kind of crap - but I was proud of the work I did to finish it. Hastings (2010) on the other hand was an excellent dissertation. In this project, the researcher aimed to do a component analysis to gauge the additive value of various intervention components paired with explicit timing.
Control
Explicit Timing (ET; isolation)
ET + Immediate Feedback (referred to as performance feedback)
ET + Goal Setting & Reward for Goal Attainment
ET + Immediate Feedback + Goal Setting & Reward for Goal Attainment
The largest effect was identified for the condition that paired all of the components together (ET + Immediate Feedback + Goal Setting & Reward for Goal Attainment). ET + Immediate Feedback outperformed ET + Goal Setting & Reward for Goal Attainment.
However, an interesting aspect of this study was the researcher evaluated which condition was the most “cost effective” in terms of how many components were included given the effects observed. See here:
The evaluation of the strength ratio revealed that the ET + Immediate Feedback + Goal Setting & Reward for Goal Attainment offered the largest strength ratio. When the isolated components were broken into their individual strength ratios and summed, the sum of the strength ratios was actually larger than the combination intervention’s strength ratio indicating that the effects were considered an example of diminishing returns. Therefore, the ET + Immediate Feedback offered the most cost efficient or “strong” intervention.
Duhon (2015) compared three conditions (a) a control condition, (b) explicit timing, and (c) explicit timing + immediate feedback. I felt like this graph is a helpful visual for the observed effects.
Immediate feedback can be a salient, beneficial intervention component for multiple reasons. In the studies included, many set an inclusion criteria for students at 80% accuracy or above. Thus, many students that were included in the study were not performing at 100% accuracy. Thus, the immediate feedback following each item would help provide corrective feedback for any items students were providing incorrect responses too. A second reason immediate feedback can be beneficial is the reinforcement property it provides to students. Getting affirmative feedback following correct responses can be reinforcing - it feels good to be right.
Conclusions
What intervention components can we use?
Goal Setting. One element to consider is who is setting the goal.
Performance Feedback. One element to consider is how the performance feedback is provided. The individual score can be shared with the student or perhaps the data can be shared via a graph with the goal line present. In some instances, researchers have had the student self-evaluate their own work and then self-graph their own progress (versus giving all of this over to the teacher).
Reinforcement. Identify ways to motivate students to put forth effort practicing. There are potential reinforcers everywhere within the environment, so find ways to leverage these. As mentioned previously, students beating their own previous score and observing their own learning can be a potential reinforcer. Teachers providing praise can potentially serve as a reinforcer. Providing students access to a preferred activity or item may be reinforcing to students - although this might be harder to consistently do.
Immediate feedback. With access to a computer-based system, this is the easiest component to implement without requiring additional time commitments (compared to performance feedback or goal setting).
Teachers can collect data on student performance and try adding a component or components to see if it accelerates growth. Don’t forget other variables (i.e., see the post on dosage) that also can be manipulated to accelerate growth.