Analysis scripts for: Evidence-Based Adaptivity Decisions for Educational Interventions

Digital technologies enable adaptive support based on students’ behavior. However, determining optimal support for each student is methodologically challenging: If adaptations depend on prior behavior, adaptivity decisions are not fully randomized, which complicates causal inference. We present a methodological approach based on sequential experimental designs and directed acyclic graphs (DAGs) to evaluate adaptivity decisions. We illustrate this approach in a mobile intervention promoting frequent vocabulary study. Students received planning prompts during a 20-day first-stage intervention, followed by an adapted prompting frequency in the second stage. Using a DAG, we formalized causal assumptions, identified confounding, and adjusted for it. The results illustrate the importance of this methodological approach to obtain valid causal conclusions: an unadjusted analysis would have led to the incorrect conclusion that scheduling more prompts leads to reduced study activity. Instead, we found that the effect of prompting frequency differed between students, highlighting the value of adapting support.

Digital technologies enable adaptive support based on students’ behavior. However, determining optimal support for each student is methodologically challenging: If later adaptations depend on prior behavior, adaptivity decisions are not fully randomized, which complicates causal inference. We introduce present a methodological approach based on sequential experimental designs and directed acyclic graphs (DAGs) to evaluate adaptivity decisions. , and apply it toWe illustrate this approach in a mobile intervention promoting frequent vocabulary study. Students received planning prompts during a 20-day first-stage intervention, followed by an adapted prompting frequency in the second stage. Using a DAG, we formalized causal assumptions, identified confounding, and adjusted for it. The results illustrate the importance of this methodological approach to obtain valid causal conclusions: an unadjusted analysis would have led to the incorrect conclusion that scheduling more prompts leads to reduced study activity. Instead, we found that the effect of prompting frequency differed between studentsThe number of scheduled prompts showed no significant effects, likely due to low engagement. In contrast, the number of prompts students actually received had a significant positive effect, particularly for students who studied less frequently and those who received fewer prompts in the first stage, highlighting the value of adapting support.

Digital technologies enable adaptive support based on students’ behavior. However, determining optimal support for each student is methodologically challenging: If later adaptations depend on prior behavior, adaptivity decisions are not fully randomized, which complicates causal inference. We introduce present a methodological approach based on sequential experimental designs and directed acyclic graphs (DAGs) to evaluate adaptivity decisions. , and apply it toWe illustrate this approach in a mobile intervention promoting frequent vocabulary study. Students received planning prompts during a 20-day first-stage intervention, followed by an adapted prompting frequency in the second stage. Using a DAG, we formalized causal assumptions, identified confounding, and adjusted for it. The results illustrate the importance of this methodological approach to obtain valid causal conclusions: an unadjusted analysis would have led to the incorrect conclusion that scheduling more prompts leads to reduced study activity. Instead, we found that the effect of prompting frequency differed between studentsThe number of scheduled prompts showed no significant effects, likely due to low engagement. In contrast, the number of prompts students actually received had a significant positive effect, particularly for students who studied less frequently and those who received fewer prompts in the first stage, highlighting the value of adapting support.