Designing and implementing experiments within local bureaucratic systems: A cautionary tale from an educator incentive program

Asad, Saher; Habib, Masooma; Karachiwalla, Naureen; Kosec, Katrina; Leaver, Clare; and ur Rehman, Attique. Designing and implementing experiments within local bureaucratic systems: A cautionary tale from an educator incentive program. Education Finance and Policy. Article in press. First published online June 21, 2024. https://doi.org/10.1162/edfp_a_00435

Collaborating with governments to co-develop pilot programs and integrate them into local administrative systems is increasingly regarded as a key strategy for achieving scalability and sustainability. This study examines a pilot initiative designed in partnership with the Elementary and Secondary Education (E&SE) Department in Pakistan's Khyber Pakhtunkhwa province, offering important lessons for such collaborations. While the intent to operate within existing bureaucratic frameworks was commendable, it also limited the design and execution of a randomized controlled trial (RCT). This paper explores institutional factors that hindered the RCT's implementation, resulting in missed opportunities to evaluate critical design elements. The pilot centered on promotion-based incentives for educators and had null effects on a variety of educator and student outcomes. However, using unique video-based evidence, it objectively identifies the implementation challenges and describes the insights gained for more effective co-design and integration of future pilot programs within governmental systems.