Rapid Evaluation for a Reshaped World
One of the most important lessons for social impact organizations and entrepreneurs is that good intentions alone do not create impact. That’s why we’ve worked hard to build an impact-obsessed culture that aims to create meaningful, measurable impact on life outcomes for youth. Our rigorous evaluations help us better understand Educate!’s impact as well as how to strengthen it. When the coronavirus began to spread, we had to adapt our monitoring and evaluation approach to a new set of program strategies. School closures presented a pressing need to develop new tools and systems for real-time data collection and rapid model development and improvement. Adapting our proven, in-school model to distance learning has required us to experiment, iterate, and evaluate our solutions much faster than ever before. We believe our models will be strengthened by these new evaluation strategies, and we will emerge from the pandemic with even better tools to create and measure impact for youth.
Recognizing a Need to Learn Faster
Rigorous, externally-run Randomized Controlled Trials (RCTs), like those we have completed in Uganda and Rwanda, are the gold standard for impact evaluation, but the process of collecting and analyzing data takes a very long time. As a result, it is difficult to make evidence-based design updates to our model as quickly and frequently as we would like. We identified the need for more rapid evaluation strategies before the pandemic, and the need became even more pressing when the pandemic forced schools to close. We began designing our new distance learning model knowing that our ability to learn and make changes as quickly as possible would be crucial, and that our changes would only be as good as the evidence we had to support them.
Our team set out to construct a Rapid Impact Assessment system to facilitate faster, more iterative learning. This tool allows us not only to run traditional experiments comparing a treatment and a control group, but also A/B testing — a comparison of the impact of one design option against another. This lets us see what works and how it is working, and target the most successful elements to generate the greatest impact on youth. Where a typical RCT could take years to return results, this system can provide valuable model insights in just a few months. In the era of COVID-19, this tool has generated critical information about how to quickly transition core components of our proven model to a distance-friendly format. Part of changing how quickly we can collect and evaluate data was identifying variables we could measure quickly — indicators of future skill development, improved gender equity, increased education attainment, and improved livelihoods. For example, when youth set goals and make business or community development plans, express ideas clearly, and work together with peers and community members to achieve their goals — this suggests they are forming the skills that will facilitate improved life outcomes. In measuring these variables, we can learn more about our possible long-term impact on youth in a shorter amount of time.
Targeting Pandemic-Era Impact with Rapid Assessment
Since launching the new Rapid Impact Assessment system, we have been able to test the design and delivery of our new distance learning solutions as we develop them, rather than waiting for the pilot model to end, awaiting impact evaluation results, and then making updates. Instead, we set up pilot experiments to generate data that helps us iterate and make improvements almost immediately. For example, we learned through an A/B test in a product development sprint that calling participants’ caregivers to tell them about Educate!’s remote learning activities could increase youth participation by 29%, so we quickly integrated this targeted outreach into the model.
The main purpose of the Rapid Impact Assessment system going forward is to understand the impact of our programs on youth — what does participation in Educate!’s distance learning model mean for skill development and other life outcomes? With this tool, we are developing and tracking indicators of success that directly map to the life outcomes we are targeting and have observed in our RCTs: improved transferable skills, increased educational attainment, greater gender equity, and improved incomes and business ownership. While Educate! has always been a learning organization, the unprecedented challenges of this year have strengthened our team’s ability to design new systems to help us better manage and monitor our impact on youth. So far, we have discovered that young people have been able and excited to learn remotely if they receive adequate support. As we continue to navigate this challenging time, we are looking forward to iterating and refining this new system so that we can strengthen our impact on youth, both during the pandemic and after.