Education operates in long cycles which explains why measuring the effect of a product can be complex. Implementing Randomized Controlled Trial (RCT) is expensive and difficult to settle in classes where environments are uncontrollable. To overcome these issues, the Californian Department of Education has defined 4 tiers of evidences, from the more (RCT) to the less experimental tier, based on external-to-the-product research studies. This last tier of evidence is the one defended by Educapital.
Regardless those startups operating in the Lifelong Learning segment - where impact is shown via employability - other Edtechs should at least base their product on science in order to get financial support to implement quantitative analysis.
When it comes to measuring the impact and outcome of an Edtech product, companies refer most of the time to their customers’ feedbacks. If they must of course be part of the evaluation, they are nonetheless an uncomplete and insufficient tool to objectively measure learning outcome.
Very much like the health industry, education works in long cycles, and this makes it much harder to measure its immediate but also long-term effects. The closest to the trade-market a service is, the easier it is to measure its impact, insofar as we can directly look at professional insertion. What about Edtech tools that play a role long before professional integration and cannot be evaluated on the basis of an employability criterium, in the pre-K12 and K12 segments for instance?
We review below some different and relevant ways to assess the impact of Edtech solutions.
Why does it seem so hard to demonstrate the efficiency of an Edtech tool?
Demonstrating that an Edtech product is working and has a positive impact is tricky. Most of the time, when possible, an RCT (Randomized Controlled Trial) is used to prove the effect on a treatment by comparing two exactly identical groups to one another. One group receives a treatment (the edtech tool in this case), and the other one doesn’t. This method ensures that the learning outcome difference between the two groups is exclusively the result of the treatment.
However, as an article on Edsurge puts it: “The conditions of a classroom are too different from those in a medical clinic. Learning environments, for instance, have a myriad of variables that are difficult to control. In an RCT, doctors control patient medical records, monitor health conditions and ensure that patients take the prescribed drugs. By contrast, teachers use interventions in multiple class sizes, with different levels of training and varying degrees of leadership support. Students are also not “prequalified”—in other words, they frequently begin a course of study with radically different levels of knowledge.
What’s more, parents rightly object if their students are not given every opportunity to succeed—and may protest if their child receives either an education “placebo” or an “untested” intervention. Finally, RCTs take literally years to complete and are costly.”
So, is there any other relevant way to assess the outcome of an edtech tool?
Yes, there is, even if they are not as reliable as the RCT method.
The Californian Department of Education defined 4 tiers of evidence that schools should look at before using funds for products and services, as part of the Every Student Succeeds Act (ESSA) :
Tier 1: Strong Evidence: [Edtech products are] supported by one or more well-designed and well-implemented randomized control experimental studies; So, these cases are very rare as mentioned above.
Tier 2: Moderate Evidence: [Outcomes are] supported by one or more well-designed and well-implemented quasi-experimental studies; Considering that we cannot always deal with a clinical environment (i.e. the exact same context and characteristics of students in each class/group of study), this moderate evidence methodology is less regarding on characteristics of participants and class context, but tries to process evaluation on as much students as possible to find the most accurate results as possible. The results will not entirely depend of the edtech tool but are still relevant to assess its impact.
Tier 3: Promising Evidence: [Outcomes are] supported by one or more well-designed and well-implemented correlational studies (with statistical controls for selection bias); The evidence provided by using this methodology is similar to the 2nd one but is less accurate. Nonetheless, studies are conducted, and they can show some correlation between the behavior of the student and the use of the edtech tool.
Tier 4: Demonstrates a Rationale: Practices associated with [the edtech product] should have a well-defined logic model or theory of action, are supported by research, and have some effort underway by an SEA [state education agency], LEA [local education agency], or outside research organization to determine their effectiveness.
Furthermore, as mentioned above, proving efficiency and finding evidence is very costly for companies and entrepreneurs. In this regard MBZ Labs and Reach Capital have identified the efficacy expectations regarding the growth of a company : The more mature and funded a company is, the more able it is to demonstrate the learning impact of its product or service. Thus, only companies at a Series B stage, or later, could afford to implement studies that could provide Tier 3 or 2 evidences.
At Educapital, we think that collaboration with entrepreneurs, researchers and teachers is a relevant and powerful first step to develop impactful product. It allows for the emergence of innovative solutions, in line with users’ real feedbacks while helping to develop a product in line with the contribution cognitive sciences. Technology is a great way to concretely develop a product/service based on what sciences tell us. This is especially true in the segment of early education, where learning cycles are long and where innovations can nurture an efficient learning process based on the four pillars demonstrated by neurosciences: attention, active engagement, feedback, consolidation. This scientific validation helps ensure that the product has a positive learning impact on its users, even before settling quantitative cohort analysis.
For example, Lalilo has been built in collaboration with the Paris Descartes University research laboratories and is based on adaptive learning (i.e. the use of AI to provide personalised learning), Lalilo adapts identification, written comprehension and oral expression exercises to the level and rhythm of each student. Concretely, in line with neuroscience research, Lalilo lets children hear a phoneme while writing or reading it, to accelerate the reading and writing learning processes.
From a different standpoint, we consider at Educapital that financial performance and societal impact are intimately interrelated in a virtuous dynamic. It is by proving its real usefulness for the greatest number of people that a product ensures its adoption and long-term existence.
Accessibility and efficiency are two key objectives that we seek and evaluate in order to fund projects that are sustainable and can be widely adopted.
Accessibility is measured by a simple quantitative criteria : the number of learners. Efficiency in lifelong learning means employability.
For initial learning, we seek to identify indicators to assess the learning outcomes, at an individual level, and we encourage our portfolio companies to collaborate with research laboratories to prove the efficiency of their solution.
As an entrepreneur, where can I find a model to follow to measure the impact of my solution?
NewschoolVenture has developed a guide designed for entrepreneurs at any stage of their research journey.
Four research practices are explored in this guide: the definition of the intended impact, the iteration based on feedbacks, the evaluation of evidence of student outcomes and the share of the discoveries.
At Educapital, we share the most important points they want to make, and their words will be our conclusion: “many types of research have value, and there is often something that can be learned and integrated into strategic decision-making. A user feedback session can uncover valuable information, as can a randomized controlled trial, or any of the many options that fall in between.” They also want to emphasize that the entrepreneur’s research journey is not linear. “Research practices don’t always fall perfectly into a sequential order, and each can have value as an independent undertaking”.