I am a big fan of evidence in education – I think it has unique potential to improve education in a systematic and scalable manner.
However, I also regularly see evidence misused in education, despite good intentions. Using evidence well is not straightforward. Often teachers have a sense of how evidence has been missed but cannot quite put their finger on the issue.
As a biologist, I like taxonomies. I think a system to classify the different ways that evidence is misused could help us recognise and avoid where evidence is misused. To my knowledge, no such system exists. A comparable tool is the detailed ‘catalogue of bias’ that is available for medical researchers.
Here are four issues that I commonly encounter.
1. Starting with a solution
Evidence is used to justify a decision that has already been made. I often see this when school leaders feel under pressure to justify their choices and show rapid progress.
Perhaps the most common version of this is with digital technology where the technology itself is seen as the solution. I cannot do better than Sir Kevan Collins who asks:
If digital technology is the answer, what is the question?
Sir Kevan Collins
The problem with starting with a solution, is that we simply lose sight of what we are trying to achieve and waste precious time. More broadly, I worry that this misuse of evidence undermines the wider evidence-informed movement in education.
2. Ignoring context
Context is crucial in education, but it can be too easy to overlook it. Here, I think an extreme example is instructive.
The Piso Ferme programme involved replacing the dirt floors of homes in rural Mexican villages with concrete floors. Piso Ferme delivered striking health and education benefits and we can be confident in these findings due to the rigour of the research.
So, should we load up the cement mixer and drive around villages in England?
No.
Piso Ferme worked by reducing infections spread through the dirt floors, such as worms that can cause a range of diseases. While Piso Ferme is an ‘evidence-based’ programme that ‘worked’, it is not suitable for the UK because the problem it is designed to address simply does not exist.
Beyond thinking about if the mechanism works, it is also important to consider the fit and feasibility. For instance, the alignment of values and norms of the approach with the intended recipients. An interesting lesson about this comes from the failure of the Intensive Nutrition Programme in Bangladesh described by Jeremy Hardie. Failure to understand the nature of family structures meant that an ‘effective’ programme failed because it was not adapted to the local context.
3. Cherry picking evidence
Perhaps the most common issue is selectively picking evidence. This often happens in combination with issue number one to justify a decision that has already been made.
Cherry picking is particularly problematic as there is usually some research to support any idea in education – especially if you are not picky about the quality.
Even if you do focus on high quality evidence – different studies reach different conclusions and unless you look at all the relevant evidence, you can be led astray. We can see this by looking at the impact of the studies featured in the EEF’s Teaching and Learning Toolkit about mentoring. Although the overall headline is that – on average – mentoring makes limited difference, individual studies differ.
If you want to claim mentoring is rubbish, point to the red study. If you want to convince a school to buy your mentoring programme, showcase the green study. Only by looking at all of the available evidence can we get an honest picture.
In addition to using all of the relevant evidence, we should avoid taking a confirmatory approach – if we actively look for contradictory evidence to our favoured ideas, we are likely to reach better decisions.
4. Using the wrong types of research
‘Hierarchies of research’ have an interesting and controversial history.
While it’s not true that there is a single ‘best’ type of research, it is true that different types of research excel at answering different types of questions.
The most common issue I see, is when causal claims are made from research, which is simply incapable of credibly supporting them. The classic of this genre is the case study that makes sweeping claims about impact.
Personally, I like this table from the Alliance for Useful Evidence, which highlights the importance of matching types of research questions with different types of research.
These issues are the tip of the iceberg when it comes to evidence misuse, but they are probably the most common ones that I encounter. Hopefully, by recognising these issues, we can more effectively realise the potential of evidence.