Thomas Martell

The ITT Market Review aims to ensure consistently high-quality training in a more efficient and effective market. It is currently out for consultation and could dramatically reshape how we prepare teachers in England.

The main recommendation is that all providers should implement a set of new quality requirements and that an accreditation process should ensure that providers can meet these requirements.

The new quality requirements cover:

• A greater focus on the curriculum linked to the Core Content Framework
• The identification of placement schools and ensuring that these placements are aligned with the training curriculum
• The identification and training of mentors, including introducing the new role of lead mentors
• The design and use of a detailed assessment framework
• Processes for quality assurance throughout the programme
• The structures and partnerships needed to deliver a programme and hold partners accountable for the quality of their work
• An expectation that courses last at least 38 weeks, with at least 28 weeks in schools

How could the review succeed?

Instead of focusing on the nature of the proposals – should courses last at least 38 weeks? Is the Core Content Framework appropriate? – I want to analyse the proposals in their own terms: are they likely to achieve their stated goals? To do this, it helps to think about potential mechanisms that could lead to improvements and potential support factors and unintended effects.

Mechanism 1: Removing less effective providers

Less effective providers could be removed from the market, and this could raise average quality. This could happen in multiple ways: providers might decide it is all too much not re-apply, which is problematic if they are a strong provider. Second, some providers may merge. Third, some providers may try but fail to meet the requirements. Time will tell how many of the 240 accredited ITT providers fit into each category.

How do we accurately assess the quality of provision?

Getting this right is fundamental if removing less effective providers is a crucial mechanism for strengthening the market. However, we should consider for every less effective provider we remove, how many strong providers we are willing to sacrifice because of the fundamental trade-off between false positives and negatives in any selection process.

The distribution of provider quality and how the assessment is done will influence the relative trade-off between sensitivity and specificity. Do we know the distribution of provider quality? My hunch is that most providers are similar, but there are long tails of stronger and weaker providers. If this is the case, do we draw the line to chop off the tail of weaker providers, or do we cut into the body of similar providers?

The second consideration is how to judge provider quality. The consultation offers a high-level process on page 29 involving a desk-based exercise with providers responsible for submitting evidence. But who will apply the quality requirements to the evidence submitted? Civil servants supported by some expert input? This might work well for some aspects, such as assessing quality assurance processes, but the heart of the reforms – the curriculum – is much harder to assess.

To maximise the accuracy of judgements, it makes sense to do it in phases: an initial separation of those that very clearly do or do not meet the criteria and then a more intensive stage for those that might meet the requirements. Otherwise, an appeal mechanism might be wise. Using a phased approach could improve the assessments’ accuracy while making the most of everyone’s finite resources.

While still thinking about the distribution of provider quality, it is worth asking if there is enough meaningful variation. If most providers are pretty similar, then at best, we can only make relatively minor improvements by removing the least effective providers. There might be more meaningful variation at the subject level or even at the level of individual tutors. If true, and we could accurately measure this variation, this would hint at a very different kind of market review (licences for ITT tutors, anyone? No?). For context, eight providers each developed over 500 teachers last year, and UCL almost reached 1,600 – should we look at a more granular level for these providers?

The less effective providers are gone; what next?

We now need to replace the capacity we removed by introducing new providers or expanding the remaining higher-quality providers. Removing lots of less effective providers is a promising sign the mechanism is working, but it poses a challenge: can we bring in new capacity that is – quite a bit – better? This may depend on how much we lose: is it 5, 15 or even 50 per cent?

Do we know if new providers will join? It would probably be wise to determine if this is likely – and the potential quality – before removing existing providers. The quality requirements set a high bar for new entrants, so a rush of new providers seems unlikely. That said, some Trusts and Teaching School hubs may come forwards – especially if given the grant funding for the set-up work advocated in the review. Other providers like the ECF and NPQ providers not already involved with ITT, including Ambition Institute, may consider applying.

Can we replace the capacity we remove with capacity that is – quite a lot – better?

Expanding existing strong providers seems desirable and straightforward enough, but we should heed the warnings from countless unsuccessful efforts of scaling promising ideas. Spotting barriers to scalability – before you hit them – is often tricky. Sir David Carter’s observation that when Trusts grow to the point that the CEO can no longer line manage all of the headteachers, a scalability barrier has been reached – new systems and processes are needed to continue operating effectively.

What are the barriers for an ITT provider? The brilliant people and the delicate partnerships with placement schools that have often developed over several years are challenging to scale. No doubt there are many more.

Before we forget, what about those providers that merged to get through the application process? How do we ensure the best practice is embedded across their work? Again, this isn’t easy, and we will likely have to base the judgements on providers’ plans rather than the actual implementation, given the timeline. Nonetheless, it seems likely that money and time will help. An analogy is the Trust Capacity Fund that provides additional funding to expanding Trusts for focused capacity building. 

Summary

If we think that removing less effective providers is an effective mechanism for the ITT Market Review, then we should:

• Purposefully design and implement the selection process
• Plan for how to replace the removed capacity
• Ensure that time and money are not undue obstacles
• Consider phasing the approach

In part two, I explore another mechanism – programme development – that the ITT Market Review might use to achieve its goals.

Effect sizes are a popular way of communicating research findings. They can move beyond binary discussions about whether something ‘works’ or not and illuminate the magnitude of differences.

Famous examples of effect sizes include:

• The Teaching and Learning Toolkit’s months’ additional progress
• Hattie’s dials and supposed ‘hinge point’ of 0.4

Like anything, it is possible to use effect sizes more or less effectively. Still, considering these four questions will ensure intelligent use.

What type of effect size is it?

There are two fundamentally different uses of effect sizes. One communicates information about an association; the other focuses on interventions. Confusing the two effect sizes leads to the classic statistical mistake of confusing correlation with causation.

Understanding the strength of associations, or correlations, is important. It is often the first step to learning more about phenomena. For instance, knowing that there is a strong association between parental engagement and educational achievement is illuminating. However, this association is very different from the causal claim that improving parental engagement can improve school achievement (See & Gorard, 2013). Causal effect sizes are more common in education; we will focus on them with the remaining questions.

How did the overall study influence the effect size?

It is tempting to think that effect sizes tell us something absolute about a specific intervention. They do not. A better way to think of effect sizes is as properties of the entire study. This does not make effect sizes useless, but they need more judgement to make sense of them than it may first appear.

Let’s look at the effect sizes from three EEF-funded trials (Dimova et al., 2020; Speckesser et al., 2018; Torgerson et al., 2014):

• Using Self-Regulation to Improve Writing (IPEELL) +9 months’
• Nuffield Early Language Intervention (NELI) +3 months’
• Embedding Formative Assessment (EFA) +2 months’

All these programmes seem compelling, and Using Self-Regulation to Improve Writing appears the best. These are the two obvious – and I think incorrect – conclusions that we might draw. These studies helpfully illustrate the importance of looking at the whole study when deciding the meaning of any effect size.

1. Some outcomes are easier to improve than others. 

The more closely aligned an outcome is to the intervention, the bigger the effects we can expect (Slavin & Madden, 2011). So we would expect a programme focusing on algebra to report larger effects for algebra than for mathematics overall. This is critical to know when appraising outcomes that have been designed by the developers of interventions. In extreme cases, assessments may focus on topics that only the intervention group have been taught!

There’s also reason to think that some subjects may be easier to improve than others. For instance, writing interventions tend to report huge effects (Graham, McKeown, Kiuhara, & Harris, 2012). is there something about writing that makes it easier to improve?

2. If the pupils are very similar, the effects are larger.

To illuminate one reason, consider that around 13 per cent of children in the UK have undiagnosed vision difficulties (Thurston, 2014). Only those children with vision difficulties can possibly benefit from any intervention to provide glasses. If you wanted to show your intervention was effective, you would do everything possible to ensure that only children who could benefit were included in the study. Other pupils dilute the benefits.

3. Effects tend to be larger with younger children.  

Young children tend to make rapid gains in their learning. I find it extraordinary how quickly young children learn to read, for example. 

A more subtle interpretation I’ve heard Professor Peter Tymms advocate is to think about how deep into a subject pupils have reached. This may explain the large effects in writing interventions. In my experience, the teaching of writing is typically much less systematic than reading. Perhaps many pupils are simply not very deep into learning to write so make rapid early gains when writing is given more focus.

4. More rigorous evaluations produce smaller effects.

A review of over 600 effect sizes found that random allocation to treatment conditions is associated with smaller effects (Cheung & Slavin, 2016). Effects also tend to be smaller when action is taken to reduce bias, like the use of independent evaluations (Wolf, Morrison, Inns, Slavin, & Risman, 2020). This is probably why most EEF-funded trials – with their exacting standards (EEF, 2017) – find smaller effects than the earlier research summarised in the Teaching and Learning Toolkit.

5. Scale matters

A frustrating finding in many research fields is that as programmes get larger, effects get smaller. One likely reason is fidelity. A fantastic music teacher who has laboured to create a new intervention is likely much better at delivering it than her colleagues. Even if she trained her colleagues, they would likely remain less skilled and motivated to make it work. Our music teacher is an example of super realisation bias that can distort small scale research studies.

Returning to our three EEF-funded studies, it becomes clear that our initial assumption that IPEELL was the most promising programme may be wrong. My attempt at calibrating each study against the five issues is shown below. The green arrows indicate we should consider mentally ‘raising’ the effect size. In contrast, the red arrows suggest ‘lowering’ the reported effect sizes. 

This mental recalibration is imprecise, but accepting the uncertainty may be useful.

How meaningful is the difference?

Education is awash with wild claims. Lots of organisations promise their work is transformational. Perhaps it is, but the findings from rigorous evaluations suggest that most things do not make much difference. A striking fact is that just a quarter of EEF-funded trials report a positive impact.

Historically, some researchers have sought to give benchmarks to guide interpretations of studies. Although they are alluring, they’re not very helpful. A famous example is Hattie’s ‘hinge point’ of 0.4, which was the average from his Visible Learning project (Hattie, 2008). However, the included studies’ low quality inflates the average; the contrast with the more modest effect sizes from rigorous evaluations is clear-cut. However, it does highlight the absurdity of trying to compare effect sizes with universal benchmarks.

The graphic below presents multiple representations of the difference found in the Nuffield Early Language Intervention (+3 months’ additional progress) between the intervention and control groups. I created it using this fantastic resource. I recommend using it as the multiple representations and interactive format help develop a more intuitive feeling for effect sizes.

How cost-effective is it?

Thinking about cost often changes what looks like the best bets. Cheap, low impact initiatives may be more cost-effective than higher impact, but more intensive projects. An excellent example is the low impact and ultra-low-cost of texting parents about their children’s learning (Miller et al., 2016).

It is also vital to think through different definitions of cost. In school, time is often the most precious resource.

In summary

Effect sizes are imperfect but used well they have much to offer. Remember to ask:

• What type of effect size is it?
• How did the overall study influence the effect size?
• How meaningful is the difference?
• How cost-effective is it?

---

Next steps for further reading

Kraft, M. A. (2020). Interpreting Effect Sizes of Education Interventions. Educational Researcher. https://doi.org/10.3102/0013189X20912798

Piper, K. (2018). Scaling up good ideas is really, really hard — and we’re starting to figure out why. Retrieved January 23, 2021, from https://www.vox.com/future-perfect/2018/12/10/18127987/global-health-poverty-development-scaling-economics-research-yrise-yale

Simpson, A. (2018). Princesses are bigger than elephants: Effect size as a category error in evidence-based education. British Educational Research Journal, 44(5), 897–913. https://doi.org/10.1002/berj.3474

Schools are hotbeds of innovation. In my role supporting schools to develop more evidence-informed practice, I always admire teachers’ creativity and dedication. However, I also see colleagues trying to do too many things, including things likely to have limited impact based on the best available evidence.

A clear message from the Education Endowment Foundation’s popular resource on putting evidence to work is that schools should do fewer things better (EEF, 2019). This includes stopping things that are less effective in order to release the capacity to do even better things. In my experience, these messages are beginning to take hold; they also feature prominently in the new national professional qualifications.

At a system level, I think we should do more to stop ineffective initiatives. The Department for Education (DfE) is increasingly good at scaling up initiatives with promise, such as the Nuffield early language intervention (NELI), which, according to multiple rigorous evaluations, has improved children’s communication and language (Dimova et al., 2020).

What about ineffective programmes?

A recent evaluation of Achievement for All’s flagship programme – used by around 10 per cent of schools in England – provides a fascinating case study (Humphrey et al., 2020). The evaluation was concerning: it found children in the control schools did considerably better than their peers in schools using the intervention. The study received the EEF’s highest security rating of five padlocks based on the randomised design, large scale, low dropout and low risk of wider threats to validity. This is on top of the EEF’s exacting standards, involving independent evaluation and pre-specifying the analysis to reduce ‘researcher degrees of freedom’ (EEF, 2017; Gehlbach & Robinson, 2018).

In short, we can be very confident in the headline: children in the Achievement for All schools made two months’ less progress in reading, on average, compared to children in schools that did not receive the programme.

What happened after the evaluation?

Not much.

The EEF (2020) published helpful guidance for schools currently using the programme, and Achievement for All published a blog (Blandford, n.d.) essentially rejecting the negative evaluation – yet many schools continue to use the programme.

The contrast is stark: when programmes are evaluated with promising results, they are expanded; when evaluations are less positive, there are limited consequences.

What if we actively stopped ineffective interventions?

If we assume that the findings from the evaluations of programmes such as Achievement for All generalise to the wider population of schools already using the programme – a quite reasonable assumption – then investing in stopping it is an excellent investment.

A bold option is to simply pay organisations to stop offering ineffective programmes – think ‘golden goodbyes’. The government, or a brave charity, could purchase the intellectual property, thank the staff for their service, provide generous redundancy payments, and concede that the organisation’s mission is best achieved by stopping a harmful intervention.

The contrast is stark: when programmes are evaluated with promising results, they are expanded; when evaluations are less positive, there are limited consequences.

If that feels too strong, what about simply alerting the schools still using the programme and supporting them to review whether the programme is working as intended in their own school. Remember, for Achievement for All, this is around 1 in 10 of England’s schools. New adopters of ineffective programmes could be discouraged by maintaining a list of ‘not very promising projects’ to mirror the EEF’s ‘promising projects’ tool, though we may need a better name.

These ideas scratch the surface of what is possible, but I think there is a strong case for using both positive and negative findings to shape education policy and practice.

Finally, there is an ethical dimension: is it right to do so little when we have compelling evidence that certain programmes are ineffective?

This post was originally published in the BERA Blog

---

References

Blandford, S. (no date). Education Endowment Foundation Achievement for All: Years 4 and 5 Trial Programme (2016–2018). https://afaeducation.org/news/education-endowment-foundation-achievement-for-all/

Dimova, S., Ilie, S., Brown, E. R., Broeks, M., Culora, A., & Sutherland, A. (2020). The Nuffield early language intervention. London: Education Endowment Foundation. Retrieved from https://educationendowmentfoundation.org.uk/public/files/Nuffield_Early_Language_Intervention.pdf

Education Endowment Foundation [EEF]. (2017). EEF standards for independent evaluation panel members. London. Retrieved from https://educationendowmentfoundation.org.uk/public/files/Evaluation/Setting_up_an_Evaluation/Evaluation_panel_standards.pdf

Education Endowment Foundation [EEF]. (2019). Putting evidence to work: A school’s guide to implementation. London. Retrieved from https://educationendowmentfoundation.org.uk/tools/guidance-reports/a-schools-guide-to-implementation/

Education Endowment Foundation [EEF]. (2020). The EEF’s evaluation of Achievement for All: Answers to key questions for teachers and school leaders. Retrieved from https://educationendowmentfoundation.org.uk/news/achievement-for-all-answers-to-key-questions-for-schools/

Gehlbach, H., & Robinson, C. D. (2018). Mitigating illusory results through preregistration in education. Journal of Research on Educational Effectiveness, 11(2), 296–315. https://doi.org/10.1080/19345747.2017.1387950

Humphrey, N., Squires, G., Choudry, S., Byrne, E., Demkowicz, O., Troncoso, P., & Wo, L. (2020). Achievement for All: Evaluation report. London: Education Endowment Foundation. Retrieved from https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Reports/Achievement_for_All_(final).pdf

Schools are fascinating places and everyone has opinions about them.

Unfortunately, we are often blinkered by our own relatively narrow experience as pupils, parents, teachers or concerned citizens.

To fully engage with discussions – without being overly influenced by individual institutions – we need to calibrate our intuitions. We can do this using the rich data available about English Schools.

Here are five of my go to places to calibrate my intuitions.

1. Statistics at DfE

The Department for Education publishes official statistics on education and children. The upside is they hold large amounts of data; this is also the downside.

Making sense of the DfE’s statistics can be fiendishly difficult, but there are two portals that can help:

• Compare schools allows comparisons of exam, Ofsted and financial data between individual schools or groups of schools.
• Get information about schools allows access to most other basic.
  

These DfE sites are useful, but they are not fun.

2. FFT Education Datalab

FFT Education Datalab produces independent, cutting-edge research on education policy and practice.

They are especially good at making sense of the complex array of data held by the DfE and generating fascinating insights. Their blog is always worth a read. If you have never seen it, stop reading this blog and read theirs.

3. School Dash

SchoolDash helps people and organisations to understand schools through data.

I particularly like their interactive maps and their fascinating analysis of the teacher jobs market. Exploring their various tools is an excellent way to calibrate your intuitions about schools.

4. Watchsted

Ofsted visit lots of schools and are potentially a very rich data source to understand the nation’s schools (putting concerns about validity to one side).

Watchsted aims to help spot patterns in the data. Personally, I like looking at the word clouds that you can create.

5. Teacher Tapp

Everyday, Teacher Tapp asks thousands of teachers three questions via an app. These range from the mundane to the bizarre. But – together – they generate fascinating insights into the reality of schools. They also provide a regular, light touch form of professional development.

For me, two features stand out. First, how quickly Teacher Tapp can respond to pressing issues. Their insights during COVID-19 have been remarkable.

Second, is the potential to understand trends over time. Their weekly blog often picks up these trends and no doubt these will become even more fascinating over time.

How do you calibrate your intuitions about schools?

I am a big fan of evidence in education – I think it has unique potential to improve education in a systematic and scalable manner.

However, I also regularly see evidence misused in education, despite good intentions. Using evidence well is not straightforward. Often teachers have a sense of how evidence has been missed but cannot quite put their finger on the issue.

As a biologist, I like taxonomies. I think a system to classify the different ways that evidence is misused could help us recognise and avoid where evidence is misused. To my knowledge, no such system exists. A comparable tool is the detailed ‘catalogue of bias’ that is available for medical researchers.

Here are four issues that I commonly encounter.

1. Starting with a solution

Evidence is used to justify a decision that has already been made. I often see this when school leaders feel under pressure to justify their choices and show rapid progress.

Perhaps the most common version of this is with digital technology where the technology itself is seen as the solution. I cannot do better than Sir Kevan Collins who asks:

If digital technology is the answer, what is the question?

Sir Kevan Collins

The problem with starting with a solution, is that we simply lose sight of what we are trying to achieve and waste precious time. More broadly, I worry that this misuse of evidence undermines the wider evidence-informed movement in education.

2. Ignoring context

Context is crucial in education, but it can be too easy to overlook it. Here, I think an extreme example is instructive.

The Piso Ferme programme involved replacing the dirt floors of homes in rural Mexican villages with concrete floors. Piso Ferme delivered striking health and education benefits and we can be confident in these findings due to the rigour of the research.

So, should we load up the cement mixer and drive around villages in England?

No.

Piso Ferme worked by reducing infections spread through the dirt floors, such as worms that can cause a range of diseases. While Piso Ferme is an ‘evidence-based’ programme that ‘worked’, it is not suitable for the UK because the problem it is designed to address simply does not exist.

Beyond thinking about if the mechanism works, it is also important to consider the fit and feasibility. For instance, the alignment of values and norms of the approach with the intended recipients. An interesting lesson about this comes from the failure of the Intensive Nutrition Programme in Bangladesh described by Jeremy Hardie. Failure to understand the nature of family structures meant that an ‘effective’ programme failed because it was not adapted to the local context.

3. Cherry picking evidence

Perhaps the most common issue is selectively picking evidence. This often happens in combination with issue number one to justify a decision that has already been made.

Cherry picking is particularly problematic as there is usually some research to support any idea in education – especially if you are not picky about the quality.

Even if you do focus on high quality evidence – different studies reach different conclusions and unless you look at all the relevant evidence, you can be led astray. We can see this by looking at the impact of the studies featured in the EEF’s Teaching and Learning Toolkit about mentoring. Although the overall headline is that – on average – mentoring makes limited difference, individual studies differ.

If you want to claim mentoring is rubbish, point to the red study. If you want to convince a school to buy your mentoring programme, showcase the green study. Only by looking at all of the available evidence can we get an honest picture.

In addition to using all of the relevant evidence, we should avoid taking a confirmatory approach – if we actively look for contradictory evidence to our favoured ideas, we are likely to reach better decisions.

4. Using the wrong types of research

‘Hierarchies of research’ have an interesting and controversial history.

While it’s not true that there is a single ‘best’ type of research, it is true that different types of research excel at answering different types of questions.

The most common issue I see, is when causal claims are made from research, which is simply incapable of credibly supporting them. The classic of this genre is the case study that makes sweeping claims about impact.

Personally, I like this table from the Alliance for Useful Evidence, which highlights the importance of matching types of research questions with different types of research.

A picture is worth a thousand words. I’ve been thinking about this while reading an excellent book on data visualisation and it’s got me thinking about my favourite education graphs.*

In no particular order, here are some of my favourites.

Phonics Screening Check

The phonics screening check is intended as a light touch assessment to promote systematic, synthetic phonics teaching and to identify pupils who have not met a basic standard.

This graph depicts the number of pupils who score at each mark from 0 – 40.

What’s striking about this graph is the ‘unexpected’ bump of pupils who just achieve the pass mark of 32. This graph illustrates why you cannot have an assessment that is high stakes for a school and also have teachers administer and mark the assessment.

Month of birth lottery

Everyone in education is aware of various inequities. One that is often overlooked – especially in secondary schools – is the considerably worse outcomes for younger children within a year group.

This simple analysis from FFT Datalab shows the percentage of children achieving the expected standard split by month of birth.

An interesting question is what should be done at a school and pupil level to account for these differences? For instance, primary school headteachers often remark that they have an unusual number of summer born pupils in some year groups – should we adjust league tables to account for this?

At a pupil level, Higgins and Elliot Major have advocated adjusting the test scores of summer born pupils.

Finding schools that misuse ECTs

Training teachers is expensive for the state, our education system and individual teachers.

Therefore, it is concerning to hear anecdotes that some schools seem to churn through Early Career Teachers, which often results in them leaving the profession.

The question is, how do we effectively identify these schools amongst the messy noise of school data?

In a great piece of detective work, Sims and Allen apply funnel plots – often used in meta-analysis – to identify schools warranting further investigation.

Crucially, their analysis accounts for the size of the schools. In total, they identified 122 schools in England that employ and then lose an unusually high number of ECTs. If these schools were like the national average, it would have resulted in 376 additional teachers remaining in the profession.

*If you do not have favourite graphs you’re either in denial or missing out on one of the great joys in life.