Natural experiments reveal whether investments in neighbourhoods make a difference, says Richard Harries
Appearing before the Housing, Communities and Local Government Select Committee last month, Michael Gove laid out his four priorities for the new Department for Levelling up, Housing and Communities. These, he said, were to strengthen and improve local leadership and living standards, raise the quality of public services, and “restore and advance pride in place”.
For my money, the fourth item on the list is the most interesting. It comes closest to the spirit of levelling up, but it is also fiendishly difficult to determine. How do you know if local people are proud of their neighbourhoods and towns? And how can you measure whether pride is increasing or decreasing?
And, most importantly in this particular case, when government makes an investment—whether it is through the UK Community Renewal Fund, the Community Ownership Fund or the Levelling Up Fund itself—how will it know if it has made things better?
Complex world
This is not a new problem. Since at least the 1970s, successive governments have tied themselves in knots trying to show that their investments in communities have made a difference.
Scientists developing new medical treatments such as vaccines face a similar problem: how do you show your treatment or intervention has worked, given the complexity of human lives and modern society? In medicine, the gold standard for answering this question is the randomised controlled trial: some patients get the treatment, others a placebo, and their outcomes are compared.
Applied to communities, though, this raises practical and ethical challenges. How can a new library or a renovated park be given to some residents and not others?
In a report published today, Power to Change, the foundation for community businesses, sets out how it has solved this problem. Its approach uses some of the same techniques for analysing data from natural experiments that were awarded this year’s Nobel Prize in Economics.
Power to Change is fuelled by a belief that when local people come together to address an unmet need in a democratic and business-like way, they improve both their own lives and their neighbourhoods. This is a bold claim, reaching way beyond asking whether this or that community business has delivered its funded project and associated key performance indicators.
Neighbourhood watch
The report, Empowering Places? Measuring the impact of community businesses at neighbourhood level, presents results from projects run in Wigan, Bradford, Leicester, Grimsby, Plymouth and Hartlepool. Part of a multimillion-pound investment running from 2017-22, the work shines a light on the difference the programme made between 2018 and 2020.
The crucial aspect of Power to Change’s approach was to ask not only what these six places were like but also what we would reasonably expect them to be like, given their specific characteristics.
To work out these expectations, Power to Change used data from the Community Life Survey. Run by central government, this national survey asks citizens nationwide more than 140 questions about the state of communities.
At the same time as the national survey was in the field, residents in the six neighbourhoods under investigation also received hyper-local booster surveys. These covered the same questions as the national survey, administered in the same way.
The key next step was to construct a separate ‘control’ group for each neighbourhood. This was done by dividing England into 32,844 areas, technically known as lower layer super output areas, and then finding the 10 per cent of these that most closely matched each of the neighbourhoods under investigation.
The matching was based on six axes that together represent more than 40 socio-demographic characteristics, such as deprivation, accommodation type, ethnicity and age.
So, for example, in the 2018 survey, 48 per cent of residents in the Plymouth neighbourhood said it was important for them to be able to influence decisions about the local area. However, the expected figure from the control group with the same socio-demographic characteristics was 55 per cent.
Spot the difference
Power to Change repeated the entire exercise in early 2020. This allowed them to track both the real neighbourhoods and their doppelgängers over time. This showed not only whether things had got better or worse, but also the effect of Power to Change’s investment on the size and direction of any change.
In Plymouth, for example, the proportion of people seeking to influence local decisions had risen to 58 per cent, compared with just 51 per cent in the control group. In other words, Plymouth had gone from being seven percentage points worse than it should have been in 2018 to being seven percentage points better in 2020—a net increase of 14 percentage points over the two years.
The evidence from this approach, called difference-in-difference analysis, is not quite as robust as that from a randomised controlled trial. But it is certainly good enough to test whether community-based interventions really make places better for the people who live there.
It’s a radical new way to evaluate place-based programmes. The results published today are fascinating and Power to Change deserves credit for including everything, warts and all—things didn’t get better everywhere.
If Gove genuinely wants to restore pride to local places, then the novel methodology developed by Power to Change could offer a statistically robust way of measuring the success of the government’s multibillion-pound programme to level up the entire country.
His department should adopt this difference-in-difference analysis as the centrepiece in its measurement, monitoring and reporting. Tony Blair’s Labour party took ‘Things can only get better’ as its campaign theme tune. This government could be the first to show that they really have.
Richard Harries is associate director of the Institute for Community Studies, a research institute powered by the Young Foundation
A version of this article also appeared in Research Fortnight