The ICM programme was complex, ambitious and long-term – how did we assess its success?
One of the main aims of the Inspiring Change Manchester (ICM) programme over the last eight years has been to change the systems that surround people experiencing multiple disadvantage – these systems include elements ranging from policies, practices and resources, to ways of thinking, relationships and power dynamics.[i]
Assessing the success of this aim was one of the most complex evaluations I’ve worked on, over almost 20 years of conducting social research in this field. I have had to step back and consider multiple questions about how to capture and measure success in a systems change programme.
Some issues in systems change evaluation
Some important issues in evaluating change interventions within complex systems are:
– The system surrounding people experiencing multiple disadvantage is complex, large and formed of multiple ever-changing, interacting sub-systems.
– Identifying the causes of change, and the effects of actions, is not as straightforward as in many other interventions. Systems change happens collaboratively and is non-linear. Traditional evaluation methods that compare quantitative ‘before and after’ data or use control groups are not the most appropriate.
– The intended outcomes of systems change programmes are often long-term and beyond the scope of time-bound evaluations.
In partnership with ICM, I designed a qualitative systems review, based on interviews with 40 people from across the system, including people with lived experience of multiple disadvantage, those in strategic and operational roles, and ICM team members. Systems mapping conducted for ICM by the Centre for Local Economic Strategies (CLES) in 2015 provided a baseline that I could compare the current system with. I combined a broad look at the system-wide changes influenced by ICM, with a more detailed look at three case studies of change. Through the case studies – which focused on the Women’s Voices Movement, multi-agency data system GMTHINK, and the ICM partnership – I aimed to capture the detail necessary for understanding and learning.
Responding to complex questions
Here are two questions I considered when designing the evaluation, and the ways in which I tried to respond to them:
How far can we attribute any changes to ICM?
In a complex system with many players, it is crucial to take account of the influence of others on the aims of the programme being evaluated. I asked interviewees to consider what they thought had influenced any changes in the system they were describing, how far they thought ICM was responsible for these, and how far other factors were responsible.
This gave a picture of a complex inter-relationship between ICM and the rest of the system. In many areas of change, ICM was part of a broader local and national movement of change; it both influenced, strengthened and supported this movement, and was in turn influenced, strengthened and supported by it. This included changes in relation to co-production, trauma-informed, gender-informed and person-centred practice.
How do we know that systems change is sustainable?
Achieving change that was sustainable beyond the programme’s own existence was important to ICM. A key risk for a systems change programme like ICM is that it fills gaps in the system itself. When the intervention ends, the system can return to its old state and may even be weakened.
The systems change literature says that systems change is more likely to be sustained:
– When it is incorporated in policy.
– When there is a change in attitudes/ways of thinking.
– When change takes place at multiple levels within the system.
– When there are ‘feedback mechanisms’ by which the system continues to change in response to changing context, learning and evidence.
I used these criteria to assess how likely it was that the changes influenced by ICM would be sustained – and to make recommendations about what needs to happen to ensure these positive changes would last.
One of these recommendations was about the importance of building feedback mechanisms into the system, so that the system can continue to adapt and improve in response to new information about how the system is performing. Two key feedback mechanisms are, first, ensuring the voices of people experiencing multiple disadvantage are heard by and influence decision-makers; and, second, funding models which prioritise learning and innovation. Strengthening both of these elements will help to ensure that Manchester sustains and builds on the innovative, forward-thinking changes that ICM and others have created.
The full evaluation report and summary report give a more detailed picture of the changes that ICM has influenced.
[i] The research has drawn on the conditions of systems change identified in Kania, J., Kramer, M., Senge, P. (2018) The Water of Systems Change, FSG.
