Ian Burbidge

There are three scales at which we need to evaluate performance, gain insights and track the overall impact of our work, especially for any organisation that is seeking to create social impact and change the system it is embedded within. First, and the one over which we have most control, is at the scale of the organisation, where measures provide assurances on its overall health. Next are the programmes of work that the organisation delivers in service to its mission, where measures can provide assurances on its quality and effectiveness. Finally, and the scale over which we have least, if any control, is that of the system we are operating within. Here measures can track what is changing within the system, but can’t attribute causality to our work to create change within it. Let’s explore these in a bit more detail.

Organisational performance measures include overall metrics of health and direction as well as team and individual objectives that ideally clearly align with the strategy. The resulting set of measures seek to answer the question: how well is the organisation operating in pursuit of its objectives and mission? It includes measures of financial health (income generation, spend against budget, the extent to which we are a going concern, etc), performance (progress against objectives) and operational health (risk, staff turnover, etc). 

A well run and functioning organisation is a prerequisite before we can deliver our value to the world through our work programmes. Just ask any fans of American Football. It’s not enough to have great talent – poorly-run teams never make it to the Super Bowl. As the then Station Commander at RAF Marham once said to me in the context of back-office budget cuts, you can’t trim the fletchings off the end of an arrow and still expect it to hit the target. You need solid foundations to perform at your best. Just as a well-maintained arrow is more likely to hit the target,  a well-functioning organisation enables it to deliver value to the world. 

The next scale is that of the work we do to impact the world. We need measures to track the efficacy with which we deliver our programmes and projects and create value. These measures are not concerned with the outcomes of the work but rather how well we did it. This might sound counter-cultural when we are so used to focusing on achieving goals and equating that with competence and success. In ordered systems where we can discern cause-and-effect relationships this works. But for most of us working to address social challenges it is more likely we are in a complex system and here we must focus on process measures. Evaluating the quality of our processes enables us to distinguish between how well we did something (process) from what changed as a result (outcome). If we fail to see this distinction we can fall into two evaluation traps. 

One is that we judge the success of our work solely by its outcome. If we achieved a good outcome we assume it was because we had coherent processes, innovative work and great people. We also assume that a poor outcome was the result of poor processes, work and people; depending on the scale and profile of the work we will take action on that part we think we have most leverage over, and that’s usually the people. We send them on training programmes, find them a mentor, move them to less ‘risky’ work, or in the worst case of failure we’ll fire them. 

The other trap is to assume direct causality, that it was solely because of what we did that the change happened. Yet there are so many other variables at play that in a complex system we can’t predict with any certainty the impact of our interventions. Doesn’t it instinctively feel wrong, for example, to hold a local charity accountable for obesity levels in a town even if its focus is delivering healthy eating and exercise programmes? The only thing we can hold them accountable for is the quality of their programmes: how many they run, how well they run them and whether anyone is better off as a result^⁠1. That’s it. Their work addresses only a small part of the wide ranging set of issues (health, culture, family, environment, etc) that influence obesity levels in a population and so to hold them to account at this whole population level is clearly misguided. 

These traps may sound counter-intuitive but our social conditioning doesn’t equip us to make sense of complex systems and emergent outcomes. In industrialised, western capitalist societies that preach hierarchy, competition, linear relationships and value extraction we are schooled in cause-and-effect logic, what Freud called aetiology. Put more simply: direct causality assumes the quality of the process is directly related to the quality of the outcome. It does not offer the possibility of a great process and poor outcome or a poor process and a great outcome. 

If we intellectually sever the link between actions and outcomes we are only left with the process over which we have any control, and so we must focus on the process of the work and not the results. Ben Stokes, the England cricket captain understands this implicitly, as he said^⁠2 after the latest and most unlikely of the team’s recent wins: “the only thing I asked of the team before the series started was can we stay committed to that process without being emotionally attached to the outcome, because that’s why we’ve been successful”. In other words, focusing on the process frees you up to perform at your best without worrying about whether you’ll win or lose, which makes it more likely you’ll perform at your best and paradoxically increase the odds you’ll win.

The crucial question, therefore, is whether we have captured the learning so that we can share it and ensure we don’t make mistakes twice, can update and improve our innovation and programming processes. Returning to the obesity example we must capture what we are learning about what does – and doesn’t – work through the delivery of the programmes. This requires the charity, those participating in its programmes, and donors/funders to collaborate. If only more funders and governments understood this and invested in the process and learning what it takes, rather than seeking to buy outcomes. 

So now we’ve taken a deep dive into measuring impact at the scale of the organisation and the work programmes that the organisation delivers. But how do these link to the system we seek to change? I’ll explore the thorny issue of how we might evaluate systems change in the next article. 

¹ Drawn from Results Based Accountability – see Mark Friedman’s Trying hard is not enough.

² Test Match Special podcast, 28th January 2024 at 25:21 https://podcasts.apple.com/gb/podcast/test-match-special/id205892240?i=1000643261015