years of IMPACT (4)-1

As we embarked upon this research project, we were interested in learning more about our client's perceptions of impact and what this might mean for our work going forward. Through experimentation, critical reflection, and years of client feedback, we've learned a lot about what it takes to support and enable leaders and organisations through complexity and change (and what can often get in the way). 

How do you measure the impact of our work?

Our work exists within a complex, interconnected, and unpredictable reality; therefore, cause and effect are impossible to perceive, making measurement in a traditional sense tricky.

We think it’s best to focus on longer-term impact, not shorter-term outcome, as it allows us to consider the broader response and shifts rather than just immediate, short-term changes.

We do this by asking clean questions, collecting stories, looking at themes, and using abductive reasoning to infer the most likely hypothesis or insight from the data. 

Risks of traditional forms of measurement

Traditionally, consultancies (including us) measure the impact of their work through immediate, quantifiable feedback mechanisms, such as post-programme surveys or Net Promoter Scores (NPS). These tools provide a quick assessment of client satisfaction and an indication of the perceived value of the work. Yet, they often emphasise short-term outcomes and recency bias and overlook the complexity of change.

While it's nice to report back to a client that participants rated a programme an average of 9/10 and share some glowing quotes, how the programme shifts the quality of leadership going forward is hard to deduce. If we only focus on how clients feel immediately after an engagement, we risk missing the broader and more profound shifts (or lack thereof) that unfold over time.

The survey design and outreach process

As part of this 10-year impact research project, we were keen to design a survey for all previous and existing clients. We wanted to ask clean questions to encourage people to share honest reflections without direction toward any hypotheses we held. 

We used a Most Significant Change method that asked about their perceptions of individual and team / organisational shifts, with additional questions for those closer to the work's account management, design and delivery.

We began by analysing our past and current clients, looking at the type of work we were engaged to support, the projects we delivered, the levels in the organisation we worked with, and the period of time we worked with them. 

We sent the survey to 355 people across 25 clients and received 78 responses—a response rate of just over 21%.  This included a process of reaching out to people via Linkedin as many had since left the organisations we had worked with them at. 

 

Acknowledging bias and assumptions

Of course, one of the major flaws of this process is that the people most likely to respond and give their time to this survey are those who have a positive regard for RISE and the work we did together. We want to acknowledge this and note that we may have missed those with less favourable or ambivalent experiences, yet to attempt to mitigate this, we made sure to ask what could have been more impactful and what didn't work so well.

We had a process of reminding people over a period of a month and we also incentivised people to respond with a contribution per response to Ecologi - our carbon footprint reduction partner. We saw the response rate increase the more we reminded people and the closer it got to the deadline.

Analysis and Write-up

We started to assess the data by looking at overall emerging themes across all respondents per question and then went through a process of cross-examining responses based on the seniority of respondents and the type of work they had been involved in. 

We intended to provide some initial analysis for each of the themes and then reflect on how they may interlink and inform each other using abductive reasoning. We also looked at the initial reason for engagement and the perception of actual impact and the links (or not) between these. 

A small project team did the initial analysis with light-touch support from an external moderator. Then, we shared the insights with the wider consultant community for engagement, integration, and learning ahead of publication to our clients.