Key Takeaways:
|
Measuring impact is a vital, but sometimes neglected, element of supporting strong social movements. Evaluation allows us to understand the strengths and weaknesses of our campaigns and plan more strategically for the future. More than that, it can help us to celebrate wins, build knowledge together, and create deeper connections in our teams.
In February, Holly Hammond sat down with three strategists and evaluators and asked them how they go about measuring impact in social movements. Amanda Tattersall, Tony Mohr and Danny Hutley, coming with hefty campaign and impact evaluation experience, were keen to share their insights.
1. Measuring Social Change Impact
Going back to the plan but being flexible
Measuring impact can be a difficult task for any movement evaluator. For Tony Mohr, going back to what the group, organisation or movement was trying to achieve is an important first step. Looking at organisational goals and strategies, and comparing these to real-world changes can help to illustrate movement impact.
“That approach has manifested in an impact report with vignettes, little stories of change, where it’s literally this was the situation before anything happened and then this was the situation after, and here’s our hypothesis about why that change happened. Stressing that it’s still a hypothesis. You’re never going to be able to prove that something changed for a particular reason. But it’s our explanation of what might have led to that.” – Tony Mohr
There are limits to assessing impact against stated plans, as the external context and campaign strategies shift over time, requiring flexibility in evaluation.
“Campaigners use all sorts of tools all the time to pivot in their campaigns, and they don’t call it evaluation, or measurement, but it is! Things like power mapping, scoring politicians, strategy documents where they say this is happening in the external world, therefore we need to pivot to this. That is really key information to consider.” – Danny Hutley
External and internal measurements
For Amanda Tattersall there are two key components to measuring the impact of a social movement:
- Measuring social change: the outward facing impact of the social movement, identifiable via policy wins and a shifting political climate.
- Measuring organisational strength and its capacity for democracy: internal factors such as leadership development and strengthening relationships between organisations.
“You trade off internal capacity building for wins and wins for movement capacity . You can never get all elements at once, they’re in flux. They’re dynamic. There is no perfect form of social change. It’s about understanding how different dynamics are present.” – Amanda Tattersall
Short term and long term wins and losses
Importantly, the campaigners interviewed emphasised that there is an ever-present and necessary tension between short-term and the long-term wins in social movements. Immediate policy wins can be relatively straightforward to measure. If the movement is campaigning for a piece of legislation to be enacted, tracking if and when that legislation passes through parliament can be an obvious measure of success. However, the movement’s impact on broader systems change, or shifts in the political climate, might only become apparent in hindsight, and can be more difficult to measure with any degree of certainty.
“In movements, it’s even more so the case that the structured way of evaluating doesn’t quite fit. Movements have a life of their own, they’re huge. They are not just about a policy win but they shift narratives, they try to change the rules of the system in different ways.” – Danny Hutley
In fact, in many cases, failure in the short term is a prerequisite for success in the long term. In this way, if a movement encounters a failure in a short-term policy aim, that’s not necessarily an indication of movement-wide failure, or a lack of movement strength. Instead, social movement evaluators and researchers need to keep both levels of analysis (immediate wins and broader systems change) in mind, looking at a movement as a long journey towards social change.
“It’s a mistake to just see things in terms of policy wins. I actually think, like Grace Lee Boggs, and Ghandi, there’s this theme in social change scholarship, that talks about: we never actually do win. There is not a moment where you go ‘it’s over’, that’s not the point of social change. The point is to be able to have the capacity to always contest and fight… Of course we want to win, it matters if we get those results, but there’ll always be another fight. If we reify externalised political wins, and take our eyes off how we contest power and work together, then the very process of looking at impact will take away our impact.” – Amanda Tattersall
Evaluating the movement against established social movement frameworks can be helpful, for example, through tools such as Bill Moyer’s Movement Action Plan.
Additionally, Danny Hutley emphasised that we need to think about whose voices we’re elevating when assessing the impact of a social movement. There will always be oppositional voices who want to downplay the impact of the movement on social change, and highlighting and trusting the voices of campaigners is important in movement evaluation.
“It all comes down to who we’re listening to, and who we’re believing, and which voices we’re elevating, in movements.” – Danny Hutley
Holding tensions
As Amanda Tattersall articulated, “impact is an ecosystem”. Social movement evaluation needs to be mindful of different approaches to change and be clear-eyed about their strengths and limitations.
“There’s not one form of change that is effective. That’s a false measure. The measure has to be how an ecosystem of change is created, and how each particular strategy will have strengths and weaknesses. So mobilising is great at scale but bad at leadership development. Organising is great at leadership development, great at relationships across difference, but struggles at scale. Parties are great at broad based agendas, but they can have a terrible culture of demobilisation. All of them have great strengths. All of them have great limits. But the trouble has been in most social movement cultures, a culture of competition, a culture of competitiveness, often induced by funders, that’s led to people trying to pitch themselves as perfect and pitch themselves as better. So we’ve ended up with this skewed false understanding of impact and success.” – Amanda Tattersall
In addition to the tension between systemic (transformational) change, and incremental (transactional) change, the campaigners interviewed also pointed out that in any broad social movement, there will be competing goals between different organisations and actors. For Tony Mohr, tracking complementary versus competing goals can be a valuable measure of social movement impact and an indicator for future movement-wide planning. However, he emphasised that there needed to be some flexibility in this, and a rigid evaluation method might not allow for inherent diversity within the movement.
“It might be useful in a learning sense, noticing where there is difference and being able to think about what that might mean for movement planning in the future.” – Tony Mohr
Staying with the trouble, and holding these tensions in movement-wide evaluation is a challenge for any researcher. However, as Danny Hutley articulated, these tensions and diversity within movements are their strength. Our methods of evaluation therefore need to account for the inherent tensions, complexity and competing aims that can make up a social movement. Trying to box social movements into any one formula won’t demonstrate the rich ecosystem of social change.
Sometimes engaging in measurement can highlight tensions in movements too.
“Around 2015 there was a small backlash against measurement. Some papers came out about vanity metrics, and it was right that this approach was called out. But I actually found in the campaigning sphere, it led to people saying, therefore, it’s impossible to even think about measurement, which was not useful. I’m hoping things move back towards somewhere in the middle, where we’re happy to use these measures, but we keep them at arm’s length. If something is definitely a vanity metric, how can we deepen that metric? How can we add more judgement to that? Can we create a framework instead of a metric?” – Danny Hutley
2. Measuring Movement Health
Holly and Danny discussed the ‘moment of the whirlwind’ as explored by Mark and Paul Engler in This is an Uprising: a dramatic time of movement take-off when awareness of an issue skyrockets and many more people move into action. Rather than measuring movement impact Danny emphasised the value of measuring movement readiness.
“All the factors in terms of the readiness for the whirlwind moment, you can find ways of measuring those. That’s a useful exercise in reflection. Like: How much shared purpose do we have between organisations and informal groups? How strong is the network or the relationships? Are the people here ready to work through conflict? Do they have processes set up to be strategic? To try things and learn from them? And to consider all the external factors?” – Danny Hutley
A social movement’s health and organisational strength is an important indicator of the movement’s capacity to enact change. In addition to measuring external impact or wins, the campaigners interviewed also offered some useful ways of measuring the internal health and strength of a social movement. Some of these included factors such as:
- Network health: shared purpose between organisations and informal groups; strength of networks
- Capacity for holding difference and managing conflict, including differences between theories of change and movement roles
- Training capacity, willingness to learn and try new things
- Ability to consider external factors and develop strategies accordingly
- Overall sustainability of the movement
- Ability to shift narratives over time
There are challenges in evaluating any of these factors. Particularly, movement-wide measurement needs methods that capture smaller groups or submovements, and those that aren’t well resourced for evaluation. For Danny Hutley, understanding how effectively smaller groups and organisations are able to engage with larger organisations can be a powerful measure of movement strength.
3. Approaches to measurement and outputs
“I’m really into letting go of this idea of evaluation as this objective, scientific, unbiased reporting method. Really leaning into trusting campaigners’ judgement and finding different ways of finding nuance and subjective judgement in assessment of campaigning and movement building. If you look at evaluation as a kind of collective sense making, like a discussion, rather than a scientific objective thing, then it really frees you up to think more creatively about methods.” – Danny Hutley
When it comes to picking up their research tools, Amanda, Danny and Tony are keen on methods that work with campaigners and communities, allow for nuance, and avoid abstract claims of objectivity. Danny Hutley described evaluation as a process of collective sense making, where researchers and campaigners work together towards a common analysis. In this way, the campaigners interviewed emphasised the power of qualitative research tools, such as interviews, workshops and focus groups. Amanda Tattersall argued for co-design methods, where knowledge is built as a collective. Tony Mohr was interested in adapting the critical path process. Where critical paths are often used by campaigners to provide a vision of the future they are trying to enact, Tony suggested we could undertake reverse engineering to look back at social change as it happened in a past campaign.
“It’s like the Most Significant Change process but using the terms and the structure that comes with a critical path process might be more relatable for some campaigners… Just going back and saying, ‘Okay, ignore the critical path that we thought we might have. Now that we’re here, what has actually preceded this, to get to this point?’” – Tony Mohr
Some of the useful methods that Amanda, Danny and Tony referenced were:
- Interviews
- Including interviewing people “outside the bubble” of a social movement
- Focus groups
- Careful, intentional co-design workshops
- Working with campaigners and campaigners’ tools (trusting campaigners judgement)
- Power mapping
- Scoring politicians
- Strategy docs
- Asking campaigners: “What has driven the strongest collective action in this case?”
- Reverse engineer a critical path
- Most Significant Change
- Informal conversations with academics
“I think that good evaluation requires qualitative methods. Interviews, but even better than interviews are co-created workshops, where people are building the knowledge together.” – Amanda Tattersall
“I really like focus groups, if they’re facilitated well, and if they’re not called ‘focus groups’. One group might be people from similar backgrounds, and then you might purposefully put in different people, people with lived experience, or other experts. That group dynamic often brings up new information that people wouldn’t have thought before.” – Danny Hutley
What makes a good evaluator?
In addition to these methods, Holly asked the campaigners: what makes a good social change evaluator? Interestingly, a common theme was critical distance. An evaluator needs to understand the movement, and build trust with those involved, but having a certain level of distance from the movement can help to see the bigger picture. There were a few core qualities that came up during our conversations:
- The ability to be surprised
- Expertise in both quantitative and qualitative methods (but with an emphasis on qualitative methods)
- Expertise in collective research methods and co-design
- Someone who understands the movement, but isn’t too close
“I think having someone who wasn’t involved in the campaign can really help. But if they don’t have much knowledge of the campaign, or the people, if they don’t have much experience of the people and organisations involved, it can be hard to build rapport. I have found trust between whoever’s doing the evaluation and whoever is effectively being evaluated is pretty crucial.” – Tony Mohr
“I don’t think a good evaluator is someone who’s separate, there’s no such thing as objective knowledge. We’ve all got to fess up to our positionality and in doing so, I think someone who is familiar with the environment is helpful. You don’t want someone who’s personally invested in it though, because that’s when the perspective starts to narrow… Evaluators need certain attitudinal capacities where they’re prepared to be surprised. They’re not coming in with a preset idea of what’s going on.” – Amanda Tattersall
Valuable research outputs
Finally, Amanda, Tony and Danny emphasised that any research outputs should be made for the social change community. They need to:
- Be designed for best use by the community
- Be easily absorbed, disputed with and argued around
- Include vignettes or stories that demonstrate impact
- Provide useful guidance for future campaigning
“I would love to see more use of evaluations in the planning for the next campaign. So when you sit down and do background research or situational analysis, the evaluation of the preceding campaign gets fed straight into that. That might happen from time to time, but so many evaluation reports get used in the moment and then shelved, forever forgotten.” – Tony Mohr
Conclusion
Some of the biggest takeaways from these conversations were that social movement evaluation needs to be nuanced, allow for the inherent tensions of social change work, and use methods that work with and for campaigners to build knowledge collectively. In this way, the work of evaluators and researchers is not to offer prescriptive analyses of success or failure, or make claims to some kind of external objective truth. Instead, we need to provide the tools and opportunities for people engaged in social movements to come together and build knowledge collectively, in a way that works best for them. This process of collective sensemaking can help changemakers to build stronger, more strategic campaigns. As we continue on with the Movement Monitor project, we’re keen to explore how the tools we use can support these processes of collective sensemaking, and how we might adapt them to better serve the movement.
About the interviewees
Danny Hutley takes a learning approach to planning for social change, measuring impact and sharing what works. He is the Director of Strategy at Nguvu Collective which supports women and nonbinary change leaders in India, Kenya, Nigeria, and South Africa to drive systemic change. Danny was previously the Director of Strategy & Evaluation at the Foundation for Young Australians, Director of Learning Analytics at Change.org, and International Policy & Strategy Officer at the UK Youth Climate Coalition.
Tony Mohr has extensive experience in the environmental and social justice not-for-profit sector, including as Campaigns Director at Australian Democracy Network, Executive Director of the Alliance for Gambling Reform, and Senior Climate Campaigner at Australian Conservation Foundation. As a consultant Tony has worked for CANA on the independent review of the Safeguard Mechanism campaign, and for Boundless.Earth on funding program evaluation.
Amanda Tattersall is a leading thinker and educator in community organising and community-led research methods. She is an Associate Professor at the University of Sydney and the host of the ChangeMakers podcast. She is the founder of the Sydney Alliance, a co-founder of GetUp, the author of Power in Coalition, and co-author of People Power in Cities (forthcoming from Oxford University Press).