Measure Better
Thinking about evaluation inevitably leads to thinking about measurement: what can we measure that can form the basis for evaluating our efforts?
Ideally, we’d evaluate the impact of our efforts. Did our work change minds? Did it change policy? Did it raise money or volunteer hours? Did it feed hungry people (or improve delivery on whatever the core mission of your organization might be)?
Unfortunately, in the communications space what’s easy to measure – website traffic, email opens, number of people at a rally – isn’t always directly linked to impact. What we can easily measure often amounts to inputs, as opposed to outputs, or activities rather than outcomes.
We’ve worked quite a bit with our clients this summer on identifying new and better measurements to inform more meaningful evaluation. While we won’t pretend to have solved the problem, we offer up here Three Things to consider:
1) Identify critical process points that have a direct effect on outcomes. We work with one organization that is investing heavily in its grassroots advocacy capacity via its affiliates, and is trying to work out meaningful measures to determine their progress. One of their key challenges is getting all of the affiliates in alignment on national advocacy priorities and messages. The staff spends signficant time every week chasing down advocacy messages that are off-topic or tangential to the organization’s primary mission. Clearly, this organization can and should measure the number of advocacy actions emanating from their affiliates and how many of those are in alignment with national priorities. That measurement alone ignores critical information that would enable them to improve: what processes, incentives, protocols, etc. can they institute to ensure their affiliates are aware of and bought into the national advocacy plan? Adding measurement criteria to points in the process – perhaps participation in a weekly conference call, responses to staff outreach, or compliance with reporting mechanisms – will help the organization pinpoint where the internal failures are that ultimately lead to disappointing outcomes, and work on fixing them.
2) Find indirect indicators. I was in a recent meeting when someone brought up the example of the Cleveland Orchestra as described in Jim Collins’ 2005 Good to Great and the Social Sectors monograph. According to Collins, the orchestra measured not just ticket sales and donations, but the number of standing ovations. Members of the orchestra were instructed to play for ovations, and as the number and length of ovations increased, so too did ticket sales. It’s brilliant: the measurement speaks to the values and motivations of the staff executing on the mission – in the orchestra case, musicians delivering fantastic performances – in ways that indirectly but undoubtedly have an impact on the more obvious, easily measured outcomes. Corollaries in the non-profit and advocacy worlds require some creativity, but I think it’s an idea worth exploring. Anecdotally, what kinds of things happen when your organization performs it’s mission with excellence? What’s your organization’s version of a standing ovation? Talk to the people on the “front lines” of delivering on your mission and find out what they see when when do their jobs perfectly, then cull their responses for measurable ideas.
3) Evaluate in a vacuum. While it’s counter-intuitive, don’t always evaluate in the context of the people and politics of your organization. If nobody’s job depended on it, would you call the number of advocacy email messages you send out a “performance measure”? I’d bet not; the performance measure is, at the very least, the number of email recipients that took the action requested of them in those emails. Similarly, do you care about the number of website hits, or do you really want to know what people are doing when they land on your website, and how those actions contribute to your organizational mission? Every so often, review what you’re measuring and evaluating without context: consider it as if you were considering another organization’s reporting. Does it still look like the right approach?
There’s much more work to be done on measurement and evaluation, and we’re lucky to have our clients as fantastic partners in wading through that work. We’d love to hear from you – what have you done on measurement and evaluation that breaks fertile new ground?