I was tickled to be invited to speak.  I gave a presentation on measuring advocacy – an elusive and difficult task.  I’ve posted my presentation slides on the blog, and this week’s Three Things offers up the core of the approach I recommend: focus on inputs, outputs, and outcomes.

1. Inputs - these are the things we put into an advocacy campaign.  Emails sent, tweets tweeted, Facebook posts made, social media interactions, coordinated actions in coalition with other organizations or individuals, etc.  Inputs might be best understood as those things we control – these are the things we do. Inputs are also the easiest things to measure, so they’re among the most commonly measured things.

2. Outputs - these are the immediate results of all of those inputs: Volunteers recruited, petition signatures gathered, email addresses added to the list, Twitter followers, “Likes” on Facebook, emails or calls to legislators, etc.  Outputs are the meat of most advocacy campaigns.  Outputs are the results that are relatively easily measured and easily reported.  Outputs have become the bread and butter of the advocacy world, and are generally among the metrics studiously measured by sophisticated advocacy organizations.

3. Outcomes – these represent the finish line, the win or the loss, and the reason for bothering with all of those inputs and outputs.  Did the legislation pass/fail? Did key decision-makers change their minds? Did the campaign change the world in its intended ways?

As I discussed in my presentation. outputs at the most important level are impossible to measure in ways that trace directly back to any campaign.  Maybe it was all those calls to the Hill that made the difference, or maybe it was a single call to the staffer writing that section of the budget.  The process of policy making  is complex and opaque, and it’s not possible to know exactly what combination of factors led to the outcome.

However, the best advocacy campaigns have a strategy that reflects a solid political theory of change and make decisions about inputs and important outputs with that strategy always at the fore.  It’s critical to measure progress along that theory of change, test it for validity, review, decide, and repeat.  Without taking measurement beyond easily quantifiable input and outputs to frustrating difficult to quantify outputs, decisions about investments of resources – time and dollars – into advocacy activities are impossible to make in an informed way.

So, be honest: if you ponder the metrics you reported on your last advocacy campaign, how many were inputs, how many were outputs, and did you get around to taking a good hard look at outcomes?  Can you make a strong case that your advocacy efforts are connected to making the change you’re fighting for?

Let us know (and, as always, we’re here to help you figure it out if you need us).

-Shayna

One line in particular struck me:

“In general, post-disaster studies ten to measure “throughput indicators” like how much food was distributed, or how much shelter got provided, instead of “output or outcome metrics” like lives saved or suffering alleviated.”

It’s another situation where measuring the wrong things has led very good people doing incredibly good work down wrong paths… paths that ultimately veer off from their goals.

In the communications space, the equivalent is measuring email messages sent (instead of actions inspired by those email messages), Facebook “likes,” or even classic clip books that measure the number of mentions.

Are you measuring what you mean to?  Are your metrics helping you make decisions that take you closer to your goals?

This week we bring you another guest 3 Things.  Meet veteran online campaigner and analytics fanatic, Shabbir Safdar of truthypr.com:

Measuring social media is very big right now.   The ways people suggest they’re measuring social media don’t always stand up to what people typically think of when they think of measurement elsewhere on the web.  If you’re trying to measure social media, or about to embark on a process to do so, it would behoove you to stop and consider the technique you’re applying before you get your answers.  Let’s examine the three ways to measure social media.

Approach #1: Measure the direct benefits with your analytics package

Measurement Technique: In almost any analytics package, it’s trivially easy to see if someone who clicked on a link from twitter or Facebook came to your website and gave money, bought product, etc.  This is the most direct form of social media measurement.

Pros: It’s really easy to measure with almost any analytics package.  People across the organization, even those who don’t understand social media, will understand it if you simply compare it to any other venue.  For example “You know how we got 25 donations the last time Brad Stone covered our cause in the New York Times Technology section?  Well we have a fan page on Facebook and when we unveiled our new product there, we got 25 donations.”

Cons: Odds are good that your facebook and twitter work isn’t generating numbers comparable to the work you put into the venue.  By presenting the straight up results to the uninformed, you may be afraid of giving your social media efforts a bad grade, and someone who wants to cut your budget the ammunition to say “social media is a failure!”

Product vendors at play here: Google Analytics, Omniture SiteCatalyst, WebTrends, etc.

Approach #2: Attempt to assign the overall benefits of the site to people who have interacted with your social media

Measurement Technique: Using a little advanced and/or sneaky technology, you can figure out if visitors to your site who convert in some way (donations, purchases, etc) have visited your Facebook fan page or clicked on one of your tweets.  Then,, when they do convert, even if they didn’t click right from Facebook or Twitter, you still can attempt to ascribe credit for that donation/sale to social media.

Pros: This can give you some expanded justification for your social media presence and may find many people that validate you by looking at your social media presence, but then go type in your website address and give, which would be otherwise untrackable.

Cons: There are two big ones.  First, you’re not going to pull this off without significant in-house or vendor technical assistance.  Second, just because someone looked at your Facebook or Twitter presence doesn’t mean that’s where the credit should lie for the donation or sale you got.

This methodology can only demonstrate correlation, not causation.  Critics will say “Sure you got a donation from someone who looked at your twitter messages, but how do you know the motivating factor wasn’t a piece of direct mail we also sent that drove them online to give?  Or perhaps a friend mentioned that they had just donated to us, why aren’t you measuring that?”
And they’ll have a point.  Since you’re not measuring the entire media intake of your donors, you’ll never really know what motivated them to give.

Product vendors at play here: Tealium + analytics vendors above

Approach #3: Skip actual results, and only measure soft numbers

Measurement Technique: You could just throw your hands in the air and say “Oh to hell with it, I’m just going to measure how many visits I get from twitter and Facebook and how many followers and fans I have”.  This is typically the most common approach used today, and frankly it drives me bonkers.   Non-monetized followers aren’t worth anything on their own, though it’s good for your ego.  (Monetized followers, as demonstrated in the Theory of 1,000 True Fans, however, is worth a good living.)

Pros: Making improvements in this area is very easy, just ask Ashton Kutcher.  Anyone, with time on their hands, can amass followers and fans.

Cons: You’ve actually not accomplished anything that serves the bottom line of the organization yet, and until then, you’ll always be at risk of someone cutting your social media budget or asking you to do more work in addition to social media work, since you haven’t demonstrated that twitter or Facebook has value yet.

Product vendors at play here: EVERYBODY.  Seriously, who doesn’t have a product that measures social media?  However most of them don’t measure much of anything that matters.

What’s your recommendation?
Since I don’t really enjoy being a curmudgeon let me be clear and say that I think that the answer is to treat your social media work like your email marketing funnel.  And by that, I mean a funnel that looks like this:

•    Top of funnel: number of followers/fans
•    Middle of funnel: number of visits you generate to conversion (sale/donation) pages
•    Bottom of funnel: number of conversions (sales/donations)

I can tell you, without even looking at your data, that your start-to-finish conversion rate is probably weaker than any other channel you’ve got (email or website visits).  That being said, this is still a young medium and experimentation here merely costs you content.  It isn’t as if you have to give the US Postal Service $2,000 every time you want to try another appeal, just craft it and post it online.

What you will need to pull this off is story telling and writing talents.  If you don’t have those in your organization right now, you need to get them right away. Social Media is a conversational media, and if you’re still using the old, broadcast style of communicating your organization’s priorities, you’re going to get ignored pretty quick.

(Note: this was originally posted on truthypr.com.  We loved it so much we begged to to cross post it here.)

Michael was on the ground for the climate movement’s efforts in Copenhagen a couple of months ago, and came away with an impression of a policy making process largely immune to the outputs of the movement’s work.  He proposes a refocusing, noting, “it is incumbent upon all of us to avoid getting so caught up in the art and craft of our online engagement and online campaigning work that we’re blinded to the reality of the people we’re trying to influence — or the landscape in which we’re operating.”  It’s a brave and insightful post and I hope you’ll read the whole thing.

In support of refocusing, we’re offering up a trio of ideas that we think can guide the way toward more effective more world-changing work, online and offline.

1. Measure for outcomes, not just inputs and outputs. It sounds obvious, but as a sector we focus relentlessly on inputs (how many emails did we send, how often did we tweet, and what’s on our campaign website?) and outputs (how many calls to legislative offices did we generate, how many people visited our website,  and how many of them signed up for our email list?), but very little on outcomes.  As we noted in a December Three Things on measuring outcomes, we tend to focus on inputs and outputs because it’s incredibly difficult to figure out how to measure outcomes in a complex environment with many moving parts and murky causality.  Difficult as it is, it’s a vital next step.

2. Remember that organizations don’t have friends. The buzzwords in online organizing are “community” and “engagement.”  Communities are great and cultivating and engaging them is a time-tested touchpoint of grassroots organizing, but mission-driven organizations need donors, activists, volunteers, grasstops leaders and sound connections within well-thought power maps.  Engagement for engagement’s sake can lead us to generate a ton of outputs (ie. Facebook fans and Twitter followers) unconnected to meaningful outcomes. We wrote about this before, too, here and here.

3. Become a power mapping pro. Most, if not all, of the policy change work we do depends ultimately on the decisions of a very few people, be they members of Congress on key committees, policy makers in the US administration, business leaders, high profile members of the media or a city council member or two.  Online campaign strategists should bring to the table a commitment to understanding what really influences those people, and endeavor always to keep investments and activities narrowly focused on those pathways of influence.  Sometimes mass grassroots actions are the right way to go, other times they’re not.  For a terrific example of power mapping in practice, check out this write up of the successful campaign to get Lou Dobbs off of CNN, in part using Facebook ads mictrotargeted to CNN/Time-Warner employees.

Clearly I’m missing some things in this list – and it might be the wrong list altogether, I’d love to know what you think! – but if we can get more focused on outcomes, on relationships that move our mission, and on focusing our efforts only where they can have an impact on decision-makers, I think we’ll see more of the change we’re working for.

2009 was all about finding and using new tools to identify numbers to attach to communications activities. The sector grew more sophisticated in terms of thinking about numbers to track and tools to track them. From Google Analytics to Facebook Fan Page Insights, information that used to be utterly unavailable or accessible only to tech geeks very comfortable with the programming back end of the inter-tubes is now readily available to most anyone with a login.

2010 will be about figuring out what those numbers mean, and how to apply them to measuring the inputs, outputs, and results that matter for non-profits and advocacy efforts.

Our friend Shabbir Safdar is ahead of the curve. Last week he published, “How are my efforts on facebook contributing to my organization’s overall goals?” While I’m not sure he’d describe it this way, I read it as a salvo on the notion that our various outputs – status updates, wall posts, exhortations to “become a fan” – are necessarily distinct from desired outputs – donations, volunteer hours, petition signatures, etc.

Quoting Shabbir:

While PR measurement experts like KD Paine (one of my heroes) can measure things like “message retention” of your social networking efforts, the easy way to measure the effectiveness of this work on facebook is to track how it hits your bottom line.  When your efforts results in an organization-wide goal (money, volunteers, sponsors) that is easy to understand even by people who don’t understand the Internet, there’s no debate about the benefit or budgeting of Facebook.

You can create a few metrics from a couple of basic data points:
1. the number of fans you have on a given day you make a fan page status post (gathered from your Fan Page Insights);
2. the number of fans that click on the link to your landing page; and
3. the number of conversions these visits from Facebook generate.

From these data points, you can then build and track the following actionable metrics:
1. clickthru rate of your facebook posts;
2. conversion rate of facebook traffic; and
3. conversion rate of all other “organic” traffic.

Once you have these metrics, you should be able to examine them and look to connect improvements or degradations in the results to specific actions. Do all my status updates without a photo perform worse than those with a photo of the land being conserved? Then we should always use a photo!

And he goes on to take on one of the sacred cows of social networking more generally – that engagement is what really matters (we took this cow on awhile back, too: Organizations Don’t Have Friends, and Three Things: Engagement? But I’m Already Married!)

Your first reaction to seeing this might be “Hey, wait, I’ve got lots of people clicking ‘Like’ on my status updates, and ‘Share’ and playing my videos.  Doesn’t that count for something?”

Well, to be concise about it, not really.

Shabbir closes out his post with detailed instructions for applying analytics to measuring results – outputs that matter when it comes to furthering your organization’s mission.

So, our first (and maybe our only) prediction for 2010: more of the communications sector will go the way of Shabbir, away from analytics per se and toward improved measurement. At least, we hope so.  (And we hope you’ll read his whole post, especially if your organization is expending resources on Facebook.)

Elections are won and lost based on a wide variety of factors, many if not most of them impervious to the efforts of individual organizations trying to influence elections. Major social problems – hunger, illiteracy, unaffordable health care, joblessness, etc. – are complex, long term, and decidedly not the responsibility of a single organization’s or coalition’s efforts. Legislating takes a long time and is a messy undertaking under the best of circumstances – advocacy and lobbying can’t change that reality.

So, what should organizations, coalitions, and the sector at large measure in the context of unrelenting complexity, widely dispersed responsibility, and, often, a virtual blackout when it comes to timely, reliable data? We’re working on some larger pieces on this for the coming weeks, but in the meantime, three things to get the ball rolling:

1) Measure progress against an articulated strategy.
If your issue depends on key legislators taking a leadership role and you’ve heard from their staffs that they fear retribution from their constituents, then it makes sense to generate support – and grassroots actions demonstrating that support – in those legislators’ districts. That’s your strategy.

The ultimate measurement is whether you successfully swayed the legislators to take the action you wanted them to take, but that might be a larger ballgame than your at-bats can reasonably deliver. You can measure how well you delivered on your strategy. How many new advocates in the district did you move to action, how many in-district alliances did you build, how many and which grasstops influencers did you bring on board, how many of the legislators’ donors did you bring on board, etc.?

“Was your strategy the right one?” is an important question that’s hard to measure. Knowing with certainty how well you delivered against that strategy is vital, and you can and should be measuring it.

2) Measure capacity building.
Changing the world is a long term endeavor. After you win the current battle, the next one’s probably coming up right behind it and you’re more likely to win that battle if you’re expanding on capacity you’ve already built and tested. Identify the two or three critical capacities you’d like to see extended, and measure how well your efforts build on them. Examples of important capacity questions: how many partners did you coordinate with and what types of coordination happened, how many advocates did you move up the engagement ladder, how many new relationships did you build, how did your structures for communications or accountability work?

3) Measure learning.
Particularly when an organization is taking on something new – new partnerships, new tactics, new tools, new messages – lessons learned are a critical and measurable outcome. When you tested message A versus message B to your email list, which one worked better for which segments? Are there any larger lessons learned from that? Maybe the lesson is just that it’s important to test, because the results were counter to your expectations. Did you bring on new staff to manage elements of a campaign, or assign existing staff new roles? What worked and what didn’t about the arrangement?

It’s easiest to measure inputs: how many emails did we send, how many people signed the petition, etc. We’d never argue that organizations should stop measuring those inputs, but that outputs and outcomes are important, too. In our view, changing the world demands it.

My very first post on this blog in January was a response to a post from Sasha Dichter about separating “program” expenses from “overhead” expenses.  The non-profit world generally looks down on high overhead relative to program expenses.  While program delivery is the key component for fulfilling the organization’s mission., my main point was:

“I think my point is that we need to view all expenses as investments.  You don’t invest in anything unless you at least have a target return in mind.  If your annual report can articluate the return on all of your investments, including overhead, and that number improves over time, then you’re well on your way to making an impressive case for an increase in funding from investors.”

As usual, Dichter is back on topic and says it even better:

“Let us not, as a sector, fall into the trap of listening to critics who say that we should minimize the dollars, effort, brain power, and ingenuity that goes into everything but the “real” work (programs).  In so doing, we risk forgetting that our role is BOTH to find solutions to the persistent problems of inequality and injustice and malnutrition and infant mortality and safe drinking water and AIDS and malaria…AND to figure out how to explain to the world that these problems matter, that we have the tools to solve them, and that if was have the tools to solve them, then we must all act.”

I think a snapshot financial summary that separates overhead from program expenses but does not spell out the return on your overhead investment misses the mark.  And when your data is incomplete, the decisions based on that data is likely to be less than optimal. Good enough -maybe.  But room for improvement – definitely.

So the question for the week: what’s the return your getting on your marketing, fundraising, communications and administrative expenses (aka investments)?  How do you know?  Can you demonstrate it?

At Englin Consulting, we’re lucky. We work with extraordinarily smart and talented people who bring those assets to bear to make the world a better place. We love our clients and the work we get to do with them.

One thing about working with extraordinarily smart people: we’re often helping them and their teams work through “analysis paralysis” – thinking and talking something through to the extent that it prevents actually doing much of anything. We’ve learned a trick or two (or three, presented below) that seem to help. So, without further thought or conversation, this week’s Three Things: working past analysis paralysis.

1) Clearly articulate the consequences of inaction. Spend a few more minutes talking, but not about what you need to do. Talk for a bit about what happens if you’re unable to push through and take action. What if you don’t launch that new campaign? What if you don’t revise the website content? What if you don’t head out with clear new messages in your press releases? Set a specific period of time to walk through the consequences of analysis paralysis, voice the fears driving the analysis. Then post the notes somewhere in the room/on the wiki/in an email. Refer back to it when you’re unable to make a decision, and ask – are the consequences of getting this wrong worse than the consequences of inaction?

2) Commit to testing. Analysis paralysis is often fear of the known – what will happen if we do whatever we’re contemplating? What if it’s irrevocably bad? Acknowledge that bad is a possibility, commit to testing the possibility, and stay open to having to change course. We live in crazy fast-paced times and the odds are good that even if you make a perfect decision now, you’ll have to revisit it before you know it. Approach decisions about action with a commitment to test first and decide based on results. Then action isn’t a result of guesses or even good instincts; it’s a result of data.

3) Be comfortable with failure. No matter how well analyzed, tested, and discussed, sometimes things fail. Clay Shirky, one of the gurus of modern organizing and communication, has famously suggested folks should “Fail informatively – Fail like crazy.” Have contingencies, be prepared to get up, dust off, learn from the experience, and go again. Part of this comfort is nailing down the consequences of failure. What if the new PR and messaging push doesn’t change the conversation in the media? What if it irritates our friends, even though they said it wouldn’t? What if nobody pays attention to our foray into social media and we’re friendless? What then? Sketch out the consequences, internal and external, so the possibilities are neither as scary nor as daunting should they come to pass.

Changing the world is serious business, and we love that our clients take it so seriously. We love that they analyze and discuss and make informed, well-thought decisions. We also love it when they act – and more often than not, they succeed. We hope these three ideas can help you move your team from analysis to action.

Thinking about evaluation inevitably leads to thinking about measurement: what can we measure that can form the basis for evaluating our efforts?

Ideally, we’d evaluate the impact of our efforts. Did our work change minds? Did it change policy? Did it raise money or volunteer hours? Did it feed hungry people (or improve delivery on whatever the core mission of your organization might be)?

Unfortunately, in the communications space what’s easy to measure – website traffic, email opens, number of people at a rally – isn’t always directly linked to impact. What we can easily measure often amounts to inputs, as opposed to outputs, or activities rather than outcomes.

We’ve worked quite a bit with our clients this summer on identifying new and better measurements to inform more meaningful evaluation. While we won’t pretend to have solved the problem, we offer up here Three Things to consider:

1) Identify critical process points that have a direct effect on outcomes. We work with one organization that is investing heavily in its grassroots advocacy capacity via its affiliates, and is trying to work out meaningful measures to determine their progress. One of their key challenges is getting all of the affiliates in alignment on national advocacy priorities and messages. The staff spends signficant time every week chasing down advocacy messages that are off-topic or tangential to the organization’s primary mission. Clearly, this organization can and should measure the number of advocacy actions emanating from their affiliates and how many of those are in alignment with national priorities. That measurement alone ignores critical information that would enable them to improve: what processes, incentives, protocols, etc. can they institute to ensure their affiliates are aware of and bought into the national advocacy plan? Adding measurement criteria to points in the process – perhaps participation in a weekly conference call, responses to staff outreach, or compliance with reporting mechanisms – will help the organization pinpoint where the internal failures are that ultimately lead to disappointing outcomes, and work on fixing them.

2) Find indirect indicators. I was in a recent meeting when someone brought up the example of the Cleveland Orchestra as described in Jim Collins’ 2005 Good to Great and the Social Sectors monograph. According to Collins, the orchestra measured not just ticket sales and donations, but the number of standing ovations. Members of the orchestra were instructed to play for ovations, and as the number and length of ovations increased, so too did ticket sales. It’s brilliant: the measurement speaks to the values and motivations of the staff executing on the mission – in the orchestra case, musicians delivering fantastic performances – in ways that indirectly but undoubtedly have an impact on the more obvious, easily measured outcomes. Corollaries in the non-profit and advocacy worlds require some creativity, but I think it’s an idea worth exploring. Anecdotally, what kinds of things happen when your organization performs it’s mission with excellence? What’s your organization’s version of a standing ovation? Talk to the people on the “front lines” of delivering on your mission and find out what they see when when do their jobs perfectly, then cull their responses for measurable ideas.

3) Evaluate in a vacuum. While it’s counter-intuitive, don’t always evaluate in the context of the people and politics of your organization. If nobody’s job depended on it, would you call the number of advocacy email messages you send out a “performance measure”? I’d bet not; the performance measure is, at the very least, the number of email recipients that took the action requested of them in those emails. Similarly, do you care about the number of website hits, or do you really want to know what people are doing when they land on your website, and how those actions contribute to your organizational mission? Every so often, review what you’re measuring and evaluating without context: consider it as if you were considering another organization’s reporting. Does it still look like the right approach?

There’s much more work to be done on measurement and evaluation, and we’re lucky to have our clients as fantastic partners in wading through that work. We’d love to hear from you – what have you done on measurement and evaluation that breaks fertile new ground?

Go back to the Three Things main page.