Measurement

Tuesday, January 12, 2010

If you do your job right, nobody will ask about social media ROI

Listening to David Meerman Scott’s epic rant on ROI, a few thoughts crossed my mind:

• David is able to exercise remarkable self control. There’s little doubt that “freaking” wasn’t the word he wanted to use.
• The ROI issue continues to inspire debate. Some of the 45 comments left to David’s post insisted that ROI calculations are necessary for social media. More to the point, though, Google’s Blogsearch reveals nearly a quarter of a million posts on social media and ROI.
• I’m more convinced than ever that ROI is simply not a relevant metric for social media, although the results of a social media effort certainly need to be measured.

The most important thought to cross my mind, though, was this:

 

If your management or client asks about the ROI from your social media efforts, you’re not doing your job.

It’s a point I wound up articulating—after some discussion with my co-host, Neville Hobson—on yesterday’s episode of FIR. You can listen to the segment below (it runs a little over 10 minutes and inlcudes David’s original rant):

Any communicator worth his or her salt will build measurement into a communication plan. Even the simplest, quickest, down-and-dirtiest communication effort needs to be based on the need to achieve a business objective. You need to know whether—and to what degree—you’ve achieved that objective. So any communication you undertake needs to begin with what the objective is and how you’ll know whether you’ve achieved it.

If you can show those two items to management, nobody will ever wonder about the ROI of your efforts.

The reason measurement must be a fundamental element of any communication is simple. Companies fund communications with the expectation that communications will help drive the business. If you can’t show how your efforts are helping the company achieve its goals, why should the company waste money on it? I’d give a long look at anyone who claims to be a social media consultant who doesn’t bake this kind of measurement into his or her social media schemes.

Measurement is important. ROI is the wrong measure. I explained why back in early November. Still, nothing helps explain a point like a good example. When Dell got started in social media, it was because the company was taking massive hits in online discussions. The business objective was to reduce the amount of negative conversation and increase the amount of positive and neutral discussion. Few executives would argue with the benefits to the business of having people say good things instead of bad things. And, in fact, Dell’s social media efforts resulted in negative sentiment declining from 48% to 23% from August 2006 to mid-2008. (That’s when Dell’s Richard Binhammer and I co-presented a session on social media to Comcast’s communications staff.)

You can’t point to the value of that improvement on a balance sheet. But nobody will ask for ROI if your plan goes like this:

1. We’re being hammered in online discussions. It’s surely affecting purchase decisions. Right now, nearly half of all comments about us are negative.
2. We plan to initiate a variety of social media efforts designed to turn the situation around.
3. We’ll monitor the space to assess whether our effort is having an impact. Our goal is to reduce negative commentary to 20% within two years.

Measurement of communication—any communication—is all about the degree to which you achieved the measurable objective that drove the communication in the first place.

Tuesday, November 10, 2009

What the hell is up with Technorati?

Despite a wealth of channels for searching the blogosphere, Technorati remains important because it’s the only tool that offers a clue about the reputation and influence of each blog. By evaluating a number of factors, including inbound links on a rolling six-month basis, Technorati is able to assign a rank and an authority level for each blog. When identifying bloggers for outreach, these metrics have proven invaluable. You can’t find anything comparable on any other blog search engine, including the popular Google Blogsearch.

But like any tool, Technorati’s usefulness vanishes if it’s broken. And Technorati has been broken in a big way for some time. What’s more, they don’t seem to be very interested in fixing it.

For years, this blog has included two Technorati links. One is a widget that displays the blog’s Technorati authority; the other is a link to a Technorati page that lists blogs that have linked to mine. A while back, Technorati relaunched its site with a new design. Suddenly, the long list of blogs linking to mine disappeared; the page now shows zero blogs linking to this one. (Google Blogsearch shows more than 4,000.)

What’s more, the authority on my widget shows 132, but a visit to my blog’s Technorati page reveals an authority of 525.

My rank is showing up as 3,930—not bad if it’s accurate, but given the other issues, to say I’m skeptical is putting a positive spin on the situation.

And that’s the problem. It’s not about my ego and needing to know that people are blogging about my posts (I get that information from Google’s Blogsearch). It’s about trusting Technorati when looking up information about other bloggers. I have no confidence in Technorati, and therefore can no longer use it in my work. I trust the Oakland Raiders to score multiple passing touchdowns in a single game more than I trust Technorati to give me usable results for any blog I’m researching.

A couple weeks ago, when this situation really started to bug me, I did some searching and found that Technorati has a Get Satisfaction page addressing the problem. The explanation from Technorati deals with a long-standing problem with producing good results and the solution, which involves “work(ing) from a clean data set.” The post continues:

Sites that we have added to the clean index are being crawled, having detailed authority calculated, display recent posts (one at the moment) on our site, and contribute to other sites authority. We are continuously adding more and more sites to this index, and are working on ways to do so faster, but as you can imagine, the volume of sites to qualify is enormous.

An impressive number of comments follow the post, none of them happy. A typical comment reads something like, “My blog has lost all of its 2 years authority, plus it’s not showing recent posts and its screenshot is very old. Please help.”

Technorati isn’t responding to any of the comments (as they did occasionally at the outset) and reports of the problems being fixed are more rare than sightings of wild condors in downtown Manhattan, leading to comments like this one: “I’d suggest that either Technorati or Get Satisfaction are ignoring this thread as there don’t seem to be any recent responses to points raised by bloggers.”

I understand that processes can take time, but for a professional service, the response from Technorati is abysmal. The problem is knowing where else to go for comparable metrics on blogger influence.

It goes without saying: Technorati FAIL.

The ROI label and the credibility of communications

When your CEO, CFO, and other C-Suite executives gather in the board room and talk about ROI, they’re talking about the accounting term. Return on Investment, when raised in the board room, is the amount, expressed as a percentage, that is earned on a company’s total capital, calculated by dividing the total capital into earnings before interest, taxes, or dividends are paid. ROI is synonymous with ROR—Rate of Return. A series of precise mathematical formulae are used to calculate ROI.

Of course, it’s common to talk about return on investment more casually. You invest time, money, and resources in an effort and want to know what you got out of it. It’s easy to refer to the results of your effort as ROI. And I understand that, outside of the C-Suite, ROI has assumed a variety of different meanings, all of which come down to the perceived value of an expense or investment.

I’m a huge believer in measurement as a foundation for any kind of communication effort, whether it’s PR, marketing, whatever. Budget-conscious executives are increasingly less likely to fund communications if they can’t see how the effort is helping the business achieve its goals, so it’s vital to be able to demonstrate the results our efforts have produced. It’s also critical so we can figure out what adjustments we need to make in order to improve the results.

While I do recognize that, in some cases, it’s possible to tease the actual ROI from a communication investment, most of the time it’s a guessing game to determine just how much money was generated in sales from the social media dimension of a company’s communications efforts. Ultimately, of course, any communication should be designed to support business goals, which at some point translates into money in the bank. But it’s usually impossible to determine exactly how much money, to the penny, fell to the company’s bottom line as a result of social efforts to bolster the firm’s reputation or tout a new product.

Thus, every time we go to executive management and claim we’ve achieved ROI using calculations that are out of synch with their understanding of what ROI is, we undermine our own credibility and reinforce the perception that communications people just don’t understand business.

The thing is, I don’t understand why we’re so obsessed with needing to prove ROI. We don’t. What we need to prove is that we have set objectives for our efforts that support business goals and that those efforts produced measurable results. That’s the kind of reporting that earns management respect and support for investment in communication.

It’s also the approach IABC—the International Association of Business Communicators—has been taking for at least as long as I’ve been a member (since 1977). To assess excellence in communication, IABC wants to know what objectives you set and how you can prove you achieved them. Nowhere does the label “ROI” appear in these criteria. Nowhere does it need to.

Consider, for example, the ongoing participation of several Dell staffers—Richard Binhammer and Lionel Menchaca, for instance—on Twitter. Their availability, their insights, their passion certainly have an impact on perceptions of Dell and the company’s reputation. These undoubtedly factor into decisions to buy. But is there a way Dell’s communications team can express the value of these contributions in precise dollars and cents? Clearly not.

On the other hand, Dell does an amazing job of assessing the shift in the sentiment of online discussion from negative into neutral and positive. The correlation between sales and negative sentiment has been long accepted by the most senior executives in almost every organization. Hence, proving that the communication effort has shifted sentiment is accepted as a valuable effort that supports business goals that lead to sales.

What about cost-avoidance, which I’ve always seen as a useful communication measure? Building strong relationships can prevent tthe need to spend money to address strikes, boycotts, costly legislation or regulation, and the like. Again, occupants of the C-Suite recognize this as valuable, but there’s no way to assign it a formal ROI calculation: Money we didn’t spend doesn’t show up anywhere on a P&L.

So I’m not suggesting that we don’t measure the impact of our social media efforts, nor am I suggesting that we can’t prove that there’s value produced for the resources invested. But for the sake of our own credibility, unless we can come up with the numbers that reflect the way management perceives it, we need to stop trying to claim that it’s ROI.

Tuesday, June 02, 2009

GM and social media: damned if you do, damned if you don’t (even if you do)

We hear that most companies still haven’t jumped on the social media bandwagon and we roll our eyes in dismay and maybe even a little contempt. But there are reasons companies resist getting engaged with communities. It can be seriously perilous.

Look at General Motors. The magnitude of the company’s problems have inflamed peoples’ passions; our emotional reactions to its situation—and how GM responds—will forge its reputation for years to come.

In the midst of this classic institutional crisis, GM has committed to engage in social media at virtually every level. Say what you will about other dimensions of General Motors, from labor practices to product innovation to financial management. The companies’ communication efforts have been sincere and wide-ranging:

They were pioneers of the corporate blog. Members of the communications team participate in the auto blog communities. Communication staff have reached out to answer questions and participate in conversations wherever they are found. Employees throughout the organization have been encouraged to talk about the company’s future in conversations they encounter during their day-to-day online activities. The public was invited to join GM leaders in open conversations about controversial issues. They have hosted mommy bloggers and podcasters on a retreat. They’re on Twitter and Facebook and YouTube and Flickr. To record the company’s 100-year history, they created a wiki to which anybody can contribute.

In other words, GM has put into practice the kinds of actions most social media consultants would have advised. Still, even people engaged in the space are oblivious to those efforts, even as they insist upon them. In a comment to Joe Jaffe’s blog (in response to something I wrote), Viveka Weiley wrote, “People are already having these conversations, we don’t need GM to facilitate, centralise and filter them. It’s up to them to join our conversation, not the other way around.”

Exactly what GM has been doing.

Jaffe’s post about which Viveka and I were commenting, by the way, is a savaging of GM over a 60-second spot the company unveiled concurrent with its bankruptcy filing. In the commercial, the company brands the bankruptcy as a turning point and acknowledges that a massive rethinking of the company is required. It ends with the URL for GM Re: Invention, the repository of all things related to GM’s turnaround effort.

The site shows an understanding of the networked world, with…

• Sharing links
• RSS feed
• Links to Twitter accounts of GM designers, engineers, and other front-line employees
• Link to a Facebook fan page where critical comments are a part of the conversation

I saw the video as an invitation to come to the site, one channel for engaging consumers among many. Joe things “Somebody deserves a real hefty bitch-slap” because (among other things) “advertising is not the answer….especially during times where cathartic healing needs to take place via honest…authentic, transparent and open dialogue.”

Which, again, GM has been doing to a degree few other companies—and even fewer outside the technology world—can claim.

Damned if you do and damned if you don’t.

Perhaps the best example of this peril comes courtesy of the Huffington Post, months before the bankruptcy filing. Back in February, Huffington blogger Allison Kilkenny tweeted a message to her followers: allisonkilkenny: sees GM is phasing out the small, fuel efficient Saturn. Oil companies: 1, Earth: 0.

Kilkenny was nonplussed when she got a reply: @allisonkilkenny we don’t have indiv trash cans at ofc cubes at hq, just an ex, not sure total $ saved from small ideas, but likely large

Kilkenny was bewildered. Why would a company needing to focus on its recovery invest in people who respond to Tweets, especially those that weren’t a specific request for help or information? “No one likes that in your rush to modernize and embrace the technology of the internet (complete with Twitter experts,) you forgot how to compete with foreign car companies,” she wrote.

So Kilkenny’s complaint, on the highly-visible Huffington Post, is that GM is doing exactly what Joe Jaffe and Viveka Weiley (and scores of others) say they must do.

Damned if you do and damned if you don’t.

Make no mistake, GM is doing the right thing, even if they’re making mistakes along the way. But knowing the kinds of hits you’ll take from both sides for stepping up as GM has would deter many a CEO from taking the social media plunge.

Measurement is key. If we cannot convince business leaders that the business results of community engagement will outweigh the kinds of risk on display with GM, it’ll be hard to condemn them for their obstinence. I have no doubt that GM’s leaders are getting regular reports on the payoff for their commitment to community engagement. Given the current climate, it’s a good thing they’re in it for the long haul.

Sunday, October 12, 2008

Change is the ultimate measure of online influence

What is influence?

I’ve been mulling over this question since reading Steve Rubel’s post asserting the Google Page Rank is the ultimate measure of online influence. I drr Steve’s point, particularly when comparing Google Page Rank to other metrics that draw on server-based data.

Like Technorati’s authority rankings, your Google Page Rank improves the more people link to you. There’s a nuts-and-bolts problem with this as a measure of influence: You don’t know why people are linking to you. Sure, you hope they’re directing their own readers to what they consider to be high-quality content. In any given case, though, it’s also possible that they’re linking to you while telling their readers, “You won’t believe what this idiot has written now.” I frequently follow links deliberately directing me to examples of bad content.

Another problem with page rank is the ease with which the unscrupulous can game the system. Not too long ago, I started moderating comments to this blog so I could reject Akismet-proof comment spam that includes a link designed to boost a site’s Google Page Rank.

But these technical issues aren’t at the core of my discomfort with Page Rank as a measure of influence. It’s the definition of influence, which has nothing to do with your popularity. Influence happens when you cause something to happen. Page Rank is an outcome of your efforts, the social media equivalent of counting the number of newspapers that pick up your press release. Influence occurs when you produce outcomes, not outputs.

Katie Paine, in her excellent book, “Measuring Public Relationships,” defines outcomes as “quantifiable changes in attitudes, behaviors, or opinions that occur as end results of a PR program.” It’s a definition I agree with. The highest possible Google Page Rank cannot determine whether your site has produced such a quantifiable change. That’s what influence is—the ability to alter someone’s attitudes, behaviors, or opinions.

Measuring your ouputs—along with outtakes (the perceptions or understanding created by your work)—is important, but it’s a communication goal, not a business goal; you measure it to determine how effective your tactics have been at meeting the business goal. Ultimately, companies have business goals in mind when they employ PR.

Unfortunately, the ultimate measure of online influence isn’t accessible from any of the online metrics or analytics available. You can’t plug a URL into a search field and produce the answer. There are three basic ways to assess your influence online:

• Read and analyze what people are saying about you to determine whether attitudes or opinions have changed as a result of your online efforts
• Apply some kind of survey mechanism to ask people whether your content drove some kind of change in the people who consumed it
• If a direct link can be made, measure the impact of your content on the business goal; for example, where the goal is to get people to sign up for an online service, you could show that a blogger outreach effort produced a measurable increase in signups

The difference between Page Rank and these three approaches should be pretty clear when you look at what you report to your client. Which would you rather say?

Option #1: Sixty-seven percent of the people who read your blog were more likely to do business with than they were before they started reading it, and 28% said they’ve already done business with you because of the thought leadership you’ve established on the blog. That’s significant, given that our online efforts have generated a Google Page Rank of 7, which means a lot of people are linking to the blog, dramatically boosting the number of customers and prospective customers.

Option #2: We’ve generated a Google Page Rank of 7. That means a lot of people are linking to the site. Isn’t that awesome?

If we’re not working to achieve our clients’ or employers’ business objectives, there’s no reason for our clients or employers to pay us. If that’s not what we’re measuring, we’re not demonstrating the value of our work. Yes, assess your Page Rank. But for goodness’ sake, don’t stop there.

Thursday, August 28, 2008

This is fresh?

The chatter about Technorati‘s rapidly diminishing credibility continues to intensify. Dan York talked about it on last Thursday’s FIR, prompting Joe Thornley to write about it, which in turn prompted Neville to write about it. Dan revisited the subject on today’s show, leading Neville and me to discuss it further.

After recording, I took a quick look at my own listing on Technorati, and found that the first five posts linking to my blog were posted 14,119 days ago. In case you don’t have your calculator handy, that’s 38-1/2 years ago. Blogging is older than we thought. (See the screen shot at the end of the post.)

Actually, the first item listed was posted July 6 of this year. The fourth was posted August 11. All this despite the fact that Technorati showed the results as “sorted by freshness.”

This is far from the worst problem Technorati is having, and doesn’t even begin to touch on the issues raised by people like Chris Brogan about the validity of Technorati’s methodology. Taken all together, these problems present a real opportunity for someone to step in and provide the kind of metrics Technorati does—and nobody else—or for Technorati to get its act together and reclaim its leadership position.

Sunday, April 20, 2008

FIR Reviews: Measuring Public Relationships - 04/20/08

The issue of measurement arises over and over again in the world of public relations. A new book, Measuring Public Relationships, provides a roadmap for both upfront benchmarking and measurement to assess the impact of organizational communications.

Written by Katie Delahaye Paine, Measuring Public Relationships covers everything from the reasons to measure to the tools, outlines the steps required for measurement of any kind of communication, then applies them to each major constituent audience PR is typically tasked with addressing.

FIR co-host Shel Holtz provides an overview of the book in this review.

Download the review here (MP3, 5.5MB, 12:08 minutes), or sign up for the FIR Book Reviews RSS feed to get it and future reviews automatically. For automatic synchronization with your iPod or other digital player, you’ll also need a such as the free Juice, DopplerRadio or iTunes, or an RSS aggregator that supports podcasts such as FeedDemon. To receive all For Immediate Release podcasts including the twice-weekly Hobson & Holtz Report, sign up for the full RSS feed.

Listen to this podcast now:

If you have comments or questions about this podcast, or suggestions for future interviews, email us at fircomments@gmail.com; or call the Comment Line at +1 206 222 2803 (North America) or +44 20 8133 9844 (Europe); or Skype: fircomments; or comment at Twitter: twitter.com/FIR; or at Jaiku: fir.jaiku.com. You can email your comments, questions and suggestions as MP3 file attachments, if you wish (max. 3 minutes / 5Mb attachment, please!). We’ll be happy to see how we can include your audio contribution in a show.

Podsafe intro music - On A Podcast Instrumental Mix (MP3, 5Mb) by Cruisebox.

Measuring Public Relationships by Katie Delahaye Paine
Publisher: KDPaine & Partners, LLC
Paperback, 204 pages
Published in December 2007
ISBN-13: 978-0978989903
Available online now from Amazon.com, Amazon.co.uk and other outlets.