The heretic and the number cruncher

Share This!

John Wagner wasn’t too impressed when Katie Paine introduced a new metric she dubbed PR Value Ratio. Wagner, owner and principal of Houston-based Wagner Communications, has posted several subsequent items that elaborate on his position that communication measurement is overrated and will never be a priority. While Wagner finds some measurement valuable, he sums up his dismissal of measurement as a standard PR component on several grounds:

• The tools are no good
• The tools don’t measure the right things
• Clients don’t want to pay for it
• It’s too much work
• Some PR efforts don’t lend themselves to measurement

Wagner summed up his initial salvo with this gem: “When measurement is possible, great. But it will likely always play second fiddle to good old-fashioned intuition.”

Katie responded by suggesting Wagner isn’t the heretic he dubbed himself, but rather a menace and a dinosaur. While Katie echoes the sentiment of a lot of PR pros—measurement is a path to a seat at the management table—Wagner responded:

If public relations professionals are to be credible “at the table,” we have to make sure that the data we provide for our programs can withstand the scrutiny of bottom-line oriented business managers. In my opinion, that’s impossible to do with most measurement tools based on circulation or viewership.

The debate has turned into a he-said she-said exercise (paraphrased below):

He said: The number of eyeballs viewing placed content isn’t adequate because it doesn’t assess whether the communication influenced the target audience.

She said: Well, duh. Have you never heard of opinion research?

He said: Sure, I’ve heard of opinion research. But who’s gonna pay for that?

And so it goes. Ultimately, though, I have to wonder how much investigation Wagner has done into communication research tools. I reported here last month about a tool Procter & Gamble developed to measure the value of its PR efforts, a tool that showed PR generated higher return on investment than other marketing channels in four out of six brands the company’s PR department tested. The instrument, called PREvaluate, “incorporates detailed analysis, including information on cost, scope, audience, geographic markets, and possible synergy with other marketing tactics,” according to Hans Bender, the company’s manager of external relations. A review of Angela Sinickas’ manual on internal communication measurement and Lou Williams’ book on external communication measurement reveals a treasure trove of solid tools. So are the basic PR textbooks, which leads me to my next point:

My biggest objection targets Wagner’s assertion that only some PR efforts lend themselves to measurement. Any PR effort, no matter how large or small, should be based on achieving some kind of outcome. John, are you suggesting that outcomes cannot be measured? Here’s what Cutlip, Center, and Broom write in the widely-used PR 101 textbook, “Effective Public Relations”:

Surely knowledge, outcomes, predisposition changes, and behaviors can be measured. So what excuse justifies not knowing if the action and communication strategies are making progress toward achieving program objectives? What justifies not documenting how the program worked? What justifies not being able to say whether or not the problem has been solved?

The book lists 13 (count ‘em, 13) types of measurement to conduct along three stages of a project:

• Preparation—Adequacy of background information base for designing the program; appropriateness of message and activity content; quality of message and activity presentations
• Implementation—Number of messages delivered to appropriate channels and activities designed; number of messages placed and activities implemented; number who received messages and activities; number who attend to messages and activities
• Impact—Number who learn message content; number who change opinions; number who change attitudes; number who behave as desired; number who repeat behavior; social and cultural change

All of which can be measured; the tools exist. Gut instinct, which Wagner suggests as an appropriate measure, simply won’t cut it in the boardroom. Every other aspect of business—including PR’s cousins in advertising and marketing—is expected to produce measurable results. With attitudes like Wagner’s, it’s no wonder PR gets so little respect! I would point John to a comment posted to this blog in response to an earier item dealing with measurement:

I work at a large R&D driven company and it wasn’t until we started measuring our comms performance (especially internally) that leadership started to take us seriously. Any number of humanistic theories meant little to the leadership, but show them a multivariate analysis and we’re talking! Since then our comms organization has nearly doubled in size, management are aware that they need to do much better in their personal presentations, we are more often on-message, we have better focus on our channel management and finally we can follow long-term trends in the organization.

That’s the power of measurement. As for the notion that clients won’t pay for it, that’s why it needs to be integrated into all programs rather than itemized as an optional service. Perhaps the cost of measurement should be built into hourly billable rates, an assumption that a set percentage of a practitioner’s time will be devoted to assessing the effectiveness of his or her work.

But measure we must. Whether it’s content analysis, opinion surveys, cost avoidance (one of my favorites), eyeballs (if that’s what matters to the client), we need to be able to show that our efforts provide value and we need to be able to show it in a way that’s meaningful to management. Claiming it’s too hard, too expensive, and too much of a challenge to convince clients to undertake measurement are just excuses. Remember what the Holmes Report said about why measurement isn’t happening among the top 100 PR agencies:

In general, (agency) responses suggested that an failure of commitment—rather than the absence of necessary tools and techniques—is behind the industry’s poor (measurement) performance.

Until the industry—including Wagner Communications—makes that commitment, we’ll continue to be viewed as lower-tier, lower-funded alternatives to marketing and advertising.

1. Oh boy, here we go again. I like your term heretic to describe Wagner’s view of measurement. Katie couldn’t have said it better. We are not voices in the wilderness though. There are some excellent professional practitioners and some enlightened academics who have provided a treasure trove of material on PR measurment for free. It can be accessed on the Measurement and evaluation section of research page of the Institue of Public Relations web site
   http://www.instituteforpr.com/measurement_and_evaluation.phtml
   The listings contain many very well research pices including more of Katie’s valuable contributions to the knowledge pool on this subject.
   It is time for Wagner and others sympathetic with his view to do their research and get with the real world. They are giving the rest of us a bad name.

   Posted by Tudor Williams  on  12/30  at  10:42 AM

2. Shel:

   I appreciate your take on this subject but I certainly don’t think my attitude towards research is holding back the PR industry!

   As I said, I’m not against research AT ALL ... just the kind that uses manufactured or false numbers to try and show outcomes.

   In that vein, I didn’t see much difference between advertising equivalency and Katie’s PRV.  It sounded like any other false process out there designed and sold by a consultant to try and justify a PR department’s existence.

   Any despite Tudor Williams’ comment above, I DO live in the real world, the one where PR professionals and clients intersect on a daily basis with time, budget and focus constraints that hinder the ability to do the kind of research that a Proctor & Gamble or other Fortune 500 companies swear by.

   Having said that, I am not at all against using whatever tools are available to measure success, even for small companies.  I’m a two-time Gold Quill winner and a one-time finalist, and I know that you know you don’t achieve that level of success without measurement.

   What I DO have a problem with is a lot of these hokey ratios or valuations that to me are nothing more than garbage in, garbage out.

   If that makes me a heretic in some people’s eyes, so be it.  At the same time, I’m not selling measurement consulting tools, either.

   Posted by John Wagner  on  12/30  at  01:27 PM

3. John, I appreciate your response (truly). The difference between Katie’s PVR and AVEs was pretty clear to me. AVE suggests at its core that you get the same VALUE from column-inches of media and what it would cost to buy advertising in the same publications. Or, as the IPR put it, “AVEs are calculated by measuring the column inches (in the case of print), or seconds (in the case of broadcast media) and multiplying these figures by the respective medium’s advertising rates (per inch or per second).” Katie’s measurement, on the other hand, is based on impressions—what would it have cost to get to the right audiences by buying advertising versus the cost of getting media placement that effectively conveyed the desired message. Now, we both would agree that, by itself, this measure is of limited value. However, remember that Katie came up with the measure as a more desirable alternative to AVE for a client that was insisting on that kind of metric. If I had a client who wanted to know that the PR efforts were producing the desired reach, I’d be far more comfortable with PVR than I would with AVE—and I wouldn’t see anything “hokey” about it! I wouldn’t see anything manufactured or false in these numbers (or those Katie used in her example).

   But, as Katie also noted, you also need to assess outcomes. There are, in fact, several dimensions of measurement, including monitoring (which is where PVR comes in).

   Congratulations on your Gold Quills, by the way. I have a few myself, and it’s always a thrill to win. In fact, Neville Hobson and I plan to submit “For Immediate Release” in the brand-spanking-new podcast category.

   Posted by Shel Holtz  on  12/30  at  03:32 PM

4. In fairness to John, I can empathize with him the challenge of trying to convince a client to undertake measurement. I’ve been there when the client says, “If we have to pay your firm more money to prove that what you did, and have been paid for, has worked, we’ve hired the wrong guys!”

   However, measurement should be included, no questions asked, if for no other reasons than to help professionals continually get better and avoid repeating mistakes.

   The solution, IMHO, lies in a comment you made Shel, about building measurement into the billable hourly rate. I’d also not even list it out as a separate activity.

   When my old agency made advertising media buys for clients, at the end of the campaign the media department always delivered a reach and frequency report based on data we pulled from the media outlets. It was just a standard part of the service; a “just the ways things are done around here.” The costs for running the research was embedded in the agency fees.

   What about taking this a step further? As Katie explained, the IPR has a policy prohibiting the use of AVE, right? What if the professional assocations implemented a standard stating that only programs/campaigns with a measurement component would be deemed “professionally produced.”

   This would give agencies the cover to build them in automatically and justify them by saying, “ACME public relations is a member in good standing and strictly adheres to the professional standards proscibed by the Association of PR Gurus.”

   and it would provide benefit for the corporate practicioner too…

   “Well J.B., by industry definition and standards a ‘professional’ public relations program must contain a measurement component. Do you really want us to put together an ‘unprofessional’ or ‘substandard’ effort here?

   Doesn’t anyone remember or still use the RACE methodology? That used to be a big deal back in the 80’s and 90’s in terms of the components of a PR plan.  If it, or something like it could be formalized and infused into the nuts and bolts of developing, conducting a program, it would go a long way to solving this challenge.

   Posted by Craitg Jolley  on  12/30  at  07:24 PM

5. Shel:

   I honestly wasn’t trying to brag about the GQs ... I won those a long time ago (in what seems like a galaxy far, far away), but I knew that those would have meaning to you as an explanation of my background.

   When I worked in corporate communications, we did a lot of measurement, especially in internal communications.  Key message analysis, reader surveys, in-depth communication audits, focus groups, etc.

   At one company, we had a fairly detailed scorecard of specific objectives that we measured against and it was taken very seriously.  So I understand and appreciate the value that information can provide.

   Media relations—to me—is a different animal.  And that’s where I get off-track with discussions about PRV and similar tools.  The “manufactured” part comes in terms of trying to put a number on how many people viewed a story.  I just think that any calculation that includes a readership or viewership number is patently false because of the way people consume media, especially today.

   In my experience on the agency side, clients are quick to dismiss these types of measurements because they are so easy to poke holes through.

   Posted by John Wagner  on  12/31  at  02:40 PM

6. John, my congrats on your GQs were sincere! And I agree that traditional media relations measurements—generally the AVEs against which the Institute for Public Relations has taken a stand—are meaningless. Hence, Katie Paine’s effort to identify a metric that has some meaning. Offering no metrics at all is as unacceptable as offering bad ones. What client would spend a considerable amount of money on a media relations effort based on an agency saying, “I can’t tell you what the coverage is worth, but my gut says it’s paying off”?

   Posted by Shel Holtz  on  01/02  at  08:29 AM

7. This is timely given the discussion here:

   http://mediainsider.prnewswire.com/blog/_archives/2006/1/4/1589310.html

   Posted by Craig Jolley  on  01/04  at  08:31 PM

8. Thanks for pointing it out, Craig. Now we can only hope that PR Newswire gets it right instead of suggesting the same meaningless measurements that John Wagner objects to.

   Posted by Shel Holtz  on  01/04  at  09:44 PM

9. Well, according to their online promotion, worthless AVE is one metric which is unfortunate. However, some of the other metrics look interesting, if the data is accurate.

   Particularly the ability to track up to five key messages simultaneously. Also being able to evaluate what items about competitors are being covered could be valuable.

   Would have to really understand how the message tone (positive, neutral, negative) was determined, especially having done this with Mark Weiner (at the time with PRData Systems) related to our Media Bias study we conducted following the 1992 U.S. presidential election.

   Of course the devil is in the details and the true value can only be assessed after having the service applied to a real, live communications program.  Perhaps we can find someone who actually uses it and ask for their review (perhaps an FIR interview?).

   Posted by Craig Jolley  on  01/05  at  11:15 AM

10. It’s interesting to read this thread.  A few comments:

    First:  The IPR’s position on AVEs are actually relatively moderate as compared to what this discussion suggests.  Read the white paper written by Bruce Jeffries-Fox.  While I believe that AVEs are an incomplete and inadequate measure most of the time, I’ve seen them correlate to sales when no other PR measure could.  Even if they are not the preferred measure and even if I believe that AVEs should be highly qualifed in their use if they are used at all, I’d rather be partially right than totally in the dark.  AVEs do provide some gauge to the amount of coverage (minutes/column inches) and the reach/prestige of the medium.  Even an incomplete measure can be abused and I think this is what so many of us find offensive.

    Second, the irony is that AVEs greatly undervalue PR’s unique role within the marketing mix. As the research provider who worked with P&G;and helped to create PRevaluate, and after having done similar marketing mix analysis in dozens of categories for scores of companies over the years, we’ve discovered that PR consistantly delivers the best ROI of any marketing agent.  On average, mass marketing advertising delivers about $1.25 on a dollar; price promotions only $.75 on the dollar and PR between $3.00 and $8.00 on the dollar.  The trouble is that most companies don’t use marketing mix modeling, and even those who do prefer to track only those marketing elements on which they spend the most money (TV and promotions) rather on those elements which deliver the best ROI (PR).  When viewed within the context of the complete marketing mix, PR is the only marketing option that provides a lift to all other forms of marketing (meaning that when the PR is good, the advertising is more effective, the direct marketing more efficient, etc).  It’s what every PR person hopes is true, and maybe in their gut believes is true...now the truth is being revealed time and time again.

    The question to ask is not “what will marketers do with this information,” but rather “what will the PR profession do with this information?” Will this be another lost opportunity for PR or will the leaders of our profession and the professional associations which represent us break free from conventional wisdom?  Will they take the lead in promoting PR not because it “generates buzz” or “delivers a high equivelant ad value” but because PR delivers on meaningful business objectives?  The story of P&G;, Wachovia and Miller Brewing broke in the advertising and marketing trades rather than the PR trades.  The evidence of PR’s unique power to drive businesses forward should be our call to action: Instead of squabbling among ourselves and finger-pointing, we should unite to promote PR simply because it works in meaningful and measurable ways.

    Posted by Mark Weiner  on  03/26  at  09:07 PM

11. >>Will they take the lead in promoting PR not because it “generates buzz” or “delivers a high equivelant ad value” but because PR delivers on meaningful business objectives?<<

    AMEN, brother Mark!

    You’ve, correctly IMHO, pointed out that PR stands at a watershed moment in time. Given the demonstrable evidence of PR’s value as a powerful BUSINESS driver, with real, bottom-line benefits, what the profession does with it will be telling.

    I think that if the profession doesn’t capitalize upon this opportunity (or a single association such as IABC, PRSA, etc.) then it doesn’t deserve to be taken seriously. Individuals can pontificate all they want about how valueable PR is, and what it brings to the table, yada, yada, yada. But if the profession doesn’t use this to start positioning itself then it really is a collection of people who are lightweights.

    The question is, who in the industry has the courage and leadership mettle to stand up and be heard? Ideally, it should be the president of either IABC or PRSA - preferrably both - to beat this drum. Sadly, from past experience I doubt that we’ll see anything more that the status quo.

    Posted by Craig Jolley  on  03/27  at  06:16 AM

12. “Clients don’t want to pay for it”

    Very true, but why would you expect a client to pay for an evaluation of effectiveness of PR for which they have already paid. Furthermore, self-evaluation of effectiveness is like a school where the students grade themselves. An impartial 3rd party should make the effectiveness and ROI measurements.

    Posted by Graduate Student  on  02/27  at  09:32 PM