This post is written by Ann Feeney, Information Retrieval Specialist at Cision.
The secret of measuring for impact is knowing what is meaningful and can guide decisions. Not every metric matters in every situation—so here are some common metrics and a guide to situations where they make a difference, along with the situations where they’re potentially misleading.
1. Share of voice: In all the articles that cover a topic, how many mention us? This is one of the easiest to measure and to calculate. When does it matter? Share of Voice matters mostly for organizations that are starting out or need to raise their basic profile, in general or in a specific market. A new company needs to know if its audience is at least developing name familiarity. An organization that needs to increase its hiring and retention diversity (and what organization doesn’t?) wants to see how often it’s mentioned in publications that target a certain demographic. Share of voice can also matter in discussions about ethical issues. If your organization’s name comes up in outlets about corporate social responsibility, transparency and environmental stewardship, that’s a good indicator.
When doesn’t it matter much? It’s less important for organizations that have fairly stable name recognition.
When is it misleading? When your monitoring results include press releases or when the content carries mixed or negative messages about your organization. During a scandal, your organization’s share of voice might go up quite a bit, but that’s not good.
2. Sentiment: how positive, negative, or neutral coverage is. It’s surprisingly tricky to assign sentiment; inter-rater reliability (how likely individual raters are to agree) can be as low as 70 percent, depending on how complex the material is and how nuanced the measurement scales are. To be truly useful, sentiment either has to be tied to a model that links business objectives to sentiment—for example, if sentiment reliably correlates to how likely a movie is to be profitable or how many high-potential students apply to a college—or to provide insight for a business decision; serious hobbyist photographers like the vibration reduction for this camera but dislike the weight, the next product development cycle needs to focus on weight.
Sentiment doesn’t matter much when you can’t tie a business decision to it and is misleading when it aggregates too many factors. For example, if comments are positive about a restaurant’s location but negative about the food, the overall sentiment score might rate as neutral but that restaurant will be in trouble.
3. Publicity value: Cision’s publicity value calculations are based on length of the story, the type of media in which it appeared, and the number of impressions for the individual article (not the site as a whole, since that can be misleading). This assigns a dollar value to publicity so it ties to business outcomes very closely and the methodology avoids the issue of fake traffic from bots, unlike counts based just on page views. However, because it does not calculate sentiment, it can be misleading when used as a standalone tool; it’s best used with sentiment or semantic analysis.
4. Semantic measurement: based on the meaning of what is being said. There are countless ways to perform semantic analysis, ranging from the simple to the mind-blowingly complex. One fairly basic but powerful way to perform semantic analysis is to search within coverage for terms related to a particular concept or emotion. For example, a famous study showed how differently people respond to articles talking about crime as a “virus infecting” a city and articles describing it as a “wild beast preying on a city.” Those who read about crime as a virus were more likely to want public investment in systemic prevention while those who read about it as a beast were more likely to want to invest in enforcement. Measuring the prevalence of terms related to various concepts can reveal a great deal about how to shape decisions and communications.
5. Projections: Using projections is an excellent way to measure the value of public relations. You can use Excel or other spreadsheets to run a trendline of any of the values that you’ve chosen for measurement. This chart shows a basic logarithmic trendline in red for a semantic measurement for anger about a particular topic. It’s projected four weeks into the future. However, the actual data measured after those 16 weeks shows a different pattern. This indicates, though it doesn’t prove, that a PR campaign executed in Week 12 actually changed the conversation. Of course, this is a fairly basic trendline over just 12 data points, chosen for an easy display, but it’s possible to run calculations with R values over 0.8, giving it a fairly solid reliability. (Talk to your favorite stats friends for more details).
Photo credit: @kokuziu via flickr