The Numbers Don’t Lie. Or Do They?

Anna Fahey is political communications theorist, language analyst, and self-proclaimed opinion research junkie. She recently joined Sightline’s staff as Communications Strategist and will be a regular contributor to the Daily Score.

Even though I’ve read report after sobering report indicating Americans’ increasing disconnection with government, it was still a little shocking to read Gallup Poll numbers released this month. Apparently, given three choices—big labor, big business, or big government—3 out of 5 Americans surveyed in December 2006 named big government as the greatest threat facing the country.

Gallup has been asking this question since 1965 to track long-term attitude trends and, as it turns out, this feeling is nothing new: Americans have consistently perceived big government as a big threat for decades.

Perhaps more surprising is the fact that in 2006 so-called big government is considered the biggest threat no matter where political allegiances lie. That is, among majorities of Republicans (63%), independents (58%), and Democrats (56%) alike. Even among self-identified liberals, 49% say big government is the number one threat.

But I’m not sure that we should take these findings at face value…

First, the obvious: The wording of a survey question can distort results. Just imagine how respondents’ answers might have been different if the word “big” were removed from their list of options. If asked to choose whether “government, business or labor” was the most threatening, respondents might give very different answers. Or think of the difference if Gallup had asked about “our government,” or just “the government.” Changing a single word seems subtle, but it can produce powerful results.

The phrase “big government” summons up unsavory connotations—and, more importantly, those connotations are negative and threatening, no matter where you fall on the political spectrum. A conservative worried about high taxes, and a liberal worried about warrantless wiretapping would probably both say that “big government” is a threat, even if they disagree in every particular about the nature of that threat. And at this point “big government” clearly carries more baggage than “big labor” or “big business.”

My point: adding the word “big” to the question completely changes the “frame” of the question. There’s been a lot written about “framing” (see, e.g., here, here, and here). But in a nutshell, frames are the mental shortcuts we all use  to make sense of our world, and are embedded in all the words we encounter—not just the ones that jump off the page. In effect, Gallup may be gauging the strength of people’s feelings about the “big government” frame, rather than their feelings about the government as they actually know and experience it.

Second, and perhaps less obvious: answers to one question on a survey can be affected by all the questions that come before it. In the 2005 version of the Gallup poll in which our “big government” questions were posed, these questions came on the heels of a barrage of queries about people’s sense of the state of the nation (right track / wrong track, presidential approval ratings, most important problems facing the nation), questions about the economy (inflation, unemployment). Intentional or not, this is called priming. Priming, as you may already know, brings certain thoughts to the foreground of our minds. It sets dominant frames in our head as we make sense of the information coming our way.

In this case, it’s likely that by the time you get to the “big government” question, you’ve been mentally cued to think about a bunch of stuff that’s not going so well in the country. This could affect your answer.

Priming works both ways. If instead you’d been warmed up with a series of questions about government activities and services generally viewed as positive—national parks, roads and transit, state universities, clean air and water, disease control—that is, positive frames about government, government as a whole would likely come out looking a good sight more beloved of the people. (Demos and Frameworks have done all kinds of research on this very thing.)

Don’t get me wrong. I’m not suggesting we scrap public opinion research altogether. Quite the contrary—this stuff can teach us a lot. I love this stuff. I love Gallup. Just take these Gallup numbers—and others—with a grain of salt. And don’t take summary reports at face value without a little digging into the context.