Recently Huff Pollster published THIS article about the 2012 American National Election Study, an ongoing study representative of the political attitudes of Americans. What they found seems to fit the common narrative, "Disappointed by President Obama’s unfulfilled promise of bipartisanship, Americans have retreated into their partisan corners and are now more bitterly divided than ever before." But as political scientists Lori Bougher and Markus Prior go on to say, that may not be the whole story... or even the right story.
One of the most popular debates in this industry is mode differences. For those of you who are not pollsters, it is essentially arguing about the differences between conducting surveys and research on phones, or on the internet (or in-person, or via text, etc.). The big deal here being that 2012 is the first year that ANES has used a multi-mode survey to conduct their study. They incorporated BOTH in-person and online surveys into their study, using the sampling for both. So why then, if they used the same sample type, would one mode result with a highly divided America, while the other would show Americans actually leaving their partisanship behind?
The most telling thing from Bougher and Prior's delve into the ANES study is the following graph which shows the ideological consistency and partisanship of respondents. It measures, in essence, respondents whose ideology matches up with their party affiliation (Democratic liberals, and Republican conservatives).
What it shows is that after the Presidential election in 2008 which elected President Obama, people were as partisan as ever. More than 20% of respondents were, what I like to call, "ideologically pure".
What's really interesting though, is that 4 years later, after the most bitterly divided Congress in history, in-person respondents' ideological purity had actually DECREASED, meaning more Democrats and Republicans are identifying as moderates... not as hard-line partisans. This would completely contradict the narrative that the partisan-division between President Obama and Congress (the House of Representatives and Ted Cruz) is causing people to move back, further into their partisan corners. So then, has the narrative changed, or not?
This brings us to the crux of my blog post. In 2012, the first year ANES utilized online surveys, online respondents actually recorded the highest level of ideological purity since they've been conducting the study. When averaged with the modest decrease of hard-line partisanship among in-person respondents, it still creates an upward trend. The debate here is over how accurate the online surveys are. Some people think that because respondents are able to sit in front of their computers and hash out their inner-monologue before putting fingers-to-keyboard, that the partisanship in internet surveys appears inflated. Whereas an in-person or telephone survey requires an immediate reaction, requiring the use of all the context and schema available to you inside that thick head of yours. With in-person, you also have the interpersonal dynamic of being more reluctant to disagree or emphasize your distance from the purpose with whom you are speaking. The continuing question is which, if either, survey mode is the best representation of "the truth" -- are American voters getting more partisan, or less partisan?
Trick question, because I don't really care about the answer, what I care about is highlighting that different interviewing techniques can yield different answers to the exact same question. Nearly 14 years ago I went to work for a firm that did solely online research (great firm, btw -- check out www.sotech.com) because I was convinced telephone interviewing would be dead before I hit forty. Now I'm (ahem) well past forty, and bought a dang phone bank in 2012 because quality interviewing counts more than ever, and we've gotten a second lease on life due to the proliferation of cell phones (and federal law forbidding computer dialing to mobile phones). It is getting harder to reach a representative sample on the telephone (landline or cell or both). Multi-modal studies that incorporate online with other methods are here, and there is no doubt that we will be using them more often, but it is a huge mistake to think that you can simply mash all the resulting data together and get a better, more accurate picture than a single-mode study.
It seems natural, you interview people, they give you answers, you compare those answers. BUT, some modes don't play as nicely with other modes as we'd like. One thing I've always stressed to my clients is that there are lot of options for doing public opinion research, but they all do different things, and give you different results. I think it would be a mistake to look at the 2012 ANES report and say that because of the disparity between the two modes, one of them must be wrong. Instead, I think it's important to look at the data from both and consider the context around them. It's a brave new world out there for political prognosticators. The victor will be the one not looking for simple answers -- cuz there ain't any.