Revisiting 2004: Assessing the Polls

This represents a diversion from the normal topic of this blog. But it struck me that with all the back and forth on the polls - one suggesting McCain gaining, most not, that it would be good to re-visit an assessment that was done after the last presidential election. What they found is that a majority of the polls - the same ones that are showing Obama with commanding leads - were essentially correct within the margin of error.

But don't take my word for it. Check out it out below. And note, if you will, how close it was. The difference now is that Obama is polling far ahead of where Bush was at this time. Believe the numbers.
--------------

From the Pew Research Center:

November 23, 2004

Pre-Election Polls Largely Accurate

Lessons From Campaign '04

Facing an array of new and difficult challenges, the pre-election surveys performed fairly well in predicting the outcome of this month's election. Once again, the final surveys by most national polling organizations fell within the margin of error, in spite of widespread concerns over rising refusal rates, "missing" cell-phone-only voters and supposedly flawed likely voter screens. In the wake of such a demanding election, there are a number of lessons to be drawn from the performance of the polls in Campaign '04:

Follow the Data. The campaign showed the dangers of making a priori judgments about voter opinion. This concern arose, most notably, with the practice of weighting polling data to meet predetermined parameters of party identification. Until recently, this approach has been mostly used by partisan pollsters but it also has been adopted by some media polling organizations.

As we noted in an earlier commentary, weighting results on the basis of party in the same manner that pollsters weight data to meet demographic parameters is misguided for a number of reasons (see "Party Affiliation: What It Is and What It Isn't," Sept. 23, 2004). It represents a fundamental misreading of party identification, which is an attitude that changes over time and not a demographic characteristic.

The election exit poll underscored the problems inherent in weighting by party. According to the exit poll, the electorate had equal numbers of Republicans and Democrats (37% each). That represents a change from the past three presidential elections, when Democrats held a slight edge in party identification. Consequently, pre-election surveys that adjusted the partisan composition of likely voters to the exit poll results from 2000 would have misrepresented the partisan make-up of the actual electorate.

Follow the Data (Part II). Among the staples of conventional wisdom in this campaign was that, in an incumbent election, voters who are undecided in the late stages of the race break overwhelmingly against the incumbent. Historically, this so-called 'incumbent rule' has proven to be predictive.

But this year's exit poll showed that Kerry had only a modest advantage among the late deciders. Among those who decided in the final three days before the election, Kerry led by 53%-44%. That was a far cry from what many analysts and pundits had predicted. Pew's analysis of undecided voters for its final pre-election estimate which was based on statistical modeling and follow-up questioning of the undecideds predicted they would break about evenly with a small advantage for Kerry.

This proved more reliable than history in gauging their intentions. There is no doubt, however, that predicting the vote among the undecideds and estimating the final outcome are among the most formidable challenges confronting pollsters.

Cross-Pressured Voters Are Hard to Read. The national polls were fairly stable during the summer, with most showing the race about even or Kerry with a slight to modest lead. But in the wake of the GOP's successful convention in late August, voter opinion was less stable and much harder to track. Similarly, the polls fluctuated after the three presidential debates; most polls showed Bush with a lead, though they differed over its size.

Much of the movement in the polls came among cross-pressured swing voters. These voters, who still comprised about 14% of the electorate in Pew's final pre-election survey, were not happy with the president yet still could not comfortably support Kerry. And they wreaked havoc with the polls, as Pew Research Center Director Andrew Kohut observed in the New York Times (Oct. 21, 2004):

"The round of national surveys taken after the third presidential debate indicates that the polls are not going to give us a clear picture of who will win the election until the final days of the campaign, if then. This is not because polling no longer works it's because voter opinion is highly unstable."

This turned out to be the case. Pew and Gallup had similar pre-election results before making their final estimates Pew showed Bush with a three-point lead among likely voters (48%-45%), while Gallup had Bush up by two points (49%-47%). The CBS News/New York Times survey also showed Bush with a slight 49%-47% advantage.

Although the three polls ended up in pretty much the same place, they all got there differently. The Pew survey showed Bush gaining some ground since mid-October, while Gallup showed Kerry gaining over the same period. The CBS News/New York Times survey showed voter opinion as more stable. Consequently, one could draw very different conclusions about the direction of the race by looking at these polls over the last few weeks of the campaign.

State Polls: Helpful But... The number of horse race polls has been rising for the past several elections, and this year brought an increasing number of polls in the battleground states. Facing a difficult political terrain by definition, all of these states were closely divided the state surveys had varying degrees of success.

In Michigan, for instance, most of the final week polls showed Kerry with a slight advantage. Of the six polls conducted between Oct. 25 and Nov. 1, the website RealClear Politics found that, on average, they showed Kerry with a 3.5% lead; Kerry won Michigan 51%-48%.

By contrast, the polls did not fare nearly as well in Florida, which Bush won by a fairly comfortable margin (52%-47%). Four polls conducted in the campaign's final weeks had Bush ahead anywhere from a point to as many as eight points; three others showed Kerry leading, with Fox News showing Kerry with a five-point advantage. The main shortcoming of the state polls is that typically they have small samples. But even some that had sizable samples missed the mark in Florida. The highly diverse make-up of the Florida electorate may present special challenges to polling.

Cell Phones: Not a Major Problem Yet. Throughout the campaign, pollsters were frequently asked about the potential bias created by the fact that cell phones are not reached in telephone surveys. A growing number of people, especially young people, are believed to be relying solely on cell phones for phone service.

The exit polls indicated that approximately 7% of all voters are cell phone-only; this group skews heavily toward younger voters, with about half of them (48%) falling under age 30. But although cell-only voters were somewhat more Democratic than the electorate as a whole, the difference was relatively small. Cell-only voters went for Kerry by a margin of 54% to 45%, compared with Bush's 51%-48% advantage among all voters. This difference, and the relatively small size of the cell-only group, was not large enough to create an error in pre-election poll estimates.

Moreover, young voters with and without cell phones were virtually identical politically, suggesting that young people reachable by conventional land-line service remain representative of their cell-only counterparts. About one-in-five (19%) voters under 30 years of age indicated they relied solely on cell phone service. These voters favored Kerry by 58%-41%, close to the advantage he had among the rest of this age cohort (56%-43%).