Re: Well so much for the polls

The Persistent Myth of the Bradley Effect

Many commentators -- the preponderance of them conservative but also some liberals -- take it as an article of faith that the current polling numbers overstate Barack Obama's position because of the so-called Bradley Effect: the notion that some material number of voters will lie about their intentions to pollsters, claiming that they will vote for a black candidate when in fact they will vote for the white guy.

A fairly typical example comes in the form of a blind quote from a Democratic strategist this morning at The Politico: "A huge challenge for Obama, insiders say, is simply determining how much skin color will matter in November. Race is nearly impossible to poll – no one ever says “I’m a racist” – and no campaign wants it revealed they are even asking questions on the issue."

“It’s the uncertainty that kills me – we know it’s going to be factor, but how big a factor?” asks a Democratic operative with ties to the Obama camp. “How do you even measure such a thing?"

Is there really so much uncertainty as this "operative" implies? Black candidates run races every cycle for the Congress and for the Governor's Mansion, and academics have spent copious time dissecting those results. And while we've never before had a major party nominate a black man for President, we did just finish an exceptionally competitive primary campaign in which a black candidate ran against an extremely popular white candidate with more than 35 million voters participating.

As we have described here before, polling numbers from the primaries suggested no presence of a Bradley Effect. On the contrary, it was Barack Obama -- not Hillary Clinton -- who somewhat outperformed his polls on Election Day.

The table below reflects 31 states in which at least three separate polls were released within 14 days of that state's primary or caucus. We compare the final trendline estimate from Pollster.com against the actual results from that state:





On average, Barack Obama overperformed the Pollster.com trendline by 3.3 points on election day.

There are some important differences by region. Using regions as defined by the US Census Bureau, Barack Obama overperformed his polls by an average of 7.2 points in the South. This effect appears to be most substantial in states with larger black populations; I have suggested before that it might stem from a sort of reverse Bradley Effect in which black voters were reluctant to disclose to a (presumed) white interviewer that they were about to vote for a black candidate.

Obama also outperformed his polls in the Midwest and the West (although there is not much data to go on in the latter case). The one region where Hillary Clinton overperformed her numbers was in the Northeast, bettering the pre-election trendline by 1.8 points. Recall that the Bradley Effect phenomenon describes covert rather than overt manifestations of racism. It may be that in the Northeast, which is arguably the most "politically correct" region of the country, expressions of racism are the least socially acceptable, and that therefore some people may misstate their intentions to pollsters. By contrast, in the South and the Midwest, if people are racist they will usually be pretty open about it, and in the West, which is nation's most multicultural region, there may be relatively little racism, either expressed or implicit.

The good news for Barack Obama is that, among the Northeastern states, only New Hampshire appears to be competitive -- and Obama would gladly trade a Bradley Effect in New Hampshire for a reverse Bradley Effect in a state like North Carolina. (Pennsylvania, it should be noted, is also defined by the Census Bureau as being in the Northeast, but in terms of political demography, it shares far more in common with the Midwest).

So why do we keep hearing so much about the Bradley Effect? Apart from the fact that it is a good way to fill column space on a slow news day, it seems that there are three or four reasons why the myth perpetuates itself:

1. Misunderstanding the Bradley Effect. Denying the existence of the Bradley Effect does not mean denying that some people vote on the basis of race. I have no doubt that some people will vote against Barack Obama because he is black. Indeed, I suspect that almost all of us either know such people, or know people who know them (friends and relatives of friends). I also have no doubt, by the way, that some people will vote for Barack Obama because he is black.

But the Bradley Effect is not an argument about whether people vote based on race. It's an argument about whether people will lie to pollsters. So long as race-based voters are honest about their intentions, Barack Obama's position is no worse than it appears to be in the polls.

2. Confusing Past with Present. There is fairly strong academic evidence that the Bradley Effect used to exist back in the 1980s and early 1990s. However, the evidence is just as strong that it does not exist any longer. The people who vouch for the existence of the Bradley Effect are not wrong so much as they are relying on dated evidence.

3. Confusing Exit Polls with pre-Election Polls. Unlike the normal, pre-election polls, exit polls conducted on the day of the election did substantially overstate Barack Obama's margins throughout the primaries. This is something to keep in mind at about 5 PM on November 4, when Matt Drudge and Jim Geraghty begin to leak exit poll results. It is not anything to worry about now, when we are trying to forecast the outcome from pre-election polling.

Nor is it clear the the discrepancies in the exit polls have anything to do with race; John Kerry, somewhat infamously, also underperformed his exit polls. The mechanics of conducting an exit poll are rather haphazard, involving a bunch of college kids and temp workers running around outside a polling place with clipboards and attempting to pass out survey forms to every Nth voter who leaves the ballot booth. This is not much easier than it sounds, and introduces a lot of human error and other forms of sample bias. For this reason, exit polls are not really intended to be used as they so frequently are in the panicked hours before ballot counting begins -- the results need to be calibrated and weighted, and exit polling firms rely on comparing their polls against actual voting results in order to do so.

4. Cherry Picking Results. The notion of the Bradley Effect gained a lot of currency after the New Hampshire primary, when Hillary Clinton did much better than anyone expected and won the state. However, the 8.9-point gap separating the pre-election polls and the actual results in New Hampshire represented only the seventh-largest error in the primaries. There were bigger discrepancies in Iowa, South Carolina, Alabama, Georgia, Wisconsin and Mississippi, all of which favored Barack Obama. These discrepancies did not receive as much attention as New Hampshire because they did not change the outcome of the election. But mathematically speaking, they were just as important.

A related phenomenon is that the media often cherry-picks polling results within a given state. The Zogby poll that had Barack Obama ahead by 13 points in California received widespread attention; the SurveyUSA result that had Clinton 10 points ahead did not. Over the course of the primaries, polling results that had Barack Obama performing well generally made for better copy, since until at least mid-February, Obama was considered the underdog. But an informed reading of the polls, such as the Pollster.com method, reveals that Clinton did not overperform in states like California and Ohio nearly so much as the media tried to imply.

-- Nate Silver at 6:50 AM

http://www.fivethirtyeight.com/2008/08/persistent-myth-of-bradley-effect.html