538.com, predictions on 8/11/08: SSE of 0.072

Historical regression, using economic data available in mid-August plus the August Palin pick: SSE of 0.075

From that, you might say “Nate wins!”, but I never claimed that one should rely completely on historical trends, just that one should incorporate their information. As I mentioned in comment #3 above, I recommended 2 parts historical, 1 part polling in August (with this ratio shifting toward polling later in the campaign).

2 parts historical, 1 part 8/11 polls: SSE of 0.065, much better than either of the two alone.

I could have provided slightly better info to the Obama campaign by recommending 1 part each, which had a SSE of 0.064.

I used the 2004 polls (and historical regressions) to inform the 2:1 August ratio, and since that’s an n-size of one campaign, I admit that I didn’t (and still don’t…now n-size is two) have a good grasp of the correct ratio.

I see two reasons that this ratio is more weighted toward polls in 2008 than 2004. First, in my 2004 calculations I used the Bayesian model developed by Simon Jackman, which is less sophisticated than Silver’s model. So, Silver is probably teasing more information from the polls. Second, in 2004 an incumbent was running leading toward more stability in terms of vote changes from the previous election to the current one.

Also, yes, I agree that Nate put himself in a tough position. I would have preferred he focus solely on forecasting, but that may have resulted in fewer web hits.

I agree with you. I’d also say that Nate put himself in somewhat of an impossible position, because the goals of poll aggregation and election forecasting are different. A convention bounce is meaningful in terms of short-term public opinion but does not say much about the ultimate election outcome.

First, credit to Nate Silver for popularizing these methods. I would point out that I posted/presented this paper before 538 was in existence.

Though my method and Silver’s are similar in many respects, the one important distinction is that I use historical regressions to downweight state-specific information contained in polls. (Silver downweights national leads by candidates early in the cycle, but even in this case, he regresses the lead to a somewhat-arbitrary mean of zero rather than using national historical regressions.)

North Dakota is the most striking example of the difference between our models. Silver, in August, looked at the impressively pro-Obama polls in ND and did not discount them–ND was projected to be only 3.8 percentage points below the national Obama vote share (estimate from 8/11/08). The historical regression, using only past results and current demographic and economic forecast ND to be 10.9 percentage points below the Obama average. My paper helps inform how much importance you should put on the historical regression vs. the polls (in August, about 2 parts regression, 1 part polls; this ratio reverses later in the campaign). So, in August, combining the historical regression and the polls, I projected ND to come in 8.6 percentage points below Obama’s vote share. (I have email evidence of this projection.)

On Election Day, Obama received 53.7 percent of the 2-party vote nationally and 45.6 percent of the two-party vote in North Dakota, for a difference of 8.1 percentage points. And that’s why Nate Silver should discount spring/summer polls more than he does now.

Caveats: 1) You might say I cherry-picked ND; I did, but I cherry-picked it in August. Silver’s projections and mine lined up well in many states in August; North Dakota was the biggest discrepancy between our models so I kept my eye on it over the course of the election. A more in depth analysis is forthcoming in early January now that (nearly) all the ballots have been counted. 2) While my model is more sophisticated in the above manner, Silver’s model out-does mine when it comes to things like pollsters’ house effects. And I tip my cap to him for that.