Donald Trump has taught us something about prediction models
Nate Silver is getting trolled. For those who don’t know, Nate Silver and his organization 538 make election predictions. A day before the US presidential elections, their elaborate model said that it was 50–50 for the two candidates. However, in the actual elections, not only did President Trump win the electoral college, he also won the popular vote. His performance improved from the last time in every state. The results were closer to a blow out than to 50–50. If you are in the business of making predictions for one-off events, you would do well to learn from this: Stop taking your prediction models seriously.
Nate Silver could well argue that Trump’s win is consistent with a prediction of 50% chance of winning. But by that logic, Trump’s win is consistent with any prediction that gave him more than a 0% chance of winning. If you had predicted that Trump had 1% chance of winning, you could still claim that your prediction was right. This is a problem with one off events.
When weather forecasters predict that there is a 50% chance of rain, their forecast is not necessarily wrong if it does not rain. To test the accuracy of their forecast, we would need to compare their forecast over many days with reality. As a crude illustration, we could take 100 days when the forecast was 50% chance of rain. If it did rain on 50 of the 100 days, we would say that the forecast was damn good. You cannot do that with a presidential election in the United States.
These elections happen only once in four years. The candidates are usually different. Even when one candidate is the same (as was the case with Trump), they may be perceived differently. The electorate is different because of additions and deletions. The social and economic conditions of individual voters are different. The perception of the voters of their economic and social conditions are different. This list of differences can go on and on.
You can believe that your model captures these differences but you have no way of testing your belief. In absence of such tests, you would have no way of correcting for your own biases. Especially, as you may not even know your own biases. No model or modeler is bias free and it begins with inputs. Let us go back to 538’s model.
The company’s website says, “538 uses polling, economic and demographic data to explore likely election outcomes.” These inputs are adjusted. For example, “They adjust for whether polls are conducted among registered or likely voters and house effects. They weight (sic) more reliable polls more heavily. And they use national polls to make inferences about state polls and vice versa.” These adjustments are necessary because we know how bad polling can be. However, there is no objective way of making these adjustments because past history of a pollster itself is not a great guide to the quality of the poll in a one off event.
Not only the inputs, the relationship between inputs and outcome is entirely at the discretion of the model makers. In theory, simulations are supposed to allow a model maker to explore a range of inputs and a range of relationships between the inputs. However, this range will be limited by the model makers. The probability distribution of outcomes will also be decided by the model maker. After all these judgment calls, the model maker has no way of testing the model.
Nate Silver claims that they simulated the US presidential elections 80,000 times (Kamala Harris won 40,012 of them). But what good are these simulations? They just tell us what happened in the model. Not whether the model was a good approximation of reality. We cannot judge that. Neither can they.
Donald Trump has broken the faith people have in complicated modeling. I suspect that he will break many more things in the coming months.
___________-
This article is part of the series — Tips for early / mid career analytical types.
“How to spot a bad expert” in this series has received a lot of attention.
Follow me on medium. Choose the option to get the stories delivered to your Inbox.
You can follow AskHow India (@AskHowIndia) or me (@YogeshUpadh) on twitter or on LinkedIn
