Big Data’s Election Failure

This election was a perfect example of how data could lead you astray if you are not careful. Before the results of the election came in, all of the polls predicted Clinton had around 85% chance of winning. Well, the 15% chance that he would win happened. This was a black swan event, and I think it will become more evident as the years passes by. The poor middle class in the Midwest who voted for him will not know what has hit them (although I really hope I'm wrong).

I had my concerns about the polls because I wondered if the pollsters were talking to the right people. Too many people were willing to support Trump, no matter how demeaning he was to various groups of people. It was like these groups of people didn't care, had no empathy, and were willing to blow up the system and so they voted for him as the Republican nominee. I couldn't reconcile the polls with the people's continuing support in the face of atrocious behavior.

So if you don't collect your data properly, you can be led astray. There are three main areas where the pollsters could have gone wrong (I'm not a polling expert or an expert on gathering such data but these are the three possibilities that came to mind):

1. The first one is a "duh" one and I would not presume to know how to fix this one - the polling sample was not representative of the likely voters. Certain set of voters may have been left out of the polling samples, such as the first time voters. How do you identify these voters?
2. Some folks may not have been truthful during the polls. Trump was loudly reviled by both the Republicans and the Democrats, as well as the world at large, that maybe some Trump voters were quiet about their support. In this instance, being non-judgmental will be needed, if that even works.
3. It may be that the undecideds were undecided up until the moment of pulling the switch, and enough of them went for Trump. I don't know how big of a group the undecideds were and whether they were large enough to sway the election. During the last week, I read that most people had already decided but what if they hadn't? What if they kept switching right up until it came time to pull the lever?

This election strongly showed that it is not yet time to trust the big data hagiography.

The results of this election reminded me of an article, "Why Executives Don't Trust Their Own Data and Analytics", that I read, I think from LinkedIn, about senior executives trusting their guts rather than big data. It is kind of funny: executives say big data is vital to their success and are thus spending quite a bit of money but then they don't trust it. The election debacle will just add fuel to their distrust. In the LinkedIn comments section, a consultant said something along the lines of senior executives needing to get onboard with this automation of decision making while on the road to progress. I don't know if he was being sarcastic (because it did seem so) or if he was tone deaf just like senior executives can be tone deaf about their people.

So executives are willing to automate the little people's jobs but when it comes to theirs....