This report from McKinsey adds to the list of similar reports coming from the HBR blog and numerous other sources, emphasizing that Big Data is here, and is going to have a huge impact on all business. What has become known as “Big Data,” refers to the terabytes and zettabytes (that’s a 1 followed by 21 zeros) of data that have been collected and stored on each one of us…for better or worse. Data mining is the industry that is emerging to make commercial sense of Big Data. It is a well-known fact that more data has been collected on us in the last two years, than in all of the previous years of computing combined.
Last week, PBS Newshour featured an election eve story about the Obama campaign‘s use of data mining. In the story, Obama election canvassers revealed how they were using Big Data and data mining to target their efforts. Rather than slog through neighborhoods knocking on every door, the Obama team had devised an ingenious field system using smartphones, GPS and mountains of data about the people living at each address. Each home that fit a precise profile that indicated their tendency to perhaps vote for Obama, or would be open to persuasion, was marked with a blue flag on the smartphone. The propensity to support Obama was determined by statisticians and mathematicians at Obama HQ, house by house, using a highly complex database of household information. This was not just income and age. It included information about magazines, organizations, that had already been shared publicly by the household. This dramatically improved the effectiveness and productivity of the field workers pounding on doors. From the election results, we now know that this system is worth its weight in gold.
Last term, UBC Faculty of Management 3rd year students investigated the emerging new “data mining” industry. We learned from example videos that data mining was being used to solve highly complex management, operational and marketing problems that had hitherto seemed unsolvable. Also, traditionally, assessment of industry trends had been highly subjective gut level judgments by expert researchers. Things like fashion trends, and video gaming preferences were thought to require mostly observation and guesswork. However, Harvard Business Review and other journals are seeing that Big Data is being mined to solve even these seemingly intractable problems.
The emergence of Big Data and data mining is essentially very similar to Chaos Theory, and the emergence of mathematical algorithms that solved the problem of apparent chaos in nature, which was discovered to actually follow an elegant order.
One Reply to “Big Data: The Next Frontier in Competition, Innovation and Productivity”