Big data vs Small data

Human decision making is truly irrational. While we strive to get conversions by using linear algorithms or banking on tons of data to understand consumer behaviour, it's the small things that truly matter - in this case, small data.

Two recent experiences reaffirmed the same :

🎤 At a recent product talk, the presenter talked about how Google's flu program failed when a simple heuristic approach worked wonders

Google Flu Trends (GFT) which used big data analytics and a black-box algorithm tried for years to predict flu outbreaks without any success (was wrong for 100 out of 108 weeks) and the program was finally terminated in 2015. However a surprisingly simple recency heuristic forecasted the flu outbreaks more accurately - based on psychological theory of how people deal with rapidly changing situations.

📖 Read a book on Small Data by a renowned branding consultant who has transformed major brands like Lego, Lowes etc using ethnography and understanding the entire ecosystem the users interact with.

Excerpt from the book :What helped turn around Lowes wasn't a local or regional solution. It didn't come out of a Harvard Business School or Wharton case study. "Big data never told us to build SausageWorks" a member of Lowes' executive team told me later. "In fact, the opposite is true." In every single Lowes, we hired a store manager whose only task was to determine whether or not shoppers were happy. They did, too. When people shopped at Lowes, they told me afterward, they felt 'at home'.

Small data involved Lowes Foods representatives actually going out and meeting with customers in their homes and speaking with them to learn more about what they want, need, and expect from the brands they choose to shop with.

Big data is important for decision making, reporting and assessing the overall health of the systems - it's effective in rational domains (what is already known). Small data has outperformed big data models in predicting outcomes, such as U.S. presidential elections, or other uncertain events, such as consumer purchases, patient hospitalizations etc.

A combination of both would be a powerful toolkit to have.