Tony Bain

Building products & teams that leverage data, analytics, AI & automation to do amazing things

The Maturing Field of Big Data

May 17, 2016 | Tony Bain

I remember when I was 19, I was working for an electricity utility as a DBA.  I was putting in lots of hours and partly as a reward, and partly because they didn't know what to do with me, then sent me off to a knowledge management conference.  Well, when I got back I was an “expert” in knowledge management and it was going to change the world forever.  I convinced my boss to get me in front of the executive team during their next board meeting.  I was on fire, delivering what could only be considered a stellar presentation on knowledge management.  At the end of the presentation I looked around the room expecting excited faces and many baited breath questions. Instead, you could have heard a pin drop.  They were staring at me not knowing what to say, until one of the executives jokingly asked if I had learnt why Microsoft Word keeps crashing while I was at the conference.  Then they moved on with their meeting, and knowledge management was never discussed again during my tenure.

For a long time after I was thinking how foolish they were, to ignore the technology which was going to change their business. I was literally handing the insight to them on a plate.  However, over time as I got more experienced my view started to change.  At some point many years after I had left I realised I was trying to sell them a solution for a business problem they didn’t have. 

As a wide eyed techie I had an assumption of perceived value, they had thousands of user files on network servers, knowledge management allowed you to structure, access and understand these files in a better way and to me that sounded like it had immense business value, although I wasn't exactly sure what this business value was and couldn't articulate it any deeper than “insight” or “understanding”. And because of this I had failed to put this into any context they cared about, how KM was going to sell more electricity or prevent outages.

Fast forward to present day and I have seen a similar repeat of my experiences across the industry in relation to Big Data, and certainly some of the commentaries on Plantir resonate.  Big Data has been primarily IT lead on the assumption that if you get enough data into one spot there is significant inherent value in it.  You can find lots of web articles about this, about finding “needles in haystacks”, about discovering previously unrealised relationships, about “monetizing” data, about understanding customers better.  But when you try and dig into the detail of what this actually is, there is certainly less information available.

Thankfully, Big Data is starting to move out of the hype phase with the related, but separate fields of Machine Learning and AI starting to take over as the topics generating internet buzz. 


But the hype is always a good thing, for a period of time, as out of it we now have awareness, technology, toolsets and capability to develop business solutions using Big Data.  But as the field has matured we also need a mature view to be successful. This includes:

  • Big Data must be business led rather than IT led.  They must be attempting to solve problems that the business cares about and has meaningful impact.  IT is an important part of the solution but not the driver.
  • Solutions must identify value that is not easily identifiable using simpler/less costly methods.  For example, say you have a factory that makes red doors.  Sales have been great but over the last few months sales are declining.  To solve this you using Big Data to identify that customers are growing tired of red doors and now they have a preference to buying blue doors.  Did you really need Big Data to solve this problem?  Do you think if you spoke to your #1 door salesperson they wouldn't be able to give you the exact same information?
  • The Big Data solutions must lead to actions that the business can undertake.  If they have a red door making plant, perhaps they can modify to make blue doors.  But they might struggle to start making cheese.  Big Data has to provide insight within business context.

This doesn't mean that Big Data is becoming boring, far from it in fact.  Instead this maturing means we are more focused on delivering data driven solutions that are going to have a real impact on the world around us.  For any analyst/data scientist that has to be more exciting than simply churning data for data’s sake.