Tony Bain

Building products & teams that leverage data, analytics, AI & automation to do amazing things

Big Data innovation marches on

September 21, 2010 | Tony Bain


With IBM intending to acquire Netezza the predicted consolidation in the distributed analytics market is well underway.  Recent deals include EMC/Greenplum Teradata/Kickfire and now IBM/Netezza.  A good breakdown of this deal is on Curt’s blog.  There is still more to go of course with one of the crown jewels, Vertica, still ripe for the picking. 

What this indicates is that MPP analytics has moved from the innovative edge into the mainstream market and now the more risk adverse large caps and now willing to invest substantially in growing this market.  Interestingly Microsoft made this move early with the acquisition of Datallegro in 2008, I doubt this has paid dividends yet but 5 years out this might be a different story as the explosive growth of machine generated data continues. 

While it is probably a bad time to start building another MPP query processor of course innovation in big data core technology continues to be strong.  Key areas of innovation relate to Flash/SSD optimization & caching, Graph databases, stream processing & CEP, Hadoop optimization, massive shared nothing (cloud) scalability & SQL/NOSQL convergence.  These technologies will come to market in a variety of different product forms some of which will later be picked up by the large caps.  Rinse, repeat.