I have been busy travelling recently but thought I would jot down a couple of NoSQL myths that are fresh in my head from my recent discussions.

• Twitter use Cassandra internally but have not migrated their tweet store, despite their earlier plans to.  For now tweets are still stored in MySQL.
• Despite the widely accepted view that the use of Cassandra led to Diggs issues a couple of Digg engineers have apparently discounted this.
• Despite the widely accepted view that NoSQL databases all use eventual consistency this is not so.  HBase, for example, offers full consistency.
• Despite the widely accepted view that NoSQL is only about unlimited distributed scalability this is also not so.  Some of the most popular NoSQL platforms have fairly rudimentary (traditional RDBMS like) scalability options.  Such as CouchDB and MongoDB which use sharding + replication to achieve scale.
• Despite being commonly reported as “easy to install” or “easy to use” the benefits of a document object model are much more significant.  Why did we spend so much time during the 90’s trying to build ORDBMS?  Because the object-relational impedance mismatch is major and this translates into significant development overhead.  It is not uncommon to see 30-60% of all code in some applications purely “plumbing” to deal with mapping data to and from the RDBMS.  This is not something necessarily well understood by DBA’s or even some long time database designers, something I will write a follow up post on.

“NoSQL”, for lack of better name, is a generic term that describes any data management system that does not use SQL as a query interface.  Generally this means any data management system that is non-relational, but the term also has also been stretched as far to include the boundaries of what constitutes a data management system at all (such as Hadoop).

Early on (a couple of years back in NoSQL time) when the term was coined I think the positioning was much more aggressive, but more recently this has been softened so now NoSQL is commonly quoted as meaning of “Not only SQL” or “next generation databases” (whatever that means).  The common message you get now is something along the lines of NoSQL systems are more “specialized”, each being designed to solve a smaller number of problems than the generic RDBMS sets out to.  NoSQL is another tool in your toolbox.  A better option in certain cases where the RDBMS doesn’t fit well.  A different hammer for a different type of nail.  All makes sense in theory, but in reality this brings its own set of troubles.

There are now dozens of NoSQL systems available for a developer to choose.  From MongoDB, Cassandra, Voldemort, Hbase, CouchDB, Riak, Neo4J, HamsterDB and so on.  And there are several different orientations of NoSQL system including document, key/value and graph.  It seems the same energy we saw open-source hackers 10 years ago putting into MySQL has now been transferred into a myriad of NoSQL systems.  Again the argument, more choice, better for everyone.

The problem, and I am putting it out there as a problem so we can think of ways to fix it, is that while that is fine in reality, in practice many choices also creates difficulties.  Real world development projects have certain skills bases they draw on, with experience and ability to “make things work” based on years of hard slog cobbling things together.  And there are very few surprises left when deploying an application on a mainstream RDBMS (of course they will, like any software, will still have issue from time to time).

One of the key reasons the RDBMS has been so dominate is the fact that you could use it pretty much for any requirement.  And using it for any requirement meant that your developers had lots of experience building applications and your DBAs had lots of experience running it.  But also you knew that you could almost always make any requirement work “good enough” by buying extra hardware and/or indexing the heck out of it etc.  Regardless of whether it was technically the best fit or not, when all things were considered the RDBMS was a stable constant given short project timeframes and limited development budgets.  It was exactly its generic nature, its ability to do most things good enough, that has led to the RDBMS to become the default option for any new development project (with the various flavors of MySQL, Oracle, DB2 ,SQL Server being less relevant).

As humans, we all have limited brain capacities and most of us can only be experts in a small number of things.  And our expertise typically come from our history, making mistakes learning what works and what doesn’t through the hard yards of experience.   So given a buffet choice of specialized NoSQL systems how on earth do we choose the most appropriate tool for the job, while at the same time dealing with the lack of expertise we will invariably have?  Also what will be the impact to development projects in choosing the wrong tool for the job?  The RDBMS is very very forgiving to poor design, poor implementation and the subsequent addition of unforeseen application requirements (you want to run OLAP now we have built you a busy OLTP database – sure but do it overnight).  Will a specialist NoSQL system have the same tolerance for our incompetence?

So now I return back to the point that is really the keystone of the NoSQL motivation, “there are requirements which a RDBMS doesn’t work at all well for”.  I agree with this, but I have yet to see any quantification of what this actually means.  Is it 5% or 10% of current development projects?  And should the question really be “what percentage of development projects is the RDBMS unusable for”?  Technical purity, and even reducing license costs, needs to be balanced against one of the largest costs, re-skilling development and production teams to understand this new data platform. 

There are some clear cases, the Googles, Twitters, Facebooks etc where scale alone is clearly outside the boundaries of what is possible on today’s RDBMS platforms.  But in terms of today’s development projects, what percentage would these scalability requirements quantify?  1%?  Less?  Sure, we are going through somewhat of a data explosion and by all counts the volume of data we collect and manage in our databases is growing at an alarming rate.  So the demand for scale will continue, but let’s also not forget that the big RDBMS vendors are very market driven, and as the market changes their products will also continue to change with it.  It is very unlikely they will be asleep at the wheel and lose their dominate share of the ~$30b market without a fight.

Contrary to how it may appear, I am actually supportive of a number of NoSQL initiatives and I am even hands on with a few.  But I do have concerns about how we quantify the market, how we ensure that people are making the right decisions in choosing a NoSQL platform.  And also how do bridge the gap with skill sets and experience for developers who will have years upon years of RDBMS experience but, by nature, only have exposure to NoSQL systems periodically based on certain application requirements. 

With IBM intending to acquire Netezza the predicted consolidation in the distributed analytics market is well underway.  Recent deals include EMC/Greenplum Teradata/Kickfire and now IBM/Netezza.  A good breakdown of this deal is on Curt’s blog.  There is still more to go of course with one of the crown jewels, Vertica, still ripe for the picking. 

What this indicates is that MPP analytics has moved from the innovative edge into the mainstream market and now the more risk adverse large caps and now willing to invest substantially in growing this market.  Interestingly Microsoft made this move early with the acquisition of Datallegro in 2008, I doubt this has paid dividends yet but 5 years out this might be a different story as the explosive growth of machine generated data continues. 

While it is probably a bad time to start building another MPP query processor of course innovation in big data core technology continues to be strong.  Key areas of innovation relate to Flash/SSD optimization & caching, Graph databases, stream processing & CEP, Hadoop optimization, massive shared nothing (cloud) scalability & SQL/NOSQL convergence.  These technologies will come to market in a variety of different product forms some of which will later be picked up by the large caps.  Rinse, repeat.

There is a lot of research being done currently around optimizing DBMS storage layers for Flash based devices.  And many seem to be coming to the same conclusions, especially in relation to the append-only log-structure format.  Microsoft’s work in this space is called FlashStore and while the VLDB paper isn’t available yet this video gives a pretty good overview.

While FlashStore is a key/value store, of course keep in mind that under most DBMSs (including relational) there typically is a key/value store at the storage layer.

Back in 2008 Stonebraker & DeWitt published a paper and associated blog post titled “MapReduce: A major step backwards”.  Their key points being Map Reduce is:

1. A giant step backward in the programming paradigm for large-scale data intensive applications
   
2. A sub-optimal implementation, in that it uses brute force instead of indexing
   
3. Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago
   
4. Missing most of the features that are routinely included in current DBMS
   
5. Incompatible with all of the tools DBMS users have come to depend on

This turned out to be one of the most contentious postings in the DBMS community at the time drawing widespread criticism.  The “old men of DBMS” didn’t get that a database was not the solution for every problem and some problems just required a different type of mallet.  Even Vertica (who Stonebraker founded) seemed to distance themselves from the comments a little issuing a post affirming their commitment to Map/Reduce.  

If you read through the comments of the original Stonebraker/DeWitt post and the follow on post you will see how vigorously people were defending it.

The key example quoted when hailing the benefits of the Map/Reduce was that of the company which popularized it in the first place, Google.  Google used Map/Reduce to build its search indexes processing the immense volumes of data in batch fashion using MR jobs run across thousands of nodes.  No matter how the arguments for MR broke down the final word could always be – “Google does it” for which there wasn’t a great comeback.

Now however things have changed.  It has been reported that Google has moved away from Map/Reduce for search indexing due to time constraints in processing updates to the index and instead has opted/reverted to a, wait for it, DBMS centric approach to the problem (Google Caffeine).  Let me quickly point out that this DBMS is not a RDBMS but instead is their own BigTable distributed database (over GFS).

So, some questions are begging to be asked.  

Firstly, was Stonebraker and Dewitt right?  It is red faced time for those who came out and aggressively defended the Map/Reduce architecture?

And secondly what impact does this have on the future of Map/Reduce now those responsible for its popularity seem to have migrated their key use case?  Is the proposition for Map/Reduce today still just as good now the Google don’t do it?  (Yes I am sure Google still use Map/Reduce extensively and this is a bit tongue in cheek.  But the primary quoted example relates to building the search index which is what, reportedly, has been moved away from MR).

Finally, this no doubt will provide a shot in the arm for BigTable like open source implementations such as HBase and Cassandra.

UPDATE: Daniel mentioned he posted a similarly themed paper at HPTS last year which I recommend you take a look at.

Image by Délirante bestiole [Lumpen river] via Flickr

Data Management is an area that I work in and follow with a passion.  “Big Data” is really the bleeding edge of this, focusing on the cloud and the requirements for the high end of scale, performance and data volume. 

The Big Data field itself is rapidly evolving, maturing and broadening in focus.  It is still going through the process of finding itself, working out what it is supposed to be.  While 12 months ago Big Data was, to me at least, a categorization for the platforms that provided data scalability I think that is less so today.  Big Data is becoming more about the layers built on top of those platforms and the value added to the data in those layers.  This is not an unexpected move, it follows path in the direction of the data-as-a-service vision that I and others have shared for some time.

I see this shift being reflected in the companies that are finding success.  While true killer innovation will almost always find funding, killer innovation today often has to be more than just n+1 scalability.  Some companies I know that have built “faster transaction processing” or “more scalable analytics” have found getting a foothold difficult in a crowded market.  The “more scalable” mantra on its own is starting to not be enough to gain and keep attention.  So many platforms in both transaction processing and analytics (both SQL & NoSQL) are delivering high scalability today.  Many of these are open source, and on the closed source side of the fence it appears consolidation needs to happen for sustainability.  Some has happened already and I expect more will follow. 

I think moving up the stack provides some clear air.  A unique point of difference based around the value added to the underlying data seems to me to offer a more clearly defined proposition.  A unique Big Data platform may be built in the process, but how that platform is applied to enrich information can be more interesting than the platform itself.

Don’t get me wrong.  Killer innovation in Big Data layers form the hardware to the user are important (flash, hadoop, MPP etc) and should continue to exist in their own right.  But a difficulty is launching a Big Data platform in a busy space means the platform may only get a small following.  And platforms with small followings, I think, are difficult to sustain.

Cassandra is one of the most interesting NoSQL platforms at the moment.  And by most interesting what I really mean is the most clearly justifiable.  Some NoSQL platforms offer new data models, improved query interfaces and/or good single node performance through relaxed consistency models.  As a database guy however, the justification for throwing out the RDBMS baby and bathwater is still difficult at this point as NoSQL platforms tend to be highly focused in one aspect of data management, and very immature in all other areas.  Cassandra is somewhat different as it is more mature in a number of key areas (albeit still immature in others).  Areas that can make Cassandra more justifiable for the right project, when compared with a more traditional RDBMS based solution.  This is because Cassandra’s primary capabilities can’t easily be replicated on those traditional mainstream platforms.

Cassandra’s primary focus is on scalability.  More specifically that is scalability combined with reasonable functionality and performance & availability when at scale.  While some other platforms are trying to bolt on scalability/availability to their functionality rich data engines, Cassandra already has proven real life examples running 150 node clusters.  Notable uses of Cassandra include Digg, Facebook, Twitter, Reddit & Rackspace.  And the feedback from these sites is very good; commonly Cassandra has been expressed as the hands down winner for transaction processing performance at scale.

One of the key contributors to Cassandra has been Jonathan Ellis and until recently he has been working on Cassandra while employed by RackSpace.  But, I was pleased to hear that Jonathan, and business partner Matt Pfeil, have taken the step of setting up their own Cassandra focused company, Riptano.

Riptano are providing the commercialized support services around the open source Cassandra that are necessary for the platform to survive and grow.  While such services may be less important for adoption from the techie rich Web 2.0 crowd, for any platform to become mainstream there needs to be an escalation path for companies uninterested or unable to tinker with the code themselves.  Riptano provides those services which can allow Cassandra use to start to grow further.

Just as importantly, this move gives representation to Cassandra and provides an entity whose best interests will be served through advocacy of the platform.  While Jonathan and others had been doing a fine job of this to date personally, another corporation investing commercial dollars into advocacy will be important to ensure Cassandra’s message isn’t drowned out by more highly funded alternatives.

Riptano has received some early funding from RackSpace and I believe already has a few customers signed for their support services.  Best luck Jonathan & Matt.

I work in all markets of the database industry, from web & startup through the largest and most established enterprises.  And to be completely honest, the name Ingres has not come up in conversation very much at all.  10 years ago maybe more often, but recently not all that much.  But Ingres has been quietly ticking away.  Despite being largely off the radar, they still have a sizable and loyal customer base, global offices and a focused & dedicated management team.  And importantly they have an open source business model which actually appears to be working.

I wrote last year that their "behind the scenes" status had the potential to change.  Ingres had been very clever and worked out a partnership relationship with Peter Bonzc’s Vectorwise.  And that relationship was promising big things for data analytics from a price/performance perspective.  But at the time it was all promise and little in the way of substance had been produced.

But that has been changing.  A month or two back Ingres somewhat quietly launched their Beta program for the Ingres Vectorwise technology.  This technology, if you have not read about it before, combines an analytical column store and “vectorized processing” to give much greater throughput rates than previously possible on your existing hardware (Vectorwise is a single node solution i.e. not MPP) .

And I have started hearing feedback, and it is good.  Very good.  While Ingres Vectorwise isn’t fully baked yet, I have heard it is producing astounding performance results in early testing.  In one case I heard of <10TB real life production comparison test and Ingres Vectorwise smoked everything else they had tested.  And they have tested a lot of different market leading analytical platforms.

So I think this is the start of an Ingres’s comeback.  Certainly anyone looking at <10TB analytical platforms will be getting the recommendation that they at least look at Ingres Vectorwise from me.  I am looking forward to seeing what 2010/2011 brings for them.

While I have written about NoSQL generating a lot of buzz recently I have also written that when compared with the activity that is occurring day in, day out on relational databases it is very minor.  I would suggest those working with NoSQL databases are still a fraction of a percent of those working with more traditional relational databases.

Which is why I was surprised to read recently over on Intelligent Enterprises blog an interview with 10gen founder Dwight Merriman:

“When his company first started making MongoDB available for free downloads last year, they numbered a few hundred a month. But traffic has rapidly built up to a level of 30,000 downloads a month, he said.”

This high number peaked my interest so I quickly did a little checking on the MongoDB site.  Couldn’t find any download stats but I did notice some stats relating to the number of people who had signed up to the support forums:

• The  mongodb-user Google group has 1682 members
• The mongodb-announce group, the “This group is for releases and important updates to MongoDB that anyone running MongoDB in production  should subscribe to” has 53 members.
• The MongoDB site blog has 916 subscribers in Google Reader

I could be wrong and the figure might be accurate, but perhaps this may have actually been page views on the MongoDB site rather than software downloads?  It if is accurate then I will take my hat off to 10gen, they have come much further than I thought.

I am one of the biggest proponents of “the right tool for the right job”, and I think NoSQL databases can be the right tool in a lot of cases.  But we need to keep our heads about us also.  We still have a very long way to go before any of this NoSQL stuff is considered mildly mainstream.

BTW, I will ping Charles Babbock for comment.

* Edit: I have removed the MySQL stuff.  I understand MySQL has 70,000 downloads a day for comparison.

I have noticed a sharp change of focus in venture funding for data orientated companies over the last six months.  Many VCs have lost some interest in funding data start ups that are doing anything around relational data management.  Instead the interest is in  NoSQL technologies, from key/value stores through to Hapdoop based data management layers.

I am highly supportive in the development, and therefore the funding, of a more diverse set of big data technologies than those based on the relational model alone.  However I also advise caution to not throw the baby out with the bathwater.  Relational data management technologies continue to be a focus of innovation. There are companies working on game changing step forwards which have relational under-pinnings.

The relational model is going to continue to be the underlying model of most of the worlds structured data for the foreseeable future.  Many opportunities for innovation exist and will continue to exist around this fundamental model into the future.

A mindset that relational is yesterdays technology and non-relational is tomorrows defies conventional wisdom and will lead to great opportunities being missed.