Microsoft has announced its Roadmap for integrating the various DW acquisitions it has been doing recently, including Datallegro.  Part of that roadmap includes their “Gemini Project”.

Gemini is the codename for the next major update to Analysis Services.  It includes an in memory, column based storage mode which allows analysis to be performed predefined models, but without the requirement to pre-aggregate or build cubes prior to querying.  Gemini is expected to support the XLMA and MDX query languages so applications that work with existing analysis services cubes could in theory be made to work with Gemini.

Microsoft is again pushing Excel as the query tool of choice for Analysis Services.  This was the approach a few years back also, but the interface was far too limited and troublesome to be used for anything but the simplest requirement.  The “grab and go” ad-hoc reporting ideal was nice in theory but wasn’t really been successful to date due to the complexity in business logic required to roll figures up into neat little KPIs.

More details on Gemini aren’t yet available but will begin to surface now the announcement has been made at their BI conference.

A week after all the hype of Oracles/Amazon’s EC2 cloud offering, Amazon has also announced that it will be providing an EC2 SQL Server Cloud offering shortly.  As previously discussed these aren’t really what most people think of as “cloud databases” instead they are just self provisioned hosted solutions. 

Microsoft is probably saving the fanfare until the service is actually available.

Mongo is a proprietary object database developed by 10Gen as part of their 10Gen application stack.  10Gen is a startup that is trying to position its application stack as cloud platform for Web 2.0 style applications.

Like many of the emerging Web 2.0 database platforms (SimpleDB, BigTable, SSDS) Mongo looks to improve scalability by simplifying functionality over a traditional RDBMS database.  But unlike SimpleDB and BigTable, Mongo aims to do this as a true object database rather than the key/value pair system of the previously mentioned databases.

Mongo is yet unproven and has many hurdles to jump before it becomes an accepted database platform.   The first is convincing the development community that they have solved all of the major object databases issues which have prevented them become popular over the last 20 or so years.  Additionally a significant challenge will be getting Web 2.0 startups to build their services on top of a platform which itself is a start up (building your stack of cards on another stack of cards) rather than a mainstream database vendor.  The available technical documentation is very light on detail and will need to be significantly improved before adoption increases. 

The alpha version of their Cloud platformbecame available last week.

There has been a lot of recent talk about MapReduce, particularly in relation to its addition to several of the specialized data warehouse platforms.  In this post I will try to answer the question of what is MapReduce and how does it relate to a relational RDBMS (at a high level).

MapReduce

In simplicity MapReduce is a framework that allows developers to write functions that process data.  There are two types of key functions in the MapReduce framework, the Map function which separates out the data to be processed and the reduce function which performs analysis on that data.  MapReduce is a logical concept, it is not a technology owned by any one vendor but it was made popular by Google as it is part of their search engine core technology.

A very simple example of MapReduce that is commonly used is a process that counts the occurrence of a particular word in a document.  The “Map” function in this example would produce a set of data that contained all occurrences the desired word from the source data, the “Reduce” function would then count the number of items produced by the Map function are return a numeric value indicating the total number of word occurrences.

While MapReduce seems very simplistic in its structure, it has been found that this type of 2 stage processing can be used to answer a large number of varied data analysis questions.

MapReduce Scalability

The real benefits of MapReduce start to occur when the framework for the execution of its functions is implemented in a large scale, shared nothing data cluster.  The platform that implements a MapReduce framework can abstract the complexity of running distributed data processing functions across multiple nodes in the cluster.  This allows a developer with no specific knowledge of distributed programming to create their own MapReduce functions and have the platform run those functions in parallel across multiple nodes in the cluster, and then also handle the gathering of the results from across the cluster to return a single result or set.  The platform can also abstract important cluster functions such as dealing with nodes that fail during execution, or nodes that are slow to respond during execution.

Again Google is a common example of a large scale implementation of a MapReduce framework.  Google is said to run clusters of 10,000 + shared nothing nodes with PetaBytes of data, yet a developer can reportedly write a query that performs analysis across data in these clusters within a relatively short space of time (e.g. 30 mins) as they are purely concerned with the data analysis details not the physical execution details.

MapReduce and RDBMS

MapReduce’s integration with traditional RDBMS’s is now occurring, with some initial implementations been undertaken by some of the specialist data warehouse platform vendors such as Greenplum.

Initially the benefits of MapReduce within a RDBMS are less clear as a distributed shared nothing RDBMS already has a mechanism for abstracting the execution of requests across nodes from the developer, through the use of SQL and the underlying query processor.  There is a lot of debate occurring in both academic and commercial fields as to if the inclusion of MapReduce in a relational RDBMS is a positive measure or not.

For SQL and the query processor to be effective, the data has to be structured and have a pre-existing, well defined schema (tables, columns etc).  Once this is done optimization techniques, such as indexing, can be implemented making SQL an optimal method for accessing and processing that data.  However when the data is unstructured (i.e. without predefined schema) the ability to process that data using SQL becomes a lot more limited.  MapReduce on the other has no assumes no predefined schema which allows the developers to create functions that make their own assumptions about the schema definition of the data they are accessing.  It is common for a RDBMS to store unstructured data in the form of a BLOB (binary object), examples of this being documents, images, XML files, binary files.  A MapReduce function can be written to perform data processing across that unstructured data, for example counting words in a document or finding nodes in XML files.

Much of the debate is centered on if the data itself should be unstructured or not, i.e. with or without schema.  If the data does not have a pre-defined schema (unstructured), then processing on that data cannot be validated by an external mechanism, such as the RDBMS.  Therefore it is up to the individual functions that make use of that data to verify its integrity.  Also without predefined schema, optimization mechanisms such as indexes cannot be created which means the processing of the MapReduce function uses a “brute-force” approach and scans all relevant data across all relevant nodes during the execution of the Map function.

However when the data is structured into tradition database tables and columns, the benefit of MapReduce over traditional SQL for processing that data is less clear.   Some of the discussion in this area focuses on the fact that MapReduce functions can be written in native programming languages (perl, java etc) which are more familiar to developers than the SQL language.  This argument seems a little weak as due to the universal implementation of SQL across all RDBMS platforms, any experienced developer will have considerable SQL experience.

Summary

MapReduce appears to be a simplistic framework for data analysis across shared nothing clusters.  The clear benefit of MapReduce is when that analysis is taking place across unstructured data.  When structured data is being analyzed the use of SQL and the query processor seems preferable as this makes use of countless optimization techniques, such as indexing and join processing.  And finally, much of the debate against the use of MapReduce is less focused on MapReduce itself and more focused on if the underlying data should in fact be structured into table/columns within a traditional RDBMS.

References

http://en.wikipedia.org/wiki/MapReduce

http://www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html

http://www.dbms2.com/2008/08/26/why-mapreduce-matters-to-sql-data-warehousing/

http://www.greenplum.com/resources/mapreduce/

During this week’s OpenWorld Oracle and HP announced their Exadata offering, a hardware/software solution for offloading I/O processing to proprietary storage nodes.  Thanks to Curt Monash for the heads up, everyone was expecting “a major innovation in database” to be announced this week, I almost missed this.  Silly me, I was looking for the “major innovation” as being the first time someone has done something, rather than the marketing version of “major innovation” which is the first time “we” have done something.

Exadata is essentially this.  Take an Oracle Database node or RAC and instead of plugging it into a dumb SAN, plug it into a bunch of intelligent storage “cells”.  These storage cells are optimized for disk throughput but importantly they also offload much of bulky database server processing.  They are able to pre-filter datasets (filtering rows & columns) so that a lot less of this processing has to be done on the database node(s) themselves.  Then the whole solution is tied together using a highspeed interconnect, infiband, which also allows the storage cells to directly insert their resulting datasets into the right spot in the databases servers process bypassing many layers in the stack.  The end result, much higher I/O scan speeds, a claimed 1GB a second per cell.

Because the solution is so hardware dependant, Oracle has partnered with HP to make this solution available as an “appliance”.  The first such available is the HP Oracle Database Machine, which is a single rack solution that comes with 14 Exadata Cells (1TB each so 14TB), 8 Database servers and the necessary infiband interconnect.  Racks can then be joined up using the infiband interconnect to give much larger storage potentials.   The infiband interconnect is also used as the interconnect for RAC.

This solution has its own unique points but the fundemental approach is in vain with those which have been happening elsewhere such as with Netezza, DATAllegro etc.  Interestingly since Microsoft announced that they intended to acquire DATAllegro they have had positive feedback for being the first major vendor to progress down this path.  With Oracles product largely done and Microsoft’s due 2010 this has changed and raises a few questions as to if the acquisition may have instead been a late reaction to Oracle’s undertakings.

Also during OpenWorld Oracle announced its Amazon EC2 Cloud offering.  “Cloud” has really now become a buzz word that really just means any service that is available across the web.

Oracle’s Cloud offerings seem to be their standard apps hosted by Amazon wrapped up with provisioning tools and so you can quickly create a new hosted environment.  But it seems no new licensing models have been created, you pay Amazon for the hosting fee but you still have to acquire traditional Oracle license for their software.

This is quite different to what else is commonly called a Cloud Database offering, such as BigTable, SQL Server Database Services, SimpleDB etc which are functionally stripped down but massively scalable databases (offered as service or DBaaS) targeted to Web 2.0 style workloads.

Not trying to be negative on this, just I think this is not anything particularly new (it used to be called ASP).

The US Department of Labor has published statistics that show demand for Database Administrators (DBA) is growing at a rate of 29%, a much faster than the rate across other occupations.  This is fitting with what we see in Australasia and also as observed in Europe.  This demand is growing due to the data explosion affecting most business, with data becoming a key primary assets and competitive advantages being defined through timeliness in analyzing and leveraging that asset.  With the increase focus in data assets the volume of data and the number of database servers have proportionally grown, with the RDBMS market revenue growing at around the same 29% rate between 2005 and 2007. 

Traditionally a database administrators role was to implement, manage and maintain the systems that contain an organisations data assets.  More recently due to the volume, scale and performance requirements of the data an organization possess , a DBA is required to provide expertise into projects that are looking to drive new value from these data assets.  This is effectively converting the DBA role from a role that has a $0 return (a business as usual (BAU) support role doesn’t generate new value, it just prevents loss of existing value) to a role that is value creating.  This two pronged demand is pushing DBA rates higher and also bringing down the average experience level of a DBA available as the supply of new DBA talent ramps up to meet the increasing demand.

One of the key traps that many enterprises fall into is their DBAs lack the availability to participate in value generating projects.  Predominately this is occurring because of the increase in data assets has had a near linear increase in operational BAU requirements.  Meaning more data requires more DBAs to simply keep things running.  While DBA teams often have the directive to introduce improvements to reduce BAU requirements (thereby giving them availability) these measure often only reduce operational requirements at the same rate as growth in the data assets is increasing, effectively cancelling out any tangible benefit.

This is the focus of RockSolid.  With RockSolid our goal is to mitigate as much as possible the BAU requirement that goes with managing an increasing number of SQL Server data assets.  Through virtualization of management and automation we have been able to reduce the BAU requirement of managing a SQL Server instance to a nominal factor, meaning the DBA time requirement to operationally managing 150 production instances for example, is very similar to that required to mange 10, 20 or 50 effectively making DBA operational overheads a near constant.  This results in freeing existing resources to focus on value creating initiatives and any further investment in DBA resources can be balanced against the value that will be produced from that investment.

IP protection in software has always been a difficult area to manage largely due to so much of software development occurring on the basis of iterative innovation rather than outright invention.  That is, a developer often creates software that does something clever to address a shortfall in existing software.

The reason I am brining this up is due to a former XPrime founder filing an accusation of patent infringement on a Datallegro founder.  This filing occurred with a couple of weeks of Datallegro announcing that it was to be acquired by Microsoft.  Those of you who don't remember XPrime, they were around between about 2003 and 2006 building an appliance query accelerator for SQL Server.

The patent at the centre of this issue really relates to distributed data and distributed query processing.  This patent seems to be related to horizontal partitioning (rows) not vertical partitioning (columns) as reported by some media outlets.  This patent is in the area of distributed databases, commonly called distributed partitioning, database federation, parallel databases or shared nothing databases.  The actual claim of the patent relates mostly to proportionally distributing tables between nodes (round robin) and a method for transmitting the relevant section of each table to each node for the purpose of joining those tables, then transmitting back and constructing a resultset.

On a side note, 2004 is very late in the day for patenting what appears to be a very simplistic distributed database strategy.  When this patent was filed in 2004 most major RDBMS vendors already some form distributed database processing in their products (I couldn't find reference in XPrimes patent to this prior art).

In Datallegro’sown patent application they have referenced a similar stratergy as above in discussion of the existing parallel database methods in the lead up to the discussion of their new innovation (prior art).  In my view, the Xprime patent comes across as a very basic high level conceptual patent for a shared nothing database, the Datallegro patent application is more practically focused and introduces concepts to address real world scalability issues (vertical partitioning, query results caching, distribution of dimension tables, hash partitioning rather than round robin).

If these “iterative innovations” are enough to justify new invention in a legal sense, we will find out.

I was on major domestic airline last night, flying from Melbourne to Sydney (a pretty standard flight for me). Notice in front of me a couple taking their seats. Then a man got walked down the isle, looked at his ticket and questioned the couple as if they were in the wrong seat. No they weren’t it appeared the two people had been allocated the same seat. Moments later another gentleman walked down the isle ticket in hand, and again questioned the couple as to if they were in the wrong seat. Again no, so now three people had been allocated the same seat. A few minutes passed as a slightly stressed flight attendant tried to work out how to fix this when a woman who was running late hurried down the isle, politely excused herself around the queue of people standing there, looked at her ticket and questioned the couple if they were sitting in the wrong seat. 4 people had all been allocated a single seat which of course was a major issue on a full flight.

Now the interesting thing is why this happened. You may put this down to software glitch, but it is more than that. Without doubt all of the check-in information is sitting on-top of a relational database platform. Now the crux of the matter is, if the application developer of the check in system was using the most appropriate isolation level and undertaking proper transaction control, this could not have happened. It would have been possible for 2 people to end up with the same seat, let alone 4 people.

But it did happen. And why is that. Well firstly, it could be just a stuff up of the application vendor, however double ups happen all the time so if this was the case then likely this particular issue would have been fixed long ago. Instead, a much more likely cause is that the application vendor and/or customer has made the decision to forgo strict transaction control for better concurrency, that is better performance during peak times. Strict transaction control results in reduced concurrency as one transaction often has to wait for another transaction to complete processing before it can in turn complete processing. So when you are checking in loads of people on loads of airplanes, all within a short period of time your primary concern is ensuring you do this as quickly as possible. Obviously the decision has been made somewhere that doing so quickly is more important than the occasional double allocation of a seat which the flight attendants can usually fix up without too much issue.

Why I am bringing this up, is that this is the model that is currently in favor and more and more systems used on scale are moving towards. In my “Performance Trumps Everything” article I spoke of how the desire for mass scalability and performance is causing the deprecation of age old features such as strict transaction control. While this model does present benefits for many scenarios in the data tier, this is a clear real world example that if you go down this route you have to take on extra responsibility in the application tiers for ensuring data integrity is maintained.

With an estimated 80% of SQL Server installs still on 2000 over the next 6-12 months there will be large numbers of upgrades if for nothing more, 2000 and the hardware 2000 is running on reaching end of life.

The question will be when doing these upgrades, do they go 2005 or 2008.  Features will come into it for some of the upgrades but not most.  The real trick for Microsoft is to get as many major application vendors certifying their products for 2008 as quickly as possible, so when the upgrades start most of the core apps are dealt with.  And also they would be wise to get Seervice Pack 1 out early, as so many sites wait for that as a physiological barrier.