Inspired by TechCrunch I have put together my own “can’t do without” list as we head into 2009.  This of course skips over all the boring by necessary productivity software (Office, Visual Studio etc).
The following is my list of technology products that I use on a daily basis:

• Blackberry Bold 9000
• Facebook
• FeedDemon
• Flickr
• Google Search
• TweetDeck
• Twitter
• TwitterBerry
• Typepad
• Wikipedia

BlackBerry Bold – I have tried a lot of mobile devices and the BB Bold is the one I have liked best so far.  I would not use another Windows Mobile device at this point, some major improvements would have to be made before I ever touch one of these again.  In saying that, the BB Bold has crashed quite frequently but I have just updated the firmware which apparently fixes this – here’s hoping.

Facebook – I had a Facebook account for years before I started really using it.  It wasn’t until after a lot of non techie friends signed up that I really used it and now it is a good way to stay in touch with people who are all around the planet.

FeedDemon – I think I must be one of the few people still using this instead of Google Reader.  I haven’t actually taken the time to look at Google Reader yet.  Shame on me I know.  Feed Demon is what I monitor all feeds in.  Works ok but sucks memory up from time to time.

Flickr – Have a pro account and keep a copy of all my photos here.  Mix public and private for everyone, family and friends.

Google Search – What can I say, I Google things probably 50 times a day on average.

TweetDeck – This isn’t as good as what it could/should be but the interface is better than what comes natively with Twitter.  Real time tweets stop me having to hit the refresh button constantly and groups allows some sort of order to be applied.

Twitter – It took me about a year to get into Twitter.  I now see a clear distinction between blogging and micro-blogging and Twitter is now the only place to micro-blog.

TwitterBerry – A lot can happen on Twitter in a short space of time, the only possible way to stay up to date is to keep an eye on things regardless of where you are.  TwitterBerry allows me to do this from the Blackberry.

Typepad – For some reason I chose Typepad as my blogging platform in 2002 and I have used this for all my blogs since.  Typepad hosts everything and allows me complete control over layout, does everything I need and I am very happy with it.  Has an API so I even integrate some of the blogs with twitter.

Wikipedia – I have a pretty specific area of knowledge which means I need to look other things up, often.  Regularly I note that when I Google something if there is a Wikipedia link for what I am searching for I go straight to that.  Also more often I am searching directly on Wikipedia rather than Googling.  Wikipedia is great.

And that’s it.  I will do this list next year and see how much stuff changes.

Image via Wikipedia

JournalSpace, a blogging website which had been around for 6 years but failed recently when their server died and they realized they didn’t have any backups of their users data.  This is bad for them and bad for the users of their services as presumably thousands of blog posts (their users data) have been lost.

Their defense was nonsense, they essentially blamed someone who didn’t work there any longer.  Phftt, a business whose sole purpose is to manage data which doesn’t have a sound backup strategy is the fault of the entire business not of one no longer employed IT guy.  Ultimately it doesn’t matter whatever the excuses are because regardless it seems the data is gone.  As we move more and more data to web services we are going to see more and more occurrences of this.  The problem is, the difference between a ‘web site’ going boom in the 2000 crash and a ‘web service’ going boom in the current climate is that web services are much more likely to eat data that matters to you.

This highlights a key issue as we move more data into a cloud services world.  While we are beginning to transition the management of information from our direct control to the control of a service provider, at the end of the day who has the ultimate responsibility to make sure your data is backed up?  Should you or your business trust your service provider to undertake a recovery strategy that will ensure you will always be protected from data loss?  You can’t see what is happening behind the scenes in their data centers so do you trust that they always remember to put the tapes in or do you trust that they are doing off sites in case of fire?   Do you trust that they actually have a data center and that their web service isn’t just been served up from someone’s PC under a desk somewhere?  Maybe you do, but do you also trust them to pay all their bills on time and not be shut down and disappear overnight?

I think the key learning here is that even in a web services world the only person who is going to have the same level of care and responsibility for your data as you do is you.  If your blog posts are important then you should have them backed up somewhere.  If your Amazon S3 data files are important then you should have them backup up somewhere.  If your Flickr photos are important these should be backed up somewhere.  If your tweets are important, or your Facebook friends list, and so on…

Good web services providers will provide good data recovery but will also understand that users are going to want the ability to provide additional protection to their own data.  They should in fact make it easy for users to do this by be providing the necessary API’s or interfaces to allow third party web services to hook into the users data and make copies of it elsewhere.

There are a lot of great web services out there that allow you to publish, store and share your data in ways like never before.  But remember that your data is yours and it is up to you to make sure it is protected whatever circumstance may arise.

So it’s that time of year again when everyone puts out their predictions for the year ahead.  I think predictions are a bit of a waste of time because to be interesting predictions have to be big, but a year really isn’t all that long so actual changes over the course of 2009 are likely to be just small progressions.  So instead I have been thinking about the top issues that we face heading into 2009 and here is my Top 10 list for issues in Data Management.  In this post I avoid offering solutions to these issues, while I have several ideas on solutions these can be the subject of subsequent posts.

10 – Limits on Scalability

While scalability is on my list it is at number 10 because against popular belief, scalability is only an issue for a very small number of data based applications.  Almost all data based applications in use today can be scaled without major issue by increasing the underlying hardware resources.  But for those applications for which it is an issue it is usually a major issue, and the most common category of data based application that has such scalability limitations is internet applications and web based services.

The problems with scaling this type of application are firstly, the scalability requirements are hard to predict in advance and secondly they can change instantly based on sudden popularity (aka the Slashdot effect).  So currently you either over invest in infrastructure and hope for the growth or you under invest and hope you have time to rapidly add capacity if/when required.  When planning “corporate” applications you usually have the benefit of capacity planning projections and the ability to manage rollouts to ensure scalability limitations can be managed and while you try to do this in a web world by nature any projections you do today can be meaningless tomorrow. 

Some cloud database platform vendors are looking to help alleviate this by providing solutions that offer on demand scalability.  These solutions come in two flavors, firstly running existing database platforms on a virtualized infrastructure and secondly providing a non-relational key/value pair repository (SimpleDB, SSDS etc) where all underlying infrastructure and database operations are abstracted.  While these represent positive steps forward I think the issue of completely abstracting infrastructure scalability to an application remains an issue that will receive attention in 2009.

9 – Constantly changing landscape

Vendors are changing/updating platforms faster than organizations and their application vendors are willing or able to respond with migrations to the new platforms.  When combined with applications that only work with a particular vendor many organizations have multiple data management platforms with multiple versions of each platform in production use.  For example, enterprise organizations that use SQL Server often have SQL Server 2000, 2005 and 2008 servers in production (as well as multiple flavors of Oracle, DB2 and so on).  Different vendors and different versions within vendors create a major management overhead.  This impacts areas such as consolidation, required skill sets, management processes/policies, tool sets.   It also restricts the level of value adding exercises that are taking place by data management teams due to their limited resources being consumed with functions surrounding operational management.

8 – Data Recovery

Recovery is something that people have focused on since we first starting collecting electronic data so how is this still an issue?  Well as the demands for electronic data grow the factors that control how data is recovered in the event of failure become more complex.

It is easier to recover an entire data set than just a specific part of it.  This is fine if you lose an entire data set but not so fine if you need to recover only a part of a data set.  New factors such as pressure to have 24×7 availability and an increase in purely electronic transactions with no paper “backup” mean that recovery of individual records/transactions presently can be a very difficult and time consuming process.  Due to the complexity of doing transaction based recovery, most database applications no longer “delete” any data instead you a status flag to indicate that data is in a deleted state.  Changes to the data are also logged as change history which is also infrequently actually deleted.  This allows for application based recovery of that transaction but significantly contributes to the increasing data volumes being experienced.   Methods for ensuring individual pieces of data can be rolled back without keeping it within the highly used data set perpetually is a complex issue that will take time to resolve.

7 – Increasing Data Volumes

Data Volumes are growing rapidly.  This graph shows the average database size across 3000 databases for ~ 30 random organizations.  This graph shows a 12% increase in the average database size in the last 6 months of 2008.  

While storage is cheap such rapidly increasing data volumes brings issues other than just pure storage costs.  Backup/recovery time frames, maintenance timeframes, query/access performance, increased CPU & Memory requirements (for batch processing etc).  This would be less of a concern if the data was adding value (see the low leverage of data topic above) however today in most database applications, much of the data contained is non active historical data that is there solely for the purpose of future reference should it so be required.  Managing the vast volumes of data to ensure the balance between data availability and manageability is maintained moving forward presents an interesting issue.

6 – Lack of trained people

Finding qualified and experienced people in IT has always been an issue, in the area of data management this is especially true.  Data management is a niche field and the rapid increase in requirement for data management experts has not been met by an increase in the number of skilled people in the workforce.  This has led to skills shortages and organizations being forced to take progressively less experienced and qualified staff onboard.

Lack of skilled resources has a flow on effect which impacts all aspects of data management including performance, security, availability and recoverability.  Managing an increasingly complex set of data management problems with a depleting set of expertise is another interesting issue heading into 2009.

5 – Inefficient use of Resources

Modern database systems are typically implemented in a manner which sees a physical server provide a platform for one or a small number of applications.  This approach has had its benefits from an implementation perspective including, the complete costs (hardware, software, license, management) can be easily calculated and assigned to a project/department, the risk of impacting other applications is low, sizing an environment is easier as only a single workload is taking into consideration and so on.  However due to the continual increase in applications this has lead to large numbers of database servers being implemented.  There are commonly dozens of such servers in small organizations, hundreds in medium sized organizations and even thousands of database servers being implemented in large enterprises.

One of the several issues with this approach is that it leads to significant inefficiency in resource utilization across the entire infrastructure.  If we define “resource usage” to mean the percentage of available CPU, Memory and I/O capacity in a server we can graph a typical analysis of usage in an enterprise: 

This shows us that a typical picture is around 50% of all servers have a 10% or less average resource usage (i.e. half the servers are only having at most 10% of their available resources used).  30% have between 10 and 20% usage and so on down to 5% of all servers having 90% or more of their available resources utilized.  This means across the entire environment there is an average resource usage of 32% meaning 68% of resources are not being put to work.

4 – Auditability

Organizations are under increased pressure to audit every action that a user performs within a database.  This is due to increased focus on security, risk, accountability and avoidance of fraud and corruption.  While security prevention measures (logins, firewalls, tokens etc) are important to prevent unauthorized access to the data in the first place, as this survey shows, most breaches occur by users who are authorized but are either negligent or malicious.

The problem with auditing is that itself generates a lot of data, potentially much more than even the database that it is monitoring.  Also the act of auditing a database can cause a significant performance load on the database being audited.  I think this is a particularly important issue facing enterprise organizations especially heading into 2009.

3 – Data Security

Security is always an issue for data.  The biggest issue facing data security I think is not actually securing the information that is contained within a database, but ensuring security is maintained on the data once that is has its original data source.  As we expand ways in which a piece of data may be consumed or modified, through methods such as API’s, web services and other integration means, currently it is up to each step in that integration chain to ensure appropriate security is provided. 

To date this is typically controlled by users making access decisions for services based on the data they own, and a mixture of specific copy/access protection systems in some cases for data the users don’t own (such as DVD’s, music and software).  However heading into 2009 a significant issue around how universally groups of data can be secured and have that security survive distribution remains open.

2 – Decentralized Data Management

When we used to talk about data management we almost always were talking about database management.  Even today this is mostly the case.  But this is changing.  Data is becoming more distributed and the source of data to a “data consumer” may not necessarily be a database but instead may be a “data service”.  How the data service is made up and if it has a database under the covers is often irrelevant to the consumer as the service rightly abstracts the underlying architecture.

Data services are commonly used today as data sources or destinations and data is pushed and pulled from service A to location B though integration processes (e.g. old world ETL processes).  Due to data volumes and the need to have timely data, pressure is mounting to instead build applications that use the data services directly as the data source and perform any necessary integration in real time.  This reduces unnecessary data duplication and increases the timeliness but creates a whole set of issues to do with consistency, recoverability and availability that will require resolution.

1 – Low leverage of data assets

Organizations have a lot of data.  A quick survey across some random business finds that on average a SMB has about 500GB of data in total in their database systems.  An “enterprise” is much more difficult to average as this will range from the 10’s of TB well into the PB’s (PetaBytes).  And of course there is large volumes of data outside of the database systems in files, email etc also.  But at the moment for most organizations much of this data is used purely operationally, which means the data is being used for the application in which it was created but no additional value is being derived from this data through wider analysis.

It used to be that competitive advantages were gained by moving inefficient manual processes to more efficient alternative through the use of technology, heading forward I think these gains will be less and the larger competitive gains will come from using the collective knowledge to understand and serve customers better.

And that is my list of the top 10 issues in data management.  I had a starting list of about 30 issues which I whittled down to 10, I am reasonably happy with this list but on a different day maybe a few in here would have been swapped out for some alternatives.  Anyway, I look forward to your comments.

NOTE: The reference data I use in this post is just data I have quickly pulled together or observed, it hasn’t been formally researched or validated and should not be considered fact.  This is a blog post not a research project!

Recently I have seen several articles about Solid State Drives (SSD) and some articles are picking 2009 to be the year of the SSD.  While I agree SSD’s will bring some great benefits in the years to come, I wanted to clarify the current state of SSD’s in relation to usage in the database field.  A primary misconception is that because SSD’s are a type of “memory chip” storage that they are super fast, like system RAM and using them to house a database will make the database super fast.  SSD’s are very different from system RAM because of course their data must be persisted and this requirement carries a major performance impact. 

SSD’s offer high read performance benefits over current mechanical (magnetic) HDD’s.  As much as 20x’s as fast.  However write performance on SSD’s is slower than current HDD’s, in fact HDD’s can be up to 10x as fast to carry out random writes when compared to SSD’s.  This is due to a number of limitations involved in writing to persisted solid state memory, including the fact the SSD sectors cannot be overwritten but instead the whole sector has to be erased and rewritten. 

The benchmarks available are a problem and it is difficult to get clear results.  This is for a few reasons.  Firstly, because both the SSD and HDD are two technologies undergoing rapid evolution any benchmark published quickly becomes out of date and irrelevant.  Secondly the benchmarks are often published by advocates of either technology rather than completely independent bodies.  Some benchmarks I have seen are showing the speeds of high end SSD’s compared with low end HDD’s and vice versa.

So when considering SSD use for improving Database performance, make sure you consider the following points. 

• SSD’s generally have better read performance than HDD. But make sure you are purchasing current SSD technology as you may find older SSDs have slower read performance than current HDDs.

• SSD’s generally have worse write performance than current HDDs.  If you have a write heavy workload and your silver bullet is SSD you may be disappointed.  There are exceptions but ensure you are comparing current with current.

• If you are able to split data based on usage patterns such as partitioning or file placement, moving highly read, low write data to SSD storage can provide significant performance improvement.

Some proponents are picking that before we see pure SSD storage replace existing HDD’s we will see the emergence of hybrid devices that contain SSD and HDD technologies.  These devices will have hardware algorithms that make a determination about where a particular sector should site based on its usage profile.  Highly read low write sectors would sit on SSD and highly written sectors on HDD, of course all of this transparent to the system using such a disk.  If such a hybrid device actually becomes mainstream or is skipped altogether due to rapid advancement of SSD is still to be seen.

Image by akakumo via Flickr

There has been some debate on a few sites the last couple of days about what a cloud service actually is. 

My definition is as follows:

• Infrastructure is abstracted (from applications and users)
• Scalability is unlimited (near linear scale out)
• Environment is multi tenant, but provides isolation

 

While there have been hundreds of posts recently about how to stay positive and how to survive a down turning economy in the tech sector, I thought I would also add my own perspective as someone involved in the technology innovation for the past 15 years.

So a significant crisis in the global economy is going to affect the dynamics of almost all industries.  You all know change is a constant force in business and in the tech sector change is also a constant reoccurrence over much shorter cycles than a lot of other industries.  But in a significant economic event such as we are seeing now the change is even more rapid and unexpected than usual and those who are too big to meet the change quickly, fail to adapt or can’t adapt for other reasons are going hurt significantly if not fail completely.

Too many businesses try and ignore change.  I have countless organisations that were previously successful fail when change occurred, by attempting to ignore the change or ride it out.  Their approach was to keep trying to do what they had done before.  And when that didn’t work they continued trying to do what they were doing before but cut more and more staff, slash marketing and new initiatives until eventually this approach killed their ability to produce, innovate and generate sales and they went out of business.  On the flip side radicalism is risky also, I once observed a large timber company try and become a web site company when the bottom fell out of the lumber business with dire consequences.  Recognizing change, having the guts to acknowledge change and making changes in strategy to adapt to change is a difficult but an important process that is required to survive long term.  Of course slash the fat and cull unnecessary expenditure, this is also what your customers are wanting to be and will be looking for providers who can help them achieve this. 

Creating technology that significantly helps organisations do more with less should do well right now.  As an example, since the start of the current downturn we have seen interest in our RockSolid product steadily increase.  Our customers are trying to do what everyone is trying to do, improve efficiency through cost reductions and increasing value from existing assets.  For our customers DBA staff are significant costs and reducing/re-deploying or gaining additional value from them is a primary issue.  Our product helps remove the high staff cost component from the operational roles (low value producing roles) allowing the staff to be used in a higher value generating capacity.  The managers in the field themselves are driving this initiative as they know if they don’t they will likely just receive a blanket order to cut staff by n% which will result in reduced costs but also a reduced ability to deliver the current level of service.

Doesn’t all technology look to reduce costs or increase value in some way?  Sure in theory, but the characteristics of what will do well right now is more specific.  The current climate has come on quickly and a lot of businesses weren’t expecting it and weren’t prepared.  Therefore people are looking at improving efficiency (without destroying value) using solutions that have an immediate or short term path to ROI.  It is too late to put in a new system that will improve efficiency in 2 years if you have significant cost pressure now.  The ROI that you are offering to your customers has to be proven, significant and measurable.  By nature SaaS offerings are well positioned to gain ground.  Most SaaS offerings have a low capital outlay (a monthly fee usually) and a quick path to producing a positive ROI so are by nature much lower risk than a full perpetual purchase of a product that may take 2-3 years to see that realized.

So my advice at this point is simple.  Improve your own efficiency but don’t lose your ability to produce/compete.  Put a focus on your existing technologies that provide significant short term returns.  If those returns are not clear then invest in making them clear through customer case studies, market analysis etc.  If your current technologies have a high initial investment cost (infrastructure etc) look at ways you can help mitigate that by, for example, providing your customers a SaaS alternative.  Also if you have “mammoth” products that have a long implementation timeframe, look at ways you can componentize these and provide your customers small, quick and practical measures to help them achieve their own efficiency goals quickly but can be built upon longer term.

In this previous post I wrote a section on Web 2.0 data platform scalability, in it I mentioned the importance of scale out.  I also mentioned that scale out was important but currently it is hard.  There really aren’t a lot of “elegant” or “natural” ways to scale out a database if you are using any of the mainstream database vendor products (SQL Server, MySQL, Oracle has grids but isn’t shared nothing scale out so is limited).

A question I was asked was “Surely this has all been sorted by now through their widespread use in the enterprise?”.  Good question but unfortunately the answer is no.  The reason behind this is simple.  There are very few enterprise workloads that cannot be satisfied with a scale up (big server) solution.  Maybe 1 in every 20,000 databases will have a workload so great that it can’t be satisfied by a single server, in fact most enterprise databases will happily share servers with other databases.  The biggest ERP’s and online share trading applications that I am aware of all are provided for by a single big server solution.  Quite simply, none of these applications have the workload requirements of tens of thousands of concurrent users that a Twitter or Facebook has.

So this issue really hasn’t been addressed in any adequate way by the mainstream vendors.  Later on I will post on why the current products are not suited for scale out, and the efforts in progress to rectify this.

Image via Wikipedia

For the purpose of this article “A Data Platform” is what I use to describe an implementation of servers, database management software, a data model, and any surrounding classes that separate data from user facing application logic and functionality.

I have noticed that sometimes startups are choosing a data platform based on factors such as purity of the development model, and the familiarly of the coding model to developers which in theory leads to increased speed of development.  While these might seem like important things, this really isn’t a great method of deciding what will be the foundations of the entire offering you build.  I mean would you choose a location, design and construction method for building a house based on what suits your builder best or what will serve your daily domestic needs and be flexible for you and your family now and into the future?

Your counter argument to the above might be that time to market is all important, therefore going with what is the quickest is the best idea and you can always retrofit things later.  Sure I agree time to market is important but probably more important is, once you have hooked people in, the speed at which you respond to user feedback, adapt your application based on user usage and add new and advanced features to stay ahead of your eventual competition. 

You might be building a user focused web application, so you might want to use Java or .NET or some other OO language, so you might think it is a good idea to choose data platform X or Y because that is the best data platform to rapidly build this style of application with your chosen development toolset.  But really that is making decisions based on a subset of requirements, your applications features and functionality.  While no one can predict the future you do need to look a little bit past the end of your nose and think about a few other key requirements that will have to deliver on if people actually start to use your application.

Scalability

If you are actually successful then you will need to be able to scale.  Not over months and years, but over days or weeks.  You will need to be able to rapidly add capacity to your data platform to continue to meet the performance expectations of your rapidly growing user base.

Unlike application layers, scaling the data platform is hard and complex.  Usually the application layer can be easily scaled by just adding more and more application servers into the pool.  Doing so at the data layer is much more difficult as you have all sorts of issues around data availability, consistency, latency, redundancy and so on.  If your cunning plan is to work this all out later you may be unhappily surprised because it could take you months to retrofit your application to a more suitable data platform and while you are doing this Joe the Competitor may scream past you at a hundred scalable miles per hour.

Anyway, while we are on the subject, you don’t care about scaling up (i.e. getting the biggest server you can get).  The type of scalability (and availability) solution you will need isn’t going to be possible with a single mega-server so don’t worry about this approach.  You need to be able to scale out the data platform, to be able rapidly and continually add more and more capacity into your data layer.  This approach also has the plus side of allowing you to start small while you are still paying for this all on your Discover Card, but when you take off and get your first million from Fred Wilson you can start rolling in capacity day and night.

And of course you need to think about availability while doing this.  Most Web 2.0 applications are globally used so there is no such thing as down time, maintenance windows, or quite time when you can make a cup of tea.  It is all on, all day, every day.  And did I mention data loss?  Don’t even think about losing anyone’s blog post, tweet, comment or photo of their kids eating cake.  If you lose data and lose it routinely you will be dead to your users from that point on.

Flexibility

So you have spent a lot of time thinking about what your application will do and which features the user will love.  You thought of them, you love them so of course everyone else will think they are brilliant.  Sure, but what will actually happen is when you will launch you will find that you were almost completely wrong all along, the general concept of what you were trying to do might remain (or might not) but your ability to continue to exist will be driven by your ability to rework your application in response to user criticism, feedback and ideas and completely change what you were doing into something that someone actually wants to use (do I need to mention Facebook and Twitter here?).

The problem here is if you have implemented a data platform specifically based on your initial utopian view of your application.  You see a good data model shouldn’t actually model a specific part of an application but instead it should model the data that it is contains in a more “natural form”.  At the core you should have definable entities (such as person, vehicle, garment), attributes (color, size, taste, hair length), relationships and constraints that represent the data itself not necessarily and applications use of that data.  If you have implemented a data platform that is heavily tied to a misguided V1 view of things then your ability to rapidly adapt for V2, the version people will actually like, is going to be heavily impeded.  This lag will of course be a blessing to all your new competitors who will take all the good ideas that you have sitting in your in-tray and implement them as a more successful rip off of your idea.

Another major downside of tying your data platform heavily into individual parts of the application layer is, well, take a look at where we are heading.  Web 3.0 (we are at 2.5 now!) is coming and as fancy and as pretty as your web site is, it is going to matter less and less.   Integration is the key.  RSS, API’s – data portability and interoperability is the future (man, I feel like I should be saying man there?).  How easy will it be to build integration interfaces to your bulging bag of data if your data model only makes sense to Tom the web site developer?  Don’t get me wrong, Tom is a nice guy but he doesn’t really understand that his site isn’t the only piece of the puzzle.  As we go forward many applications interfaces are going to float around the central data platform,  sure his web site will be there as one of them, but also you will have various API’s and interfaces that allow your users to integrate their data into the 78 other social network applications they use.  They might use your application 10 times a day but never log into your web site at all.

Leverage your Asset

It doesn’t matter what your application does, whether it tweets or trades or pokes or woofs or whatever.  At some point you will reach critical mass and the game will change.  You have to remember the only asset you will have is the data.  Your $1b valuation consists of a very big pile of bits and bytes stored within your data platform, and the associated eyeballs that check every 15 minutes to see if their bits have changed slightly.  To make money you will need to sell those eyeballs stuff.  Whether it is your stuff or someone else’s stuff, you need to your maximum potential ensure every time those eyeballs are checking their bits they are also been tempted with some tantalizing relevant product, service or new feature.

The only way you can do this is to analyze the data you have at hand and work out a trend, profile, pattern or segmentation that will allow you to interact with each user personally like you are an old school chum, despite that fact that they are just one of a trillion or so users who have use your application.  You will of course need to do data analytics.

The data platform you use for running your application day to day isn’t necessarily the same data platform that you use to do the analytics with but you do need to think about how this process is going to work up front.  If you are processing thousands of transactions a second how are you going to feed this data out to your analytical platform and how are you going to feed all the wonderful tidbits back in? 

What will be the timeliness of information doing this?  Is analyzing last month’s data going to be good enough?  Well no, of course it isn’t.  What would your recommendation for your stock application be today if that was the case, hold GOOG?  You will need to be able to do this in real time in response to real time changes of your user base.  Keep in mind all the big successful online players generate their core revenue, or enhance their revenue using advanced analytics (Google, Amazon, EBay).

Analytics will also help you listen through the noise and evolve your application to ensure you are adding things people want, not screwing up things people actually use and that you’re staying ahead in usability shifts as people evolved with your service.  Micro changes in behavior can result in massive changes in usage patterns when you are multiplying everything by a squillion.

Summary

What is interesting after reading this post is that the points I am making aren’t new.  In fact they are old, really old, wind back the clock 10 years and we could have been having the same conversation but instead of Web 2.0 I would have been saying client/server or just Web App.  Basically all I am saying is when building a Web 2.0, 3.0 or whatever, start with the data, focus on the data model and build up from there ensuring you address scalability and availability requirements in your design as you go.   You may not need to action these parts of your design right away but when you get your break on CNN or Oprah you know you will be able to roll in the capacity to see you grow and grow.

And don’t forget everything you were ever taught in development school.  Data models still matter.  Data independence matters.  Platform scalability matters.  Data integration & Interoperability matters. 

So if you get all this data stuff right, you will be able to work out all the minor bits like features and functionality of your web application later on once the users tell you what on earth that want to do with it.

Ok so you are setting out to build the next Google and are considering using a Map/Reduce based data access strategy over traditional SQL.  Just as you need a database server to process SQL queries you also require the underlying infrastructure to manage your data and to execute your Map/Reduce routines.  Hadoop is one such system that is gaining acceptance, being co-developed and implemented for data analytics purposes at Yahoo and Facebook amongst others.

Hadoop is the system that allows unstructured data to be distributed across hundreds or thousands of machines forming shared nothing clusters, and the execution of Map/Reduce routines to run on the data in that cluster.  Hadoop has its own filesystem which replicates data to multiple nodes to ensure  if one node holding data goes down, there are at least 2 other nodes from which to retrieve that piece of information.  This protects the data availability from node failure, something which is critical when there are many nodes in a cluster (aka RAID at a server level).

So will Hadoop outperform a RDBMS?  Well unless you are dealing with very large volumes of unstructured data (hundreds of GB, TB’s or PB’s) and have large numbers of machines available you will likely find the performance of Hadoop running a Map/Reduce query much slower than a comparable SQL query on a relational database.  Hadoop uses a brute force access method whereas RDBMS’s have optimization methods for accessing data such as indexes and read-ahead.  The benefits really do only come into play when the positive of mass parallelism is achieved, or the data is unstructured to the point where no RDBMS optimizations can be applied to help the performance of queries.  Indeed benchmarks from the Hadoop site show performance significantly slower in straight line query performance when compared to a relational DB on small scale tests.

	MySql 5.0.27	Hadoop-0.15.2
Data	B-tree disk table (MyISAM)	Text files (access_log)
Machine	1	2
Rows	5,914,669	5,914,669
Results	100	100
Time	4.43 sec	172.30 sec

But with all benchmarks everything has to be taken into consideration.  For example, if the data starts life in a text file in the file system (e.g. a log file) the cost associated with extracting that data from the text file and structuring it into a standard schema and loading it into the RDBMS has to be considered.  And if you have to do that for 1000 or 10,000 log files that may take minutes or hours or days to do (with Hadoop you still have to copy the files to its file system).  It may also be practically impossible to load such data into a RDBMS for some environments as data could be generated in such a volume that a load process into a RDBMS cannot keep up.  So while using Hadoop your query time may be slower (speed improves with more nodes in the cluster) but potentially your access time to the data may be improved. 

Also as there aren’t any mainstream RDBMS’s that scale to thousands of nodes, at some point the sheer mass of brute force processing power will outperform the optimized, but restricted on scale, relational access methods.

So while Hadoop and Map/Reduce are gaining more popularity it shouldn’t be considered a like for like alternative to a relational RDBMS for most applications.  It is a specialized tool with a specialized set of criteria that need to be fulfilled to achieve benefit over more traditional approaches.

Image via Wikipedia

My response to the following post from Billy Bosworth:

The surge to the cloud feels like a certainty (as much as anything can be a certainty in technology).  It reminds me of the momentum shifts felt when things were moving to client/server computing, and more recently around virtualization.  I think the media is largely in a frenzy precisely *because* it’s the newest thing we’ve had to talk about of “mass” interest since virtualization.  That frenzy feeds on itself (as evidenced even  by my own posts on the topic).

This highlights why I think it will be difficult for a startup or even a well established but small time player to launch a successful cloud data service.  I have posted my concerns about this in the past in relation to 10gen.  Building a key part of your business on top of a shaky or unproven foundation is a never a great idea, and this holds true regardless if you are a Web 2.0 application startup or a more traditional business.  How will you suffer if part of your stack disappears due to their infrastructure or process failures or they just run out of funding and turn everything off?  If this also takes all your data with it then you will be doubly screwed.  An off cloud, locally controlled backup in a non-proprietary format will go a long way to reduce such concerns.  But still if you have your data in hand it is going to take you at least a week or two to port to another provider.

So while I think Cloud data services will have a broad appeal and their acceptance will continue to grow and grow, I think we will see a natural attraction to the big and robust vendors, meaning overtime the vendors owning the market share of data services in the cloud will be largely the same as those owning the market share of data services in the enterprise (of course with a few key exceptions).