Blockchain, Blockchain, Blockchain, Oi, Oi, Oi!

Sydney_bridge

One of the fastest moving technologies of 2016 is Blockchain.  Put simply, Blockchain is decentralised trust system for ensuring transaction validity without a central authority.  The use-cases for Blockchain are far reaching as it is essentially a data platform on which any applications that require trust for the “exchange” of information between multiple parties can be built.  And it has just been revealed/confirmed that it has Australian origins!

The Blockchain methodology was designed to underpin Bitcoin but has since gained momentum in its own right, in fact arguably Blockchain is a more important innovation than Bitcoin itself.  Certainly banking and FSI in general are all over Blockchain, with CBA and 10 of the world’s largest banks simulating trading using Blockchain, and the ASX working on a Blockchain based system for Australian equities.  But these are just a couple of examples. The interest, momentum and pace surrounding Blockchain is quite astounding.

Blockchain is still in its early days and doesn’t have all the issues solved yet.  Scalability can be a challenge but these problems will be resolved as Blockchain technology is evolved.  In terms of resource, there is a lot of technical detail on the web about how Blockchain systems work, however if you’re looking for resources on the “why” for Blockchain a couple of good ones include:

If you’re involved in building or operating trust based systems that exchange information between multiple parties you absolutely need to get up to speed with Blockchain.

Will Automation take my Job? Well, Maybe….

Will Automation take my Job?

Automation is a business transformation technology that involves innovations in the field itself, but more recently leveraging innovations in the areas of AI, Machine Learning and Big Data. And as all of these fields gain maturity, pundits are naturally playing forward the impact and making predictions about job losses across various industries directly as a result of automation.

Reiterating the title of this post, “will automation take my job” I think the answer is a clear “maybe”. But job loss isn’t the only outcome of automation. My experience has shown that many organisations are seeking to increase the value of the output of their internal workings, and often key employees are constrained with low value tasks. In IT this is particularly true, where many employers are seeking proactive innovation and thought leadership from employees in their respective areas. But often this is not being realised as they are consumed with lower skill, high occurrence tasks that are important – but are not producing an ROI to the business. IT is just one example, the same problem can cross many industries and skill sets.

Automation of Today

Today, automation can be good at undertaking pre-planned actions when pre-defined conditions occur. Which means certain types of roles, that are formulative in nature, lend themselves to automation. But trying to improve the efficiency of these roles is not necessarily new. Many organisations have already spent effort reducing the associated costs, sometimes replacing higher cost resources with lower cost alternatives. This transition typically required organisations to document the process aspects of these roles in detail, naturally this feeds well into the foundations of an automation drive. And this is not necessarily limited to the lower end of the pay scale, I am sure there are a number of people in high paying roles in FSI, trading, banking etc. that are beginning to see components of their role replaced by automation.

Automation of Tomorrow

Looking forward, automation is beginning to become more adaptive and use machine learning and AI more broadly to make judgement calls. Bots may understand typed and spoken language as input. Routines may use analytics and prediction to select the best cause of action to a specific situation. This broadens the scope of the application of automation from tasks, which have clear black/white outcomes to those with shades of grey requiring intuition calls.

"If you are doing the job of a robot today, then it is logical to think that computers may one day replace you. But the question is, do you want to be doing the job of a robot to begin with?"

So is this all doom and gloom? I think this is definitely an approaching wave of change that is going to impact on areas of the workforce. Over time this will phase out some roles, and aspects of others, but it will also result in creation of new roles and the improvement of others. Contrary to how if can sometimes seem, most organisations are not just trying to cut costs. They are instead usually focused on ensuring value is being created for both their customers and their shareholders, and driving their competitive advantage. While this does mean reducing costs where practical, it also means making investment in areas that continue to drive growth. This should therefore also mean new jobs, new opportunity and more innovation across the board.

What to do?

But it does mean change is likely for some, and change can be very unpleasant. To ensure you are ready for change I think you need to take an honest look at your current role to determine if it fits the model of a function that overtime could be automated. If so take the opportunity to begin preparing for the change, developing skills and experiences that will ultimately be of higher value if/when organisations begin to adopt automation as a means to increasing value.

Is Amazon about to Disrupt the Database Market?

My LinkedIn feed shows me Shawn Bice, former GM of database systems (SQL Server) at Microsoft is joining Amazon AWS as VP of Analytics. Assuming accuracy, Shawn has joined the likes of Hal Berenson (on the former Microsoft luminaries behind SQL Server), Raju Gulabani and Sundar Raghavan.

While AWS has for many years provided support for common database platforms via their EC2 and RDS options, more recently they have released their own transactional database platform AWS Aruora, and the AWS RedShift data warehousing platform. And to get you there, they have also recently released their database migration service for on-mass on-premise to cloud migration.

AWS seem to have realised that a keystone in winning in the cloud is winning the database. In the data centric world ahead, the data platforms are going to become core to how applications are architected and ultimately deployed. Within the cloud providing a comprehensive set of data services with (semi-) seamless integration, rapid deployment and op-tap scalability will be compelling in convincing developers and organisations to “buy into” that vendors stack.

AWS are actively hiring some of the best and brightest in database for what could be a double whammy if they can get it right. The last time I looked I think the database market on it’s own was a $30b+ market, but in the cloud winning with the database also likely means winning a customers complete cloud stack.

Of course, Microsoft and Oracle are formidable opposition and are arguably ahead of the in terms of developer and enterprise buy in. So it is not necessarily and easy path for ahead.  

I think I have been saying this continuously for the last 15 years; but it is (still) an interesting time to be in database.

What is the biggest challenge for Big Data? (5 years on)

Five years doesn't half fly when you’re having fun!  In this post from 2011 I highlighted some of the challenges facing the “big data revolution” centring on a lack of people with the right skills to deliver value on the proposition.  Fast forward to 2016 and this not only remains true, but is likely the key issue holding back the adopting of advanced analytics in many organisations.

 While there has been an influx of “Data Scientist” titles across the industry, generally organisations are still adopting a technology driven approach driven by IT.  The conversations are still very focused on the how rather than the why, it is still all very v1.0.  There is still a lack of the knowledge required to turn potential into value, value that directly affects an organisations bottom line.

This will start to sort itself out as the field matures and those who understand the business side of the coin become fluent with big data concepts, to the point they can direct the engineering gurus.  IBM with Watson is looking to take this a step further by bypassing the data techies and letting analysts explore data without as much consideration for the engineering/plumbing involved.  This is a similar direction that services such as AWS and Azure Machine Learning are heading, in the cloud.

In 2016 the biggest challenge for Big Data is turning down the focus on the technical how, and turning up the focus on the business driven why.  Engaging and educating those who understand a given business in the capabilities of data science, motivating them to lead these initiatives in their organisations.

About Tony Bain

Tony Bain

I am the co-founder of the RockSolid SQL business, the primary developer of the technology, and have built the business to include some of the largest and most well known customer logos.

My area of expertise is building solutions that deliver customer value via leveraging big data, machine learning, AI and software automation technologies. I have written numerous books, articles and posts on data driven business and have presented at conferences globally.

As a Director for RockSolid SQL I am responsible for:

  • Creation of the core RockSolid technology and technical innovation and development of the product
  • Data and analytics stratergy/implementation
  • Financial performance and growth
  • Customer satisfaction, retention and growth
  • Business development, sales and marketing and leadership on key opportunities
  • Service offerings, pricing and go to market strategies
  • Partnering relationships
  • Product strategy and direction for our core RockSolid technology
  • Technical leadership in key development initiatives
  • Team leadership and retention

The SQL/NoSQL war is over. The winner is… wait, was there a war?

SQL_NoSQL

We are approaching 7 years since the term “NoSQL” re-entered the popular tech vernacular, and 7 years since I wrote the post “Is the Relational Database Doomed?”. During this time, we have experienced a tidal wave of non-relational data management technologies. So, time for an update to my prior article.

At the start of the decade, when the NoSQL buzz was in its heyday, some were predicting the end of the dominance of the relational database platform (RDBMS) within a decade.  The reason for this seemed somewhat sound. That being, the relational database is based on what is now 40+ year technology and things are so much more advanced now than back then, so clearly this was a technology ripe for disruption.

So how has this disruption gone?  Well, all my metrics show there are more relational databases in existence today than at any point in history.  It may be hard for many people to comprehend the volume.  Often, mid-size enterprises operate hundreds of relational databases.  Many large enterprises have thousands to tens of thousands.  These represent the data stores of everything from ERP's, financial systems, content sharing apps, IT tools and so on.

So despite the noise surrounding NoSQL, in a head to head comparison of volume of use, NoSQL use seems so very small.  At a guess, I would predict that for every NoSQL database in existence there would be at least 1000 relational databases.  Probably more.  You would be forgiven for thinking NoSQL use was almost insignificant. 

So why has there been so little disruption?

  • The relational database has such a massive legacy. The IT world is full of people whose front of mind solution for a new data management requirement is a relational database.  To demonstrate this I looked on LinkedIn. My search showed that over 1,000,000 people list “SQL” as a skill in their profile.  In comparison only 16,000 listed NoSQL and 30,000 listed MongoDB.  That’s a massive skills gap.
  • The RDBMS is very general purpose.  There are very few day-to-day data management requirements that cannot be met without a run of the mill RDBMS.  So why would you not go with what you know, if what you know is suitable?
  • Relational databases are solving very complex problems in a balanced approach.  There is 40+ years of learnings on how to balance consistency, concurrency, scale and performance.  Many NoSQL initiatives focus on improving some objectives (such as scale or performance) at the expense of others (such as consistency or redundancy) solving their own problems but also lacking a general purpose appeal.
  • Relational database vendors have also kept innovating.  With some RDBMS vendors you can now combine SQL and XPATH, support JSON natively and support other non-structured data types.  Also, many RDBMS platforms now support in-memory databases and others are quickly adding this support.

So with the continued dominance of the relational database, what future is there for the NoSQL alternatives?  Well that is clear, the same opportunity as they have been filling over the last 7 years.  Edge cases.  Sure, enterprises have many routine data in/data out applications and these belong on the RDBMS but modern enterprises are trying to do more with data than ever before and leverage data in new ways for a competitive advantage.  Maybe they need data to be captured at a massive scale, much greater than what is possible with a traditional RDBMS.  Maybe they are looking to deep mine data to identify complex relationships between entities, or make predictions about how scenarios will transpire.  Maybe they are trying to learn from large and diverse data sets and discover key new ways to improve productivity.  This new world of requirements is where new world platforms have the opportunity to shine, focus on improving a specific set of key objectives, potentially at the expense of others, and then find the market that needs those objectives.

To summarise, I cannot see a world in the near future where any non-RDBMS gains any dominance in supporting the data management needs of most applications.  The vendors people use may change, and the location of those databases may change (on premise to cloud) but they will be relational. However, what are the edge cases of today will of course become more mainstream.  When this happens it will not be at the expense of the RDBMS, but instead they will be in addition to it.  The playing field is getting bigger.   Organisations desire to do more with data, via big data or data science initiatives that is fuelling a market ripe for vendors with clever, yet tightly focused, data management solutions.

So as it turns out, relational (SQL) and non-relational (NoSQL) technologies were not at war at all.  They are in fact allies, working together to deliver organisations both general purpose and special purpose data management solutions.

The Business is the Database

IT budgets are under more scrutiny than ever before.  While organisations continue to realise benefit in becoming more IT centric, the pressure to demonstrate value and compelling ROI's is increasing.   This increased pressure, while generally positive, can also have a negative impact across other areas of IT, especially those which struggle to articulate the value they provide to the business.  Operational management is one of these areas, where costs are seen as an expense rather than an investment.  Many organisations are driven to reduce these “costs” as much as practical, this often leads to outsourcing resources, reduction or removal of budgets for tools and a lack of drive to be innovative in this space.

One subset of IT operations, database management, falls very much into this category and is often subject to reduced investment and focus as a result.  At RockSolid SQL we engage with organisations who value the contribution operational database management makes to their organisations, but many of these organisations have come out of a period of underinvestment.  All too often the drive to reduce costs has been ineffective. This has led to organisations shooting themselves in the foot, as concurrently they have been becoming increasingly data hungry.  The volume of data generation and consumption has grown at a rapid pace yet the lack of investment in database management can cause a significant flow on impact which overshadows any expected cost savings.  Some of the reasons for this follow.

"The volume of data generation and consumption has grown at a rapid pace yet the lack of investment in database management can cause a flow on impact which overshadows any cost savings."

Businesses are their Databases

It doesn't matter if your organisation is in construction, logistics, financial services, mining, retail or virtually any industry.  You may have buildings, equipment, and employees but a key business assets is the information that is unique to your business, and this is more often than not stored in various databases.   Without your databases you have nothing that ties all your assets, your stock, plant, people and knowledge – all this business stuff – into the actual business.  Without your databases you don’t know who your customers are, what you've sold, and what your profit is.  Without your ERP, CRM, financial, HR, document management or other application databases you don’t have an operable business.

Performance is a factor of Productivity

Productivity is key to maintaining competitive advantage, and a factor in productivity is how much time your employees are working, and how often they are waiting.  Databases with performance issues can cause hospitals to treat less patients, retailers to sell less stock, logistics companies to ship less freight, call centres to hire more staff and government agencies to be less responsive to their constituents.

Poor database performance hurts productivity, but yet it becomes one of the first areas to become neglected when organisations pull back on database management.  For most businesses there are very few data requests that, with today’s modern hardware, should take hours or even minutes to process.  Yet countless businesses have hundreds or sometimes thousands of staff waiting for reports, searches, screens to refresh and data insights to be generated simply because the organisation does not have a sufficient investment in maintaining database performance.

Do you lock the door at night?

We have long ago stopped printing things out and keeping paper records, now data contained within an organisations databases is often the only “copy” of that information.  Keeping that information recoverable, secured and accurate should be of utmost concern to any organisation, but again this is an area that quickly becomes neglected when savings are sought.

Thankfully most organisations maintain a reasonable backup strategy for their databases.  But the same cannot be said in terms of a robust security strategy.  Security of “the business” is often overlooked, becomes unmanaged, unmaintained and processes focus only on granting access rather than managing security.

Examples of locking the door, but leaving the window open are all too frequent.  Security breaches have the potential to cause untold harm to both your business and your customers, and the lack of proper controls and auditing can make knowing who did what virtually impossible.  Businesses assume that somewhere in the IT department alarms will go off, and screens will flash it security breaches occur.  But in reality, this is usually not the case. Without proper monitoring systems inappropriate access can occur without warnings being raised, copies of sensitive data could be made without record being made.  Security breaches could occur frequently without anyone knowing.

Agility is a factor of competitive advantage

Maintaining competitive advantage requires organisations to be dynamic, respond to change and launch new initiatives in response to changing customer demands.  The ability to understand customers and identify changing demands is driven from an organisations data.  This data can be a wonderful thing as it contains what you know and have already learnt, but it also can contain important things that your business doesn't yet know about itself.  It’s called data science, and organisations doing it well have a distinct advantage over those dragging their heels.

"data can be a wonderful thing as it contains what you know and have already learnt, but it also can contain important things that your business doesn't yet know about itself"

A reduction of investment in database management often results in a loss of agility and instead sentences the organisations to a life with a stagnant mix of systems unable to interrelate.  A lack of focus on upgrades and migration limits available functionality to outdated mechanisms. A lack of investment restricts a business hungry for data and “insight” to an environment where this may be exceptionally difficult to glean, and potentially time consuming to try.

So what’s the Alternative?

So all well and good arguing that reducing investment in database management may have a significant impact on a business.  But other organisations may also argue that they have relatively high investment but still struggle to realise the value.  Well this is really the key, it is not about a high or low investment.  It is about efficient use of an effective level of investment.  It is about using appropriate resources, well for lack of a better time, appropriately.  It is about using tools which demonstrate a compelling ROI.  It is about valuing the organisations data assets, and assigning them suitable, yet cost effective, care.

Driving Efficiency

RockSolid SQL has been helping organisations do this for the last 12 years.  To aid organisations in achieving this we have developed the RockSolid Database Management Efficiency Framework (RDBEF).

Key aspects of the RDBEF approach are as follows:

  • Define operational standards, turn them into policies. Use software to audit and identify exceptions.  Manage and resolve these exceptions.  Rinse and repeat.  Stabilising an environment through standardisation is in my experience, the single most effective thing you can do to reduce operational database management costs.  But yet concurrently improving quality of service and reliability.  It is a true win-wine.
  • Define provisioning, patching, operational management and resolution processes. Use software to automate these at Level 1 response and some Level 2.  In fact, automate everything you can.  The cheapest employee you can find, is still orders of magnitude more expensive than a few CPU cycles.  Plus automation does it the same way every time, it doesn't have bad days.
  • Monitor everything. You will never be able to predict what you will need to know, when you will need to know it and how retrospective you will need to be.  You will not believe how much time is saved by knowing, on demand, everything that has happened, when it happened and who made it happen.  It is like night and day.
  • Use appropriate resources for the tasks at hand. Don’t have your expensive top guns doing junior level work.  Don’t have your juniors working outside their skillsets creating as many issues as they resolve.  Outsource if required, or mix-source.  Use vendors who share your vision.  Use software to assign and escalate issues appropriately.
  • Become nimble. Make the things that are currently the hardest the easiest.  Patching, provisioning, upgrades and migrations should be routine.  Nothing to do with operational database management should be scary or time consuming.  It is all just routine.
  • Refocus and re-architect towards agility. Know that your organisations desire for data is likely going to increase.  And both the rate at which they demand access, and the methods of consumption will both likely grow exponentially.

"Automate everything you can.  The cheapest employee you can find, is still orders of magnitude more expensive than a few CPU cycles"

Wrapping it up

IT as an industry has a checked history of ineffectiveness and sometimes extravagance and I suppose the response to this has been for organisations to drive cost cutting initiatives, especially where the perceived value is low. But few organisations I know want to eliminate investments that generate a real and measurable positive return.

Database management has been undervalued by many organisations and we have seen the sobering impact of that.  Many organisations have poorly managed, secured and maintained systems.  Databases suffering performance issues, issues which may have productivity impacts to the business which far outweighs any initial cost savings.  

We are advocates of efficiency and effectiveness.  Use software to monitor, automate, audit and escalate.  Automate where at all possible, use skilled people for difficult exceptions, and use lower cost resources for easier exceptions.  Become nimble and agile, bring your data assets forward to modern standards and provide an environment where your organisation can stand on a level playing field with both current and future competitors.

About RockSolid SQL – RockSolid SQL is an innovative software and services company. We set out in 2004 to create a solution that allows customers to cost effectively manage their databases, regardless of the scale of their environment.  For more information visit www.rocksolidsql.com

Webinar: NoSQL, NewSQL, Hadoop and the future of Big Data management

Join me for a webinar where I discuss how the recent changes and trends in big data management effect the enterprise.  This event is sponsored by Red Rock and RockSolid.

Overview:

It is an exciting and interesting time to be involved in data. More change of influence has occurred in the database management in the last 18 months than has occurred in the last 18 years. New technologies such as NoSQL & Hadoop and radical redesigns of existing technologies, like NewSQL , will change dramatically how we manage data moving forward. 

These technologies bring with them possibilities both in terms of the scale of data retained but also in how this data can be utilized as an information asset. The ability to leverage Big Data to drive deep insights will become a key competitive advantage for many organisations in the future.

Join Tony Bain as he takes us through both the high level drivers for the changes in technology, how these are relevant to the enterprise and an overview of the possibilities a Big Data strategy can start to unlock.

https://redrockevents.webex.com/redrockevents/onstage/g.php?t=a&d=869100422

 

What is the biggest challenge for Big Data?

Often I think about challenges that organizations face with “Big Data”.  While Big Data is a generic and over used term, what I am really referring to is an organizations ability to disseminate, understand and ultimately benefit from increasing volumes of data.  It is almost without question that in the future customers will be won/lost, competitive advantage will be gained/forfeited and businesses will succeed/fail based on their ability to leverage their data assets.

It may be surprising what I think are the near term challenges.  Largely I don’t think these are purely technical.  There are enough wheels in motion now to almost guarantee that data accessibility will continue to improve at pace in-line with the increase in data volume.  Sure, there will continue to be lots of interesting innovation with technology, but when organizations like Google are doing 10PB sorts on 8000 machines in just over 6 hours – we know the technical scope for Big Data exists and eventually will flow down to the masses, and such scale will likely be achievable by most organizations in the next decade.

Instead I think the core problem that needs to be addressed relates to people and skills.  There are lots of technical engineers who can build distributed systems, orders of magnitude more who can operate them and fill them to the brim with captured data.  But where I think we are lacking skills is with people who know what to do with the data.  People who know how to make it actually useful.  Sure, a BI industry exists today but I think this is currently more focused on the engineering challenges of providing an organization with faster/easier access to their existing knowledge rather than reaching out into the distance and discovering new knowledge.  The people with pure data analysis and knowledge discovery skills are much harder to find, and these are the people who are going to be front and center driving the big data revolution.  People who you can give a few PB of data too and they can provide you back information, discoveries, trends, factoids, patterns, beautiful visualizations and needles you didn’t even know were in the haystack.

These are people who can make a real and significant impact on an organizations bottom line, or help solve some of the world’s problems when applied to R&D.  Data Geeks are the people to be revered in the future and hopefully we see a steady increase in people wanting to grow up to be Data Scientists. 

SQL Server to discontinue support for OLE-DB

ODBC was first created in 1992 as a generic set of standards for providing access to a wide range of data platforms using a standard interface.  ODBC used to be a common interface for accessing SQL Server data in earlier days.  However over the last 15 years ODBC has been second fiddle as a provider for SQL Server application developers who have usually favoured the platform specific OLE-DB provider and the interface built on top of it such as ADO.

Now in an apparent reverse of direction various Microsoft blogs have announced the next version of SQL Server will be the last to support OLE-DB with the emphasis returning to ODBC.  Why this is the case isn’t entirely clear but various people have tried to answer this, the primary message being that ODBC is an industry standard whereas OLE-DB is Microsoft proprietary.   And as they are largely equivalent, it makes sense to only to continue to support the more generic of the two providers.

After years of developers moving away from ODBC to OLE-DB, as you would expect this announcement is being met with much surprise in the community.  But to be fair I suspect most developers won’t notice as they user higher level interfaces, such as ADO.NET, which abstract the specifics of the underlying providers.  C/C++ developers on the other hand may need to revisit their data access interfaces if they are directly accessing SQL Server via OLE-DB.