How Data Science may change Hacking

How_ml_change_hacking

Malicious hacking today largely consists of exploiting weaknesses in an applications stack, to gain access to private data that shouldn't be public or corrupt/interfere with the operations of a given application.  Sometimes this is to expose software weaknesses, other times this is done for hackers to generate income by trading private information which is of value.

Software vendors are now more focused on baking in security concepts into their code, rather than thinking of security as being an operational afterthought.  Although breaches still happen.  In fact, data science is being used in a positive way in the areas of intrusion, virus and malware detection to move use from reactive response to a more proactive and predictive approach to detecting breaches.

However, as we move forward into an era where aspects of human decision making are being replaced with data science combined with automation, I think it is of immense importance that we have the security aspects of this front of mind from the get go.  Otherwise we are at risk of again falling into the trap of considering security as an afterthought.  And to do this we really need to consider what aspects of data science open themselves up to security risk.

One key area that immediately springs to mind is “gaming the system” specifically in relation to machine learning.  For example, banks may automate the approval of small bank loans and use machine learning prediction to determine if an applicant has the ability to service the loan and presents a suitable risk.  The processing and approval of the loan may be performed in real-time without human involvement, and funds may immediately available to the applicant on approval. 

However what may happen it malicious hackers became aware of the models being used to predict risk or serviceability, if they can reverse engineer them and also learn what internal and third party data sources were being used to feed these models or validate identity?  In this scenario malicious hackers may, for example, create false identities and exploit weaknesses in upstream data providers to generate fake data that results in positive loan approvals.  Or they may undertake small transactions in in certain ways, exploiting model weaknesses that trick the ML into believing the applicant is less of a risk than they actually are.  The impact of this real time processing could cause catastrophic scale business impact in relatively short time frames.

Now the above scenario is not necessary all that likely, with banking in particular having a long history of automated fraud detection and an established security first approach.  But as we move forward with the commoditisation of machine learning, a rapidly increasing number of businesses are beginning to use this technology to make key decisions.  When doing so it becomes therefore imperative that we not only consider the positive aspects, but also what could go wrong and the impact misuse or manipulation could cause. 

For example, if the worst case scenario could be, for example, that a clever user raising customer service ticket has all their requests marked as “urgent” because they carefully embed keywords causing the sentiment analysis to believe they are an exiting customer, you might decide that while this is a weakness it may not require mitigation.  However if the potential risk is instead incorrectly granting a new customer a $100k credit limit, you may want to take the downside risk more seriously.

Potential mitigation techniques may include:

  • Using multiple sources of third party data.  Avoid becoming dependant on single sources of validation that you don’t necessarily control.
  • Use multiple models to build layers of validation.  Don’t let a single model become a single point of failure, use other models to cross reference and flag large variances between predictions.
  • Controlled randomness can be a beautiful thing, don’t let all aspects of your process be prescribed.
  • Potentially set bounds for what is allowed to be confirmed by ML and what requires human intervention.  Bounds may be value based, but should also take expected rate of request into consideration (how may request per hour/day etc.).
  • Test the “what If” scenarios and test the robustness and gamability of your models in the same way that you test for accuracy.

The above is just some initial thoughts and not exhaustive, I think we are at the start of the ML revolution and it is the right time to get serious about understanding and mitigation of the risk surrounding the potential manipulation of ML when combined with business process automation.

Will Automation take my Job? Well, Maybe….

Will Automation take my Job?

Automation is a business transformation technology that involves innovations in the field itself, but more recently leveraging innovations in the areas of AI, Machine Learning and Big Data. And as all of these fields gain maturity, pundits are naturally playing forward the impact and making predictions about job losses across various industries directly as a result of automation.

Reiterating the title of this post, “will automation take my job” I think the answer is a clear “maybe”. But job loss isn’t the only outcome of automation. My experience has shown that many organisations are seeking to increase the value of the output of their internal workings, and often key employees are constrained with low value tasks. In IT this is particularly true, where many employers are seeking proactive innovation and thought leadership from employees in their respective areas. But often this is not being realised as they are consumed with lower skill, high occurrence tasks that are important – but are not producing an ROI to the business. IT is just one example, the same problem can cross many industries and skill sets.

Automation of Today

Today, automation can be good at undertaking pre-planned actions when pre-defined conditions occur. Which means certain types of roles, that are formulative in nature, lend themselves to automation. But trying to improve the efficiency of these roles is not necessarily new. Many organisations have already spent effort reducing the associated costs, sometimes replacing higher cost resources with lower cost alternatives. This transition typically required organisations to document the process aspects of these roles in detail, naturally this feeds well into the foundations of an automation drive. And this is not necessarily limited to the lower end of the pay scale, I am sure there are a number of people in high paying roles in FSI, trading, banking etc. that are beginning to see components of their role replaced by automation.

Automation of Tomorrow

Looking forward, automation is beginning to become more adaptive and use machine learning and AI more broadly to make judgement calls. Bots may understand typed and spoken language as input. Routines may use analytics and prediction to select the best cause of action to a specific situation. This broadens the scope of the application of automation from tasks, which have clear black/white outcomes to those with shades of grey requiring intuition calls.

"If you are doing the job of a robot today, then it is logical to think that computers may one day replace you. But the question is, do you want to be doing the job of a robot to begin with?"

So is this all doom and gloom? I think this is definitely an approaching wave of change that is going to impact on areas of the workforce. Over time this will phase out some roles, and aspects of others, but it will also result in creation of new roles and the improvement of others. Contrary to how if can sometimes seem, most organisations are not just trying to cut costs. They are instead usually focused on ensuring value is being created for both their customers and their shareholders, and driving their competitive advantage. While this does mean reducing costs where practical, it also means making investment in areas that continue to drive growth. This should therefore also mean new jobs, new opportunity and more innovation across the board.

What to do?

But it does mean change is likely for some, and change can be very unpleasant. To ensure you are ready for change I think you need to take an honest look at your current role to determine if it fits the model of a function that overtime could be automated. If so take the opportunity to begin preparing for the change, developing skills and experiences that will ultimately be of higher value if/when organisations begin to adopt automation as a means to increasing value.