It has been a while since I wrote here.
During that time artificial intelligence has gone from being an interesting area of technology, largely discussed in terms of machine learning and prediction, to something almost everyone now interacts with. Generative AI can write, analyse, reason over information, create software, interpret images and increasingly interact with other systems on our behalf.
The technology has changed enormously.
But looking back at some of the things I was writing about AI in 2017, I am struck by how little some of the underlying problems have changed.
AI has always been probabilistic
In 2017 I wrote about a recurring problem I was having with a bank's fraud detection system. A legitimate monthly transaction was repeatedly classified as suspicious and my credit card was blocked.
The interesting part wasn't that the machine learning system made a mistake. Any predictive system will make mistakes.
The interesting part was that the people at the bank could recognise that the prediction was wrong, but appeared to have very little ability to influence the automated system that was making the decision.
I argued at the time that systems using machine learning to influence real-world outcomes needed some form of sanity override. High overall accuracy doesn't help very much when you are the individual sitting inside the error percentage.
That remains true today.
Large language models are considerably more capable than the machine learning systems we were deploying a decade ago, but they haven't removed uncertainty. Generative AI is still probabilistic. It can misunderstand information, make incorrect assumptions, select the wrong tool or produce an answer that sounds convincing while being wrong.
The difference is that we are now beginning to give these systems much greater authority.
From prediction to action
Traditional enterprise machine learning generally produced a prediction.
A fraud model might determine that there was an 87% probability that a transaction was suspicious. A churn model might calculate the probability that a customer was going to leave. A classification model might decide which category a document belonged to.
Something else then decided what to do with that prediction.
Generative AI is starting to collapse those layers.
An AI system can now understand a request, retrieve relevant information, reason over that information, decide what needs to happen, call software tools or APIs, update business systems and then communicate the result back to the user.
That is a much more significant change than simply having a better chatbot.
We are moving from AI that says things to AI that can do things.
The important question is becoming authority
For enterprise AI, I think one of the most important design questions over the next few years will be less about intelligence and more about authority.
There is a meaningful difference between an AI system that can:
- answer a question;
- recommend an action;
- prepare an action for somebody to approve; and
- perform the action itself.
The consequences of being wrong increase substantially at each level.
If an AI produces an imperfect summary, a person can usually recognise and correct it before anything significant happens.
If the AI is able to change a customer's account, approve a payment, issue a regulatory document, alter a business record or trigger another automated process, the error has escaped the model and entered the real world.
This doesn't mean we shouldn't give AI systems the ability to act. In fact, I think connecting AI with automation is where a large proportion of its eventual enterprise value will come from.
But the controls surrounding those actions become just as important as the intelligence performing them.
Automation still matters more than the demo
Another article I wrote in 2017 was called Automation trumps AI.
My argument was that organisations often became fascinated by sophisticated AI problems when relatively simple automation could deliver enormous value. Automating mundane but poorly attended tasks could free people to work on progressively more valuable problems.
I still believe that.
What generative AI has changed is the range of processes that can realistically be automated.
Traditional automation works extremely well where the process can be expressed as deterministic rules: if this happens, do that.
But many business processes contain a layer of human interpretation. Somebody reads an email, understands a document, looks at several pieces of information, applies experience and then decides what to do.
Historically that interpretation created a natural boundary for automation.
Generative AI can cross much of that boundary.
It can interpret unstructured information and turn it into something a deterministic workflow can act upon. That potentially allows us to automate processes that previously required a human simply because the input wasn't sufficiently structured.
But it also introduces something unusual into automation: uncertainty.
Traditional automation is generally predictable. The same input and rules produce the same outcome.
AI-assisted automation may not.
So we are combining the enormous scalability of automation with the probabilistic behaviour of AI. That is incredibly powerful, but it deserves careful engineering.
The guardrails are part of the system
This is why I don't think enterprise AI architecture should treat governance, approvals, permissions and audit as things that get added once the AI functionality is working.
They are part of the functionality.
For an AI agent operating inside an organisation we should be able to answer questions such as:
- What information is this agent allowed to access?
- What actions is it allowed to perform?
- Which actions require human approval?
- What limits apply to those actions?
- What source information was used to reach a decision?
- Which model, instructions and tools were involved?
- Can we reconstruct what happened afterwards?
- What happens when the AI is uncertain or wrong?
Those questions aren't barriers to using AI. Answering them is what allows us to use AI for increasingly important tasks with confidence.
The technology changed. The responsibility didn't.
I also wrote in 2017 that every data project has ethical considerations and that people building these systems need to think about the real-world consequences of their work.
That seems even more relevant now.
AI capabilities that once required specialist teams and significant investment are becoming available to almost any developer. Increasingly, a developer can give a model access to a collection of tools and APIs and create something capable of taking meaningful actions in a surprisingly small amount of code.
That democratisation is a very good thing, but greater capability also means greater responsibility.
We shouldn't only ask whether an AI system can perform an action.
We should ask whether it should perform it autonomously, what happens when it gets it wrong and whether the surrounding system is capable of detecting and managing that failure.
Where I think enterprise AI is heading
The last few years have understandably been dominated by the capabilities of the models themselves. Every few months models became better at reasoning, coding, understanding documents, working with images or using tools.
That progress will continue.
But I suspect the next stage of enterprise AI will increasingly be about everything surrounding the model.
The most useful enterprise AI systems may not necessarily be those using the smartest model.
They will be the systems that understand what the model is allowed to know, what it is allowed to decide, what it is allowed to change, when a human needs to be involved and how we can understand exactly what happened afterwards.
In other words, after spending several years asking how intelligent we can make AI, the next important question may be:
How much authority are we prepared to give it?
That is a question I expect we will be working through for quite some time.