Practical considerations for AI adoption: Part 1

Ash: I think there’s a big disjoint in terms of what organisations think AI is and a lack of understanding, and often a misunderstanding.

Miriam: For sure. Yes.

Ash: I know from my own experience of running roundtables, everybody thinks of AI but often that means different things to different organisations.

In our minds, anyway, what that means to us is – this is how we group it just to make it a little bit easier – three broad sectors. So you’ve got Machine Learning, where you are using models that have been trained to be able to identify patterns or forecast data, and then to be able to infer from that at run time and, off the back of that, to be able to make predictions. That’s been around a long time.

You’ve then got the ‘middle section’, Agentic AI or Intelligent Automation or an AI agent. Using large language models to automate workflows and systems in that decision-making process – to introduce autonomy. And then the third thing, I think this is what a lot of organisations are talking about, is really using AI at a personal or team level to improve productivity and to allow people to do more.

Is that broadly what you’re seeing Miriam?

Miriam: Yes, and I think the business-side uses that we tend to deal with, they often don’t have that clear distinction. They muddle things up and they tend to range from thinking about AI as purely automation (meaning that whole processes could be taken away, handed over) and very often they’re not clear that they will still need supervising.

They think: ‘the process can be handed over, AI will solve this and we don’t to deal with that part anymore’, which for many sophisticated processes is not quite right. On the other hand, you have many business users who think: ‘well, if we just increase personal productivity, surely then everything will improve automatically because nobody has enough time. If we have more time, we’ll get more done and create more value.’ Which also doesn’t quite work, because even using AI at that level only gets you 60-80%. You still need that last bit where you need the human to actually take charge.

Ash: So that human in the loop aspect of it.

Miriam: Very much so, yeah. That’s what we are seeing quite a lot. I’d love to hear your views on how you can blend the three types that you just described to create better outputs, but also more assurance and more confidence in the AI tools. We’re seeing, on the business-user end, that many users dabble with AI, whether that is the AI chatbot or whether that is a bit of Agentic AI that’s been built for them. And, they see that it’s not 100% and they think: “I could have done it faster myself” or “is it really aiding us that much? Is it really worth it?” So I’d love to hear how you blend those different types.

Ash: I think you get more value from blending them. But as well as that, there also needs to be recognition that [for] those three different pillars, the skills and the people required to do them are completely different. So whilst it’s all AI within those three different pillars, that Machine Learning side is very much focused around data science. Very much built from maths. And that Agentic AI side is more of, exactly as you said, that automation of workflows. So that’s much more around business analysis and understanding what the business wants to achieve. And I think that’s maybe a common thing that’s often lacking in terms of defining upfront what it is that the business actually wants to achieve from this, and what the ROI looks like, and what success looks like.

Miriam: That’s a really, really good point. If you could dig a bit deeper in it, because I know, from having spoken to many leaders, they think they need to hire data scientists. But from what you’re saying, I’m guessing that’s not actually necessarily true. If they are running certain businesses within a consultancy, then professional services or something like that, they don’t necessarily need data scientists on the team.

Ash: I’d agree with that completely. If you’re doing that Machine Learning aspect of it, then yes, there is definitely a data science element to that. Depending on what you’re doing with it, a lot of the tools nowadays are actually pushing down the skills required so more people can do that.

I guess it’s similar to what we saw in software development of the last few decades, where 30 years ago you needed somebody who was a developer who really understood at a deep level the code and how it communicates with the operating system. And even just to get it to do a trivial thing, that’s got to produce a window with a couple of buttons on it. Now we see that’s pushed down. You don’t need those super hardcore skills anymore to be able to do that. But at the same time, we’re expecting developers to do more in terms of: they’re expected to talk to the business, to work out what the requirements are, they may be expected to plan, run that project, test it, maybe get involved in more deployment.

So they’ve been asked to move up the food chain. And I think that’s what we’re seeing, particularly in the Agentic AI space, where you need individuals who are capable of running this as a tech project, but it’s also a change project. It’s a lot of involvement with people to be able to understand what those workflows are, and in fact, are they optimal? There’s no point in going away and just automating wildly something that’s evolved and probably shouldn’t look like that in practice.

Miriam: This is a question that we see a lot: when leaders are asked, “What would you like?” They say, “We want everything. We want the whole shopping list. We want the whole Christmas wishlist.” They want everything. And when you then probe into [it], you find out that actually there is a lot there, there are a lot of items on that wishlist, which is good, but do you need it? Do you really need 15 reports on the same topic, formatted in these different ways for different stakeholders? Or are there better ways of, for example, communicating with the stakeholders and still fitting to their requirements, but, you know, actually enhancing the relationship that you build with the reporting as just a tool. So that’s just one example where there are processes that we see in place that probably shouldn’t be around. And it’s not just reporting. There are many other, kind of processes that should probably be looked at.

Ash: Completely, yes. Rather than just blindly automating it and ending up with the same thing. One thing that the automation does lend itself to, bizarrely, is the fact that you can then easily specialise the reports that it produces for many different stakeholders, rather than having to hand-crank through all that data and to be able to generate those graphs.

Whether you should or not is another question, but the skills required to communicate with the business and say, “Do you need five different sets of reporting? Wouldn’t it make more sense to align everybody and combine that into one?” That’s a different skill set to what is seen as that traditional, technical developer kind of role.

Miriam: Yes, and having been involved in many of those projects even before AI came around, I definitely know that it’s a hard part, because you find different parts of an organisation speaking different languages. They’re just not speaking the same language, and they’re having difficulty to see. What do you do in your side of the business? Because I know that you’re working really hard with the people that you’re developing.

Ash: When you’re putting junior people, particularly in an organisation, there’s always an element of even if they’re exceptionally bright when they land, you need to orientate yourself towards a business in that sector. Now, there will be certain terminology and certain phrases and certain technologies which are common, and they could be expected to know or expected to Google it.

Equally, you’ll find yourself in an organisation where there’s 1,000,001 different acronyms that are all used, some of which will be internal to that organisation, some of which will be internal to teams in the organisation. And if you really are lucky, you may find that different teams will use the same acronyms to mean different things! But unpicking that is really, really tricky.

What we found the best way to do that in practice is when the first people land in an organisation, they actually create a set of assets off the back of that, which is almost an orientation guide, essentially. So what we say to people is that when you land, you’re almost in a unique position where you don’t stay new for long, and you’d be amazed at how quickly you’re no longer the new person and people are coming to you. So that’s a unique opportunity with a fresh pair of eyes to be able to note all this stuff down, and then to be able to define it so that when the next people come along, they’re given a headstart. In fact, we work with some organisations where they then use that internally because they haven’t been able to create that.

Miriam: Yeah, I can absolutely see that. And it’s also when you’re the new person, you can ask all the questions and you get to listen to a lot more when you come in with new ideas. We’re seeing, being on the other end of that, we are seeing how appreciative business users are when they have team members on the more technical sides who make that effort, who speak to them, because it’s a bit like nobody ever wants to speak to finance.

Ash: Oh, that’s harsh! Well, you can say that as an ex-CFO!

Miriam: Yes! I’ve been that person! And finance functions all the last few decades have made great progress of embedding them into the business and therefore adding real value. And, the same happening for the technology piece is incredibly valuable, particularly with AI now, because even if I leave the automation side, having the ability for a business user to tap into somebody with a deep technological know-how on how to ‘tweak’ an agent that people thought was great. But actually, in the real world, things may need to be tweaked, things need to be changed, and being able to speak to that person quickly is incredibly valuable. Rather than having to go through formal processes and being held back and saying; “We’ll get back to you in three weeks time” in which that, you know, so-called automated agent is sitting there doing nothing.

Ash: Or potentially worse – doing it completely wrong.

Miriam: Oh gosh, yes!

Ash: I think that that’s a really good point there, particularly around the engineering rigour that has to go into a lot of these things. So yes, you deploy that agent, it may be that that agent works well, you know, but over time, that may drift. The outcomes may not be as good as they were when it first started so that observability and monitoring is really important. Pre- this AI age, if you had a software platform deployed on some infrastructure, if nothing changed to get the same outcomes that you can’t, I’m not in that space anymore. You know, it’s now it can drift. It’s not deterministic in its nature.

So it requires much more engineering rigour. And I think that’s one of the challenges when you have teams of individuals who maybe don’t come from that tech background. So again, in that development space, you know, in recent years we’ve seen the rise of tooling, which allows business users to be able to create apps. Which is really powerful because as business users know exactly what they want, you know, they’re able to create apps using that but sometimes the challenge is, without that engineering rigour, they can find themselves painting themselves into a corner fairly easily. Maybe it doesn’t scale or heaven forbid, it’s not secure. Or, they start off well, they’re getting through the work that needs to be done. And then they found they got a maintainability nightmare, you know, to make changes. Suddenly, lots of things break when they try and make a single change.

All those things require engineering rigour. But at the same time, to your point, what you don’t want to do is add a whole lot of process, which just slows the whole thing down.

Miriam: You said in organisations pre-AI, nothing changed. But actually, what did happen – and I’m guessing the whole thing with AI is just, you know, shining a huge spotlight at that – what did happen in organisations pre-AI is that people created their little workarounds. I remember working in very large organisations, and because something in the tech stack didn’t quite work or didn’t quite deliver, everybody had their own little Excel spreadsheets and their own little analysis tools, and later on their own apps, which obviously created huge silos everywhere.

That was my pet hate as a CFO: that there was no common source of truth, no single source of truth. And you really had to then go and dig down, spend sometimes weeks and months trying to come to an answer that was acceptable to all the various stakeholders in an organisation, particularly when it came to money. Which you’d think would be a straightforward answer. So you have that already. And now with AI, the opportunity for things going even further apart, without that engineering rigour, without keeping things together, without pulling the reins somewhere.

Ash: I think that point around the workarounds is a really key thing. We come across that a lot in automation, where you have a workflow which is defined, and there’ll be a diagram beautifully explaining how it was created in the first place. But then on the ground, it doesn’t look like that anymore. And that’s often not captured. And it’s often not documented. So that comes back to the people side that’s been able to elicit those requirements and actually understand.

Miriam: It’s the shadow organisation. It’s like the official one has the nice org chart and all the workflows and responsibilities and then how things really get done, because somebody over here needs something done over there. So they talk to each other, which in many ways is really important because:

A) Sometimes the structures impede progress. I have seen many projects being delayed purely because some boxes haven’t been ticked.So, that’s a bad thing in itself.

But also, B) it creates conversations and it creates ideas. Innovation comes from there. I chatted to [someone] from the IT team and say, “I wish it could do this”. And he’s like, “Why do you want this? You really want X over here?”. So you do want to not lose parts of that shadow organisation.

Ash: Oh, completely. It’s a valuable thing. It’s and it’s a balance to be struck between that innovation and how it’s created. But at the same time, capturing it so that shadow organisation isn’t quite so shadowy.

I remember a few years ago working with a university, and that was a very interesting scenario, because they had their enterprise tooling, but then they had applications that were created by particularly the Comp-Sci and Engineering departments that were running on machines under people’s desks. And when they came in and did an audit of what was actually running end-to-end across the state, there were more than 300 apps. But nobody could actually say exactly where these things were.

Ash: I think when many companies think about AI, they really focus on that personal productivity part, and they often don’t think of the automation, let alone that Machine Learning side of it. Is that your experience as well?

Miriam: Yes, very much so. And the really interesting paradox that we’re seeing, and that is creating a lot of disillusionment with AI, is that personal productivity does go up, whether it’s with the Agentic AI, whether it is with the chatbots, even, it absolutely goes up for individuals. But companies are not seeing that individual productivity flowing through to a company-wide impact. There’s been research recently that shows that impact on organisations – and we’re not even talking bottom line, just impact in terms of increased productivity companywide – is only at around 1 to 3%. Which obviously makes the CFO wonder, “Why am I paying more to increase productivity but not seeing any returns?” So there is a big paradox going on.

Ash: It’s probably worth us unpicking some of that a little bit as well, because there’s that point of an individual focused on that, and then the right framework to allow that to be surfaced up to a team level, and then they see the advantages that they’re seeing to be shared sort of throughout the organisation. But at the same time, you also need some top-down design now as well, in terms of policy organisationally, particularly focusing on some of those automated workflows. What can be done with chatbots and Agentic AI to be able to drive productivity for customers as well as for the internal users, but also for those internal users within the business.

Miriam: Absolutely. On the one hand, you need that – and it’s crucial that you have that rigour that you always talk about – that technological rigour that keeps all these AI tools together that allows the scaling. Whether it’s going up to team level, whether it’s flowing into other areas. But it also means that you have to think very carefully about what you do with that improved productivity.

For example, if in the past it took you two days to create a particular stakeholder report and now it takes you half an hour, what do you do with the rest of the time? The inclination for many business users is to jump like, “Let’s do more reports.” Whereas the real strategic value comes when you then say, I’m going to take another two or three hours and think deeper, analyse deeper, collaborate with others, get more opinions, and really take a more profound understanding of what these trends, what these insights are telling me, and how that can add strategic value to the work we’re doing, rather than just doing more of the same thing.

Ash: Completely, and if you’re just generating more and more reports, for example, what’s the actual value? What business decisions are you actually using from those reports? Then drive that change through. If they’re just being generated and you’re going to generate more of them, that actually just adds more cognitive load further down as well because of this whole concept and idea of ‘AI slop’. If you’re able to generate more at pace, but that still needs to be consumed, then that still needs to be useful in order to be able to do something with it.

Miriam: This focus on productivity, only in particular individual productivity, is a race to the bottom. Nobody’s going to maintain a competitive advantage to develop a competitive advantage because of it, because it’s just going to be the baseline if implemented correctly, of course.

What mindset shifts enable businesses to use AI as a competitive edge, especially in industries resistant to change?

Ash: I think there are two key strategies here: there’s a top-down and a bottom-up approach.

In terms of that bottom-up approach, those are the people on the ground who are using processes in these systems day-in-day-out, to the point you made earlier, who have all those workarounds in place. For them to be able to be empowered, to be able to build things, they need the skills, they need everything else in place, but once they have that, that’s only useful if there is a forum and a mechanism whereby they’re able to actually scale that up so they’re able to share that with their teams. Their teams are then able to share that with other wider teams. And then there’s a view across that organisation, in terms of what people are doing, and sharing that knowledge, and really understanding it. But then also making it available.

You don’t want these little silos of work to be going on. It has to be made clear and communicated that this is happening. Whether that’s done through ‘show-and-tell’s or whether that’s done more formally. You know, that works. But the key thing is really identifying somebody in each team who can be an evangelist around this, who’s an early adopter, who’s very keen on it. To be able to find that individual, train them, give them the skills they need and then have them commence work on this.

Miriam: Absolutely, and I would suggest it goes even a step further. Because it is great to have these evangelists, and that’s something we always highly recommend and support people to develop – just like you do – you also then need to create the business dynamics to allow that to happen so this person doesn’t have evangelist as a little side job in their already incredibly busy job. And that the KPIs that people are measured with reflect that desire of the organisation to have innovation in that way, to scale up, have development shared, have time that is set aside to ensure that solutions are robust, that they can scale, that it’s recognised and not seen as a little side hustle for that person.

That is a big ask. And only when you create those kinds of dynamics and really ensure that individuals see themselves as “Yes, part of my role is to circulate information of how I developed a tool and how amazingly it can help other people too. That is part of my role, not something that I just happen to do – and it’s fine”. And as well as leaders recognising that and supporting their people in that way and seeing themselves as leaders not just of ‘task doers’, but actual learners and developers, not just in the technical way, is really crucially important.

Ash: Absolutely, absolutely. I think the other approach is equally important, if not more important, it’s more of a top-down approach. At the organisational level, you need to understand what processes are being run. And sometimes it’s a focus on the bigger processes. Whereas in actual fact, if you have many people running small processes lots of the time, that’s a lot of time spent. To be able to really capture that objectively. Once you understand that, then you’re in a position to be able to see what the best cases are for return on investment on this.

You want to choose your pilots correctly, in terms of which ones you want to go for. Start with something reasonably simple, where you can really demonstrate value and get everybody on board with a process. Don’t go for something mission-critical and really big and complex first, because you’re making life far more difficult for yourself.

Once that’s done, once that pilot has been achieved, that needs to be communicated within the wider business. One thing that there is a lot of at the moment, and there was actually a government whitepaper which reflects this directly, is this concept of ‘pilot-itus’, which is what you have when you have organisations where those kind of citizen developers, if you like, are off generating things without that engineering rigour. If you’re not careful, you then end up in a scenario where you have a proliferation of these things, which are unmanaged, often not maintained, [and] different people doing the same thing.

So it’s really important that you build these things right the first time, on platforms which can scale, which are secure, which do have high availability, and that you’re able to see and plan the costs for. Because this is changing so quickly but at the same time, there are a number of key players in this field where you can build these platforms on.

And I think that’s something we’re going to discuss in the next podcast.

Miriam: Yes. Looking forward to that.