Managing speed vs quality with your project stakeholders
Navigating the delicate balance between speed and quality is essential in software delivery – made even more complex by project stakeholders. Their growing targets, evolving business goals, and need to keep pace with competitors mean that their approach to a project can change mid-flight. So, how should you discuss the speed vs quality debate with them?
Here’s what Ten10’s expert panel had to say:
Emma Hargreaves
I think essentially it comes down to risk. Protection of production is a massive deal for every organisation. There’s a correlation between the willingness to accept risk and the confidence in being able to fix things if they go wrong. So (for example) bringing the website down, if you can bring it back up within 10 minutes, that’s not catastrophic, especially if you time it at quiet times. You commit to mitigating to an extent.
A lot of projects that I come across, the projects that they’re trying to deliver have been a long time in the making. They’ve gone through long budgetary approval cycles. Everyone’s desperate to get that there, but they kind of know that once this version is live, that’s it for a while because the investment is done and they’re going to have to wait for more money, more investment, to get the next iteration out. So I think that that drives a lot of nervousness about getting it right and having to make it perfect before it goes to production because if it’s not perfect when it gets there, it’s never going to get fixed. Whereas other projects where they know they’re releasing weekly or daily, it’s okay to go live with a load of SEV 2 bugs because I know I’m going to get the next batch next week, and the next batch after that, and it informs the decision.
Robbie Falck
We’ve recently been on a bit of a journey where we’ve reshaped our QA planning process. So we start with Three Amigos and we focus that more on risk: asking the right risk questions. We also focused it more on who needs to be involved. So if we don’t have to test everything, we can get other people to test things. We have Operations departments that deal with our internal systems. They know it better than us. Do we have to test it? Can that help us speed things up if they can take a bit of the load off of us?
Before we go to the test plan side of things, we put a new step in between which we call a ‘balanced approach session’. We basically broke down every QA activity that we do throughout the testing lifecycle. It starts with resource and kind of question of ‘who?’ But then it’s ticket testing, end-to-end, UI automation, API automation. Instead of seeing it as ‘we must do all of these things every time’, we change it from ‘Activities’ to ‘Levers’. That means that every activity that we do is a lever that you can pull to change the risk and change the speed.
So, for example, end-to-end testing. We could do no end-to-end testing, in which case we get a lot more speed but then the risk goes up because we’re not checking everything that’s together. We could do full end-to-end testing which takes two weeks, a lot of time, but then we get much lower risk because we make sure that absolutely everything works together. Or we could go for something in the middle, like I said earlier, where we only do the P1 scenarios, and leave the P2s and the P3s, and that gets us medium level of speed and a medium level of risk.
We work in eight-week orbits – it’s like an eight-week sprint – there’s six weeks of development in there and we shape projects to fit into those six weeks, which means we shape our testing to fit into that time too. If people are willing to accept the level of risk that comes with that, if we have to turn all the levers off, for example, then we need to say ‘We’re doing almost no testing here. You’re going to be accepting a lot of risk if we take this approach.’ We found that works quite well [and it] makes us have that conversation, think about everything we’re doing, and rethink the traditional approaches. We can be flexible in anything, as long as we make the risks clear and you’re willing to accept them.
Mala Benn
I worked in a team that was focusing on the sales journey, so there was not as much regulation around TV products as they might have been on broadband or mobile. And you could play with those levels of risk. So we did things like canary releases, where we would deliver or launch things to a small percentage of the customer base and be able to then watch and learn and see how things ran.
So there were things like that that we could do after conversations with our product teams to discuss certain features [that] we could go out with a slightly lower level of testing. You would still get a certain level through your automation testing, but some of the more manual, exploratory, or end-to-end testing or regression test levels would reduce, and we’d be able to get something out there quicker. Through conversations and agreements and discussions on the level of risk, we could get things out quicker and we had mechanisms like using our AB testing abilities to be able to just get stuff out there. It was trying to work out where those boundaries lie and the amount of risk that people could take and, as you say, reducing some levels of our test process.
With big launches, we’d always go through a really lengthy trial process. We’d have internal staff trials and then we’d go to friendly trials, that sort of thing. And after quite a lot of discussion and negotiation with the business, we would be able to reduce some of that intense and lengthy levels of testing to be able to just get stuff out there. It was all about conversation and discussion.
Vernon Richards
I agree with everything that you’ve all said. It all seems to be based on risk. So if you can understand what that risk is, there are a bunch of different people that risk might apply to.
- There’s the people building the product or the service
- There’s people consuming that as customers
- There’s the business who are trying to monetise that thing and make sure that everyone’s okay
So when you start talking about how a particular risk impacts those three different sets of people, there’s a constant tension there that you’re trying to make sure doesn’t skew in one way for too long. But if you can do it intentionally because you’ve got a clear reason, I think that makes a lot of sense.
Really what you’re trying to ask people is: what are you worried about the day after we release this thing? What you scared is going to go wrong? And then they will give you an answer to that. And then you can start talk about ‘how can we mitigate that?’ And sometimes it is very typical testing activities, but sometimes it isn’t. Sometimes it’s a canary release – how can we get this in front of a subset of people to get a clue, get some kind of data, and then we can take another decision?
Stuart Day
The risk question is an interesting one. What is the real impact of risk? Often we’ll think about the immediate risk – if this thing breaks, what’s the impact to the customer at that moment in time? But then also, there’s a ripple effect as well.
As an example, working at Capital One, credit cards are our thing. So if somebody’s going to pay with their credit card and they’re buying a week’s shopping and their card doesn’t work, it’s our fault they can’t buy their week’s shopping. When you’re in a conversation, it might be ‘the customer can’t purchase’. That’s quite simple, but what you don’t realise is that person can’t buy food for their family for that week. That has a ripple effect. That’s just one customer, what if it’s two customers? Three customers? Four customers? Then calls go to the call centre.
Robbie Falck
I work for a parking company and if you break people’s apps and they can’t pay for parking it’s amazing how angry people will be. We all would [be]. I can’t get out of my car to go do the things that I got here for because I can’t pay for it, and if I leave I’m going to get a fine. And the fines are ridiculously expensive. So any app that you build or website, you’ve just got to continuously think about what the person is actually doing in their day-to-day life and the impacts on that.
Stuart Day
So you’ve got the risk of something happening versus the real impact of if it happens. That’s what I’m hearing here, a lot around how we can mitigate those things. Some of those are technology – how we can implement canary releases – some of it’s just conversation and trying to guide people towards sort of that outcome and get them comfortable. And a point you picked up on, Emma: how comfortable are you in your release mechanisms? If your pipeline takes an hour on a good day but it doesn’t always work, you’re less likely to try and push something out really quick because you can’t fix it really quick.
How I’m looking at this from this conversation is you’re only as quick as your slowest thing, whatever that might be, and your quality is only as good as your lowest quality thing, because all those things are going to impact you in some way. But there isn’t a silver bullet to it either. You’ve got to have those conversations. You’ve got to understand your customers. You’ve got to understand your technology. What are your limitations? There’s no point in just saying ‘we need to go quicker’. Slamming your fist on a desk to say, we need to go quicker. It’s not going to make you go quicker. If anything, it’s probably going to make you go slower.
Breaking it down to its simplistic form, what I’m hearing is:
- Why? Understanding why we’re doing this in the first place.
- Who’s the customer? Who are we doing this for? What problems are we trying to solve?
- What are we worried about? What are we concerned about? What’s the impact?
Learn how to balance speed and quality in software delivery
Read all of our panellists’ insights from our Speed vs Quality panellist roundtable
Or listen to the event in full on our Ten Minutes with Ten10 podcast feed