Essential questions to evaluate your test automation strategy

Does your organisation have a strategy ready to make the most of test automation?

A robust strategy is the backbone of successful test automation implementation. While interest in all things automation is high, and automation technology continues to evolve in exciting ways, putting the technology to work properly always comes down to effective strategic thinking.

Ask yourself these questions to think critically about your tools, applications, objectives, test environments and more essential elements of your test automation strategy:

Have you assessed your existing tools and technologies?

Before you think about your test automation strategy, you need to think about the programming languages you’re using and your infrastructure – what you’re doing at the moment and how your team currently works. List the tools your organisation uses, think about what integrations are available, and whether your staff need training or support to use them.

What tests are you trying to automate?

Assess your project timelines and milestones – what tests should be automated, and which should remain tested manually? Consider a risk-based approach to decide what to automate and how this can improve your project’s ROI.

Too often, people move to automate everything then feel overwhelmed and lose confidence when regression tests pass but basic issues are not caught. When getting started, consider adopting the 80/20 rule: aim to automate the most important 20% of test cases, first covering the areas which are more likely to be interacted with by 80% of your users. Start small and build up as confidence in the suite develops.

Which applications can you apply automation testing to?

Establish which technologies they are based on and whether your test automation platform supports these technologies. Typically, implementing automation testing will involve diverse application types: web-based, desktop-based, mobile apps, etc. Therefore, it’s important to have tools that can meet all your automation testing requirements.

Are you considering every layer of the test automation pyramid?

When organisations consider test automation, they often concentrate on the top of the pyramid – the User Interface tests – as those tests validate the controls that an application’s user directly interacts with (such as menus, buttons and icons). This is a mistake. Test automation concentrated on this level can prove brittle and likely to break as the application continues to be developed. Instead, effective test automation starts at the bottom of the pyramid with Unit/Component tests.

How will you integrate automated testing into your teams?

Do you need a dedicated automation team or do you need to integrate automation as part of your in-sprint testing? Whatever approach is used, all members of your teams need to know who is responsible for each element of the automation project.

What are your primary objectives?

Clearly define the parameters against which you can measure the success of your test automation strategy. Be specific when defining your goals. Common goals in test automation can include reducing testing time and cost, increasing the speed of the testing process, improving test coverage, reducing redundancy, improving quality, and reducing manual intervention.

You should also set key performance indicators (KPIs) for these metrics. For example, you can set a target of a 40% reduction in testing costs or 70% faster regression testing. These should always align with the broader company’s goals and objectives.

Have you evaluated your test environments?

Assess the current state of your test environments and consider the data you will use as part of your tests. Does the environment allow test cases to be repeatable with the same data? What happens when you change to a different environment to run the tests? Are your tests linked to data available in a specific environment? What integrations do you have in place that may restrict an end-to-end journey?

How will you execute your tests?

A robust test suite provides confidence in the application, how will you execute it? Will you only be running it locally, or will you run it on a remote executor? Will this remote executor be a dedicated virtual machine or one provisioned on the fly as per the demands of the team? Will this be integrated in a CI/CD pipeline to run as frequently as possible as development changes are made, to quickly catch any regressions?