A large international luxury retailer
Point of Sale (Oracle XStore)
Automation Review, Automation Factory
The client is a UK-based luxury retailer with an online sales platform and stores worldwide using their point of sale (POS) terminals in all their stores. Whilst a suite of automated regression tests was in place, supported by a manual regression test suite, the team had become dependent on manual tests for the different country configurations, whilst automated tests lacked visibility and were not being maintained.
Ten10 provided the following services:
- A Review of the current automation approach, tooling and implementation.
- Improvement recommendations to increase confidence, test coverage and visibility of the automated suit.
- Implementation of the recommendations via Ten10’s Automation Factory service.
What we did
The Review was conducted by an experienced senior test automation engineer and focused on the existing framework, evaluating whether it was fit for purpose and what improvements could be made to meet the client requirements. The Review found that the existing framework (Oracle XUnit) was the most suitable approach, as it was part of the overarching technology stack with a suite of existing tests, however, a number of improvements would increase the value of the automated tests. This conclusion took into account the duration of the engagement, the challenges of a new framework and the expertise of the client’s test team.
One of the main challenges was that the full capability of the existing regression framework was not being utilised. No assertions were in place to verify that transactions were completed, and the in-built database connections were not being leveraged, meaning that it was possible to get false-positive results in the tests.
Whilst the number of failing tests were shared amongst the project teams, the details and patterns in failures were not being shared or analysed, due in part to the lack of assertions which made it difficult to understand the exact point of the failures.
Although the existing tests covered a broad range of functionality, built over time using the manual regression tests as a guide, there was limited visibility of test execution and the coverage achieved. Naming conventions meant it was difficult to understand what was being tested without looking at the code. Without a visible link between manual and automated tests, duplication of work led to regression testing taking up to five days for a release.
The client has stores in several countries that are subject to different taxation and payment rules. A single suite of tests was executed against all countries, so the automated coverage was reduced to those features that were common across all countries, meaning a number of areas of functionality were only tested manually.
Finally, although the automated tests were stored on a shared confluence site, this did not support version control and led to conflicting changes and no audit trail. The execution of the tests required an intensive manual process, so there was no way of knowing if tests were broken until they had been executed and failed to run.
Implementation via Ten10’s Automation Factory Service
Working in conjunction with the client, we prioritised the workload and delivered against agreed targets. The first goal was to maximise the potential of the existing framework, especially the in-built database assertions for completed transactions. This meant the state of the database could be checked before and after transactions to ensure they completed as expected.
With a suite of assertions tailored to the functionality of the POS, we were able to embed assertions throughout the tests and increase the number of assertions from 0 to over 400, providing greater confidence in the tests, and aiding the debugging of failing tests. This had the added benefit when transitioning the suite back to the client, as it identified which test step was failing, enabling quicker debugging and defect reproduction.
The tests were also split out into a suite of “base” tests that were consistent across all country configurations, alongside custom tests which were configured for each location. The client could now state the location configuration for the POS terminal being tested and the custom tests would be added to the base suite, ensuring localised tests could be run and functionality was not overlooked.
Our next goal was increased traceability and visibility. By renaming tests to reflect functionality, prioritising a backlog and having two-way mapping between the automated tests and the manual regression suite in Jira, we were able to reduce the duplication of work during regression testing, as well as making it easier to understand the level of coverage. This coupled with more detailed, easier to understand HTML reports provided clearer visibility of failing tests and enabled better root cause analysis.
Once these improvements were implemented, the final goal was to remove the manual overhead of automated test execution. Using the client’s repository and CI tool meant working on the tests could be more collaborative, and the latest version could be remotely deployed and executed on the POS terminals. This simplified approach meant that anyone in the team could easily execute the suite, but also enabled scheduled execution of service connectivity to be tested prior to the team starting work, with failing tests being shared in an email.
POS regression tests are now completed in ~1.5 days, a reduction from 5 days prior to the automation improvements.
Ten10 worked with the client to maximise the potential of their existing automated tests rather than starting again and losing the investment already made in test automation. This collaboration reduced the demand on the manual test team, bringing the time required for regression testing on the POS terminals down from 5 to 1.5 days, allowing the testers to have a greater focus on higher value exploratory testing.
API testing runs as a scheduled job every morning providing meaningful feedback in time for the start of their day reducing the time taken for defects to be discovered.
A subset of the automated tests has been used to support non-POS changes (e.g. Windows updates), further reducing the workload on the manual test team and increasing the return on investment in the automated test capability.
This greatly improved integrated and Agile approach to automated testing and has been complemented by the introduction of user-friendly visual reporting, ensuring key stakeholders have access to information in a timely manner.