Remote quality assurance for a Windows upgrade

The Project

The customer had requested a Quality Assurance (QA) service to support its desktop upgrade project, which consisted of both an upgrade of Office 365 and Windows 10 to their latest versions. The validations were to take place on standard laptops and any other application and device combinations deemed high priority by the customer, including any customer or operational customisation.

Due to the covid-19 situation, this project was delivered completely remotely, from onboarding and scoping through to execution and closure. This included the provision of hardware, test data creation and environment access.

What we did

We provided a clear and concise QA test plan which could also be utilised in the future planned Windows 10 and Office updates and patches.

A principal test consultant and a senior test analyst worked on this project to help identify the gaps in the current process and provided recommendations to improve the test strategy.

The project deliverables not only helped increase the test coverage and efficiency but also uncovered some issues with the current system. Overall, we were able to gain the confidence of the stakeholders on the deliverables and also show the value created by having a proper QA process in place.
Ten10’s QA plan was clearly defined into two phases:

QA Planning phase:

A complete gap analysis was performed remotely to understand the differences between the as-is system version and the to-be system versions. Following the gap analysis, a full review of all the application and customer customisations was conducted to understand and document the applications that need validation. Based on the list of applications, a risk review was undertaken to understand their criticality and priority to ensure all high priority applications are identified.

QA validation phase:

In the QA validation phase, validation was done on the standard customer issued laptops to ensure the hardware, drivers, and manufacturer firmware were compatible with the updated Windows 10 versions.

Comprehensive validation of all the applications installed on the standard laptop and Citrix environment were conducted, including integration between apps, data and customer-specific customisations. Non-functional testing was undertaken to validate the performance of the applications against the established benchmark in the existing Windows 10 and Office versions.

Additional to all the planned validations, we also helped test a couple of other applications, one of which was going to be updated shortly and the other which was going to replace an existing application. By proactively validating these applications, we made sure the system was stable and reported any shortcomings.

Challenges

In the times of remote working, it was a challenge to get the multiple teams aligned to ship the standard laptop with correct OS versions, access and all the required applications.

The original test plan and timelines had to be drastically realigned as the Citrix test environment was not set up. The initially planned validation on one of the standard build laptops was also descoped. To facilitate these changes, we made changes to the test plan and worked with the customer on the revised timelines and suggested further validations on a newer version of Windows 10, which was planned for release by the end of the year.

Success story

In spite of the setbacks initially and the reshuffling of the original test plan, we were able to accommodate all changes and deliver to the stakeholder on time.

Using a holistic, risk-based, exploratory approach, we were able to test the system thoroughly. By testing every aspect of the system both individually and together and communicating failures and other issues to the customer’s project manager, Ten10 also uncovered defects that existed in the new system and even defects in the newer version of applications that were to be deployed to production shortly.

The Office 365 upgrade versions that were tested as part of the first cycle were unintentionally deployed to production before the actual test recommendation was sent to the stakeholder. The quality assurance validation done by us was so thorough that the stakeholders decided with confidence not to roll back the deployment done accidentally.

Using Ten10’s experience in delivering remote QA services, any challenges regarding the remote nature of this project were fully mitigated by careful planning and regular communication, coordinated by Ten10’s experienced Test Consultant.

Benefits

Not only did our testing enable the successful delivery of the service to the end customer, but it also gave confidence that it could be demonstrated successfully to its customers and therefore gain market traction. So successful was this approach that the customer engaged Ten10 on another project.