Test Management in a multi-vendor environment for a global law firm
Elite 3E Enterprise Business Management
Quality and Test Strategy Consultancy, Managed Test Services, Performance Test and Engineering
Ten10 was engaged to provide testing services for an International Law firm, who was in the process of implementing a heavily customised and integrated Practice Management System.
Ten10 was joined by multiple providers on the programme including the client, the vendor, application experts, data analysts, training providers and other testing providers. Ten10’s responsibility included governing the overall testing effort, providing quality assurance of the other test providers, implementing test processes (defect, review, rework) and tools (TFS), providing regular reporting (passive and active) to various levels of stakeholders as well as executing the testing and defect management.
The development of customisations was to be delivered in a manner leaning heavily on Agile principals however the wider programme and integrations were following a more traditional Waterfall methodology.
What we did
The primary role of functional testing involved Ten10 working with a third party test supplier who were application experts. The third party was responsible for writing test scripts. Ten10 implemented a review process for these scripts using the third party’s Leads to perform an initial peer review.
This was subsequently passed to client Subject Matter Experts to review from a business perspective. Finally, Ten10 reviewed to ensure they were written to standards before execution and uploading into Microsoft’s Team Foundation Server.
The team was also responsible for planning and coordinating a UAT phase involving over 300 scenarios, executed by 150 business users, spread over 5 different global locations.
Ten10’s total team onsite flexed throughout the duration of the programme up to a maximum of 15 onsite concurrently. The team’s structure included; Test Director, Test Management, Test Leads and Test Analysts. The role of the Test Director was to set strategy, manage stakeholders, control budget and set resource models depending on demand and constraints.
To help ensure efficient delivery of the testing, an onboarding process was set up that involved a mixture of application and process documentation, shadowing and mentoring as well as using logical functional areas of the system for initial execution.
A rotation system was implemented to remove ‘key man’ dependencies.
An integration team was established to focus on the 30+ direct and indirect integrations in the overall landscape that relied on the core product being implemented. This required definition of requirements and a formal Integration Test Plan to be defined.
The plan utilised the priority of each integration, as well as the complexity to order the testing in a risk-based approach. It also provided a resource model and forecast. One critical component of the end-to-end solution involved a US-based 3rd party testing provider. Ten10 was asked to coordinate with them to set expectations on progress and ingest their metrics to provider wider reporting.
As well as functional testing Ten10 also implemented a full automated framework using Eggplant which fully automates the 3E regression pack. This has developed to an automation first approach where 80% of all manual 3E scripts are now automated.
To support the onsite team and the client, Ten10 delivered through an account management service which involved regular account review meetings with the programme director and the client head of QA. A mid-point service review was also conducted. This involved a test expert not involved in the programme coming onsite reviewing processes and collateral, interviewing Ten10 resources and stakeholders, and delivering a health report to both the internal team and the client.
Finally, regular and ad-hoc meetings were organised for the programme director and various other stakeholders to meet Ten10’s Client Engagement Director to discuss tactical and strategic consulting needs, providing valuable expert guidance on best practice.
Ten10 had a number of challenges to overcome as part of this engagement.
There was no formal development roadmap in place which made it difficult for Ten10 to plan in terms of resourcing requirements. Ten10 worked with the development partners to identify forecasted development delivery of new code or fixes and based on this was able to identify any constraints to increasing productivity. When this constraint was resource-related, Ten10 was able to plan onboarding of additional personnel in a timely and planned fashion.
The integration team had little existing architectural and infrastructure documentation to work with. This was resolved through a discovery phase to document the entire application landscape with integration points and data flows. The team worked with solution architects, enterprise architects, application owners and the client infrastructure team to put this in place. The resulting documentation was then used to create a test plan to cover; data migration testing, point-to-point integration testing, end-to-end system integration testing and user acceptance testing.
“Partnering with Ten10 to develop a highly effective and professional testing approach has enabled us to pursue the implementation with pace and confidence.”
Steve Capon, Head of Global Finance Projects, Hogan Lovells
Through strict test governance and Ten10s ability to flexibly scale up and down testing resources in line with development forecasts, we were able to successfully support the delivery of a major PMS transformation project on time and on budget.
This has led to Ten10 being continually utilised for ongoing PMS enhancements as well as a further major upgrade which is planned through 2020.
In addition, the automation first framework that is now in place had delivered significant timesaving in terms of 3E execution. As of April 2020, 800 3E scenarios are automated which take 1 day to execute. If these were to be executed manually it would take 2-3 weeks for a single resource. This allows the client better and more regular regression coverage, improving test quality and identifying defects early.