Managed test service for a comprehensive change programme

The Project

The customer was embarking on a comprehensive change programme which aimed to fully replace its existing Intapp matter lifecycle ecosystem by the end of 2022. This included the below modules:

Whilst project management, business analysis and development were being managed by the customer they wanted to improve their test and quality assurance (QA) capability, and move away from a ad-hoc approach formed of temporary teams from different areas within the business.

They approached Ten10 due to our experience and background in delivering similar test projects in the legal sector to support the QA of the matter lifecycle change programme.

What we did

Ten10 was initially tasked to conduct a review of how QA was historically conducted by the customer. This was delivered over a five-day period which included interviews with key stakeholders and reviews of existing QA collateral, allowing Ten10 to make an assessment of the customers QA maturity, using our internal maturity framework.

Once the initial maturity assessment was completed Ten10 defined an overarching QA strategy which would be adopted for all matter lifecycle change projects.

Following the approval of the QA strategy Ten10 provided a managed service to deliver full end-to-end testing for the first wave of change programmes which included the implementation and integration of the Intapp Conflicts Management (Conflicts), Client Matter Intake (CMI) and Time Management (Time) lifecycle modules. This included:

• End-to-end test management
• System level testing
• System Integration testing
• User acceptance test support
• Compatibility testing
• Performance testing

Success story

Ten10 defined a bespoke QA strategy which addressed many of the maturity development areas highlighted in the initial QA review. It defined a hybrid test approach which considered the waterfall nature of delivery from vendors but implemented agile best practice such as backlog planning, close communication, and continuous improvement. It also defined a strict governance approach, recommended reporting structure, approach to defect management and automation and defined required environments for testing.

During each testing engagement Ten10 ensured scripting underwent full review from the customers business analysts to ensure coverage and execution was scheduled in line with expected drops from the vendors. This allowed the team to prioritise their activities effectively. Risks and Issues logs, regular defect calls and daily and weekly progress reports were implemented which allowed issues and blockers to be escalated quickly.

As part of this engagement Ten10 also defined an automation framework to automate the regression packs of each matter lifecycle project. Due to some applications operating via web and some via desktop a framework was needed which catered for both. Following the review of a number of automation tools Katalon Studio was used. This provided an open source option to the customer which was easy to maintain and provided a robust level of reporting.

Performance testing was successfully run in collaboration with the customer infrastructure and external support teams. This was the first time the customer had run formal performance testing so Ten10 walked them through the approach, helped them define the volumetrics models required for testing and ensured that the correct support stakeholders were available and understood their actions when performance test execution was conducted. Whilst no major issues were identified during performance testing a number of medium issues were highlighted which would have otherwise been unknown to the customer.

Challenges

Ten10 faced a number of challenges during the engagement:

• Deviations from the QA approach
  As this was the first time the customer had a formal QA approach defined it took time to transition to an improved way of working. In particular the definition and sign off of user stories. In many instances user stories were vague, or changed once builds were delivered from the development partner. Ten10 were able to use their legal background in testing similar applications to fill gaps and had to be flexible and responsive where user stories did change to ensure scripts remained accurate.
• Lack of collaboration tool for defect management
  Despite recommendation in the QA strategy a collaboration tool was not available for defect management. Whilst Ten10 were able to implement an excel based solution this was time consuming with collaboration limited. Ten10 continued to raise this lack of collaboration as an Issue during the weekly risks and issues meeting and Jira was eventually implemented. Ten10 supported the set up for this and documented the ways of working for both user story and defect management.
• Environment access for automation and performance tool set up
  Due to the remote nature of the engagement the Ten10 team required remote access to the customer’s desktop to perform testing. This access also required the ability to download and install various tooling to perform automation and performance testing which required admin rights to these desktops. Due to security restrictions admin access to these desktops was time consuming which could have led to significant delays. Ten10 had to be flexible in terms of team location often being required to shift work from Ten10 to customer offices to allow progress to continue.

Benefits

Ten10 delivered a number of benefits as part of this engagement:

• Improved quality assurance maturity
  During the initial maturity assessment Ten10 felt the customer was operating at a ‘managed’ level (the second of the five level maturity framework). Following the delivery of three change programmes and through the adoption of the principles defined in the QA approach Ten10 now feels the customer is operating at a ‘measured’ maturity level (the fourth of the five level maturity framework). By using Ten10 they now have a formal test organisation in place, have formal defect management across the programme delivery, have adopted automation and performance testing and have tooling in place to generate reports and QA metrics.
• Automated regression pack
  Ten10 have delivered an automated regression pack for both the CMI and Conflicts projects. This provides happy path validation following new builds being delivered and allows builds to be assessed within 10 minutes vs 0.5 days it would take to run manually. In addition to delivering the automation framework and packs Ten10 also defined an automation ‘How to Guide’ which has been shared with the customer.
• A formal approach to performance testing
  Following the completion of performance testing across three projects the customer now has an understanding of how performance testing should be run for change projects, the support that is needed and what is assessed as part of performance testing. This had not been done previously and now allows this approach to be replicated across future change projects.