Is functional testing or non-functional testing more important to your project? The reality is they go hand in hand – Joseph Michael, Ten10 Principal Consultant
So you want to set out a comprehensive test strategy for your project, but what testing do you need to focus on – functional or non-functional testing?
Testing is fundamental to the successful delivery of any IT project. It provides the framework which demonstrates that the system being built fulfils the specified capability and satisfies the expected behaviour as set out by the project sponsors and users. Intelligent and mature software delivery is intertwined with a thorough and structured testing regime.
What is the difference between non-functional testing and functional testing?
The main purpose of functional testing is to exercise individual functions or processes (sequenced aggregation of functions) and prove that they produce the expected results and outputs. Functional testing includes (but is not limited to):
- Unit testing
- Component testing
- Regression testing
- Integration testing
- API testing
- UI testing
- System testing
The primary aim of non-functional performance testing is to demonstrate that the system and the platform it is being built on, can operate effectively and efficiently under conditions that are representative of how it will be used in live service. Non-functional testing includes (but is not limited to):
- Security testing
- Reliability testing
- Recovery testing
- Stability testing
- Usability testing
- Scalability testing
- Interoperability testing
- Accessibility testing
- Performance testing
Both fields of testing have specific functions but, when delivering a project, functional and non-functional testing usually come hand in hand.
Why do people perceive a choice between the functional and non-functional testing?
Discussing why people think of ‘functional testing vs non-functional testing’ rather than the two working in harmony is rooted in the tunnel vision that can occur when planning a project. Let’s create an example business and show how this mistake is made.
Considering functional testing first
Imagine that you work for a train operating company. This company has recently bought a new franchise for which you need to develop a new ticket booking mobile application. Initial analysis is likely to conclude that the immediate priority is to ensure that the application allows for the sale of tickets to the public which must have the following high-level functionality available to users:
- Allow users to select the parameters for the journey
- Show ticket availability and prices for the chosen parameters
- Allow users to select the tickets they would like to purchase
- Take payment for the selected tickets
- Provide electronic delivery of the tickets and proof of payment
- Allow users to re-book or cancel tickets and provide refunds
As a business, you could add a suite of additional functionality (for example: allowing passengers to upgrade the class of carriage, pre-order their food, or secure bicycle storage) but as the base iteration, you are confident that the critical core functionality necessary for the business to operate and succeed has been defined.
The next logical step is to assemble the design and development teams to build the application, a test team to system test the functionality followed by user acceptance testing (UAT) by the business to confirm they are satisfied that the core business processes function as they should do. You would also plan a rigorous session of testing by end-users (the public and potential passengers) to get the rubber stamp of great user experience.
With full endorsement of the application from the functional test team, business and end-users it would appear that all the necessary positive indicators are in place for the launch of the application into live service to be a resounding success. Maybe, but probably not.
The sole focus or ‘tunnel vision’ on the functional aspects of the application will likely lead to serious pitfalls. Many vital facets of a system are non-functional. If they are ignored, they can significantly compromise the success of the application, from both an operational and compliance perspective. Failures in these areas could incur penalties, cause reputational damage to the business or even result in legal proceedings and prosecution.
Realising non-functional performance testing is needed
Revisiting the application for purchasing train tickets, if we take a step back and a broader view to include the non-functional aspects of the application, several additional considerations start coming into the field of view. Typical non-functional aspects that need to be considered include:
- How quickly does the application need to respond to users?
- Which mobile devices, tablets and browsers does the application need to work with?
- How do we ensure that the application is available to users 24/7?
- How will updates and fixes be delivered once the application is in live service?
- How will we know if something is going wrong with the application or the users are experiencing errors or interruptions?
- What confidence is there that the application is secure and not susceptible to cyber-attacks?
- Are there any legal requirements that need to be complied with relating to the data the application uses and stores?
- Is the application fully accessible to international and domestic industry standards?
As with the application functionality, the non-functional attributes and behaviour must be tested and validated through non-functional performance testing before promoting the application to live service. In terms of test delivery, it mandates that additional and specialised test types are incorporated into the testing lifecycle. Typically these would include:
- Performance Testing
- Compatibility Testing
- Security Testing
- Penetration Testing
- Operational Acceptance Testing
- Disaster Recovery and Business Continuity Testing
- Accessibility Testing
Even when the nature of an application may appear to be wholly orientated in providing functional capability, such a presumption needs to be challenged with non-functional testing tools. Both the functional and non-functional aspects must be evaluated and assessed at project inception.
Risk consideration affects your decision making
As illustrated in the ticket booking example, there is a substantial amount of analysis when trying to establish the appropriate level of functional and non-functional testing required. However, the level of testing for any system needs to ensure that the core value of the solution is delivered whilst mitigating potential risks.
Consider a system for a legal firm or government department that is required to store classified files. Security is naturally paramount, but file accessibility across all teams is still a major consideration. Access needs to be restricted and highly regulated yet enable fast downloads across encrypted networks. No loss of data or unauthorised access is acceptable. In such a scenario, testing will inevitably be focussed on security, performance, recoverability and back-ups – therefore predominantly on the non-functional aspects of testing.
On the other hand, an application for learning a new language will need to provide a rich user experience in terms of video, audio, graphics, course modules, learning progress, completion badges etc. Functional testing of the application here is key to ensuring the core benefit of enhancing the language skills of the end-user is delivered.
In conclusion, the question is not ‘functional testing vs non-functional testing’. In reality, it will always be a mixture of both functional and non-functional testing in software testing. The pertinent question is how much of each is required to deliver the expected business benefit whilst mitigating the high priority risks.
There is no uniform answer; it is a case by case evaluation. It is a difficult and complex landscape to navigate and you need the right people with the relevant expertise and experience to get the right answer.