1. Automated Testing is a key success factor for “Continuous Delivery”
Continuous delivery takes the idea of continuous integration and advances it one step further. Automated testing is an extremely important middle step to realize continuous delivery. With continuous delivery, the code is not only integrated with changes on a regular basis, but it is also verified automatically safely and quickly in a sustainable way.
2. Challenges of GUI Automated Testing
However, if just seek a 100% Graphical User Interface (GUI) automated test to fulfill the needs of continuous delivery, some obstacles affect automated verification in delivery pipeline. GUI automated test scripts development is often left to next iteration until new features are stable because of following difficulties.
- Automated test can only be started when development is done;
- Automated test need to be updated when UI is changed;
- Business logic need to be verified in multiple platform, like both desktop web and mobile web;
- GUI testing need to resolve page loading or compatibility issues of OS and browsers;
- Test automation at presentation layer is often expensive both to produce and to maintain over time.
3. Layered Testing Pyramid
Micro service architecture is very popular because of its loose coupling feature, and applications can be integrated with and sharing services flexibly. Modern End-to-end business processes have rich and complex steps that are happening below the GUI within what is sometimes called the ‘business layer’ through API calls and data bases interfaces. Testing should also take care of the quality of the smallest component and services instead of only GUI layer.
Layered testing pyramid should be like figure2, and business logic acceptance testing will affect the result of GUI testing in wired, mobile and other devices. Except exploratory test, Unit test, Service test and GUI end to end system testing is suggested to be automated for continuous delivery.
4. Layered Automated Test
Let’s introduce layered automated test from a simple scenario in training system. A trainee can register a course using mobile application, mobile browser or wired browser, but registering time and available seats of the course are limited. Its business logic is implemented in service layer as figure 3.
The business logic of registration is exposed as restful services.
- When trainee register course before registering duration, service return a failure code and message;
- When trainee register course after registering duration, service return a failure code and message;
- When no available seat is left for course, service return a failure code and message.
- When there are available seats and trainee register during the right time, service return successful code and message.
Unit test should test the smallest unit of code, like time comparison. It is always done by developer, but QA testers had better to review unit test code. The overall coverage is suggested to be over 85%.
Service test can be automated to verify course register logic with different input parameters following below verification points.
- Verify returned codes and messages for above scenarios.
- Verify registered person number is calculated correctly by course id.
- Verify seat number is calculated correctly by course id.
- When trainee need to pay for the course, system need to be integrated with Payment services.
The response of course registration service and course info service is as figure 4. Then service integration with the payment system can be verified if it is needed. Mocked service can be built to simulate the response of payment, when it is not available for testing. It helps service test shift left in the whole process. The overall coverage is suggested to be over 85%, and should cover the part which unit test missed for.
What to be automated test from GUI? When business logic is accepted, presentation layer test is to verify the integration between UI and Service, and user interaction can work as expected from presentation layer.
Click register button and verify course is registered with correct seats and attendees displayed. Because seat number calculation is verified in service and use the same web elements for four scenarios as figure 5, it should pass all with one scenarios verification. When the failure or successful message are displayed with the same text element, it is not necessary to verify message displayed in GUI for every scenario either. In this case, 75% of test cases were removed from GUI automated test in this case. If no message is shown for successful scenario, another exceptional test scenario need be covered to verify message UI element and 50% of test cases were reduced from GUI test.
In this way, when new features are applied to UI, tester maintain less test cases in GUI. Service test and unit test can be reused to verify business logic is correct. Even if business is changed, scripts maintenance in service and unit test is easier than GUI testing scripts.
5. Exploratory Testing
Exploratory testing, is all about discovery, investigation and learning based on business and testing knowledge. Tester need to know which has been covered by existing test cases, and then explore more scenarios. Sometimes, we call exploratory test as “Bug Out”, as it looks for defects hided in complicated and uncommon scenarios. The improvement of layout, color, and requirement can be detected during this phase. It uses testers’ expertise and intelligence to validate software and improve it.
6. Conclusion
Layered testing can reduce the dependencies for testing scripts development, and shift testing earlier by maximizing and reusing non-GUI testing. The defects of business logic can be detected before the whole feature is done.
It can also minimize the use of hard-to-maintain GUI tests, as we move business logic verification to service layer. As service test scripts is easier to maintain and faster to execute, layered approach can reduce response time of testing and overall cost.
Last but not least, QA testers are suggested to being involved in presentation, business, and data Layers design, so that application is more testable and extendable.
