1.Challenge of API Testing
API testing is a black box test, which is used to determine whether APIs return the correct response in the expected format for a broad range of feasible requests, as well as react properly to edge cases such as failures and unexpected/extreme inputs.
The quality of the test is often evaluated by code coverage. In terms of API testing, we are concerned not only about the success or failure of one test case, but also about describing the degree to which the source code is executed and the success or failure of all test suites on a global perspective. It needs to combine the code with the requirement, so that we can use the coverage in digital indicators to measure the quality of the tests.
2.API Testing Tools Selection
It is a JavaScript-based project, and the main technology used in the project consists of MongoDB, Express, Angular2JS and Node.js. The test framework uses the Behavior-Driven development(BDD) model.
BDD avoids the problems in the traditional development model: customers and developers feel it’s difficult to understand each other, which means it is difficult for the customers to understand the problem from the technical level, and for the developers to consider the problem from the business perspective. The common language of BDD is a form of near-natural language description software.
Disconnection between requirements and development can be solved. It starts from the needs of users, and ensures that the implementation of the program is consistent with the user’s needs.
Jasmine is a popular and easy-to-use BDD framework for testing JavaScript code, and it is recommended by the Angular2 official document. The Node.js module makes this framework available in node.js. These features make it very suitable for our project.
As we mentioned above, code coverage of the API test is an important indicator for test quality. One of the most suitable tools to get JavaScript-based code coverage of API testing is Istanbul. With Istanbul, we have a holistic view of specs’ coverage.
For Node applications, such as a web application test, it needs to cooperate with an HTTP proxy to complete the HTTP request and routing test. Super-test is chosen in this paper as it is an HTTP proxy service engine that can simulate all HTTP request behavior. Super-test can be used with any application framework to perform the test.
As a matter of fact, we originally intended to use the frisby.js framework for API testing, but this framework is developed by the individual, and has not been maintained for several years, so we changed the plan quickly.
At last, we choose Jasmine, Istanbul, and Super-test to work together for API testing.
3.API Testing Framework in Our Project
The project is deployed on the Docker and continuous integration is implemented with Gitlab CI. The API testing framework we developed is as below.
The database uses MongoDB, and data in the project is stored in the form of JSON. Each collection has their own prepared data separately. When executing test cases, the clear operation is performed first for the related collection, then we put prepared data in the collection. After the execution of each test case, the collection data should be restored.
The utility is the core of implementing requirements, which can be divided into two parts: client and server. The server part contains a main file, a configuration file, API router files, prepared data, API functions and MongoDB functions and so on.
Test scripts are all placed in the spec directory, they are client test scripts, API test scripts and unit test scripts. All test case files should be named as *spec.js to run with jasmine-node commands. Test cases should not interfere with each other, they are all independent of each other.
There is a file called package.json in the root directory of the project. It defines the dependencies that need to be installed. Input ‘npm install’ command in the terminal, then you can install all modules involved in package.json under the node_modules folder in this directory, rather than install the package manually one by one.
4.Junit Report and Coverage Report with Istanbul
After executing test cases, it generates a Junit report under the test results path, which is displayed in an xml format. The Junit report can be parsed to an HTML report easily with popular CI tools, like Jenkins, Hudson and so on. As the picture shows, it clearly records every test case in every test suite running result
After executing test cases, we want to see a general report as the result to indicate how to improve testing quality. In the root directory of the project, a folder called ‘coverage’ will be automatically created, which is used to store the Istanbul code coverage report.
For this project, it has four metrics of code coverage criteria. They are statements coverage, branches coverage, function coverage and lines coverage. A combination of function coverage and branch coverage is sometimes also called decision coverage.
As we can see from this coverage report, Statements coverage is 88.7%, Branches coverage is 63.89%, Function coverage is 83.51%, Lines coverage is 88.7%.
Each dimension has a percentage coverage form and fraction coverage form. What we can see is that src/server/controller/loginController/ branches coverage is only 37.5% and function coverage is only 25%, then I will click into this file to figure out which function and which statements in them were not tested.
