But, what about test automation? Is it worth doing it? Is it efficient? We have been developing test automation in our environment. We have tried out several tools to automate test cases. We have developed comprehensive test suite with tool called FitNesse. But was it worth it?
Following figure shows the time used to desing and execute manual and automated test cases. As we can see from the figure, manual test case is fast to desing compared to automated test case. But the difference can be found from the execution times. Automated test cases are faster and easier to execute.
As the figure shows, to have efficient test automation, tests needed to be executed many times (In this case over 100 times). That is quite a lot! If the only benefit of test automation would be the time saving I would leave test automation effor immediately. But test automation offers many benefits like visibility to software, increased testing scope, regression testing, etc.
Used together with manual testing, test automation methods can overweight the desing overhead by offering these benefits. But be careful. Test cases to be automated should be selected carefully. Only most important tests should be automated. Not all... And remember, do not forget the manual testing.
I have studied the effectiveness of the manual and automated review process. Automated review was conducted with tool called PMD. It is an easy to use plugin for many IDEs that review your Java code and finds potential problems to be fixed. It finds example possible bugs, dead code, suboptimal code, etc.
Following table summarizes the findings:
|Manual review||Automated review|
|Time||120 min||5 min|
|Relative hourly cost||100||100|
|Lines of code||201||201|
As table shows, automated review could be 25 times more effective than manual review. It found almost same number of problems than five person manual review with same priorities. If automated review can be made 25 times faster with same kind of results, it is obvious that it should be used instead.
My suggestion is that you leave your manual review process until you have fixed all problems found with PMD. After that you can find advanced problems with manual review like logic problems. And PMD is also very good solution to begin the review process to improve your product quality. So why don't you give it a try?
As working as a tester, I have found hundreds of different bugs during my work and free-time. I will demonstrate some of them for you (Mostly those found during free-time :o). Let’s start with my favourite defect: The (HTML) injection. The idea is to insert code to a free field. The magic comes from the fact that page echoes the invalid input. (Dictionary)
This kind of bug might be quite dangerous when some of the following conditions are met:
- Input is saved and viewed to other users. This enables example Cross Site Scripting (XSS)
- Injection makes possible to execute SQL statements (See example)
- Some illegal services are used through injection
I have found that this kind of defect is quite common. Even if it is common, it is quite easy to fix. As a rule of thumb, all inputs should be validated and special characters shouldn’t be allowed if they are not really necessary. In many cases characters and numbers are enough. If somebody has a cool solution for the input validation, please share it with us…