The problem of test automation overhead

Software development methods have matured during the recent years. One good example is the agile development methods. Improved development methods set requirements to the software testing. Manual testing methods do not fulfill the requirements alone. Test automation methods have promising benefits to fulfill these new requirements of software development.

But, what about test automation? Is it worth doing it? Is it efficient? We have been developing test automation in our environment. We have tried out several tools to automate test cases. We have developed comprehensive test suite with tool called FitNesse. But was it worth it?

Following figure shows the time used to desing and execute manual and automated test cases. As we can see from the figure, manual test case is fast to desing compared to automated test case. But the difference can be found from the execution times. Automated test cases are faster and easier to execute.



As the figure shows, to have efficient test automation, tests needed to be executed many times (In this case over 100 times). That is quite a lot! If the only benefit of test automation would be the time saving I would leave test automation effor immediately. But test automation offers many benefits like visibility to software, increased testing scope, regression testing, etc.

Used together with manual testing, test automation methods can overweight the desing overhead by offering these benefits. But be careful. Test cases to be automated should be selected carefully. Only most important tests should be automated. Not all... And remember, do not forget the manual testing.

Improve your review process with PMD

Do you spend lot of time to review your code? Does it feel like unnecessary work? Would you like to get rid of boring and unproductive manual review process or start reviewing your code easily? Here is one effective solution to replace the manual review process: Tool called PMD.

I have studied the effectiveness of the manual and automated review process. Automated review was conducted with tool called PMD. It is an easy to use plugin for many IDEs that review your Java code and finds potential problems to be fixed. It finds example possible bugs, dead code, suboptimal code, etc.

Following table summarizes the findings:

	Manual review	Automated review
Time	120 min	5 min
Relative hourly cost	100	100
Relative cost	200	8.33
Problems found	29	31
Relative cost/problem	6.9	0.27
Priority points	88	95
Relative cost/point	2.27	0.09
Lines of code	201	201
Rows/problem	6.9	6.5

As table shows, automated review could be 25 times more effective than manual review. It found almost same number of problems than five person manual review with same priorities. If automated review can be made 25 times faster with same kind of results, it is obvious that it should be used instead.

My suggestion is that you leave your manual review process until you have fixed all problems found with PMD. After that you can find advanced problems with manual review like logic problems. And PMD is also very good solution to begin the review process to improve your product quality. So why don't you give it a try?

Good, bad and my favourite bug

Some bugs are bad; some of them are terrifying. But not ALL software errors are bad. The importance and severity of defects varies. Sometimes the benefit derived from removing an error is less than the cost of removing it.

As working as a tester, I have found hundreds of different bugs during my work and free-time. I will demonstrate some of them for you (Mostly those found during free-time :o). Let’s start with my favourite defect: The (HTML) injection. The idea is to insert code to a free field. The magic comes from the fact that page echoes the invalid input. (Dictionary)

This kind of bug might be quite dangerous when some of the following conditions are met:

1. Input is saved and viewed to other users. This enables example Cross Site Scripting (XSS)
2. Injection makes possible to execute SQL statements (See example)
3. Some illegal services are used through injection

I have found that this kind of defect is quite common. Even if it is common, it is quite easy to fix. As a rule of thumb, all inputs should be validated and special characters shouldn’t be allowed if they are not really necessary. In many cases characters and numbers are enough. If somebody has a cool solution for the input validation, please share it with us…