The problem of test automation overhead

Software development methods have matured during the recent years. One good example is the agile development methods. Improved development methods set requirements to the software testing. Manual testing methods do not fulfill the requirements alone. Test automation methods have promising benefits to fulfill these new requirements of software development.

But, what about test automation? Is it worth doing it? Is it efficient? We have been developing test automation in our environment. We have tried out several tools to automate test cases. We have developed comprehensive test suite with tool called FitNesse. But was it worth it?

Following figure shows the time used to desing and execute manual and automated test cases. As we can see from the figure, manual test case is fast to desing compared to automated test case. But the difference can be found from the execution times. Automated test cases are faster and easier to execute.



As the figure shows, to have efficient test automation, tests needed to be executed many times (In this case over 100 times). That is quite a lot! If the only benefit of test automation would be the time saving I would leave test automation effor immediately. But test automation offers many benefits like visibility to software, increased testing scope, regression testing, etc.

Used together with manual testing, test automation methods can overweight the desing overhead by offering these benefits. But be careful. Test cases to be automated should be selected carefully. Only most important tests should be automated. Not all... And remember, do not forget the manual testing.