What test cases to Automate: (3 hurdle method)

There has been a lot of literature on selection of test cases for automation, though a lot of them regard ROI as the beacon for decision making they fall short in explaining the factors influencing the ROI.

The need for positive ROI as the elective hurdle stems from the experience where the cost of creation of test scripts is rather high, it includes tooling & maintenance cost and at lot of times the returns from ROI has not been commensurate with the investments.

The current improvement in automation technology and usage of open source tooling has helped bringing down the cost to automate thereby renewing interest in automation.

Earlier repetitive runs of a test case either due to multiple iterations or with different data sets etc was the key determinant , infact the accepted benchmarks were 5 to 7 runs as break even post which the ROI started to pay back.

We hardly saw any test case having a visibility of 1 or 2 runs getting selected, with the proliferation of software there has been a pressure on reducing development time , thus incentivizing testing to start early and in lot of cases creating a business case for automation on grounds of faster time to market.

Also test cases which are domain intensive or involve complex calculation wherein the probability of error in verification is high can provide improved outcomes on Automation.

 

With the above I believe that selection of a test case for automation has to clear 3 hurdles, the hurdles of Quality , Time and ROI. Typically automated test cases by their inherent nature pass the hurdle of time and quality when compared to their manual execution and in such cases ROI becomes the difficult hurdle, yet there are instances where time to market is the key and there is a high premium on faster execution in such cases though the test cases clear the ROI hurdle with ease we should not ignore the quality hurdle.

 

The 3 hurdle test

3 hurdle test

 

TC ROI positive

TC Quality acceptable

TC Time acceptable

 

Typical factors that influence selection of test case automation,

 

  • Cost (ROI)
    • Repetition/Frequency of test execution due to
      • Multiple builds, iterations
      • High visibility of application change/life
      • Multiple data sets
      • Different hardware or software platforms and configurations

 

  • Quality
    • Human Limitation like complexity, speed etc
      • Test cases that involve complex calculations.
      • Test that involve multiple permutations and combinations of attributes.
      • Test that are very long as a result of which probability of human error is high.

 

  • Time
    • Is there a constraint on test execution time. like testing for emergency fixes.
    • Scenarios where time to market can influence the success of the application at marketplace calls for high investment in automation to reduce qa time and hence project time.

NOTE

Though automation has its merits, while selecting test cases for Automation one should not forget the below.

  • Test automation can only find bugs which it intentionally looks for.
  • Test automation is code. Code can have defects or can be wrong. Conclusion: Test automation requires both test and review.
  • Test cases for which the requirements are changing frequently or where code is not stable should be automated with due caution.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s