Planning Good QA Automation Scope

by Vitalii Tsybulnyk 11. June 2010 13:20

It seems to me that the most important part of a successful automated test infrastructure is an elegant approach for test planning/coverage/prioritizing. Below are my thoughts and ideas on how this approach should look for a high-quality long-term software project.

In order for your automated tests to be reliable and fast enough, but also to test the product the same way customers actually use it, you should have different types of tests for different purposes. Here is a classification which looks reasonable to me:

· Feature (API) tests – each test in this category should cover one particular small product feature. You should exclude all extra product activity (e.g. pre-requisites creation, product configuration) and replace it with test code. This test should run in the fastest and the most reliable way, so API should be used instead of UI when possible. Feature tests have to be 100% reliable and fast, so you should be able to run them nightly.

· Scenario (UI) tests should cover several (not many) features each and test them the way the customer really uses them – via UI. To keep these tests comparatively short, some pre-requisites might be pre-created with test code; however all steps involved into customer scenario should be done with UI.

Each scenario test will probably cover several features, so Coverage matrix should be tracked to make sure that the feature coverage is full and even, e.g.:

  Scenario 1 Scenario 2 Scenario 3 Scenario 4
Feature 1 X X
Feature 2 X X X
Feature 3 X X
Feature 4 X X X
Feature 5 X X X

Scenario tests take much longer to execute and are less reliable then feature tests (because they use UI, a problem in one feature could fail several scenarios, etc.), so they should be run rarely.

· End-to-End (UI) tests cover many features (often the whole feature area), which makes this test the closest representation of actual user activity and provides the highest level of integration testing. E2E tests are super long and the least reliable among all test types, so they should be run very rarely.

· Stress (API) tests are in fact feature tests, but run many times each. Test infrastructure should allow for running feature tests just in ‘stress’ mode, without writing separate stress tests.

· Fuzz (API) tests are in fact feature tests, but run several times each with different prerequisites and various (random) input. Test infrastructure should allow for running feature tests just in ‘fuzz’ mode without writing separate fuzz tests.

So you should plan and run tests according to the matrix of test types and test targets, e.g.:

  Feature (API) Scenario (UI) E2E (UI) Stress (API) Fuzz (API)
Module 1 nightly weekly monthly monthly monthly
Module 2 nightly weekly bimonthly bimonthly bimonthly
Module 3 weekly monthly bimonthly bimonthly bimonthly

or

  Feature (API) Scenario (UI) E2E (UI) Stress (API) Fuzz (API)
OS 1 nightly weekly monthly monthly monthly
OS 2 nightly weekly bimonthly bimonthly bimonthly
OS3 weekly monthly bimonthly bimonthly bimonthly

Tags:

Software Quality Assurance

Add comment




  Country flag

b i u quote
Loading


Blog