Be on the same page with us
Subscribe to our latest news
By clicking the button Subscribe you give a permission for the automatic sending of e-mails

What amount of testing do you really need?

Speaking about what amount of money you want to invest in testing activities "the more the better" is not a valid approach. Consulting companies often sell QA engineers whenever there is a budget opportunity, so there is a common image of a quality budget hole in the industry.
November 22, 2017
I suggest a different approach: analyzing your risks. All the risks in software development fall into two types: technical risks (the product will not work as intended) and business risks (when a technical failure may influence the business). Common risk factors for the software development projects are:

1. Business-related Quality risks:

  1. Affect of the failure on company revenue
  2. Affect of the failure on the product churn rate
  3. Number of customers using the product day-to-day
  4. Number of other products using the output of the system
  5. Sophistication of the end users
  6. Subject-area quality standards compliance (e.g., medical or government software)
  7. Costs associated with late delivery
  8. Visibility of the product quality by senior management
  9. Etc.

2. Development-side Quality risks:

  1. Unexpected Requirements changes
  2. Misestimating Workload
  3. Turnover of project team members
  4. Engineering compromises
  5. Fresh, unproven technologies
  6. Bad project management
  7. Test environments instability
  8. Team experience (specific toolset)
  9. Lack of communication or documentation
  10. Changes in the functionality with new releases
  11. Etc.

If technical and business risks on the project are low, there is no big need in structured testing at all. The best you can do is to conduct some manual exploratory testing and review it with the stakeholders.

If your analysis indicates medium or above average risks it's time to focus on unit testing. Also add some manual black-box testing including test analysis, design, review and support of the tests.

But if your risks are high, make sure you have the following areas covered:

1. Control your manual testing:

  • Each test case is detailed and stored in the test management system, grouped in test suites.
  • All test cases are reviewed and formally signed off by the Development and Product Management team representatives.
  • All the manual testing is accounted in TMS so you always control what, when and how is tested.Whenever the issue is missed, you can analyze the reasons and identify underperforming engineers.
  • Ad Hoc testing is allowed only for product managers and developers. QA is not allowed to test ad hoc.

2. Use certain amount of automation on multiple layers:

  • Whitebox tests (Unit, Integration, System level)
  • Blackbox tests (UI and API logic)
  • Performance test automation (Load and Volume tests)

3. Assess System Security on a regular basis:

    Working approach: there is the principle of diverse half-measures: apply a bunch of methods instead of investing 100% of your budget in manual testing. This will save you much costs.

    Understanding the risks not only helps you to mitigate them, you will be more confident in the budget to spend for a certain area of the project.