QA Confidence Intervals (Part 1)

Goal: provide a simple data point(s) for managing personnel in an organization to understand the quality of the product to be released.

This post will go through my thought process for achieving this goal. I do not, at the start of writing, have a solution in mind yet.

What this is NOT:

  • deterministic value for QA quality or efficiency
  • absolute value reflecting data from tests

Potential outputs of this:

  • absolute value derived from a formula
  • value in a confidence interval
  • absolute value in multiple categories
  • mix of information in the form of buckets

Data points which I personally feel are more impactful:

  • CI build trends
  • Unit test coverage
  • E2E test results
  • Manual test results
  • Percentage of tests automated
  • Quality of tests
  • Complexity in updating tests

It appears there are several different areas of confidence I want to convey:

  • confidence in the product working now
    • in a happy path
    • in potential edge cases
  • confidence in our ability to update and scale the product

References:

Confidence’s Role in Software Testing - ThinkTesting.com
Do you hear “QA is a bottleneck” in your organization? Maybe the test team lacks confidence in the quality of code historically delivered by development
Defining Confidence in Software Testing
Trying to make sense of all the vague terminology in testing land, starting with confidence.
64 Essential Testing Metrics for Measuring Quality Assurance Success

https://www.cell.com/neuron/fulltext/S0896-6273(16)30016-2

this wasn't directly used, but is useful for context about human confidence as an abstract concept. Additionally, a much longer paper which I did not read. https://economics.mit.edu/files/14924

Show Comments