The Five QA Senses
~ by Randy Wagner
The world is defined by our perceptions which are driven mainly by our senses. What we see, hear, taste, touch, and smell give us a handle on the world around us. And we expect them to be in agreement. We become inquisitive and learn more when those senses are not in agreement. If we don’t see a predator but we hear a low growl close by, we get curious in a hurry. In software QA, we don’t have those same senses but reporting metrics can give us a similar handle on the project.
Five Reporting Metrics that Reveal QA Status
I believe there are five senses in QA reporting that give a vivid picture of the QA status, warnings of danger, and, ultimately, peace of mind, when all are in balance. These five senses—traceability, defects, split stories, regression, and time—provide the necessary range of insight. Each gives a different sense, evolves with the project, and balances against the others.
#1 – Traceability
Traceability metrics are the most conspicuous.
- Did we create test cases against each story and acceptance criteria?
- How many of those have been executed?
- How many are blocked?
- What percent passed?
- How many total test cases did we create?
If we have one test case per requirement, there is a strong likelihood we’ve undershot the mark. These are simple but useful numbers.
#2 – Defects
Defects allow us to move from blunt to finesse. There are obvious numbers like total defects, open, closed, reworked, blocked defects, etc. It’s when we move into how defects are distributed that we start seeing in more vivid color.
- What vectors are the defects coming from?
- Is one functionality more problematic than another?
- Is the test environment unstable?
- Are the requirements unclear?
- Are defects bouncing back and forth between functional areas?
- Are we seeing a higher percentage of high severity than expected?
Each data point gives us more insight and opportunity for positive action than the raw numbers. Statistics can offer even more detail if you choose to add that layer. Finally, defect leakage is the worst burn. Defects caught later in the lifecycle cost more and reflect failures of unit and QA testing. Those deserve more attention to see where we can improve.
#3 – Split Stories
“Split stories” is an area that isn’t specifically QA driven but it’s a hidden gem that tells an awful lot about the state of the project. Generally, there are three reasons for splitting stories:
- Estimates were bad
- The code was delivered late in the sprint
- Staffing needs to be adjusted
Split stories can happen but higher frequencies correlate with higher defect rates and more rework. It’s an area that requires management attention.
#4 – Regression Testing
Regression testing tells you as much about requirements stability as it does system stability. At its core, a regression test should pass every time, excluding factors like server response times.
The only reason the code itself should fail is that a new code change broke something, or later requirements altered expected behavior. Tracking regression failures can bring to light both unit testing problems and requirement deficiencies.
Paying attention here, especially when you can run regression on Dev builds, offers wonderful understanding.
#5 – Time
Finally, we have the fifth sense—time. Time gives context to the other senses. Here are some examples:
- A sudden spike in defects is expected with UAT as users apply fresh eyes to the project deliverables.
- Low traceability coverage is a red flag later in the project.
- A high volume of split stories in the first two sprints is a danger sign.
- The later the change request shows up in the project lifecycle, the more defects are likely to result.
Whether or not something happens can be less important than when it happens.
Trust the Five Senses for a Balanced Perspective of QA Status
Reporting is one of those things that is either over-done or not done well enough. Too often, it is analogous to the drunk who lost his keys and is searching under the streetlight, because that’s where the visibility is best.
We need to focus the reporting light on the areas that give us the truest representation of the project’s QA efforts. The five senses provide the balance of perspectives we need to keep QA on track in the project.
Randy Wagner is Director of Quality Assurance for CastleBay Companies. He has 20 years of consulting experience across private and public sectors, Guidewire InsuranceSuite, InsuranceNow, and Duck Creek, with specializations in quality assurance, configuration management, and automation.