When Data and Feel Collide

~ by Randy Wagner

How to Spot Problems When Metrics Fail to Accurately Represent Reality

Much of the work we do is reduced to metrics. Test cases completed, defect counts, and accepted stories drive the numbers.  We measure velocity and plan accordingly. We gauge health accordingly. But then there are times when the numbers just don’t accurately reflect reality and it takes experience to spot problems before they impact timelines. Those “feel” related inputs tend to come down to several vectors.

Watch for Verbal Cues in Your Team

Verbal cues are often the first to manifest. Everyone works to be professional but even in well-ordered teams, there may be times when people feel it necessary to restate their points in conversation, feeling the need to be heard. It’s not always about the stated issue but leadership should be keeping an eye on tension to address concerns and feel out what the deeper issue may be. If team calls are consistently running offtrack but on valid issues, it’s a good bet something else is rumbling beneath the surface.

Consider the Complications of Dependencies Early on

Dependencies are often sources of subsurface challenges. Whether the perspective is project management, business analysis, development or quality assurance, the complications that dependencies raise can lead to delays as individuals hesitate or take extra time to address the implications.  We may be showing on track but, if we’re not ahead of the curve mentally, the feel can become a genuine data point impacting the timelines.  Do we have all the requirements and expectations from the vendor?  Some integration vendors require certain validation timelines on their side that can blindside a team that doesn’t ask the right questions.

Keep an Eye on Troublesome Trends Related to Workflow Issues

Workflow is potentially one of the biggest challenges and it shows up at multiple levels.  The workflow within the team might have gaps or grey areas that cause the team to abandon or misdirect handovers.  Workflow in the systems can point to limited requirements, missing logic in the code, or clear test steps.  No matter where the workflow issue appears, it doesn’t tend to show up in story point estimates and adds more defects to be addressed in the current or future sprints.  Good metrics can spot that trend, but a lot of teams are so focused on the next deliverable that they don’t identify the trend.

Never Underestimate the Delivery Impact of Personalities

Personalities play another big role.  Whether you have bossy team members that drive off course or the silent types taking too long to solve issues because they are afraid someone will notice the problem, we have to keep an eye on deviations from the estimates. It doesn’t matter what functional area you work in; the way people behave has a huge impact on delivery dates.

Trust Your Experience, Not Just the Metrics

Obviously, there are a myriad of ways problems will not show up in the pure metrics.  We strive to use the numbers to keep us on track but simply paying attention to the why, where, when, and who can often give us the clarity that the “what” just doesn’t provide. At the end of the day, experience can offer a metric that just isn’t quantifiable, and is often what customers seek most.  When something feels off, it almost certainly is.  We usually have to look past the first layer to sort it out.


Randy Wagner is Director of Quality Assurance for CastleBay Companies. He has 20 years of consulting experience across private and public sectors, Guidewire InsuranceSuite, InsuranceNow, and Duck Creek, with specializations in quality assurance, project management, configuration management, and automation.