UAT – Comfort Level vs Zero Defects
– by Randy Wagner
Managing Expectations of UAT Becomes a High-Wire Act for Consultants
User Acceptance Testing is something of a necessary evil in Policy Admin System (PAS) development. It requires a carefully crafted system to be thrown to unpredictable users who will identify unanticipated issues or, at the very least, suggestions for future improvements. Managing expectations becomes a high-wire act for consultants who wind up between the project sponsors, who eyeball the time and cost, and the product owners, who push for perfection.
The Temptation to Start UAT Before Major Defects are Resolved Can Be Costly
As always, success starts before the first day of testing. If the project dutifully addresses defects during the Development cycle and executes a stabilization phase without significant defect leakage, the UAT will go smoothly. It’s QA’s job to push back if the project wants to “show progress” and start UAT with incomplete functionality, batch processes still coming up to steam, and numerous defects that users will have to work around. That’s not to say anyone enters UAT with a perfect system. Perfect is the enemy of good. Hold off until major defects are resolved.
Embrace UAT for the Three Valuable Insights the Process Provides
UAT provides three benefits. First, users get their hands on the system and begin learning how to use it. Ideally, they become good trainers for others to follow. Second, you should get ideas. UAT testers will see the world differently and offer suggestions. Finally, you will find defects. Imperfections will come to light. Even comprehensive testing never quite catches everything. A client should be prepared for this. Users are unpredictable and will find issues.
Prepare Testers to Understand the Context of the System for Wider Adaptation
Preparation for UAT requires a thoughtful sequencing of training and testing to allow users to absorb the system in context. Training should be done by the Product Owner because they ultimately hold the reasoning for how the system evolved as it did. The “why” questions from the users are great teaching moments that give depth and understanding to the new system and functionality. Invite all streams to test policy creation so they can go back to their stream, say billing or claims, with an understanding of the payment expectations or coverages that appear in their own tests. Model existing accounts and transactions so they are focused more on the flow and functionality of the system than explicit step by step test cases. The better stream testers understand context, and focus on their usage, the more likely they are to adapt to the system.
Manage Defects as a Team to Insure Priorities are in Check
If defects are not managed well during UAT, the project will immediately begin to creep in scope. This not only adds time and cost, it also invites new errors as the project tries to change features that may not have been worked on in months or a year. It is critical that defect triage happen often and include senior client leadership who can provide the necessary pushback on items that may not be as critical as a PO might feel.
Keep Testers Energized by Providing a Solid System, Good Training & Thoughtful Testing
In the end, the real measure of success for UAT is how energized the testers come out of the sessions. They won’t be masters of the new system, but they will have a comfort level. There will be defects but, hopefully, UAT started with a solid system, good training, and thoughtful testing, so the issues will not be serious. Oh, and always have candy in the room for best results!
Randy Wagner is Director of Quality Assurance for CastleBay Consulting. He has 20 years of consulting experience across private and public sectors, Guidewire InsuranceSuite, InsuranceNow, and Duck Creek, with specializations in quality assurance, configuration management, and automation.