In 2022, we applied generative AI to the QA pipeline on a data migration program at Canada Life. Twelve months later, we had confirmed a 50% reduction in QA effort with zero defect leakage across both a three-month and a six-month measurement window. This is what that actually looked like.
The program was migrating policy data from a legacy system to a modern platform. The data involved was decades old, structured in formats that had evolved over multiple system generations, and subject to ongoing business rules that had never been fully documented.
QA on a program like this is expensive. Every migrated record needs to be verified. The verification rules are complex. The volume is high. And the cost of a defect reaching the target system is significant: incorrect policy data affects claims, billing, and regulatory reporting.
The team was spending roughly 40% of total program effort on QA activities. That was not sustainable across a multi-year migration timeline.
We did not replace the QA process with AI. That is not what happened. What we did was use AI to automate the pattern recognition work that was consuming most of the QA team's time: identifying records that matched known defect patterns, flagging anomalies for human review, and generating the comparison reports that QA analysts were producing manually.
The AI component was trained on historical defect data from earlier migration waves. It learned what defective records looked like for our specific data model and business rules. It did not know general data quality principles. It knew our specific patterns.
The measurement framework was set up before the AI component went live. That is the thing most AI programs skip. They deploy the AI and then try to figure out whether it worked. We defined the baseline QA effort per migration wave, the defect detection rate, and the acceptable defect leakage threshold before we changed anything. The measurement told us what the AI was actually contributing.
We kept human QA analysts in the loop for anomaly review. The AI flagged; the human decided. This was not a philosophical position. It was a practical one. The AI was good at pattern recognition but not at distinguishing between a genuine defect and a legitimate exception to the standard pattern. That distinction required judgment. Judgment required a human.
AI in a QA pipeline works best when it handles pattern recognition and humans handle judgment. The moment you remove the human from anomaly review, you lose the ability to catch the exceptions the model was not trained on.
The 50% reduction was in QA effort, not in QA thoroughness. The number of records verified did not decrease. The defect detection rate did not decrease. What decreased was the time QA analysts spent on mechanical pattern-matching work that the AI could do faster and more consistently.
That time was reallocated to higher-judgment work: reviewing complex exceptions, validating business rule interpretations, and improving the defect pattern library that the AI used for future waves. The QA function became more sophisticated, not smaller.
We underestimated the time required to build and validate the defect pattern library. The AI was only as good as the patterns we gave it. The first wave of patterns was too broad. The model flagged too many records for human review, which created a different kind of bottleneck. We spent the first three months tuning the patterns before we got to the stable state that produced the 50% reduction.
If I were doing this again, I would budget six months for pattern library development and validation before claiming any efficiency gains. The efficiency is real. It takes time to get there.
The result was confirmed at 50% and held across two measurement windows. That is the number. The more important outcome was that the QA team's capacity was redirected to work that required their actual expertise rather than their time. That is what made the program better, not just cheaper.