Automated Test Driven Development for Large Insurance Company

ATDD reduces testing costs, increases Agile team velocity, and lowers overall production defects

The Business Need

Our client, a large insurance company, was tasked with replacing an aging quoting system with a modern .NET web solution that would allow frequent and rapid expansion of commercial insurance product features. Additionally, the client perceived a major opportunity to demonstrate its “Ease of Doing Business” corporate values by deploying a system readily utilized via browser, tablet, or mobile handset.

The client hired Centric to deliver the new quoting system while at the same time working to complete an expansive switch to Agile development techniques. But the successful move to Agile with 5 or more Agile sprint teams chiseling away at backlog at an increasing velocity brought a new set of challenges. It was becoming harder for Agile sprint teams to run all tests prior to committing code to the appropriate code branch. As the overall code base grew regression testing prior to major releases was taking longer. Developers needed a greater degree of automation to execute the growing bank of tests.

The Centric team adopted automated Acceptance Test Driven Development (ATDD) to improve testing efficiency. Without ATDD automation costs would rise due to the following realities: 1) increasing sprint duration, 2) adding a significant number of manual testers to sprint teams, and 3) the decreasing ability to frequently push releases into production.

Centric’s High Impact Approach

To achieve these goals Centric specified the ATDD tooling, assembled the required test team, and built the needed test infrastructure to efficiently execute and monitor tests.

The client had standardized on Microsoft’s Team Foundation Server (TSF) for critical team activities such as project management and source code control and build. But TSF was not capable of doing the level of automation required to meet project needs. Ultimately, the team settled on Ruby, Cucumber, and Watir based on its automation capability, attractive cost point, and ease of use.

The overall test team was expanded to include a program level test lead as well as an automation test architect. Each Agile sprint team was given a mix of automation test engineers and manual testers.

Finally, a sophisticated framework was built using Microsoft’s C# and Ruby that allowed for tests to be checked in and executed as part of ongoing Agile sprints or as part of an overall regression test.The framework included a test dashboard that displayed results including overall trends and allowed identified failed tests to be assigned for resolutions to Agile teams.  The current regression test suite includes over 9000 tests. Additionally, the framework allowed the tests to be executed concurrently across 20 servers shrinking automated test execution time 75% (from 36 hours to 9 hours). The test execution framework has allowed the complete automated regression test to be run every two days, not just prior to every major release.

Results

Our client received many benefits from their ATDD efforts including:

  • Overall test team size was held in check even as the underlying application and number of acceptance tests grew in complexity and size.
  • Estimates suggest the team size would need to be at least 37% larger if automation had not been incorporated. Ongoing annual savings exceed $1.45M.
  • Sprint team velocity has been maintained and improved allowing for faster regression testing and more frequent pushes into production.
  • Production code quality has been dramatically increased with the number of critical and high bugs pushed to production consistently staying very close to zero.