Insights

We look at the differences between traditional and modern performance testing, diving into the benefits of using the more modern approach.

---

Many people in the United States remember the rollout of the US government’s HealthCare.gov website. This site was a crucial part of the new Affordable Care Act, AKA “Obamacare.” Hundreds of thousands of US citizens needed to access the site to make critical decisions on how their healthcare insurance would work.

Unfortunately, the site’s rollout was an epic failure. The teams building the site missed the mark on capacity planning. Thousands of citizens flooded the site to enroll for insurance, and the site couldn’t support the load. As a result, the site was unusable for several months while the responsible teams scrambled to mitigate the numerous performance problems.

Why Performance Testing Matters

Everyone associated with the US site’s release effort suffered significant reputational damage. Worse yet, the end-users -- US citizens in need of insurance -- could not make critical health and financial decisions for months. The reasons for the system’s failures were numerous, but one central theme was clear: The teams had completely missed the basic performance needs for the site. Few projects have the size, scope and impact of Healthcare.gov. However, every project supports some critical aspect of an organization’s business. (If the project doesn’t, you shouldn’t build it, but that’s a different post.) We use “normal” testing and quality assurance activities to ensure each project meets requirements. We should add performance testing to nearly every project to ensure those systems behave correctly when under user load.

Performance Testing Is a Business Activity

We traditionally performance testing is viewed as a highly technical activity. That’s 100 percent true, but it’s not the complete picture. At its root, performance testing is a vital business activity that helps your organization understand if your most critical user activities are successful — are you able to complete your most basic mission? Perhaps that mission is selling products to customers. Perhaps it’s helping people find healthcare choices. Regardless, every organization should ensure its systems support these mission-critical activities. Failing to do so creates an extreme risk to your company’s mission — and most likely revenue. In the past, only a few projects used performance testing to ensure those mission-critical features. Thankfully, the industry is changing.

Traditional Performance Testing Lags Delivery

Traditional approaches for performance testing used teams of specialists who were far removed from the actual delivery teams. Those teams spent weeks or months detailing performance requirements documentation, then would take one version of the system and work in isolation. Weeks, sometimes months later, the performance team would release a lengthy report detailing system performance and calling out particular areas of concern. Numerous problems with this approach resulted in significant impacts on the delivery cycle.

• First, the performance teams were far lagging behind the current versions the development team delivered. Issues identified in one version were often superseded as work continued — and of course, development teams often injected new issues.
• Secondly, major architectural impacts were late-hitting, which meant significant rework to meet new architectural decisions.
• Finally, slow feedback cycles leave the business unable to change priorities and validate critical use cases in a timely fashion.

Modern Performance Testing is Different

Modern performance testing approaches this work in a much more agile, collaborative fashion. Performance specialists are either embedded in the delivery teams, or at least work in close, constant collaboration. Performance specialists focus on one or two use cases at a time, often tied to work underway in the current iteration. Traditional performance testing has been done through large, complex enterprise-level toolsets. Users could rarely integrate these tools into the larger development release cycle. Modern performance testing still uses complex tools — it’s the nature of the domain — however, they’re often coupled with code-level tests checking simple measurements around timing. This second approach is key, as it enables quick measurements even on developers’ local systems versus having to push everything to a specific performance testing environment. Again, fast feedback is central to good performance testing. Continuous Integration/Continuous Delivery (DevOps) has been a game-changer for modernizing performance testing. CI/CD pipelines enable teams to include segments of load testing runs in build cycles as an automated set of performance-related regression checks. These performance regression tests aren’t the complete set of the load suite, but they’re comprehensive enough to ensure no performance degradation happens as development work continues. As noted in many works such as Jez Humble’s book Continuous Delivery, companies like Etsy, Flickr and Netflix personify the success of these practices. Flickr, an online photo-sharing service, has been doing multiple deploys to production since the early 2000s. Etsy used them as a model and extended the DevOps model to include basic performance testing as part of their release model – thereby ensuring the user experience was always hitting expectations. Netflix’s online stream service can’t survive without a complex, continuous performance testing approach as a first-citizen for their release process.

Fast Feedback, Risk Mitigation, Value Maximization

Organizations moving to modern performance practices see measurable results. You can identify major architectural issues early in the project, dramatically reducing risk. Stakeholders and executives can have faith that their most critical use cases are solid or quickly have information to make appropriate changes in direction. One critical concept to keep in mind as you consider performance testing: It shouldn’t be an all-or-nothing proposition. Start small, mature the process, and iterate on how your organization handles performance testing. Stay focused, stay Lean.

Conclusion

Lessons we’ve learned in software delivery have pushed the industry to rethink many approaches to how we envision, specify, build, test and release software. These learnings help make the case for bringing an appropriate level of performance testing to nearly every project. Why “nearly every project?” Because you want to ensure each of those projects can deliver on crucial experiences that impact your client and support your organization’s mission. What’s the next project you’re going to test with modern performance practices?