BDD provides a framework for full-system testing in a CI and CD pipeline as part of a team’s agile processes using three primary principles.
Automated integration tests are an essential part of achieving Continuous Integration and Continuous Delivery as part of agile development.
We use Behavior Driven Development (BDD) to provide a set of integration tests written from the perspective of the business. This article details the fundamental principles we use with BDD.
What is Behavior Driven Development (BDD)?
Software development teams employed many different testing strategies, including unit tests, integration tests, and automated tests. Test-Driven Development (TDD) is a well-known methodology that involves writing unit tests first, then implementing the method, class, or module until the tests pass. This methodology allows the developers to minimize over-engineering the codebase. They accomplish this by implementing only enough code to meet the requirements while continuously improving the structure of the code. Unfortunately, TDD tests do not test the overall system from a user’s perspective. Teams address this issue by introducing higher-level testing in the form of Acceptance Test-Driven Development (ATDD), which is specifically set up to test the user’s requirements. Behavior Driven Development is a specific type of ATDD that uses Given/When/Then syntax.
BDD in Practice
Below is an example of a BDD test for a car-rental company. This test shows the usage of a Background block executed before each of the Scenario blocks. In this example, we showcase a single scenario. The “When” block in the scenario section creates a rental agreement for the customer and equipment combination as specified in the background section. The “Then” block in the scenario section verifies the default rates configure on the rental agreement.
The main principles we followed as part of our team’s BDD implementations include:
- Executable in most environments: Local (developer workstation or cloud), Dev, Test/QA, and more.
- Should not depend on pristine data in the database.
- Written in a language that is complimentary to the languages the team uses.
Let’s look at each of these principles.
Executable in Most Environments
TL;DR
Local: allow test-driven development and quick issue resolution
Dev: allow CI/CD automation of the BDD tests of the deployed code
Test/QA: allow additional testing of ancillary systems and testing as part of formal QA deliverables
BDD tests that execute in specific environments provide significant benefits beyond standard unit tests. Automated BDD tests pinpoint issues in the system, but only in the running environment.
We designed our BDD tests to run only in a single, fully configured “Live” environment. Our team’s developers struggled with reproducing failed tests locally (developer’s workstation or a separate cloud-based environment depending on the system architecture). And, our team wanted to run the failing test(s) locally until we identified and fixed the issue. When we couldn’t run locally, we resorted to “commit and hope” styles of checking-in, significantly reducing overall efficiency while building a narrative indicating an originally broken system. Once available, local execution allowed our team to diagnose and fix issues quickly. They also performed test-first development with the BDD tests by adding new tests locally and then coding to make the tests pass.
Since the BDD tests exercise the business value of a system, the data created provides additional benefits beyond the assertions made within each scenario. Others repeatedly ask our team to test ancillary systems as well as edge cases not easily automated. Without BDD, these ancillary tests are cumbersome to set up and perform because of the data set up, so there is a constant push to postpone formal testing until “feature complete.” Postponing the tests opposes our team’s goal of CI and CD and causes pressure to revert to waterfall implementation of features. However, by allowing and actively supporting BDD tests running informal Test and QA environments, the ancillary system testing is no longer a purely manual effort.
No Pristine Dataset
TL;DR
Running tests across different environments should not require resetting test data since we use most environments for more than just automated tests
BDD tests interact with a fully deployed system which introduces data issues when compared with unit tests that rely on mocked data to control the test conditions. The first solution many teams use is a set of pristine data requiring careful control, so the tests pass.
A pristine dataset breaks the ability to execute the tests across environments. For example, when our team used a pristine data set, none of the developers wanted to run the tests locally because it broke their manual test data. Our team employs Kanban with stories that span a few hours to a couple of days. However, we can immediately execute high priority stories using tools such as git stash. To accomplish this with minimal waste, our team members rely on the ability to run tests without side effects.
When we apply pristine data to a Dev, Test, or QA environment, it conflicts with a standard practice of periodically bringing scrubbed production data into lower environments. This step causes a constant battle within the team to determine when the “acceptable waste” can occur with the team having to rebuild the test data. When we refresh the data daily, then the team must rebuild the data daily. If we never refresh the data, then the team doesn’t have to rebuild, but the environment becomes stale. Tests that generate and configure the data removes this issue and allows teams to update environments without conflict.
BDD tests not relying on pristine data creates data, so each test execution results in a complete set of results. Since a pristine dataset isn’t required, the environment contains a running history of test results. We use these historical results to verify behavior in previous versions of the software as well as for additional manual testing.
In our example BDD test at the beginning of this article, the “And vehicle exists” step searches for the specific type of vehicle. If there is a vehicle in the system available for rent, this step uses that vehicle. However, if no car exists, it creates a new vehicle and adds it to the fleet. The test results generated by cucumber use the “world.attach” functionality to show the actual vehicle identifier used during each scenario execution. The test results become a full history of the test with the actual data used, which we can use for additional manual testing. We accomplish this process without needing to revert to using pristine datasets with fixed ids.
Complimentary Language
TL;DR
Remove barriers to BDD adoption by using a BDD implementation language approved by the team
The maturation of agile teams and the implementation of DevOps largely boosted BDD’s popularity over the last few years. This boost resulted in more pressure on testing teams, necessitating the use of automation. There are multiple tools that support BDD. For our current client, we leveraged the popularity of Typescript due to its ease of integration with cucumber-js. This step allowed our developers to start owning end-to-end tests without switching languages. All major languages support BDD using cucumber, including .NET (via SpecFlow), Java, Ruby, Python, Go, and more. Determining primary responsibility for writing and owning automation is one of the critical factors in deciding which language stack to adopt. See our other blogs on BDD tutorials.
Summary
BDD provides a framework for full-system testing in a CI/CD pipeline as part of a team’s agile processes. BDD achieves a couple of key elements to enable the agility of the team in the following ways:
- By being readable, Any member of the team can write automations, and product owners and lines of business can review both the test results and test cases.
- By using the Given/When/Then format, the focus of the test moves away from technical or user interface clicks to a straightforward outcome of a business scenario on the enterprise data. Business scenario outcomes tend to change less often, as well as be more business-critical than traditional unit tests.
- By not requiring a pristine data set of data or dedicated environment, we can run tests against the code as the user chooses to execute it.
Although a simplified example, this style of tests allowed the execution of 1000s of combinations of pricing, billing, and tax scenarios to give confidence in ongoing agile system changes to a mission-critical core system. By following the principles above the BDD tests can provide both valuable regression and ATDD benefits without overwhelming the team.
