Whenever joining a new development team, there is an inevitable debate on “branching” strategy. The branching strategy debate centers around how to manage multiple feature development streams while still maintaining the ability to release small enhancements and bug fixes continuously.
This debate elicits a lot of passion around the most effective way to achieve agility, stability, and release management control. As agile becomes a de-facto standard, and as continuous delivery and feedback is an increasingly important business and IT imperative for success, our team migrated away from a “Branching Strategy” to a Feature Toggle pattern.
This article details the fundamental principles and example usage of Feature Toggles.
What Are Feature Toggles?
In its simplest form, a Feature Toggle is a simple lever that allows the team to enable or disable a production feature without having to redeploy software. Whether a team works on a core transactional system replacement or enhances an existing application, there will be a mismatch between the completeness of different features.
Many agile teams revert to complex strategies to manage the rollout imbalance. One option teams use is complex release planning schedules where the team holds releases until they complete a certain set of features. This process leads to a long lead time for simple changes since these require a delay until the more complex features reach maturity.
Another option is creating multiple feature branches where you keep longer-running features out of the mainline deployments. This process allows the teams to deploy the small changes quickly since large changes are separated from mainline. However, this option introduces a problem of stale feature branches. Multiple feature branches can pull from mainline, but they don’t “see” the major changes occurring across the other feature branches. This step limits the amount of refactoring done in each branch and increases the complexity of merges when the feature branches complete.
To address these issues, our agile team introduced Feature Toggles across the system. The Feature Toggles allow us to control the availability of features in production which we accomplished with a few minor upgrades to our existing permissions system.
We can use the Feature Toggles at both a macro level, where we enabled full menu items and business processes for a select set of users and small features, such as a single additional button.
Finally, allowing A/B testing of replacements for existing features is another usage pattern.
Feature Toggles in Practice
Below is an example of a simple feature toggle where we added a new button to an existing UI view. The screenshot is from the same car-rental company system example I used in my previous Behavior Driven Development (BDD) blog. The first image is the view for most of the users.
Figure 1 Feature Toggle OFF
This image is the view for a select few users who are testing out the new functionality in production before it is turned on for all users.
Figure 2 Feature Toggle ON – Pre-Inspect Button
The process is incredibly simple, and it seems like it isn’t worth implementing a Feature Toggle! But, consider the effect on the process of rolling this feature out. Without the Feature Toggle, we can’t add the button to the system without determining when the build goes to production. We need to consider the effect of the new button as part of the Operational Change Management Plan. And, we must fully test the button effects before we can push the build to production.
With the Feature Toggle, we eliminate these issues. We can commit and push the code for the button to production immediately and then enable it for production use at the convenience of the business.
Feature Toggles for New Processes
In modern complex systems, the addition of new processes sometimes requires new UI screens, changes to existing screens, as well as underlying service changes. Planning the process with UI mockups, service-tier designs or architecture, and user interviews provides an initial starting point for the implementation. However, we receive the best feedback by slowly rolling out the feature.
The slow rollout allows teams to focus on the user’s true requirements instead of adding “gold-plating.” Gold-plating is a common problem when agile teams build a feature that not used actively in a production environment. It occurs when a team orders the backlog based on usage assumptions instead of actual user feedback.
Figure 3 Feature Toggle OFF
Figure 4 Feature Toggle ON – New Process Menu Item
For new processes, we use a Feature Toggle to begin the rollout to a primary set of users working closely with the team to define the new process. A new process might require a simple button Feature Toggle on an existing screen as well as a Feature Toggle on a set of menu items and UI flows.
Feature Toggles for Complex Process Changes
Below is a screenshot of a sophisticated A/B implementation of a Feature Toggle. The example shows the existing user’s “A” implementation as well as the new “B” implementation that has a dramatically different flow based on user feedback from using the initial “A” implementation.
Figure 5 Feature Toggle OFF
Figure 6 Feature Toggle ON – Complex Process Change
The team began implementation of the new feature by making a few tactical changes to the existing implementation. This step is an example of Branch by Abstraction, where we put a small glue layer in-place in preparation for a larger refactoring. By adding the Branch by Abstraction layer to the existing implementation, it freed the team to make the significant changes required for the new process without breaking the existing functionality.
Feature Toggling of complex changes also reduces the complexity of Operational Change Management since actual users champion the new functionality as well as provide actual usage-based training to new users.
Sample Implementation
We controlled the Feature Toggles through our standard permissions system. As part of our DevOps approach to development, we have the permissions committed in source control. And, we deploy the permissions with a single button push per environment. The following is one possible implementation. Feature Toggle implementations vary based on the architecture and design of the system.
The “CustomerInsurancePermission” contains the definition of the Feature Toggle as well as default Dev environment permissions.
The _beta, _release, and _test files contain overrides to set the permissions based on the specific environment but do not redefine the Feature Toggle. The _release version below shows a disabled Feature Toggle.
The developers who add the Feature Toggle also add the specific Feature Toggle file(s) as well as “empty” permissions for the other environments. The developers also disable all Feature Toggles by default in the higher environments. The business representatives coordinate with the agile team (primarily the Business Analysts, Scrum Masters, and Project Managers) to determine when we can enable a Feature Toggle in each environment. Initially, developers were responsible for updating the environment permissions by editing the individual environment permissions files. This step was a significant roadblock because it depended on normal ticket assignment. To mitigate this issue, the Business Analyst now makes the changes to the environment files using a Web-based view of our source control.
The deployment process is to deploy the permission updates and then deploy the code. Our team also performs beta rollouts where we coordinate turning on a feature for a short time (using the standard deployment mechanisms) and then disable the feature with a later deployment. Finally, if we find a critical issue in production, our team can immediately disable a feature. We rarely need this capability but can execute it, if necessary. Being able to disable a new feature temporarily keeps the team from having to make emergency hotfixes for code in production.
Summary
Feature Toggles are a straightforward mechanism for managing the complex rollout of features in a Continuous Integration and Continuous Delivery agile environment. The Feature Toggles achieve a couple of key elements to enable the agility of the team:
- Restore control of feature delivery to the business instead of relying on developers coordinating code check-ins.
- Enable feature rollout to full userbase without additional developer code changes.
- Simplify Operation Change Management by allowing early-adopter users to provide feedback for improvement of the feature without having to enable the feature for all user across the system fully.
The introduction of Feature Toggles changes the conversation for designing, building, and deploying changes to a mission-critical core system from a complex codebase merge and deployment planning to a simple business readiness check. Once a feature passes the readiness check, we enable it is without developer intervention.
