Exciting!
You have just finished developing this awesome new feature which is going to have a lot of impact on your customers. But how do you plan to roll it out?
Are you planning to use the regular deployment process? Sure, it might be safe and all, but what if you need to give it more time to bake and test it in production for a while before it is public? What if you want to experiment with the feature only on a selected number of customers and get some feedback before rolling it out to everyone? What if the newly added functionality went horribly wrong and you want to immediately disable it, without stopping and rolling back the deployment of the new version? What if the development of the feature is long and you want to push separate “self-contained” logical units of the feature in a “disabled” state until you pushed in everything you need for the feature to work fully (handling all those merges when working on a separate branch ain’t fun is it? Continuous Integration anyone?).
While there are a couple of ways to achieve the above, one practical way to achieve those goals is with the use of “Feature Flags” (or as also commonly known Feature Toggles).
Feature flags give you the ability to turn on and off “logical gates” in your code that relate to a feature’s functionality. The feature flags’ state is independent of the code and is managed in a different “store”.
This means that for each feature flag the code needs to be able to handle two states: it should do one thing when the feature flag is turned off, and another thing when it is turned on. Wherever you decide to keep the state of your feature flags, it needs to be somewhere independent of your service’s codebase and a place accessible by the service during runtime. Some examples are: database, JSON string or a config file blob kept in storage, service configuration file, exposed through an API of another service etc…
“Feature Flags” don’t have to wrap only new features, they (and should) wrap any significant functionality change. That way, the enablement of a significant change can be isolated from the code deployment, giving the owner of the change much greater control over its impact and lifecycle – with the ability to turn it off in case things don’t go as planned. Therefore, deployments of new versions to your cloud service will rarely enable any significant changes automatically with them, making your deployments even safer.
An important concept that is commonly used, is having a separate feature flag configuration for each environment. This gives you the ability to test or enable\disable a feature in a specific DC\ScaleUnit\Test environment and assess how ready the feature is before it is enabled to all customers. Similarly to how you want your code changes to be deployed gradually and safely, the feature enablement process should be as gradual. Therefore you should usually first enable the new functionality on a small set of customers, and gradually increase the number of customers who are getting the enabled functionality, until it is enabled for everyone. You might also decide to enable the feature flag gradually and randomly based on the percentage of user exposure to it. E.g. A random 1% of all requests will have the new feature flag enabled for them, then 5%, 10% and so on…
During the enablement process, you will want to monitor the usage of the new functionality to understand its impact. Therefore, you should log precise telemetry so you can later view the collected data and understand the functionality’s performance and the user’s response. Optionally, you can have your feature flags to “stick” to the user (like setting a cookie on the response) so that if a user was randomly selected to have the feature flag enabled for him, he will get consistent experience during his entire session(s) and you will be able to see if the feature flag has any wider effect than a specific request\response scenario. Some very advanced systems, use feature flags to conduct experiments on new features. We will address this kind of usage in a different and dedicated blog post.
Another thing that is possible when using feature flags, is having the ability to dynamically enabling or disabling features for a specific request. One way to do this is by sending a feature’s enablement state in a header\query parameter\cookie and dynamically deciding on the logic to perform according to the state that was sent by the request. This allows you to test your feature in production, before making it the default behavior. If this is a security concern for you, you can scope it down and only support dynamic enablements in your test\canary environment.
Automated tests should also be aware of the feature flags that are being used. You want to make sure you are testing all possible scenario’s and everything is working as it should both when the feature is enabled and when it is disabled. To achieve this, you should have the ability to selectively enable or disable a specific feature flag for a specific test, or even run the same test multiple times, each time with a different feature flag state (e.g. once when a specific feature flag is enabled, and once when it’s disabled).
One very practical way to implement feature flags in your code is by using dependency injection. In order for your code to not be filled with those nasty “if-else” statements for every feature flag, you could encapsulate both the decision logic of each feature flag and the actual functionality that each feature flag state triggers.
Here’s an example. Instead of this code:
class BackScratcher
{
public void ScratchMyBack()
{
if (Features.GentleScratching.IsEnabled)
{
ScratchGently();
}
else
{
ScratchHard();
}
}
}
Use this:
class BackScratcher
{
IScratchingTechnique scrachingTechnique;
public BackScratcher(IScratchingTechnique scratchingTechnique)
{
this.scratchingTechnique = scratchingTechnique;
}
public void ScratchMyBack()
{
this.scratchingTechnique.Scratch();
}
}
Where’s the decision point of which implementation of IScratchingTechnique to inject to BackScratcher’s constructor? Either near instantiation logic of the BackScratcher class or, preferably, when you set up your dependency injection\IoC container.
Needless to say, feature flags come with the cost of having two versions of functionality to achieve similar but different things, they make your code more complex and as a result, harder to maintain. That is why you should proactively get rid of all feature flags that have already been turned on by default for a while, or those whose functionality was “abandoned” and are not planned to ever be released. While I believe feature flags are a great quality control tool for developers, you need to be careful and keep a low overall amount of feature flags in your code base.
Nadav
Living on a Cloud 