{"id":9145,"date":"2016-09-30T16:14:00","date_gmt":"2016-09-30T21:14:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/buckh\/?p=9145"},"modified":"2016-09-30T16:14:00","modified_gmt":"2016-09-30T21:14:00","slug":"controlling-exposure-through-feature-flags-in-vs-team-services","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/buckh\/controlling-exposure-through-feature-flags-in-vs-team-services\/","title":{"rendered":"Controlling exposure through feature flags in VS Team Services"},"content":{"rendered":"<p>One question that I often get from customers is how we manage exposing features in the service. Features may not be complete or need to be revealed at a particular time. We may want to get early feedback. With the team working in master and deploying every three-week sprint, let\u2019s take a look at how we do this for Team Services.<\/p>\n<p><strong>Goals<\/strong><\/p>\n<p>Our first goal is decoupling deployment and exposure. We want to be able to control when a feature is available to users without having to time when the code is committed. This allows engineering the freedom to implement the feature based on our needs while also allowing control for the business on when a feature is announced. Next we want to be able to change the setting at any scope from globally to particular scale units to accounts to individual users. This granularity gives us a great deal of flexibility. We can deploy a feature and then expose it to select users and accounts. That allows us to get feedback early, which includes not only what users tell us but also how the feature is used based on aggregated telemetry. Additionally, we want to be able to react quickly if a feature causes issues and be able to turn it off quickly.<\/p>\n<p>To make all of this work well, we need to be able to change a feature flag\u2019s state without re-deploying any of our services. We need each service to react automatically to the change to minimize the propagation delay.<\/p>\n<p>As a result, we have the following goals.<\/p>\n<ul>\n<li>Decouple deployment and exposure  \n<li>Control down to an individual user  \n<li>Get feedback early  \n<li>Turn off quickly  \n<li>Change without redeployment<!--EndFragment--> <\/li>\n<\/ul>\n<p><strong>Feature flags<\/strong><\/p>\n<p>Feature flags, sometimes called feature switches, allow us to achieve our goals. At the core, a feature flag is nothing more than an input to an if statement in the code: if the flag is enabled, execute a new code path, and else if not, execute the existing code path.<\/p>\n<p>Let\u2019s look at an actual example. In this case I want to control whether a new feature to revert a pull request is available to the user. I\u2019ve highlighted the Revert button in the screen shot.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/39\/2019\/03\/image778.png\"><img decoding=\"async\" title=\"image\" style=\"border-left-width: 0px;border-right-width: 0px;border-bottom-width: 0px;padding-top: 0px;padding-left: 0px;padding-right: 0px;border-top-width: 0px\" border=\"0\" alt=\"image\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/39\/2019\/03\/image_thumb646.png\" width=\"1081\" height=\"365\"><\/a><\/p>\n<p>First we need to define the feature flag. We do that by defining it in an XML file. Each service in VSTS has its own set of flags. Here\u2019s part of the actual file that defines the feature flag for this button with the name of the feature flag highlighted.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: blue\">&lt;?<\/span><span style=\"color: #a31515\">xml <\/span><span style=\"color: red\">version<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">1.0<\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">encoding<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">utf-8<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">?&gt;\n&lt;!-- <\/span><span style=\"color: green\">In this group we should register TFS specific features and sets their states. <\/span><span style=\"color: blue\">--&gt;\n&lt;<\/span><span style=\"color: #a31515\">ServicingStepGroup <\/span><span style=\"color: red\">name<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">TfsFeatureAvailability<\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">\u2026<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"\" <\/span><span style=\"color: blue\">&gt;\n  &lt;<\/span><span style=\"color: #a31515\">Steps<\/span><span style=\"color: blue\">&gt;\n    &lt;!-- <\/span><span style=\"color: green\">Feature Availability <\/span><span style=\"color: blue\">--&gt;\n    &lt;<\/span><span style=\"color: #a31515\">ServicingStep <\/span><span style=\"color: red\">name<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">Register features<\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">stepPerformer<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">FeatureAvailability<\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">\u2026<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"\" <\/span><span style=\"color: blue\">&gt;\n      &lt;<\/span><span style=\"color: #a31515\">StepData<\/span><span style=\"color: blue\">&gt;\n        &lt;!--<\/span><span style=\"color: green\">specifying owner to allow implicit removal of features <\/span><span style=\"color: blue\">--&gt;\n        &lt;<\/span><span style=\"color: #a31515\">Features <\/span><span style=\"color: red\">owner<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">TFS<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">&gt;\n          &lt;!-- <\/span><span style=\"color: green\">Begin TFVC\/Git <\/span><span style=\"color: blue\">--&gt;\n          &lt;<\/span><span style=\"color: #a31515\">Feature <\/span><span style=\"color: red\">name<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\"><font style=\"background-color: #ffff00\">SourceControl.Revert<\/font><\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">description<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">Source control revert features<\/span><span style=\"color: black\">\" <\/span><span style=\"color: blue\">\/&gt;<\/span><\/pre>\n<\/blockquote>\n<p>When we deploy the service that defined this feature flag, the deployment engine will create the feature flag in the database.<\/p>\n<p>Using the feature flag in code is simple. Here\u2019s the Typescript code that is used to create the button. I\u2019ve combined contents from two files. The export is from a file that defines constants. The rest is from the code to create the button on the page. I\u2019ve highlighted the flag and the button creation. In this case, there was no prior code, so if the flag is off, nothing gets added to the page.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: blue\">export module <\/span><span style=\"color: black\">FeatureAvailabilityFlags {\n    <\/span><span style=\"color: blue\">export var <\/span><span style=\"color: black\">SourceControlRevert = <\/span><span style=\"color: #a31515\">\"<font style=\"background-color: #ffff00\">SourceControl.Revert<\/font>\"<\/span><span style=\"color: black\">;\n}\n\n<\/span><span style=\"color: blue\">import <\/span><span style=\"color: black\">FeatureAvailability = <\/span><span style=\"color: blue\">require<\/span><span style=\"color: black\">(<\/span><span style=\"color: #a31515\">\"VSS\/FeatureAvailability\/Services\"<\/span><span style=\"color: black\">);\n<\/span><span style=\"color: blue\">private <\/span><span style=\"color: black\">_addRevertButton(): <\/span><span style=\"color: blue\">void <\/span><span style=\"color: black\">{\n    <\/span><span style=\"color: blue\">if<\/span><span style=\"color: black\">(FeatureAvailability.isFeatureEnabled(<font style=\"background-color: #ffff00\">FeatureAvailabilityFlags.SourceControlRevert<\/font>)) {\n        <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">._calloutButtons.unshift(\n            &lt;button onClick={ () =&gt; Dialogs.revertPullRequest(\n                <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.props.repositoryContext,\n                <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.props.pullRequest.pullRequestContract(),\n                <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.props.pullRequest.branchStatusContract().sourceBranchStatus,\n                <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.props.pullRequest.branchStatusContract().targetBranchStatus) }\n                    &gt;{ VCResources.<font style=\"background-color: #ffff00\">PullRequest_Revert_Button<\/font> } &lt; <\/span><span style=\"color: #a31515\">\/button&gt;\n            <\/span><span style=\"color: black\">);\n        }\n    }<\/span><\/pre>\n<\/blockquote>\n<p>In addition to the web UI, the code for the MVC controller is also protected with a feature flag. I\u2019m omitting the definition of the constant and some of the code for brevity, but as you can see the only mention of the feature flag is in the attribute on the controller, making it really easy to control the feature with a flag.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: blue\">namespace <\/span><span style=\"color: black\">Microsoft.TeamFoundation.SourceControl.WebServer\n{\n    <font style=\"background-color: #ffff00\">[FeatureEnabled(FeatureAvailabilityFlags.SourceControlRevert)]<\/font>\n    <\/span><span style=\"color: blue\">public class <\/span><span style=\"color: #2b91af\">GitRevertsController <\/span><span style=\"color: black\">: GitApiController\n    {\n        [HttpPost]\n        <\/span><span style=\"color: blue\">public <\/span><span style=\"color: black\">HttpResponseMessage CreateRevert(\n            [FromBody] WebApi.GitAsyncRefOperationParameters revertToCreate,\n            [ClientParameterType(<\/span><span style=\"color: blue\">typeof<\/span><span style=\"color: black\">(Guid), <\/span><span style=\"color: blue\">true<\/span><span style=\"color: black\">)] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">repositoryId,\n            [ClientIgnore] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">projectId = <\/span><span style=\"color: blue\">null<\/span><span style=\"color: black\">)\n        {\n        }\n\n        [HttpGet]\n        <\/span><span style=\"color: blue\">public <\/span><span style=\"color: black\">GitRevert GetRevertForRefName(\n            [FromUri] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">refName,\n            [ClientParameterType(<\/span><span style=\"color: blue\">typeof<\/span><span style=\"color: black\">(Guid), <\/span><span style=\"color: blue\">true<\/span><span style=\"color: black\">)] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">repositoryId,\n            [ClientIgnore] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">projectId = <\/span><span style=\"color: blue\">null<\/span><span style=\"color: black\">)\n        {\n        }\n\n        [HttpGet]\n        <\/span><span style=\"color: blue\">public <\/span><span style=\"color: black\">GitRevert GetRevert(\n            <\/span><span style=\"color: blue\">int <\/span><span style=\"color: black\">revertId,\n            [ClientParameterType(<\/span><span style=\"color: blue\">typeof<\/span><span style=\"color: black\">(Guid), <\/span><span style=\"color: blue\">true<\/span><span style=\"color: black\">)] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">repositoryId,\n            [ClientIgnore] <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">projectId = <\/span><span style=\"color: blue\">null<\/span><span style=\"color: black\">)\n        {\n        }\n    }\n}<\/span><\/pre>\n<\/blockquote>\n<p>The FeatureEnabled attribute checks to see if the specified feature flag is enabled and throws an exception if not.<\/p>\n<pre class=\"code\">    <span style=\"color: blue\">public class <\/span><span style=\"color: #2b91af\">FeatureEnabledAttribute <\/span><span style=\"color: black\">: AuthorizationFilterAttribute\n    {\n        <\/span><span style=\"color: blue\">public <\/span><span style=\"color: black\">FeatureEnabledAttribute(<\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureFlag)\n        {\n            <\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.FeatureFlag = featureFlag;\n        }\n\n        <\/span><span style=\"color: blue\">public string <\/span><span style=\"color: black\">FeatureFlag { <\/span><span style=\"color: blue\">get<\/span><span style=\"color: black\">; <\/span><span style=\"color: blue\">private set<\/span><span style=\"color: black\">; }\n\n        <\/span><span style=\"color: blue\">public override void <\/span><span style=\"color: black\">OnAuthorization(HttpActionContext actionContext)\n        {\n            <\/span><span style=\"color: blue\">base<\/span><span style=\"color: black\">.OnAuthorization(actionContext);\n            TfsApiController tfsController = actionContext.ControllerContext.Controller <\/span><span style=\"color: blue\">as <\/span><span style=\"color: black\">TfsApiController;\n            <\/span><span style=\"color: blue\">if <\/span><span style=\"color: black\">(tfsController != <\/span><span style=\"color: blue\">null<\/span><span style=\"color: black\">)\n            {\n                <\/span><span style=\"color: blue\">if <\/span><span style=\"color: black\">(!tfsController.TfsRequestContext.IsFeatureEnabled(<\/span><span style=\"color: blue\">this<\/span><span style=\"color: black\">.FeatureFlag))\n                {\n                    <\/span><span style=\"color: blue\">throw new <\/span><span style=\"color: black\">FeatureDisabledException(FrameworkResources.FeatureDisabledError());\n                }\n            }\n        }\n    }<\/span><\/pre>\n<p>Other than some similar code for a menu entry for revert, the feature flag has now been added for the new feature.<\/p>\n<p><strong>Controlling feature flags<\/strong><\/p>\n<p>We have both PowerShell commands and a web UI to turn the feature flags off and on for different scopes.<\/p>\n<p>Our PowerShell commands are what you would expect: Get-FeatureFlag and Set-FeatureFlag. Here are some examples of what they can do.<\/p>\n<table cellspacing=\"0\" cellpadding=\"2\" width=\"989\" border=\"0\">\n<tbody>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Getting all flags and their states globally<\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Get-FeatureFlag<\/font> <\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Getting all flags and states for a single account<\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Get-FeatureFlag -Host account<\/font> <\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Getting information for one flag globally<\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Get-FeatureFlag -Name feature_name<\/font> <\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Getting information for one flag in account<\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Get-FeatureFlag -Name feature_name \u2013Host account<\/font> <\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Setting a feature flag globally<\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Set-FeatureFlag -Name feature_name -State {Off|On|Undefined} <\/font><\/td>\n<\/tr>\n<tr>\n<td valign=\"top\" width=\"311\">&nbsp;&nbsp;&nbsp; Setting a feature flag for a single account <\/td>\n<td valign=\"top\" width=\"676\"><font face=\"Courier New\">Set-FeatureFlag -Name feature_name -State {Off|On|Undefined} -Host account<\/font> <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p>We also have internal site that provides an interactive way to do the same operations. In this example, you can see that I have the feature flag for the revert feature on for one of my personal accounts and off for the other. This is a great way to be able to control the flags for accounts on demand, such as when we allowed customers to <a href=\"https:\/\/devblogs.microsoft.com\/buckh\/would-you-like-ssh-turned-on-for-your-team-services-account\/\">request SSH<\/a> before it became broadly available.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/39\/2019\/03\/image779.png\"><img decoding=\"async\" title=\"image\" style=\"border-left-width: 0px;border-right-width: 0px;border-bottom-width: 0px;padding-top: 0px;padding-left: 0px;padding-right: 0px;border-top-width: 0px\" border=\"0\" alt=\"image\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/39\/2019\/03\/image_thumb647.png\" width=\"849\" height=\"670\"><\/a><\/p>\n<p><strong>Turn it off!<\/strong><\/p>\n<p>It\u2019s important to have the right telemetry to monitor new features. If we find that a feature is causing problems, we can turn the feature flag off. Since there\u2019s no deployment involved \u2013 just a script or change in the administrative web UI \u2013 we can quickly revert to the prior behavior.<\/p>\n<p><strong>Testing<\/strong><\/p>\n<p>New features that are hidden behind a feature flag are deployed with the flag turned off. As we start turning on the feature for users or accounts, both the old and new code will be executing. We need to test both with the feature flag on and off to ensure it works. This also critical for ensuring the feature can be turned off quickly if something goes wrong.<\/p>\n<p>Tests can easily control whether a flag is on or off by calling methods like the following.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: blue\">public static void <\/span><span style=\"color: black\">RegisterFeature(TestCollection testCollection, <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureName)\n<\/span><span style=\"color: blue\">public static void <\/span><span style=\"color: black\">UnregisterFeature(TestCollection testCollection, <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureName)\n<\/span><span style=\"color: blue\">public static void <\/span><span style=\"color: black\">SetFeatureStateForApplication(TestCollection testCollection, <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureName, <\/span><span style=\"color: blue\">bool <\/span><span style=\"color: black\">state)\n<\/span><span style=\"color: blue\">public static void <\/span><span style=\"color: black\">SetFeatureStateForDeployment(TestCollection testCollection, <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureName, <\/span><span style=\"color: blue\">bool <\/span><span style=\"color: black\">state)\n<\/span><span style=\"color: blue\">public static bool <\/span><span style=\"color: black\">IsFeatureEnabled(TestCollection testCollection, <\/span><span style=\"color: blue\">string <\/span><span style=\"color: black\">featureName)<\/span><\/pre>\n<\/blockquote>\n<p>Tests can be run conditionally based on the state of a flag.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: black\">[TestMethod, Owner(<\/span><span style=\"color: #a31515\">\"buck\"<\/span><span style=\"color: black\">), Priority(1)]\n[Description(<\/span><span style=\"color: #a31515\">\"Verity that Revert works correctly.\"<\/span><span style=\"color: black\">)]\n[RequiresFeature(FeatureAvailabilityFlags.SourceControlRevert)]\n<\/span><span style=\"color: blue\">public void <\/span><span style=\"color: black\">SourceControl_Revert()\n{\n   ...\n}\n<\/span><\/pre>\n<\/blockquote>\n<p>Since most feature flags are used initially with a default state of off, it\u2019s also easy to set them in the test environment for the test run.<\/p>\n<blockquote>\n<pre class=\"code\"><span style=\"color: blue\">&lt;?<\/span><span style=\"color: #a31515\">xml <\/span><span style=\"color: red\">version<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">1.0<\/span><span style=\"color: black\">\" <\/span><span style=\"color: red\">encoding<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">utf-8<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">?&gt;\n&lt;<\/span><span style=\"color: #a31515\">TestEnvironment<\/span><span style=\"color: blue\">&gt;\n  &lt;<\/span><span style=\"color: #a31515\">TestVariables<\/span><span style=\"color: blue\">&gt;\n    &lt;<\/span><span style=\"color: #a31515\">Value <\/span><span style=\"color: red\">Key<\/span><span style=\"color: blue\">=<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">SetTfsFeaturesOn<\/span><span style=\"color: black\">\"<\/span><span style=\"color: blue\">&gt;<\/span><span style=\"color: black\">SourceControl.Revert<\/span><span style=\"color: blue\">&lt;\/<\/span><span style=\"color: #a31515\">Value<\/span><span style=\"color: blue\">&gt;\n  &lt;\/<\/span><span style=\"color: #a31515\">TestVariables<\/span><span style=\"color: blue\">&gt;\n&lt;\/<\/span><span style=\"color: #a31515\">TestEnvironment<\/span><span style=\"color: blue\">&gt;\n<\/span><\/pre>\n<\/blockquote>\n<p><strong>Stages<\/strong><\/p>\n<p>I mentioned earlier that we can use feature flags to get feedback. We also use them to allow our own team to begin using the feature to help flush out bugs (it\u2019s a great to build the service that we use). Rather than have every team invent their own process, we established a process to roll out features in a standardized way. This provides an opportunity to gather feedback and bugs early on in the development process.<\/p>\n<p>We have standard stages that we\u2019ve defined that each team can use. How quickly a feature goes through the stages depends on the scope of the feature, feedback, and telemetry. The stages include an increasingly broad group of users with increasingly diverse perspectives.<\/p>\n<p><strong>Stage 0<\/strong> &#8211; Canary<br>This is the first phase and is the account used by the VSTS team plus select internal accounts. Program managers are responsible for sending out communication to the users once the feature flags are enabled.<\/p>\n<p><strong>Stage 1<\/strong> &#8211; MVPs &amp; Select Customers<br>This is the second phase and will include MVPs and select top customers who have opted in. Program managers are again responsible for emailing the users.<\/p>\n<p><strong>Stage 2<\/strong> &#8211; Private Preview<br>Private preview is used for major new features and services and is designed to test new features with a broader set of customers that we don&#8217;t have regular contact with. There are many ways to collect a list of customers for a private preview &#8211; from forum interaction, blog comments, etc. We&#8217;ve also done invitation codes, publicized email aliases for customers to request access, as we did for SSH access, and sometimes create in-product &#8220;opt-in&#8221; experiences for customers, as we\u2019ve done for the new navigation UI. Individual teams will manage and communicate directly with their private preview customers.<\/p>\n<p><strong>Stage 3<\/strong> &#8211; Public Preview<br>Public preview is a state reserved for major new features and services where we want to gather feedback but are not yet ready to provide a full SLA, etc. Public Preview features are enabled for all VSTS customers but their main entry points in the UI are annotated with a &#8220;preview&#8221; designation so that customers understand this is not a completed feature. When a feature enters public preview, it is announced in the VSTS news feed and may also be accompanied by marketing communication, depending on the feature.<\/p>\n<p><strong>Stage 4<\/strong> &#8211; General Availability (GA)<br>GA denotes when a feature\/service is available to all customers and fully supported (SLA, etc).<\/p>\n<p><strong>Cleaning up flags<\/strong><\/p>\n<p>It\u2019s easy to accumulate a lot of flags that are no longer used, so after a feature has been in production and fully available, teams decide when to delete the feature flag and the old code path. This may happen a few weeks or a few months later, depending on the scope of the feature.<\/p>\n<p><strong>Events<\/strong><\/p>\n<p>One of our goals is to allow for features to be unveiled at events in some cases. If you want to unveil a feature at an event, when do you enable it?<\/p>\n<p>We learned the hard way that it\u2019s not the morning of the event. Several years ago at the Connect 2013 event we turned on a large set of feature flags just before a keynote and demo. The service was unusable. By turning on feature flags at the same time, we had a large amount of new code interacting with the system under production load, and the system fell apart. You can read about details of the incident in Brian\u2019s post, <a href=\"https:\/\/devblogs.microsoft.com\/bharry\/a-rough-patch\/\">A Rough Patch<\/a>. At that time, we didn\u2019t have stages, and we only had one public scale unit.<\/p>\n<p>As a result of that experience, we ensure that feature flags are enabled in production at least 24 hours ahead of an event so that the code is under full production load. That leaves us time to react, whether to fix problems or disable features, before the event starts. Of course, this means that new features could be discovered early. That\u2019s certainly a possibility, but it\u2019s mitigated by controlling the entry points, making the new features hard to find except for the customers who\u2019ve gotten early access or knowing where to look (perhaps setting a special cookie). Some of it comes down to a judgment call.<\/p>\n<p>We followed this policy when we unveiled the <a href=\"https:\/\/marketplace.visualstudio.com\/vsts\">Marketplace<\/a> at the Connect 2015 event. In contrast to the event in 2013, everything worked as it should during the event.<\/p>\n<p><strong>Summary<\/strong><\/p>\n<p>Feature flags have become a critical part of how we roll out feature, get feedback, and allow engineering and marketing to proceed on their own schedules. It\u2019s hard to imagine DevOps services without them!<\/p>\n<p>While we built our own implementation, you don\u2019t have to. <a href=\"https:\/\/launchdarkly.com\/\">LaunchDarkly<\/a> offers <a href=\"http:\/\/blog.launchdarkly.com\/launched-enterprise-feature-flag-management\/\">feature flags as a service<\/a>, including a <a href=\"https:\/\/marketplace.visualstudio.com\/items?itemName=launchdarkly.launchdarkly-extension\">LaunchDarkly VSTS extension<\/a> to integrate with VSTS work items and releases.<\/p>\n<p><em>Follow me at <\/em><a href=\"https:\/\/twitter.com\/tfsbuck\"><em>twitter.com\/tfsbuck<\/em><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>One question that I often get from customers is how we manage exposing features in the service. Features may not be complete or need to be revealed at a particular time. We may want to get early feedback. With the team working in master and deploying every three-week sprint, let\u2019s take a look at how [&hellip;]<\/p>\n","protected":false},"author":94,"featured_media":10268,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-9145","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"blog_post_summary":"<p>One question that I often get from customers is how we manage exposing features in the service. Features may not be complete or need to be revealed at a particular time. We may want to get early feedback. With the team working in master and deploying every three-week sprint, let\u2019s take a look at how [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/posts\/9145","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/users\/94"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/comments?post=9145"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/posts\/9145\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/media\/10268"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/media?parent=9145"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/categories?post=9145"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/buckh\/wp-json\/wp\/v2\/tags?post=9145"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}