Please see the Rollout Update section below for important information about brownout status and schedule change for East US 2 region.
Azure DevOps Services will support Service Tags by the end of CY2020. Azure Service TagsĀ are a convenient way for customers to manage their networking configuration to allow traffic from specific Azure services. Once a Service Tag has been set up for Azure DevOps Services, customers can easily allow access by adding the tag name azuredevops to their NSGs or firewalls either through the portal or programmatically.Ā
In preparation for this enhancement, our IP address space will be changing for outbound traffic from Azure DevOps Services to customers’ on-prem systems, effective October 5 2020. If you’re currently using firewall rules to allow traffic from Azure DevOps Services, please be sure to update these rules to account for our new IP ranges by that deadline. We will be conducting a brownout test from September 9, 2020 to September 15, 2020 as indicated below. Please note the change from originally announced date of September 8, 2020 to September 9, 2020. While most of the features will work, the below four scenarios will be impacted by the brownout test due to IP address range change. If you do not want any impact for these four scenarios during this period, please add the additional IP address ranges below for your region to your firewall rules as soon as possible.
The Service Tag does not apply to Microsoft Hosted Agents. Customers are still required to allow the entire geography for the Microsoft Hosted Agents.Ā For inbound traffic from customers’ on-prem systems to Azure DevOps Services, customers can continue to follow the guidelines here.
Determining impact
To help you determine whether this change impacts your organization, we are building an Azure DevOps IP Check Tool. The IP Check Tool is used to validate inbound and outbound connectivity between Azure DevOps Services and customers’ on-prem systems. Please use this tool prior to the brownout and after to validate your connectivity.
For inbound testing from your on-prem system to Azure DevOps Services, please make sure that the browser running the test is connected to your target network. We will attempt to contact Azure DevOps Services and report any errors we see.
For outbound testing from Azure DevOps Services to your on-prem systems, please provide us with a REST URL you expect our services to call. We will attempt to call the URL from each of our service regions. Any HTTP status code between 200 and 499 will be considered a successful connection. All 5xx status codes will be reported as an error.
If you are having issues, please post an update on this open developer community item.
IP Address Changes
To react to the changes in our IPv4 address range, users should ensure dev.azure.com is open and update their allowed IPs to include the following IPv4 addresses (based on your region). You will also be able to use the service tag name azuredevops to allow all IP ranges below but the tag will notĀ be available until November 2020. IPv6 is not supported at this time.
IP Address Ranges
| Region | IP address ranges |
|---|---|
| brazilsouth | 191.235.226.0/24 |
| asiaeast | 20.189.107.0/24 |
| uscentral | 20.37.158.0/23 |
| australiaeast | 20.37.194.0/24 |
| indiasouth | 20.41.194.0/24 |
| useast2 | 20.41.6.0/23 |
| uswest2 | 20.42.134.0/23 |
| australiasoutheast | 20.42.226.0/24 |
| useast | 20.42.5.0/24 |
| ussouth | 40.119.10.0/24 |
| europewest | 40.74.28.0/23 |
| usnorth | 40.80.187.0/24 |
| uswest | 40.82.252.0/24 |
| uksouth | 51.104.26.0/24 |
| uswestcentral | 52.150.138.0/24 |
| canadacentral | 52.228.82.0/24 |
Azure DevOps documentation will be updated with the new IP address ranges here. A complete list of Azure DevOps Services guidelines for configuring firewalls and proxy servers can be found in the Allow IP addresses and URLs to the allow list document.
Rollout plan
Over the course of the next few weeks, we will conduct a series of brownout tests to identify organizations that may be impacted by these routing changes. We will conduct our first test on September 9, 2020 and complete by September 15, 2020. Please note the change from originally announced date of September 8, 2020 to September 9, 2020 and also note the change for East US 2 region to September 16,2020 at 07:00 EDT (11:00 UTC). See below for the brownout schedule. The brownout test will take 2 hours.
Rollout update
Azure DevOps Services started the network configuration change for the East US 2 region on September 9, 2020 at 10:00 EDT (14:00 UTC) and noticed a spike of customer impacting failures during one of the deployments. The spike lasted for 1 to 3 minutes for web traffic and customers may have noticed a message with āTF400898: An Internal Error Occurredā in their browser. The brownout in the East US 2 region was halted but we completed the brownout in the Central Canada region with success. We will continue with the brownout in the South India and West US 2 regions on September 10, 2020. We have updated the brownout schedule for the East US 2 region to September 16, 2020 at 07:00 EDT (11:00 UTC).
Brownouts in chronological order
| UTC Date Time | Region | Local Date Time |
|---|---|---|
| 2020-09-09 19:00 | canadacentral | 2020-09-09 15:00 EDT |
| 2020-09-10 11:00 | indiasouth | 2020-09-10 16:30 IST |
| 2020-09-10 17:00 | uswest2 | 2020-09-10 10:00 PDT |
| 2020-09-11 12:00 | uksouth | 2020-09-11 13:00 BST |
| 2020-09-11 18:00 | brazilsouth | 2020-09-11 15:00 BRT |
| 2020-09-14 13:00 | europewest | 2020-09-14 15:00 CEST |
| 2020-09-15 00:00 | asiaeast | 2020-09-15 08:00 HKT |
| 2020-09-15 14:00 | uscentral | 2020-09-15 09:00 CDT |
| 2020-09-15 22:00 | australiaeast | 2020-09-16 08:00 AEST |
| 2020-09-16 11:00 | useast2 | 2020-09-16 07:00 EDT |
In the event we are running these tests and use cases such as service hooks, data import, and pipelines are not working during this period of time, please navigate to the status page and check that there aren’t any ongoing incidents and update your IP address allow list. We are targeting November, 2020 to make Service Tags generally available for Azure DevOps.
Reporting Issues
If you experience any issues with accessing your Azure DevOps organization after updating your IP allow list, please post an update on this open developer community item.
Hi Justin,
One thing is not clear on to what
(based on your region)refer to here.Our MS hosted Azure Devops Organization is in USEAST2 , we have people connecting from different regions across the world using on-prem self hosted agents. Can you let me know what would be the action taken by us (does adding just USEAST2 IP address range on all regions suffice?) to get ready with this change ?
This is based on your MS hosted Azure DevOps organization in the region. Since your organization is in USEAST2, you can allow IP ranges for USEAST2.
Hi Justin,
Appreciate your quick reply !
Does that mean if my OnPrem datacenters are in Europe, Australia, India, then i just have to whitelist the USEAST2 ipaddress on each of these datacenters and that fixes the problem ? Just wanted to be double sure on this š
That is correct. You just need to allow the IP address range of USEAST2 in your on-prem data center’s firewall for Europe, Australia and India.
The traffic from Azure DevOps Services in USEAST2 to your on-prem data centers needs to be allowed.
This is an important update on the brownout test in the East US 2 region on September 9, 2020 at 10:00 EDT (14:00 UTC).
Azure DevOps Services started the network configuration change for the East US 2 region on September 9, 2020 at 10:00 EDT (14:00 UTC) and noticed a spike of customer impacting failures during one of the deployments. The spike lasted for 1 to 3 minutes for web traffic and customers may have noticed...
Just to clarify, if we are using the MSFT hosted version of Azure DevOps Services (NOT the on-prem version), this change does not apply to us? Thank you!
This change applies to MSFT hosted version of Azure DevOps Services for the following scenarios only:
-Azure DevOps Services connecting to endpoints for Service Hooks,
-Azure DevOps Services connecting to SQL server in customerās on-prem systems for Data Import,
-Azure Pipelines connecting to on-prem source code repositories such as GitHub Enterprise or BitBucket Server,
-Azure DevOps Services Audit Streaming connecting to on-prem or cloud-based Splunk.
We are using Deployment Groups and we installed self hosted agents are installed on premises server, as mentioned in the article brownout test for UK south is from September 9, 2020 to September 15, 2020 (12:00 to 13:00 BST), will this effect our release pipelines to move/install the artifacts?
My scenario is self hosted agent installed on premises server will pull the code from azure build artifact store and install the components, could you please let...
Your service will be affected by the brownout test. Please follow the guidelines provided in the blog.
Will you add new IP ranges only or the current IP Pool will be change also?
We will be adding the new IP ranges only. The current IP address space will no longer work after October 5,2020. If youāre currently using firewall rules to allow traffic from Azure DevOps Services, please be sure to update these rules to account for our new IP ranges by that deadline.
Thanks a lot!
Are the IP ranges of the Azure Pipelines Agent Pool could be affected?
The IP ranges of the Azure Pipelines Agent Pool will NOT be affected.
To whom or where should the REST URL be sent?
Thank you
The REST URL can be entered into the “Target Outbound Address” box and enter “Run Test” in the IP Check Tool: https://status.dev.azure.com/_serviceTagConnectivity. This will verify the connectivity from Azure DevOps Services to your on-prem system.
Thanks for such a quick response.
Regards.
i had tried to test the Target Outbound Address by entering the REST URL : “http://on-premises Build agent IP address” like http://10.x.x.x. But i got the error “Special purpose IP addresses are not allowed”. please let me know what all we need to give REST URL.
If you are using the build agent’s internal (local) IP address, this will not work. Please provide externally addressable IP address. For example, any public IP address with REST endpoint that can respond to Git request.
Hi Team,
Date for AutraliaEast has been changed again?
It was 2020-09-09 08:00AM AEST but now its 2020-09-16 08:00 AEST
Can you please confirm is there any further changes that need to be made to plan ? We use Azure DevOps across the organization.
And just to confirm there will be no impact on ADO Boards and ADO Test Plans?
Thanks
Date for AustraliaEast has changed to 2020-09-16 08:00 AEST. We apologize for the inconvenience. There will be no further changes to the date. There will be no impact to Azure DevOps Boards and Test Plans.
Thanks for response, and apologies in advance to ask this again. Just to confirm there will be no impact on Code Reposteries as well, Our developers can continue using Azure DevOps Repos as it is and we do expect some impact for Azure Pipelines
Once again thanks in advance for all support.
Your developers can continue using Azure DevOps Repos. There should be no impact.
The article only mentions “outbound connections”. How about self-hosted agents that need to access Azure DevOps Services to pull pending jobs, code and artifacts? Do we need to whitelist the same new IP blocks listed in this article? Do we keep the current 4 CIDR blocks?
I re-read the article, and it did have the answer to my question. The 4 CIDR blocks currently documented will remain in use for inbound connections.
Would this affect deployment groups? They utilize an AzDO agent installed on on-prem infrastructure. AzDO must reach out to the agent to initiate a deployment correct?
That is correct. Azure DevOps needs connectivity to the job agents running in customer’s on-prem for deployments.
We are using self hosted agents for build and release pipelines where we have configured build agents using PAT. We have not configured any Firewall or Azure IP Address for inbound or outbound manually in build and release servers. Through Internet access the builds artifacts are deploying in release servers.
Please let us know whether any impact for such build and deployments.
You will be not affected in this use case, the on-prem Agents are using the DNS name of the ADO endpoint.
This is NOT official answer from Microsoft. š
During the two-hour brownout time window, are the regions being shut down entirely, or are you just turning off select services?
Kelsey Barrett,
We are not shutting down the entire region. We are only shutting down part of the Azure DevOps Services that will impact the following scenarios:
-Azure DevOps Services connecting to endpoints for Service Hooks,
-Azure DevOps Services connecting to SQL server in customerās on-prem systems for Data Import,
-Azure Pipelines connecting to on-prem source code repositories such as GitHub Enterprise or BitBucket Server.
Are you actually shutting anything off during the test windows?
From the article, it sounded like you were browning out access to certain things from the old ip block.
If someone isn't using Azure DevOps ipv4 block to open up access inbound or outbound, will anything actually be off during these tests? (i.e. will service still be available inbound/outbound during the test window, just from the new range, temporarily?)
Also, the test page you link... Would be nice...
š! Thanks for the clarification, Justin!