Engineering teams ideally want to spend all their time writing code to create applications and services for their users! In reality, many end up spending a significant portion of their time on other tasks, such as maintaining DevOps infrastructure.
In Azure DevOps, Microsoft-hosted agents (aka Azure Pipelines agents) provide a fully managed, low overhead way to get started with Azure Pipelines. Many customers find that these agents are not flexible enough to meet their needs – not enough power, not enough memory, an inability to connect to private networks, etc. In these cases, teams can use self-hosted agents for maximum flexibility, but at the cost of a significant increase in overhead and maintenance costs.
Azure Virtual Machine Scale Set agents addressed the overheads problem by enabling teams to create agent pools based on Azure VM Scale Sets with any SKU, image, storage or network that teams wanted. While the maintenance overheads were less than what teams were spending on maintaining self-hosted agents, they were still spending significant amount of time maintaining and troubleshooting their pools. Since the VMs were being created in the team’s subscription, it was also difficult for the user and support teams to troubleshoot when things went wrong. We saw an opportunity to further improve the experience of creating and managing custom Azure DevOps pools.
Today, we’re excited to announce the public preview of Managed DevOps Pools (MDP), a feature of Azure DevOps that enables dev teams or platform engineering teams to quickly spin up custom DevOps pools that suit your team’s unique needs. It combines the flexibility of Scale Set agents and the ease of maintenance of Microsoft Hosted agents. It enables engineering teams to establish consistency and best practices while maximizing performance, security, compliance, and cost-efficiency for their custom DevOps Pools. Managed DevOps Pools was inspired by a Microsoft internal service called “1ES(One Engineering Systems) Hosted Pools”. You can read more about the problems Microsoft solved with “1ES Hosted Pools” in Managed DevOps Pools – An Origin Story.
Managed DevOps Pools for Azure DevOps
The Benefits
By using Managed DevOps Pools, teams can expect to see the following key benefits.
Hosted on your behalf
Managed DevOps Pools is a fully managed service where VMs powering the agents are created/managed by Microsoft services in Microsoft owned Azure subscriptions. VM agents are not created in team’s own Azure subscription like it is in Azure DevOps Elastic/VM Scale Set Pools. The host on behalf model reduces the time spent in managing agents, improves reliability and won’t have other services in the same subscription competing for compute cores with your CI/CD agents.
Time spent in Management
Managed DevOps pools will drastically reduce time spent in management of agents that are based on on-premises infrastructure or manually maintained. Teams that are using Azure Virtual Machine Scale Set agents can further reduce the time spent in creating, managing and troubleshooting Scale Sets, by switching to Managed DevOps Pools. Teams that used the service in private preview were able to create new pools in under a minute and spent little time actively managing the pool.
Specific Pools
Due to the ease with which new pools can be created, organizations can very easily create multiple team-specific or workload-specific pools. Creation of specific pools helps teams tailor pools to accelerate the workloads that run on it and eliminates the noisy neighbor problem. A team in private preview created a pool with a memory-optimized SKU, for a build workload that was memory intensive. They then switched only the pipelines that would benefit from a more expensive SKU to the new pool to significantly reduce the duration of their build pipelines.
DevOps Billing
Managed DevOps Pools helps optimize a team’s DevOps bill through many features. It makes it easy for teams to find an optimal balance between a pool’s QoS/performance and cost. Managed DevOps Pools offers advanced features to manage Standby agents and therefore helps teams to better manage cost than Scale Set agent pools. We observed private preview customers that were able to reduce their Azure billing by as much as 50%.
Scalable
Teams that use on-premises infrastructure are limited to the hosts they own. Teams that create agents manually are limited in scale to the maximum agents they create. Scale Set pools perform less optimally when its scale approaches 100 parallel agents. Managed DevOps Pools solves the scaling problem by automatically orchestrating multiple virtual machine scale sets under the hood and enables Managed DevOps Pools to scale to 1000s of agents in a single pool. This also helps Azure DevOps pools better respond to bursty traffic and reduce queuing of pipelines.
Notable Features
Microsoft-hosted Quick-starter Images
Teams can create pools with quick-starter images that contain the same software present in Microsoft hosted agents. This provides an easy transition for teams currently using Microsoft hosted agents that want to use a more powerful agent, require the agents to be stateful, or need to connect securely to their private network.
Standby Agents
Teams can choose to make their pipelines start more quickly by deciding how many agents they want pre-warmed during specific hours of the week or choose the “automatic” option that uses historical data to create standby agents.
Private Networking
Teams can create pools that connect to resources on their private network, such as package registries, secret managers, and other on-premises services.
Bring your own Image
Teams can create pools with images that the team has created with pre-requisites that are unique to their scenario.
Stateful Agents
By default, MDP pools are stateless and a new agent is created for every pipeline job. However, teams can choose to reuse the same agent in multiple jobs to improve performance of their pipelines because of not needing to re-download files or not needing to re-compute operations due to local cache hits. Managed DevOps Pools implements best practices for stateful agents by auto-recycling agents based on time or the agent running out of disk space.
Pick any SKU
Azure offers a variety of compute families that are tailored for various workload requirements. Teams can pick an Azure SKU family and a size that matches their workload’s unique core/memory/disk usage profile to make them more performant or cost effective.
Additional Storage
If teams need extra disk space but are content with the number of cores and memory on their SKU size, they can just add an additional disk without needing to go up a SKU size.
Geographic co-location
Teams can create pools in a geographic location that is closest to the rest of their resources for improved performance due to low network latency or to meet compliance requirements.
Get started now
Managed DevOps Pools for Azure DevOps is now available public preview, and you can access the service directly in the Azure Portal. During the public preview period, you can get started with no extra service-related billing apart from the cost of Azure resources created as part of the environment. To learn more about pricing for compute, storage, networking, and other Azure resources, check out the Azure pricing calculator.
To learn more about Managed DevOps Pools and getting started with the service, visit the Managed DevOps Pools page. You can request features or report bugs in the Azure DevOps Developer Community portal.
Hi Suraj and Eliza,
This is great, just what we were looking for. I have one question, can the agents join a domain? We need a domain user to access some SQL databases. Is this possible?
We have a feature to add agents to private networks, but we do not currently support domain-joining as a specific feature. You can request this feature in “Dev Communities – Azure DevOps”
trying to use az mdp pool update
need some examples for –agent-profile parameter please
if the json structure is provided, it would be a huge help
thanks
Congratulations! This is a much needed feature that was missing in Azure DevOps.
I have been able to deploy and the test the Managed Agents. However, there seems to be some issues with life-cycle management of agents and resource.
– The agent (isolated VNet) was stuck in starting mode for 25 minutes before the return. It finally worked after a new agent was provisioned after another 5 minutes.
– Unable to delete agents integrated with existing VNets due to this issue – https://github.com/MicrosoftDocs/azure-docs/issues/48902
Can you share a pool name where you faced these issues so we can investigate?
It is indeed due to the subnet delegation and serviceAssociationLinks to be more specific. This issue has been around for a few years now but it only occurred when dependent resources were deleted in the wrong order. The delegation cannot be removed with the existing service association link that can only be deleted by deleting the resource. However, looks like MDP is getting into a chicken and egg situation with this and the MDP instance cannot be removed once injected into an existing VNet.
Vijay, This error might be due to not disassociating the subnet delegation to MDP. Can you try disassociating and then deleting it? Our team is looking into how we can make the experience better
The agent lifecycle was just an inconvenience / finding that I wanted to report. The main issue is with not being able to delete the agents once injected into an existing VNet nor can it be disassociated either.
Failed to delete pool : The client ‘xxxxxxxxxxxxxxxxxxxxxxxx’ with object id ‘xxxxxxxxxxxxxxxxxxxxxxxx’ does not have authorization to perform action ‘Microsoft.Network/virtualNetworks/subnets/serviceAssociationLinks/delete’ over scope ‘/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-xxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Network/virtualNetworks/vnet-xxxxxxxxxxxxxxxxxxxxxxxx/subnets/snet-xxxxxxxxxxxxxxxxxxxxxxxx/serviceAssociationLinks/ManagedAgents’ or the scope is invalid. If access was recently granted, please refresh your credentials.
Can you please retry it and see if you face the same problem again?
If you do not have standby agent mode, on-demand agent creation will take ~5-10 mins
If your maximum agents is set to 1, your pipelines will wait not have a parallelism of more than one and will wait for the 1st job to complete
Do check the self-hosted parallelism of your AzureDevOps account to see if that is throttling your parallelism.
The pool name I used was “ManagedAgents”. You can DM me using my guest account on your tenant if you need more details.
I appreciate this feature a lot! It solves the problem we have of needing agents that can communicate over private endpoints for terraform deployments.
One thing that we’ve done in the past is use Orchestrated VMSS agents, but have noticed 5 minute delay or more to scale in our agent pool.
It looks like based on another comment that this will still be the same problem with these new pools. Is there anything that can be done to make this faster? Or is this an Azure limitation?
Thanks Adam! It takes a few mins for Azure to create VMs on demand depending on the size of the image, etc. If you want to reduce the time it takes start your pipeline(to get an agent assigned), you can enable the Standby agent mode feature in MDP. https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/configure-scaling?view=azure-devops&tabs=azure-portal#standby-agent-mode
At the moment, reducing agent assignment time is a trade-off between cost and performance. That said, we are constantly looking for opportunities to reduce the on-demand assignment time.
Hi Suraj,
Thank you for your response regarding the MDP agent availability in the UK West region.
I wanted to bring another issue to your attention. We’re encountering a potential bug in the UK South region. We have several VNETs in our subscription, and when we try to select a subnet from the VNET, it doesn’t populate in the portal and shows up blank. However, when we inject the same Subnet via an ARM template and redeploy then it works fine.
Can you please confirm that the logged-in user has permission to view the VNet and Subnets on Azure Portal outside of MDP UI? It is possible that the run as account for ARM template deployment has access to the VNet/SubNet but not the logged-in user.
Looks like this might require some additional troubleshooting.
Can you please report this as a problem in Developer Communities – Azure DevOps? If possible, if you can record it as a GIF and privately share it in the post too, we can look at it too.
I’ve confirmed that I have all the necessary permissions, and I’m using the same user account to redeploy directly through the portal. The logged-in user does have access to view the VNet and Subnets on the Azure Portal outside of the MDP UI.
Let me know if there’s anything else I should check.
I notice that agents listen for jobs took so long (like five minutes). How can we mitigate this start up time without using stateful or standby agent ? My team currently use self-hosted k8s keda to provision agents, which start up fast but to save costs, we are looking for another solution.
Thanks for sharing your feedback. Our team is considering adding support for container-based agents through ACA to improve on-demand(No standby agents) agent spinup times and cost/performance tradeoffs. If you have any suggestions, please do let us know.
That would be great! My team has tried ACI but spin up time is still slow (1 min). I never heard of ACA before, maybe I should give that a try.
Shipping features like this is what keeps Azure DevOps the best developer platform out there! Great work product team!
We successfully managed to test MDP agents in the UK South region using BYO Vnet. However, we encountered an issue with the UK West region as the service is currently unavailable, preventing us from using it for releases in that region.
The limitation that MDP and virtual networks must be in the same region further restricts our ability to utilize the service in the UK West region.
Can you please provide information on when this service will be available in the UK West region?
Thanks,
We added this to our backlog and will share a note when it is available in UK West. You can expect an update on this within the next 4 weeks.
Hi Suraj,
Thank you for your response regarding the MDP agent availability in the UK West region.
I wanted to bring another issue to your attention. We’re encountering a potential bug in the UK South region. We have several VNETs in our subscription, and when we try to select a subnet from the VNET, it doesn’t populate in the portal and shows up blank. However, when we inject the same Subnet via an ARM template and redeploy then it works fine.
Very nice. Is it now possible to use service tags for this solution? What we want is not to be forced to add the IP or IP-Ranges of the VMs on any Azure Infrastructure in the IP-restriction part everytime we run a pipeline from Azure DevOps. Service tags would go a long way here!
Sure! We are happy to answer any questions you may have. Can you please request your contact in Microsoft Germany to setup a meeting with me(The Product team)?
You can achieve this already using BYO Vnet in MDP. In your BYO VNet, you can route outbound traffic through a set of public IP addresses.
Post GA, we plan to add native support in MDP for service tags/ IP addresses.
Thank you for your response! I have one additional question. We have around 4,000 users and 600 code committers in a company of 200,000 employees. We’re considering offering this as a self-service option within our landing zone, as well as a plug-and-play service. While we have a contact at Microsoft in Germany, we’ve found that our more complex questions often require input from an expert with comprehensive oversight, especially when it comes to future planning.
Would it be possible to arrange more direct contact with such an expert, if feasible?
I am also interested in testing this in Australia East.
What is the expected timeline before this will be generally available? My organisation is looking at rebuilding our aging self-hosted Azure DevOps agents and my preference would be to not have to build new VM’s. We are already using GitHub-hosted runners with private networking for our GitHub Actions, so it makes sense to leverage this new functionality for Azure DevOps.
Australia East is already available. Please give it a go and let us know if there is anything else we can help with.