Engineering teams ideally want to spend all their time writing code to create applications and services for their users! In reality, many end up spending a significant portion of their time on other tasks, such as maintaining DevOps infrastructure.
In Azure DevOps, Microsoft-hosted agents (aka Azure Pipelines agents) provide a fully managed, low overhead way to get started with Azure Pipelines. Many customers find that these agents are not flexible enough to meet their needs – not enough power, not enough memory, an inability to connect to private networks, etc. In these cases, teams can use self-hosted agents for maximum flexibility, but at the cost of a significant increase in overhead and maintenance costs.
Azure Virtual Machine Scale Set agents addressed the overheads problem by enabling teams to create agent pools based on Azure VM Scale Sets with any SKU, image, storage or network that teams wanted. While the maintenance overheads were less than what teams were spending on maintaining self-hosted agents, they were still spending significant amount of time maintaining and troubleshooting their pools. Since the VMs were being created in the team’s subscription, it was also difficult for the user and support teams to troubleshoot when things went wrong. We saw an opportunity to further improve the experience of creating and managing custom Azure DevOps pools.
Today, we’re excited to announce the public preview of Managed DevOps Pools (MDP), a feature of Azure DevOps that enables dev teams or platform engineering teams to quickly spin up custom DevOps pools that suit your team’s unique needs. It combines the flexibility of Scale Set agents and the ease of maintenance of Microsoft Hosted agents. It enables engineering teams to establish consistency and best practices while maximizing performance, security, compliance, and cost-efficiency for their custom DevOps Pools. Managed DevOps Pools was inspired by a Microsoft internal service called “1ES(One Engineering Systems) Hosted Pools”. You can read more about the problems Microsoft solved with “1ES Hosted Pools” in Managed DevOps Pools – An Origin Story.
Managed DevOps Pools for Azure DevOps
The Benefits
By using Managed DevOps Pools, teams can expect to see the following key benefits.
Hosted on your behalf
Managed DevOps Pools is a fully managed service where VMs powering the agents are created/managed by Microsoft services in Microsoft owned Azure subscriptions. VM agents are not created in team’s own Azure subscription like it is in Azure DevOps Elastic/VM Scale Set Pools. The host on behalf model reduces the time spent in managing agents, improves reliability and won’t have other services in the same subscription competing for compute cores with your CI/CD agents.
Time spent in Management
Managed DevOps pools will drastically reduce time spent in management of agents that are based on on-premises infrastructure or manually maintained. Teams that are using Azure Virtual Machine Scale Set agents can further reduce the time spent in creating, managing and troubleshooting Scale Sets, by switching to Managed DevOps Pools. Teams that used the service in private preview were able to create new pools in under a minute and spent little time actively managing the pool.
Specific Pools
Due to the ease with which new pools can be created, organizations can very easily create multiple team-specific or workload-specific pools. Creation of specific pools helps teams tailor pools to accelerate the workloads that run on it and eliminates the noisy neighbor problem. A team in private preview created a pool with a memory-optimized SKU, for a build workload that was memory intensive. They then switched only the pipelines that would benefit from a more expensive SKU to the new pool to significantly reduce the duration of their build pipelines.
DevOps Billing
Managed DevOps Pools helps optimize a team’s DevOps bill through many features. It makes it easy for teams to find an optimal balance between a pool’s QoS/performance and cost. Managed DevOps Pools offers advanced features to manage Standby agents and therefore helps teams to better manage cost than Scale Set agent pools. We observed private preview customers that were able to reduce their Azure billing by as much as 50%.
Scalable
Teams that use on-premises infrastructure are limited to the hosts they own. Teams that create agents manually are limited in scale to the maximum agents they create. Scale Set pools perform less optimally when its scale approaches 100 parallel agents. Managed DevOps Pools solves the scaling problem by automatically orchestrating multiple virtual machine scale sets under the hood and enables Managed DevOps Pools to scale to 1000s of agents in a single pool. This also helps Azure DevOps pools better respond to bursty traffic and reduce queuing of pipelines.
Notable Features
Microsoft-hosted Quick-starter Images
Teams can create pools with quick-starter images that contain the same software present in Microsoft hosted agents. This provides an easy transition for teams currently using Microsoft hosted agents that want to use a more powerful agent, require the agents to be stateful, or need to connect securely to their private network.
Standby Agents
Teams can choose to make their pipelines start more quickly by deciding how many agents they want pre-warmed during specific hours of the week or choose the “automatic” option that uses historical data to create standby agents.
Private Networking
Teams can create pools that connect to resources on their private network, such as package registries, secret managers, and other on-premises services.
Bring your own Image
Teams can create pools with images that the team has created with pre-requisites that are unique to their scenario.
Stateful Agents
By default, MDP pools are stateless and a new agent is created for every pipeline job. However, teams can choose to reuse the same agent in multiple jobs to improve performance of their pipelines because of not needing to re-download files or not needing to re-compute operations due to local cache hits. Managed DevOps Pools implements best practices for stateful agents by auto-recycling agents based on time or the agent running out of disk space.
Pick any SKU
Azure offers a variety of compute families that are tailored for various workload requirements. Teams can pick an Azure SKU family and a size that matches their workload’s unique core/memory/disk usage profile to make them more performant or cost effective.
Additional Storage
If teams need extra disk space but are content with the number of cores and memory on their SKU size, they can just add an additional disk without needing to go up a SKU size.
Geographic co-location
Teams can create pools in a geographic location that is closest to the rest of their resources for improved performance due to low network latency or to meet compliance requirements.
Get started now
Managed DevOps Pools for Azure DevOps is now available public preview, and you can access the service directly in the Azure Portal. During the public preview period, you can get started with no extra service-related billing apart from the cost of Azure resources created as part of the environment. To learn more about pricing for compute, storage, networking, and other Azure resources, check out the Azure pricing calculator.
To learn more about Managed DevOps Pools and getting started with the service, visit the Managed DevOps Pools page. You can request features or report bugs in the Azure DevOps Developer Community portal.
Hi,
Firstly, thanks for a great service addition!
Two thoughts regarding costs:
1. For our VMSS agents, we incur costs during periods even when the agent is idle or in a boot/teardown state (i.e., the VM runtime when the Azure DevOps agent isn't active or serving any pipelines). Will this be the same for MDP? Specifically, will we only be charged for compute power during the actual runtime of the agent when it's executing a pipeline, or...
Thanks Alexander!
You will pay for the entire VM lifecycle. This is because MDP is not a common shared pool but a pool of VMs created based on your pool settings. You can choose pool settings that are optimized for cost, so you are not paying for idle compute.
MDP uses a very granular scaling approach and will scale exactly by demand. If you do not have standby agent mode enabled, it will only create VMs based...
Hi,
Not sure where else to ask this, but for me it seems impossible to deploy a Managed Pool.
I'm trying to test this with a fresh Azure DevOps organization, where I'm the administrator, and the organization is properly linked to my Entra ID tenant. I've gone through all the pre-requisites, and when I finally try to create a Managed Pool I keep getting this error during deployment:
Failed to provision agent pool. Exception: The logged in...
Can you please post this at https://developercommunity.visualstudio.com/AzureDevOps?q=managed+devops+pools ? It will help us engage with you privately to ask follow-up questions to troubleshoot the issue.
Hi Guys
If i spot any issues with this feature where can i register an issue in the bicep community based on dependency issues?
The feature were we select the MDP to be available for only some specific Azure DevOps projects is not really useable, when the Azure DevOps project name contains whitespaces.
You cannot create a DevCenter -> Project with whitespaces in it's name, only display name and i dont have any other option in bicep...
Hi Simon, Can you please share context on how you went about creating your Bicep file for deploying MDP? What steps you did or which doc you followed to create a Bicep file?
I discovered that the DevCenter -> Projects versus the actual project in Azure DevOps are not related.
When i create the DevCenter Project i can just replace the whitespaces and then make sure i just use the project id for the "devCenterProjectResourceId" property when creating the Managed DevOps Pool, and then under the organizationProfile -> organizations -> projects i just define the projects it is allowed to be used in as an array where it...
Hi Suraj and Eliza,
Is “Managed DevOps Pools” service in “General availability” now?
If not, what would be tentative month for General availability for this service?
Hi Abhinav, We are aiming for GA in the month of November. Can you please share more context on why the GA tag is important to you?
@Janne, Thanks for pointing it. It looks like there was an error with the metadata. We are fixing it now. Preview banners should go up as early as today or latest by end of the week.
Also: "The Microsoft Privacy Statement explains the personal data Microsoft processes, how Microsoft processes it, and for what purposes. Your applicable Services Agreement or the Preview Supplemental Terms may specify lesser or different privacy measures for some Preview services."
source: https://azure.microsoft.com/en-us/support/legal
"Previews may employ lesser or different privacy and security measures than those typically present in the Products and Services. Unless otherwise noted, Customer should not use Previews to process Personal Data or other data that is...
Abhinav, Thanks for your response. Our team will support teams having an issue with the same SLA as we will in GA. I don’t believe there is a different SLA for GA. You can contact support if you run into any issues and they will engage us as necessary.
Thanks Suraj, we are planning to use it in production pipeline.
And as per my understanding public preview does not come under Microsoft SLA.
Do these agents also have 60 minutes job timeout as Microsoft-hosted agents? I tried them on a long-running pipeline and it failed with the error:
Jim, Luigi, Can you please post this at https://developercommunity.visualstudio.com/AzureDevOps?q=managed+devops+pools ? It will help us engage with you privately to ask follow-up questions to troubleshoot the issue.
For Microsoft Hosted agents, the 60 min limitation applies to the free job in private projects based on https://learn.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops&tabs=ms-hosted#how-much-do-parallel-jobs-cost
There is no such 60-minute timeout in Managed DevOps Pools since you pay for the usage.
I have the same error and everytime the pipeline stops for a 60minutes timeout. The pipeline regularly builds on the Microsoft Hosted agent and even specificying the timeoutInMinutes parameter makes no difference.
Hi Suraj and Eliza,
This is great, just what we were looking for. I have one question, can the agents join a domain? We need a domain user to access some SQL databases. Is this possible?
We have a feature to add agents to private networks, but we do not currently support domain-joining as a specific feature. You can request this feature in “Dev Communities – Azure DevOps”
trying to use az mdp pool update
need some examples for –agent-profile parameter please
if the json structure is provided, it would be a huge help
thanks
Congratulations! This is a much needed feature that was missing in Azure DevOps.
I have been able to deploy and the test the Managed Agents. However, there seems to be some issues with life-cycle management of agents and resource.
- The agent (isolated VNet) was stuck in starting mode for 25 minutes before the return. It finally worked after a new agent was provisioned after another 5 minutes.
- Unable to delete agents integrated with...
Can you share a pool name where you faced these issues so we can investigate?
It is indeed due to the subnet delegation and serviceAssociationLinks to be more specific. This issue has been around for a few years now but it only occurred when dependent resources were deleted in the wrong order. The delegation cannot be removed with the existing service association link that can only be deleted by deleting the resource. However, looks like MDP is getting into a chicken and egg situation with this and the MDP...
Vijay, This error might be due to not disassociating the subnet delegation to MDP. Can you try disassociating and then deleting it? Our team is looking into how we can make the experience better
The agent lifecycle was just an inconvenience / finding that I wanted to report. The main issue is with not being able to delete the agents once injected into an existing VNet nor can it be disassociated either.
Failed to delete pool : The client 'xxxxxxxxxxxxxxxxxxxxxxxx' with object id 'xxxxxxxxxxxxxxxxxxxxxxxx' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/serviceAssociationLinks/delete' over scope '/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/resourceGroups/rg-xxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Network/virtualNetworks/vnet-xxxxxxxxxxxxxxxxxxxxxxxx/subnets/snet-xxxxxxxxxxxxxxxxxxxxxxxx/serviceAssociationLinks/ManagedAgents' or the scope is invalid. If access was recently granted, please refresh your credentials.
Can you please retry it and see if you face the same problem again?
If you do not have standby agent mode, on-demand agent creation will take ~5-10 mins
If your maximum agents is set to 1, your pipelines will wait not have a parallelism of more than one and will wait for the 1st job to complete
Do check the self-hosted parallelism of your AzureDevOps account to see if that is throttling your parallelism.
The pool name I used was “ManagedAgents”. You can DM me using my guest account on your tenant if you need more details.
I appreciate this feature a lot! It solves the problem we have of needing agents that can communicate over private endpoints for terraform deployments.
One thing that we've done in the past is use Orchestrated VMSS agents, but have noticed 5 minute delay or more to scale in our agent pool.
It looks like based on another comment that this will still be the same problem with these new pools. Is there anything that can be done...
Thanks Adam! It takes a few mins for Azure to create VMs on demand depending on the size of the image, etc. If you want to reduce the time it takes start your pipeline(to get an agent assigned), you can enable the Standby agent mode feature in MDP. https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/configure-scaling?view=azure-devops&tabs=azure-portal#standby-agent-mode
At the moment, reducing agent assignment time is a trade-off between cost and performance. That said, we are constantly looking for opportunities to reduce the on-demand...
Hi Suraj,
Thank you for your response regarding the MDP agent availability in the UK West region.
I wanted to bring another issue to your attention. We’re encountering a potential bug in the UK South region. We have several VNETs in our subscription, and when we try to select a subnet from the VNET, it doesn’t populate in the portal and shows up blank. However, when we inject the same Subnet via an ARM template and redeploy...
Can you please confirm that the logged-in user has permission to view the VNet and Subnets on Azure Portal outside of MDP UI? It is possible that the run as account for ARM template deployment has access to the VNet/SubNet but not the logged-in user.
Looks like this might require some additional troubleshooting.
Can you please report this as a problem in Developer Communities – Azure DevOps? If possible, if you can record it as a GIF and privately share it in the post too, we can look at it too.
I’ve confirmed that I have all the necessary permissions, and I’m using the same user account to redeploy directly through the portal. The logged-in user does have access to view the VNet and Subnets on the Azure Portal outside of the MDP UI.
Let me know if there’s anything else I should check.