{"id":2681,"date":"2011-04-04T13:30:58","date_gmt":"2011-04-04T13:30:58","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/powershell\/2011\/04\/04\/scaling-and-queuing-powershell-background-jobs\/"},"modified":"2019-02-18T13:05:48","modified_gmt":"2019-02-18T20:05:48","slug":"scaling-and-queuing-powershell-background-jobs","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/powershell\/scaling-and-queuing-powershell-background-jobs\/","title":{"rendered":"Scaling and Queuing PowerShell Background Jobs"},"content":{"rendered":"

A couple of months ago I had asked the PowerShell MVPs for suggestions on blog topics. Karl Prosser, one of our awesome MVPs, brought up the topic of scaling and queuing background jobs.<\/p>\n

The scenario is familiar: You have a file containing a bunch of input that you want to process and you don’t want to overburden your computer by starting up hundreds of instances of PowerShell at once to process them.<\/p>\n

After playing around for about an hour on Friday afternoon, here is what I came up with… This example assumes you have a text file containing the names of many event logs and you want to get the content of each log.<\/p>\n

# How many jobs we should run simultaneously<\/span><\/span><\/span><\/span><\/p>\n

$maxConcurrentJobs<\/span><\/span> <\/span>=<\/span><\/span> <\/span>3<\/span><\/span>;<\/span><\/span><\/span><\/span><\/span><\/p>\n

 <\/span><\/span><\/span><\/p>\n

# Read the input and queue it up<\/span><\/span><\/span><\/span><\/p>\n

$jobInput<\/span><\/span> <\/span>=<\/span><\/span> <\/span>get-content<\/span><\/span> <\/span>.\\input.txt<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

$queue<\/span><\/span> <\/span>=<\/span><\/span> <\/span>[System.Collections.Queue]<\/span><\/span>::<\/span><\/span>Synchronized(<\/span> (<\/span><\/span>New-Object<\/span><\/span> <\/span>System.Collections.Queue<\/span><\/span>)<\/span> <\/span>)<\/span><\/span><\/span><\/span><\/span><\/p>\n

foreach<\/span><\/span>(<\/span><\/span>$item<\/span><\/span> <\/span>in<\/span><\/span> <\/span>$jobInput<\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/p>\n

{<\/span><\/span><\/span><\/p>\n

    <\/span><\/span>$queue<\/span><\/span>.<\/span><\/span>Enqueue(<\/span><\/span>$item<\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/p>\n

}<\/span><\/span><\/span><\/p>\n

 <\/span><\/span><\/span><\/p>\n

 <\/span><\/span><\/span><\/p>\n

# Function that pops input off the queue and starts a job with it<\/span><\/span><\/span><\/span><\/p>\n

function<\/span><\/span> <\/span>RunJobFromQueue<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

{<\/span><\/span><\/span><\/p>\n

    <\/span><\/span>if<\/span><\/span>(<\/span> <\/span>$queue<\/span><\/span>.<\/span><\/span>Count<\/span> <\/span>-gt<\/span><\/span> <\/span>0<\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/p>\n

    <\/span>{<\/span><\/span><\/span><\/span><\/span><\/p>\n

        <\/span><\/span>$j<\/span><\/span> <\/span>=<\/span><\/span> <\/span>Start-Job<\/span><\/span> <\/span>-ScriptBlock<\/span><\/span> {<\/span><\/span>param<\/span><\/span>(<\/span><\/span>$x<\/span><\/span>);<\/span> <\/span>Get-WinEvent<\/span><\/span> <\/span>-LogName<\/span><\/span> <\/span>$x<\/span><\/span>}<\/span> <\/span>-ArgumentList<\/span><\/span> <\/span>$queue<\/span><\/span>.<\/span><\/span>Dequeue()<\/span><\/span><\/span><\/span><\/span><\/p>\n

        <\/span><\/span>Register-ObjectEvent<\/span><\/span> <\/span>-InputObject<\/span><\/span> <\/span>$j<\/span><\/span> <\/span>-EventName<\/span><\/span> <\/span>StateChanged<\/span><\/span> <\/span>-Action<\/span><\/span> {<\/span> <\/span>RunJobFromQueue<\/span><\/span>;<\/span> <\/span>Unregister-Event<\/span><\/span> <\/span>$eventsubscriber<\/span><\/span>.<\/span><\/span>SourceIdentifier;<\/span> <\/span>Remove-Job<\/span><\/span> <\/span>$eventsubscriber<\/span><\/span>.<\/span><\/span>SourceIdentifier<\/span> }<\/span> <\/span>|<\/span><\/span> <\/span>Out-Null<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

    <\/span>}<\/span><\/span><\/span><\/span><\/span><\/p>\n

}<\/span><\/span><\/span><\/p>\n

 <\/span><\/span><\/span><\/p>\n

 <\/span><\/span><\/span><\/p>\n

# Start up to the max number of concurrent jobs <\/span><\/span><\/span><\/span><\/p>\n

# Each job will take care of running the rest<\/span><\/span><\/span><\/span><\/p>\n

for<\/span><\/span>(<\/span><\/span> <\/span>$i<\/span><\/span> <\/span>=<\/span><\/span> <\/span>0<\/span><\/span>;<\/span> <\/span>$i<\/span><\/span> <\/span>-lt<\/span><\/span> <\/span>$maxConcurrentJobs<\/span><\/span>;<\/span> <\/span>$i<\/span><\/span>++<\/span><\/span> <\/span>)<\/span><\/span><\/span><\/span><\/span><\/p>\n

{<\/span><\/span><\/span><\/p>\n

    <\/span><\/span>RunJobFromQueue<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

}<\/span><\/span><\/span><\/p>\n

The English version of this script is:<\/p>\n