Parallel Processing with jobs in PowerShell

Doctor Scripto

Hello everyone!  Doctor Scripto is elated to present some more great content from Joel Vickery, PFE , today he discusses using jobs in PowerShell.   Stay tuned in the upcoming weeks for some amazing regular content.   Take it away Joel!

I have to be honest, back in 2010 I was firmly entrenched in VBScript and had no interest in learning anything new.  Then fate threw me a curve ball when I was presented with a task to fix over 9,000 workstations that had lost their “parent” antivirus server.  Each machine needed to have the exact same set of commands executed on it, a new file copied and service restarted, to fix them so I created a quick  VBScript to connect to each machine and perform the fix actions. The only bad part about this script was that it took days to run against the machines, processing each one sequentially in a For loop before moving on to the next. I did some research on multi-threading with VBScript and kept getting results that referred to PowerShell “jobs”, which gave me the motivation to start learning PowerShell.

For those of you who come from a Unix or Linux background, jobs will be a familiar concept since you can background any command by placing an “&” at the end of the command.  PowerShell Cmdlets sometimes include the “-AsJob” parameter that will allow the command to submitted in the background, releasing the console to be used for other commands. Alternatively, you can use the Start-Job CmdLet to run commands as Jobs. This also means that you can submit multiple jobs in the background and continue processing without waiting for each job to complete.  Keep in mind that there are limits to everything so keep your machine’s resource consumption in mind as you test this concept out.  Submitting a large number of background commands could be resource intensive.

Below are two different ways to do a WMI Query as a job:

Get-WMIObject Win32_OperatingSystem -AsJob

Or

Start-Job {Get-WMIObject Win32_OperatingSystem}

These jobs do all of their processing in the background and store the output until you receive them.  You can check on the status of a job by running Get-Job, the status is in the “State” column, which will show if the command is Running, Completed, or Failed.  Also notice the HasMoreData column.  This indicates that there is output to be retrieved from the job.  In the example output below, notice that the job is still running.

After the job has been kicked off, you can check on the status of the job by running the Get-Job command, noting the State and HasMoreData values. The State will change to Completed when the job has finished and the  HasMoreData value will indicate if there is output.

Get-Job

Once the job has completed, you can use the Receive-Job cmdLet to get the data from the command.

Receive-Job -Id 3       # NOTE, you could also do -Name Job3 here, either will work

The output of the command is now delivered to the console:

Note that the HasMoreData value has now changed to False after running the Receive-Job command:

The important thing to remember here is that you have one chance to get the information from the job so make sure that you capture it in a variable if the output needs to be evaluated.

$JobOutput = Receive-Job -Id 1

Once you are done getting the job’s output, the job will basically hang out there until you remove it by running the Remove-Job Cmdlet.

Remove-Job -Id 1

As you can tell, there are a lot of moving parts to this.

The PowerShell script that I created to address the antivirus client remediation task had to have a mechanism to control how many jobs I could submit at a time, as well as constant monitoring of the queue to receive input from the completed jobs, remove them, and refill the queue with new jobs.

To top it off, once I finished processing the queue of items, I had to write some code to wait for the last batch of jobs was completed to make sure that I received the output from all of them.

That is a lot of overhead. Fast forward to today and we have PowerShell WorkFlows that make this process much easier to manage which I’ll cover in another post.

So that is all there is to with basic PowerShell jobs!  Pop by next week as we look into some introductory work with PowerShell Workflows!

I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Forum. See you tomorrow. Until then, Keep on Scripting!

Your good friend, Doctor Scripto

PowerShell, Doctor Scripto, Joel Vickery, Jobs

 

3 comments

Discussion is closed. Login to edit/delete existing comments.

  • Justin Grote 0

    Jobs and Workspaces? Is this still 2012?

    Please look into Powershell Runspaces (ThreadJob,PoshRSJob,InvokeParallel modules) for 5.1-6 and Foreach-Object -Parallel in Powershell 7.

    • Hans Korsch 0

      Right, and what is the value of a blog post that is just replicating what has been blogged about already? There is also nothing new here that I cannot read in the docs.microsoft.com. This is not the quality I expect of a official Microsoft blog.

  • Phyo Wai Paing 0

    Can you please make the performance measurement between Runspaces & Parallel in foreach loop ?

Feedback usabilla icon