{"id":86682,"date":"2019-12-18T01:00:53","date_gmt":"2019-12-18T09:00:53","guid":{"rendered":"http:\/\/devblogs.microsoft.com\/scripting\/?p=86682"},"modified":"2020-03-02T06:28:38","modified_gmt":"2020-03-02T14:28:38","slug":"just-go-with-the-flow-workflow-that-is-with-windows-powershell","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/scripting\/just-go-with-the-flow-workflow-that-is-with-windows-powershell\/","title":{"rendered":"Just go with the flow…. WorkFlow that is with Windows PowerShell"},"content":{"rendered":"

Doctor Scripto returns again with our good friend\u00a0Joel Vickery, PFE<\/a> who is going to touch on the use of Workflows In PowerShell.<\/p>\n

Take it away Joel!<\/p>\n

Following up on my original post\u00a0Parallel Processing with jobs in PowerShell<\/a>, I wanted to go into another method of running parallel processes in PowerShell, namely WorkFlows. WorkFlow is very similar to using the Start-Job<\/em><\/strong>\/-asJob<\/strong> functionality but it has some distinct advantages that are fully covered in many other blog postings on TechNet.\u00a0 I’ll mention them quickly below, but if you want deeper detail, see the When Windows PowerShell Met WorkFlow<\/a> blog posting on MSDN. Finally, I wanted to focus on how PowerShell handles the activity of Workflow from a process\/thread perspective so that you can factor in the way that processing workflows works into your decision to use it over other parallel and multi-threading methods. I started this set of posts with the thought “I wonder which one of these methods is the fastest?” so I will include that information as well.<\/p>\n

The Great Parts<\/h3>\n

(Not) Managing the Queue<\/h4>\n

WorkFlow takes some of the overhead of managing the jobs away.\u00a0 If you have worked with PowerShell Jobs, it doesn’t take long to realize that you have a lot of work to do to manage the submission, monitoring, retrieval, and removal of the jobs.<\/p>\n

Unique Processing Control<\/h3>\n

WorkFlow also adds some unique controls for how commands within the workflow execute:<\/p>\n

Parallel<\/h4>\n

Commands that are contained within a Parallel block will run concurrently, meaning that each command does not wait until it is complete before running the next command in the Parallel Block.<\/p>\n

Sequence<\/h4>\n

Commands within a Sequence block will execute each command and wait for the command to complete before running the next command.<\/p>\n

Checkpoints<\/h4>\n

Workflow scripts that are interrupted can be resumed even through a reboot.<\/p>\n

The Tough Parts<\/h3>\n

Restrictions<\/h4>\n

All good things come at a price.\u00a0 With WorkFlows, there are restrictions<\/a> that you need to be aware of that take this from a simple exercise to an intermediate-level development effort.\u00a0 I struggled with issues like changing parameters and scoping of variables as well as the lack of support for invoke-command<\/em><\/strong>.\u00a0 Like any learning experience, once you get the hang of it, you’ll be cranking out WorkFlow code like second nature.<\/p>\n

Debugging<\/h4>\n

Debugging WorkFlows in earlier releases of PowerShell was a challenge since the PowerShell_ISE would lose insight into the processes when they would execute in parallel. This was corrected in PowerShell v4.0, which allows full control of the code within the WorkFlow block, to include setting breakpoints.<\/p>\n

Example Script<\/h3>\n

The script below queries AD for all computers and then, using a WorkFlow, gets the most recent 4000 event 4624 events from the security event log.\u00a0 Since I am using the “-parallel<\/em><\/strong>” switch on the foreach<\/em><\/strong> loop, querying each of the computers will happen concurrently.\u00a0 I will be adding more posts beyond this one and will use the exact same basic script in the future posts, just wrapped in a different parallel processing wrapper.<\/p>\n

workflow Test-WF\r\n{\r\n\u00a0 \u00a0\u00a0 param([array[]]$ServerList)\r\n\u00a0 \u00a0 $ReturnArr = @()\r\nforeach -parallel -ThrottleLimit 10 ($Server in $ServerList)\r\n{\r\n\u00a0 \u00a0 $returnName = $Server\r\n\u00a0 \u00a0 \u00a0 \u00a0 $strCompName = $Server.Name\r\n\u00a0 \u00a0 \u00a0 \u00a0 $Count = InlineScript { (Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| '\r\nWhere-Object {$_.EventID -eq '4624'}).count}\r\n$Workflow:ReturnArr += \"$returnName,$Count\"\r\n}\r\n\u00a0 \u00a0\u00a0 $ReturnArr\r\n}\r\n$arrComputers = (get-adcomputer -filter * -server dc1.contoso.com:3268).Name\r\nGet-date\r\n$Stats = Measure-Command -Expression {$OutTest = Test-WF -ServerList $arrComputers}\r\nGet-date\r\n$outTest\r\n$Stats<\/pre>\n
Process Analysis<\/h3>\n
I have to admit that I made a mistake here, but it turned out to be a good thing since it pointed out an efficiency and performance issue that you can encounter depending how you implement your script.\u00a0 My initial script had an InlineScript<\/em><\/strong> Activity on the line of code that queries the event log and that turned out to be an important factor in how efficient and fast this script is. To get a good idea of what is happening behind the scenes, I’m using ProcMon from the Sysinternals Suite to record the process tree that results from the WorkFlow.\u00a0 In my small test lab environment, I have 8 domain controllers and a general tools server, so 9 machines that are returned from the query against AD and then will each be contacted to parse the event logs. In the screen capture above, you will notice that the initial script was started in the PowerShell_ISE, which then created several additional PowerShell.exe processes, each with their own Process ID.<\/p>\n
In the screen capture below from ProcMon, you can see the PowerShell_ISE.exe creating the final Processes and worker threads to go forth and get the information that we have requested.\u00a0 It’s becoming very clear through this capture that WorkFlows, much like Start-Jobs<\/em><\/strong>\/-asJob<\/em><\/strong> are “multi-process”, not truly multi-threaded.<\/p>\n
It turns out that this is due to the fact that I used the InlineScript<\/em><\/strong> activity, which run in their own process.\u00a0 Just something to keep in mind if resources are an issue and you cannot afford the overhead of multiple PowerShell.exe processes running.<\/p>\n
Finally, in the screen capture below, the newly created processes are querying the security event logs in parallel. To be thorough, I went back and modified the script without the use of the InlineScript<\/em><\/strong> activity, instead using the direct Get-EventLog<\/em><\/strong> call against the remote servers.\u00a0 I changed the line: $Count = InlineScript { (Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| ‘ Where-Object {$_.EventID -eq ‘4624’}).count}<\/em><\/strong> to: $Count =(Get-Eventlog Security -ComputerName $Using:returnName -Newest 4000| ‘ Where-Object {$_.EventID -eq ‘4624’}).count<\/em><\/strong><\/p>\n
Now I am seeing the single PowerShell.exe with multiple threads that I was expecting.<\/p>\n
Performance<\/h3>\n
To give a quick frame of reference, I ran a regular sequential ForEach<\/em><\/strong> loop against all of these same domain controllers and the sequential PowerShell loop took 1 minute and 23 seconds to complete the retrieval of events from the DCs. In the output below, you can see that WorkFlow with InlineScript<\/em><\/strong> beats that time by a considerable margin at 39 seconds.<\/p>\n
Performance took a hit with the InlineScript<\/strong> activity removes when compared to the multi-process, dropping to 1 minute and 17 seconds.\u00a0 It was still faster than the sequential loop but not by a lot.<\/p>\n
Conclusion<\/h3>\n
As you start using these parallel capabilities in PowerShell, it’s important to think about the correct use case for the particular parallel processing capability.\u00a0 For WorkFlows using the InlineScript<\/strong> activity, you have to make sure that you have the resources to handle having multiple full PowerShell.exe environments running, including memory and CPU utilization.\u00a0 Without InlineScript<\/em><\/strong>, resource consumption is less of an issue and the script will run in a true multithreading mode. \u00a0 I tend to use Workflows when I have a decent amount of time to create a workable script, and I need the performance increase, and I know I’ll be collecting dividends on the time spent when compared to a simple sequential loop.<\/p>\n
So that is all there is to with our quick introduction to PowerShell workflows!\u00a0 Pop by next week as we look into PowerShell Workflows a little bit deeper with Joel !<\/p>\n
I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Forum. See you tomorrow. Until then, Keep on Scripting!<\/p>\n
Your good friend, Doctor Scripto<\/strong><\/p>\n
PowerShell, Doctor Scripto, Joel Vickery, Workflows<\/p>\n","protected":false},"excerpt":{"rendered":"
Doctor Scripto returns again with our good friend\u00a0Joel Vickery, PFE who is going to touch on the use of Workflows In PowerShell. Take it away Joel! Following up on my original post\u00a0Parallel Processing with jobs in PowerShell, I wanted to go into another method of running parallel processes in PowerShell, namely WorkFlows. WorkFlow is very […]<\/p>\n","protected":false},"author":596,"featured_media":87096,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1739,690,1738],"tags":[1740,2740,377,2795],"class_list":["post-86682","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-doctor-scripto","category-getting-started","category-powershell","tag-doctor-scripto","tag-joel-vickery","tag-powershell","tag-workflows"],"acf":[],"blog_post_summary":"
Doctor Scripto returns again with our good friend\u00a0Joel Vickery, PFE who is going to touch on the use of Workflows In PowerShell. Take it away Joel! Following up on my original post\u00a0Parallel Processing with jobs in PowerShell, I wanted to go into another method of running parallel processes in PowerShell, namely WorkFlows. WorkFlow is very […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/86682","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/users\/596"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/comments?post=86682"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/86682\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media\/87096"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media?parent=86682"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/categories?post=86682"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/tags?post=86682"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}