Before stepping into the steppable<\/em> pipeline, it is essential that you have a good understanding of how and when<\/em> exactly items are processed by a cmdlet<\/a> in the pipeline. The PowerShell pipeline might just look like syntactical sugar but it is a lot more than that. In fact, it really acts<\/em> like a pipeline where each item flows through and is handled by each cmdlet one-at-a-time. In comparison to the pipes in CMD, PowerShell streams objects<\/em> through the pipeline rather than plain text.<\/p>\n The following explanation describes the one-at-a-time processing<\/strong> section of the About pipelines<\/a> document. A good analogy of the pipeline is a physical assembly line where each consecutive station on the line could be compared with a PowerShell cmdlet. At a specific station and time, some something is done to one item while the next item is prepared at the prior station. For example, at station 2 a component is soldered to the assembly while the next item is being unpacked at station 1. Items iterate through the pipeline like:<\/p>\n Iteration: Iteration: Cmdlets act like stations in the assembly line, taken a simple example:<\/p>\n In this example the To visualize the order of the items that go through the Notice that This shows the following output:<\/p>\n This proves that each item flows out of the pipeline ( The previous section explains how a cmdlet would perform if correctly implemented for the middle of a pipeline but there are a few statements that might “choke<\/strong>” the pipeline, meaning that the items are no longer processed one-at-the-time<\/strong> but piled up in memory and eventually processed all-at-once<\/strong>. This happens for:<\/p>\n Assigning the pipeline to a variable<\/strong>:<\/p>\n Using parentheses<\/strong>:<\/p>\n Some cmdlets might choke the pipeline by design:<\/strong><\/p>\n In general, a well defined cmdlet should write single records to the pipeline. See the Strongly Encouraged Development Guidelines<\/a> article.<\/p>\n Yet this is not always possible. Take, for example, the This shows the following output:<\/p>\n In general, you should avoid chocking the pipeline, but their are few exceptions where it might be required. For example, where you want to read and write back to the same file as in the previous “using parenthesis” example.<\/p>\n In a smooth pipeline, each item is processed one-at-the-time, meaning that The process cannot access the file ‘.\\Data.txt’ because it is being used by another process.<\/p>\n<\/blockquote>\n In this situation, chocking the pipeline and reading the complete file first avoids the error.<\/p>\n Objects in the PowerShell pipeline contain more than just the value of the item. They also include properties such as the name and type of the item and of all the properties. Take, for example, the .NET PowerShell converts each row into a new object, duplicating the header information for each row. The memory usage considerably increases even if the value is just a few bytes. This extra overhead shouldn’t be an issue if you stream the objects through the pipeline because there will only be a few objects in the pipeline at any time.<\/p>\n Nevertheless, there is a pitfall in using the pipeline. Consider the following two objects being output to a table:<\/p>\n Results<\/p>\n Notice that there is no As you might have noticed, some actions, like outputting a header, are only required once. As in the analogy with the assembly line, heating up a soldering gun is only required once, when the pipeline is started. Cleaning up the station is only required when the pipeline is completed. Similar time consuming or expensive actions could be required for a cmdlet, such as opening and closing a file. These actions are respectively defined in the When running this example cmdlet in a pipeline like A similar pipeline can be created with the common With this understanding of the pipeline, you can see why you shouldn’t wrap a cmdlet pipeline inside another pipeline, like:<\/p>\n Wrapping a cmdlet pipeline into another ( Unfortunately, it is not always possible to create a single syntactical pipeline. For example, you might need different branches for different parameters values or as output paths. Consider a very large But as stated this before, this will open and close each output file ( Every 10,000 ( (Note that it is important to end the pipeline but there is no harm in invoking the It is a little more code, but if you measure the results you will see that in this situation the later script is more than 50 times faster than the one with the wrapped cmdlet pipeline.<\/p>\n With the steppable pipeline technique, you might even have multiple output pipelines open at once. Consider that for the very large Explanation:<\/strong><\/p>\n processes each (One-at-a-time) item of the Takes the first character of the If the pipeline for the specific character doesn’t yet exist:<\/p>\n Closes all the existing steppable pipelines (aka <\/p>\n","protected":false},"excerpt":{"rendered":" The PowerShell pipeline explained from the beginning to the end.<\/p>\n","protected":false},"author":109964,"featured_media":77,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[13],"tags":[80,3,81],"class_list":["post-861","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-powershell","tag-pipeline","tag-powershell","tag-steppable"],"acf":[],"blog_post_summary":" The PowerShell pipeline explained from the beginning to the end.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/posts\/861","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/users\/109964"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/comments?post=861"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/posts\/861\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/media\/77"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/media?parent=861"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/categories?post=861"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell-community\/wp-json\/wp\/v2\/tags?post=861"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}One-at-a-time
process<\/code><\/h2>\nn<\/code><\/strong><\/p>\n item 3 --> item 2 --> item 1\nStation 1 | Station 2 | Station 3\n<\/code><\/pre>\nn + 1<\/code><\/strong><\/p>\n item 4 --> item 3 --> item 2\nStation 1 | Station 2 | Station 3\n<\/code><\/pre>\nGet-Content .\\Input.txt | Foreach-Object { $_ } | Set-Content .\\Output.txt<\/code><\/pre>\nForeach-Object { $_ }<\/code> cmdlet does nothing more than:<\/p>\n\n
Get-Content .\\Input.txt<\/code><\/li>\nSet-Content .\\Output.txt<\/code>.<\/li>\n<\/ul>\nForeach-Object { $_ }<\/code> cmdlet you might use the Trace-Command<\/code> cmdlet but that might overwhelm you with data. Instead, using two simple ForEach-Object<\/code> (alias %<\/code>) test commands show you exactly where your measure points are and what goes in and come out the specific cmdlet in between.<\/p>\n\n
%{Write-Host 'In:' $_; $_ }<\/code><\/li>\n%{Write-Host 'out:' $_; $_ }<\/code><\/li>\n<\/ul>\n...; $_ }<\/code> in the end of the command will place the current item back on the pipeline. In the following example, the cmdlet at the start of the pipeline (Get-Content .\\Input.txt<\/code>) has been replaced with 4 hardcoded input items (1,2,3,4<\/code>) and the cmdlet at the end of the pipeline (Set-Content .\\Output.txt<\/code>) with Out-Null<\/code> which simply purges the actual output of the pipeline so that only the two test cmdlets produce an output.<\/p>\n1,2,3,4 | %{Write-Host 'In:' $_; $_ } |\n Foreach-Object { $_ } |\n %{Write-Host 'Out:' $_; $_ } |\n Out-Null<\/code><\/pre>\nIn: 1\nOut: 1\nIn: 2\nOut: 2\nIn: 3\nOut: 3\nIn: 4\nOut: 4<\/code><\/pre>\nOut: 1<\/code>) before the next item (In: 2<\/code>) is injected into it. As you can imagine, this conserves memory as there are only a few items in the pipeline at any time.<\/p>\nChoking the pipeline<\/h2>\n
\n
$Content = Get-Content .\\Input.txt | Foreach-Object { $_ }\n$Content | Set-Content .\\Output.txt<\/code><\/pre>\n<\/li>\n(Get-Content .\\Data.txt | Foreach-Object { $_ }) | Set-Content .\\Data.txt<\/code><\/pre>\n<\/li>\nSort-Object<\/code> cmdlet, which is supposed to sort an object collection. This might result is a new list where the last item ends up first. To determine what item comes first, you must collect all items before they can be sorted. This is visible from the simple test commands used before:<\/p>\n1,2,3,4 | %{Write-Host 'In:' $_; $_ } | Sort-Object | %{Write-Host 'Out:' $_; $_ } | Out-Null<\/code><\/pre>\nIn: 1\nIn: 2\nIn: 3\nIn: 4\nOut: 1\nOut: 2\nOut: 3\nOut: 4<\/code><\/pre>\n<\/li>\n<\/ul>\nGet-Content<\/code> and Set-Content<\/code> are concurrently processing items in the pipeline. This causes the following error:<\/p>\n\n
Heavy objects<\/h3>\n
DataTable<\/code> object. The header of a DataTable<\/code> object contains the column (property) names and types where each row in the DataTable<\/code> only contains the value of each column. If you convert a DataTable<\/code> into a list of PowerShell objects, like:<\/p>\n$Data = $DataTable | Foreach-Object { $_ }<\/code><\/pre>\nMissing properties<\/h3>\n
$a = [pscustomobject]@{ name='John'; address='home'}\n$b = [pscustomobject]@{ name='Jane'; phone='123'}\n$a, $b |Format-Table<\/code><\/pre>\nname address\n---- -------\nJohn home\nJane<\/code><\/pre>\nphone<\/code> column, meaning that the phone='123'<\/code> property is missing from the results. This is due to the one-at-a-time processing. At the moment the Format-Table<\/code> cmdlet receives object $a<\/code> it is supposed to process it immediately by writing it to the console and release it so that it can process the next item. The issue is that the Format-Table<\/code> cmdlet is unaware of the next object $b<\/code> because it hasn’t entered the pipeline yet. The initial output, based on $a<\/code>, has already been written to the console. In other words, a cmdlet written for one-at-a-time processing bases its output on the first object received from the pipeline. This also implies that if you change the order of the items in the pipeline (for example, $a, $b | Sort-Object | Format-Table<\/code>) properties might appear differently.<\/p>\nProcessing blocks<\/h3>\n
Begin<\/code> and End<\/code> blocks of a cmdlet. The actual processing of items is defined in the Process<\/code> block of cmdlet. A well defined pipeline PowerShell cmdlet might look like this:<\/p>\nfunction MyCmdlet {\n [CmdletBinding()] param(\n [Parameter(ValueFromPipeLine = $True)] [String] $InputString\n )\n Begin {\n $Stream = [System.IO.StreamWriter]::new(\"$($Env:Temp)\\My.Log\")\n }\n Process {\n $Stream.WriteLine($_)\n }\n End {\n $Stream.Close()\n }\n}<\/code><\/pre>\n1..9 | MyCmdlet<\/code>, the log file is only opened once<\/em> at the start, then each item in the pipeline is processed one-at-a-time, and the log file is closed (once<\/em>) at the end. Note that when there are no Begin<\/code>, Process<\/code> and End<\/code> processing blocks defined in a function, the content of the function is assigned to the End<\/code> block. See also: about Functions Advanced Methods<\/a>.<\/p>\nForeach-Object<\/code><\/a> cmdlet using the -Begin<\/code>, -Process<\/code> and -End<\/code> parameters to define the corresponding process blocks:<\/p>\n1..9 | Foreach-Object -Begin {\n $Stream = [System.IO.StreamWriter]::new(\"$($Env:Temp)\\My.Log\")\n} -Process {\n $Stream.WriteLine($_)\n} -End {\n $Stream.Close()\n}<\/code><\/pre>\nPerformance<\/h3>\n
1..9 | ForEach-Object {\n $_ | MyCmdlet\n}<\/code><\/pre>\nForEach-Object<\/code>) pipeline is very expensive because you’re also invoking the begin<\/code> and end<\/code> block of MyCmdlet<\/code>. This will open and close the log file for each item instead of only once at the beginning and the end of the pipeline. The performance degradation can happend with any cmdlet that takes pipeline input. See also PowerShell scripting performance considerations<\/a>.<\/p>\nThe steppable pipeline<\/h2>\n
csv<\/code> file that you want to cut in smaller files. The obvious approach is to split it into files with a maximum number of lines:<\/p>\n$BatchSize = 10000\nImport-Csv .\\MyLarge.csv |\n ForEach-Object -Begin {\n $Index = 0\n } -Process {\n $BatchNr = [math]::Floor($Index++\/$BatchSize)\n $_ | Export-Csv -Append .\\Batch$BatchNr.csv\n }<\/code><\/pre>\n.\\Batch$BatchNr.csv<\/code> ) 10,000 times where it only needs to be opened and closed once per output file. So, the solution here is to use a steppable pipeline which lets you independently define the processing blocks for the required output stream:<\/p>\n$BatchSize = 10000\nImport-Csv .\\MyLarge.csv |\n ForEach-Object -Begin {\n $Index = 0\n } -Process {\n if ($Index % $BatchSize -eq 0) {\n $BatchNr = [math]::Floor($Index++\/$BatchSize)\n $Pipeline = { Export-Csv -notype -Path .\\Batch$BatchNr.csv }.GetSteppablePipeline()\n $Pipeline.Begin($True)\n }\n $Pipeline.Process($_)\n if ($Index++ % $BatchSize -eq 0) { $Pipeline.End() }\n } -End {\n $Pipeline.End()\n }<\/code><\/pre>\n$BatchSize<\/code>) entries, the modulus (%<\/code>) is zero and a new pipeline is created for the expression { Export-Csv -notype -Path .\\Batch$BatchNr.csv }<\/code>.<\/p>\n\n
$Pipeline.Begin($True)<\/code> invokes the Begin<\/code> block of the steppable pipeline, which opens an new csv<\/code> file named .\\Batch$BatchNr.csv<\/code> and writes the headers to the file.<\/li>\n$Pipeline.Process($_)<\/code> invokes the Process<\/code> block of the steppable pipeline using the current item ($_<\/code>), which is appended to the csv<\/code> file.<\/li>\n$Pipeline.End()<\/code>invokes the End<\/code> block of the steppable pipeline, which closes the csv<\/code> file named .\\Batch$BatchNr.csv<\/code>. This file holds a total of 10,000 entries.<\/li>\n<\/ul>\n$Pipeline.End()<\/code> multiple times.)<\/p>\nMultiple output pipelines<\/h3>\n
csv<\/code> file in the previous example, you do not want batches of 10,000 entries but divide the entries over 26 files based on the first letter of the LastName<\/code> property:<\/p>\n$Pipeline = @{}\nImport-Csv .\\MyLarge.csv |\n ForEach-Object -Process {\n $Letter = $_.LastName[0].ToString().ToUpper()\n if (!$Pipeline.Contains($Letter)) {\n $Pipeline[$Letter] = { Export-CSV -notype -Path .\\$Letter.csv }.GetSteppablePipeline()\n $Pipeline[$Letter].Begin($True)\n }\n $Pipeline[$Letter].Process($_)\n } -End {\n foreach ($Key in $Pipeline.Keys) { $Pipeline[$Key].End() }\n }<\/code><\/pre>\n\n
Import-Csv .\\MyLarge.csv | ForEach-Object -Process {<\/code><\/p>\ncsv<\/code> file<\/p>\n<\/li>\n$Letter = $_.LastName[0].ToString().ToUpper()<\/code><\/p>\nLastName<\/code> property and puts that in upper case.<\/p>\n<\/li>\nif (!$Pipeline.Contains($Letter)) {<\/code><\/p>\n\n
{ Export-CSV -notype -Path .\\$Letter.csv }.GetSteppablePipeline()<\/code><\/li>\nBegin<\/code> block: .Begin($True)<\/code> which creates a new csv<\/code> file with the concerned headers<\/li>\n<\/ul>\n<\/li>\nforeach ($Key in $Pipeline.Keys) { $Pipeline[$Key].End() }<\/code><\/p>\ncsv<\/code> files)<\/p>\n<\/li>\n<\/ul>\nSee also<\/h3>\n
\n