{"id":1070,"date":"2014-07-03T00:01:00","date_gmt":"2014-07-03T00:01:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/heyscriptingguy\/2014\/07\/03\/back-to-basics-part-4-more-ways-to-manipulate-data\/"},"modified":"2022-06-10T11:55:30","modified_gmt":"2022-06-10T18:55:30","slug":"back-to-basics-part-4-more-ways-to-manipulate-data","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/scripting\/back-to-basics-part-4-more-ways-to-manipulate-data\/","title":{"rendered":"Back to Basics Part 4: More Ways to Manipulate Data"},"content":{"rendered":"<p><b style=\"font-size:12px\">Summary<\/b><span style=\"font-size:12px\">: Microsoft PFE, Gary Siepser, talks more about the basics of the Windows PowerShell pipeline.<\/span><\/p>\n<p>Microsoft Scripting Guy, Ed Wilson, is here. Today Gary Siepser delivers his Part\u00a04 of his five part series.<\/p>\n<p>To follow along, you can read:<\/p>\n<ul>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/scripting\/back-to-the-basics-part-1-learn-about-the-powershell-pipeline\/\" target=\"_blank\" rel=\"noopener\">Back to the Basics Part 1: Learn About the PowerShell Pipeline<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/scripting\/back-to-the-basics-part-2-learn-about-the-pipeline-and-getters\/\" target=\"_blank\" rel=\"noopener\">Back to the Basics Part 2: Learn about the Pipeline and &#8220;Getters&#8221;<\/a><\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/scripting\/back-to-the-basics-part-3-do-something-with-your-data\/\" target=\"_blank\" rel=\"noopener\">Back to the Basics Part\u00a03: Do Something with Your Data<\/a><\/li>\n<\/ul>\n<p>In Part 3, we introduced several cool things you can do with your data. In this post, we will continue that investigation by looking at even cooler techniques. We\u2019ll focus on:<\/p>\n<ul>\n<li>Grouping your data<\/li>\n<li>Using your data in the pipeline while you use it another way too<\/li>\n<li>Advanced use of <strong>Where-Object<\/strong> filtering that we discussed in Part 2<\/li>\n<\/ul>\n<h4>Grouping data<\/h4>\n<p>Sometimes when you are working with typical IT admin-type information, it\u2019s useful to get counts of various things. Often there are common values on objects. For example, when you are looking at a list of users, they all might have an office location. It is a great thing to be able to get a quick count of users per office.<\/p>\n<p>The cmdlet <strong>Group-Object<\/strong> (the alias is <strong>Group<\/strong>) does a wonderful job of lumping together data based on common values. In our user per office example from Active Directory, you can easily pipe the user objects into <strong>Group-Object<\/strong> and tell it to group by the <strong>Office<\/strong> property. You will be pleasantly surprised at how quickly you get exactly what you are looking for.<\/p>\n<p>Using the <strong>Group-Object<\/strong> cmdlet is actually pretty easy. Identifying the properties that you want to group by is a little more challenging. If you look back at Part 2 in this series, you will see how you can use the <strong>Get-Member<\/strong> cmdlet to see the available properties for your objects. That is only half the battle though, because it only makes sense to group objects where the data values have commonality. This is logical, and it would be true even if you were working in Excel to do the same thing. When you know what property you want to group by, you can use it as a parameter for <strong>Group-Object<\/strong>.<\/p>\n<p><strong>Group-Object<\/strong> has a number of interesting parameters in addition to what property you choose to group by. If you are truly interested in only the group names and counts (and not the individual object data\u2014which is the case for me most of the time), there is a <strong>\u2013NoElement<\/strong> property that will give you only those unique group names and counts. See the Help file for more information about extra things <strong>Group-Object<\/strong> can do (Get-Help Group-Object \u2013ShowWindow).<\/p>\n<p><a href=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/2068.Capture1.PNG\"><img decoding=\"async\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/2068.Capture1.PNG\" alt=\"Image of command output\" title=\"Image of command output\" \/><\/a><\/p>\n<p><span style=\"font-size:12px\">The last example combines grouping with sorting, and it adds a parameter that we introduced in Part\u00a03 of the series from the <\/span><b style=\"font-size:12px\">Select-Object<\/b><span style=\"font-size:12px\"> cmdlet. This parameter allows you to grab a chunk from the top of your output. Recall that <\/span><b style=\"font-size:12px\">Select-Object<\/b><span style=\"font-size:12px\"> is about choosing part of your objects, but I did call it the Swiss Army knife of Windows PowerShell. The <\/span><b style=\"font-size:12px\">\u2013First<\/b><span style=\"font-size:12px\"> parameter is simply another tool being used in the pocket knife. This example is cool because it shows you the five most popular verbs used by cmdlets on your system.<\/span><\/p>\n<h4>Do two things with your objects at the same time<\/h4>\n<p>There is a cmdlet called <strong>Tee-Object<\/strong> (the alias is <strong>Tee<\/strong>). It allows you to keep your stuff flowing down the pipe, but also store it in a variable for later or even in a text file for long term keeping. I have found this to be a really cool trick for scripts (I know we are focused on the pipeline, but it\u2019s a cool trick).<\/p>\n<p>There are times when I have a nice pipeline that does some cool stuff, but it might take a while. This happens a lot when I use cmdlets like <strong>Get-CIMInstance<\/strong> or <strong>Get-WMIObject<\/strong> to get information from a lot of distributed systems. Often I want to output that data to a text file, or even to a variable to use later in that script.<\/p>\n<p>Because the pipeline takes a long time to finish, I see nothing on the screen the entire time that I am waiting. When you see nothing for too long, you start to wonder if it\u2019s even working. <strong>Tee-Object<\/strong> can make the user experience a lot better because you can see the objects being output steadily as the pipeline does its work, and still get the results in a variable or text file after.<\/p>\n<p><strong>Tee-Object<\/strong> simply splits the objects coming down the pipe in two directions (that is why it\u2019s called Tee). One direction is always to continue flowing down the pipeline, and the other is a choice between a text file and a variable. It\u2019s very simple, use the <strong>\u2013FilePath<\/strong> parameter for a file, and the <strong>\u2013Variable<\/strong> parameter for a variable.<\/p>\n<p>When you use the <strong>\u2013FilePath<\/strong> parameter, there is another parameter called <strong>\u2013Append<\/strong> that you can use to continue adding data to a file that already exists. Again, this is a particularly great trick for pipelines that take a long time.<\/p>\n<p><a href=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/0726.Capture2.PNG\"><img decoding=\"async\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/0726.Capture2.PNG\" alt=\"Image of command output\" title=\"Image of command output\" \/><\/a><\/p>\n<p><span style=\"font-size:12px\">Both of these examples show that we were able to see the results when running the pipeline, and then also in the text file and the variable. Look at these and imagine how this would be nice for a long pipeline. You get to see something the entire time it&#8217;s running!<\/span><\/p>\n<h4>Revisiting the Where-Object cmdlet<\/h4>\n<p>This time around, we&#8217;ll focus on the more traditional and advanced syntax of <strong>Where-Object<\/strong>. The advanced syntax is useful when you want more than a simple filter. You\u2019ll remember from Part\u00a03 of this series that the simplified syntax worked by simply picking the property you wanted to filter by, choosing a filter comparison type, and choosing the value you want to compare to.<\/p>\n<p>The more advanced syntax requires a little bit more in the way of special characters. First of all, we are going to use a parameter of <strong>Where-Object<\/strong> called <strong>\u2013FilterScript<\/strong>. The trick is that this parameter is the first position, meaning if you leave it off and give <strong>Where-Object<\/strong> the value, it knows you meant to use <strong>the \u2013FilterScript<\/strong> parameter.<\/p>\n<p>This is the way I normally see it used. Actually, I have found that in many cases, folks don\u2019t even realize they are using a regular old parameter when they use <strong>Where-Object<\/strong>. They learned how the syntax works and \u201cwent with it\u201d\u2014without even knowing it is a parameter or the parameter\u2019s name.<\/p>\n<p>The value we want to use with the traditional syntax of <strong>Where-Object<\/strong> is called a script block in Windows PowerShell. Don\u2019t be confused by the name, it\u2019s just a way to give Windows PowerShell some script, and have that script be the argument. What is important is to realize that the code you have in the script block is going to be surrounded by <strong>{<\/strong>curly braces<strong>}<\/strong>.<\/p>\n<p>Inside this script block, we need to tell <strong>Where-Object<\/strong> what criteria we are going to use to filter this object. Let\u2019s take a look at a couple of comparisons of the simplified syntax vs. the equivalent filter using the traditional advanced syntax.<\/p>\n<p><a href=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/6278.Capture3.PNG\"><img decoding=\"async\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/6278.Capture3.PNG\" alt=\"Image of script\" title=\"Image of script\" \/><\/a><\/p>\n<p>Both examples are functionally the same.<\/p>\n<p>Not only do you see the script block surrounded by curly braces, but you\u2019ll notice a few other differences. The <strong>$_ **automatic variable is in use. This is a pretty simple concept. It is a placeholder that represents the objects being piped to **Where-Object<\/strong>.<\/p>\n<p><strong>Where-Object<\/strong> examines each object to see if it needs to be filtered. The special variable allows you to refer to each of those objects together in one script block. You can also interchangeably use <strong>$PSItem<\/strong> in place of <strong>$_<\/strong>. Either one works the same. <strong>$PSItem<\/strong> was introduced in Windows PowerShell\u00a03.0 to give a more readable name and reduce the use of the underscore special character, if desired. I\u2019m so used to the simplicity of the single character name that I stick with <strong>$_<\/strong>.<\/p>\n<p>You\u2019ll also notice that we need to use a more traditional way to access an object\u2019s property. That is using a single dot. The single dot allows access to any object members, and the properties and methods for an object. Remember, as we learned in Part 3, you can use the <strong>Get-Member<\/strong> cmdlet to explore what is available for an object.<\/p>\n<p>The only other difference is that when using a text value (a string) with the traditional syntax, you need to place the string in quotes.<\/p>\n<p>You can see this is definitely not as pleasing to the eye, nor is it as easy for beginners who are new to Windows PowerShell. Luckily with this introduction in Windows PowerShell\u00a03.0, you can use the simpler syntax. The traditional syntax only needs to be used when you want to do more than a simple test.<\/p>\n<p>One common reason we need the traditional syntax is to combine multiple conditions at the same time. There are operators in Windows PowerShell for this purpose. The great news is they work just like you use them in spoken language. We have <strong>\u2013and<\/strong>, <strong>\u2013or<\/strong>, and <strong>\u2013xor<\/strong>.<\/p>\n<p>The <strong>\u2013and<\/strong> is pretty simple: both sides need to be True. The <strong>\u2013or<\/strong> means either side or both sides need to be True. Although lesser known, <strong>\u2013xor<\/strong> is also really easy. It means one side or the other needs to be True, but not both at the same time.<\/p>\n<p>Take a look at these examples to see how these operators can make your filters richer. In both examples, you will see I used sets of parenthesis to ensure Windows PowerShell compares things correctly in the right order. This is a good habit, though these particular examples wouldn\u2019t have run any differently if I left them off. Parenthesis allow you to control the order of operations if you need them different than the default. It\u2019s just like the algebra you learned in school as a kid.<\/p>\n<p><a href=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/1325.Capture4.PNG\"><img decoding=\"async\" src=\"https:\/\/msdnshared.blob.core.windows.net\/media\/TNBlogsFS\/prod.evol.blogs.technet.com\/CommunityServer.Blogs.Components.WeblogFiles\/00\/00\/00\/76\/18\/1325.Capture4.PNG\" alt=\"Image of script\" title=\"Image of script\" \/><\/a><\/p>\n<p><span style=\"font-size:12px\">In both of these examples, we have multiple tests going on, thus the traditional <\/span><b style=\"font-size:12px\">Where-Object<\/b><span style=\"font-size:12px\"> syntax is required. If you are new to Windows PowerShell, and you think this looks like a lot of confusing syntax, take it slowly. You\u2019ll get it all figured out in time. However, feel some pity for those using Windows PowerShell for a while. Before version\u00a03.0, this was the only way to filter with the <\/span><b style=\"font-size:12px\">Where-Object<\/b><span style=\"font-size:12px\"> cmdlet.<\/span><\/p>\n<p>There are all sorts of neat things you can do inside those curly braces. The thing to remember is that <strong>Where-Object<\/strong> is looking for a True or False result. You can literally put anything in those filters\u2014they simply need to come out as True for the objects you want to keep flowing down the pipeline. Check out the Help file for more details (Get-Help Where-Object \u2013ShowWindow).<\/p>\n<p>As you can see, there are so many cool things you can do with your data when you have it in the pipeline. Parts 3 and 4 in this series have covered the basics and a few cool things, but there are many other great cmdlets that you can use to manipulate your objects. First, check out all the cmdlets that use the noun <strong>Object<\/strong>. Beyond them, look at all the script you see around and check out what others are doing in their pipelines.<\/p>\n<p>In Part 5, the end of this series, we\u2019ll learn about a few of the ways to output your objects at the end of your pipeline.<\/p>\n<p>~Gary<\/p>\n<p>Thank you, Gary! I invite you to follow me on <a href=\"http:\/\/bit.ly\/scriptingguystwitter\" target=\"_blank\" rel=\"noopener\">Twitter<\/a> and <a href=\"http:\/\/bit.ly\/scriptingguysfacebook\" target=\"_blank\" rel=\"noopener\">Facebook<\/a>. If you have any questions, send email to me at <a href=\"mailto:scripter@microsoft.com\" target=\"_blank\" rel=\"noopener\">scripter@microsoft.com<\/a>, or post your questions on the <a href=\"http:\/\/bit.ly\/scriptingforum\" target=\"_blank\" rel=\"noopener\">Official Scripting Guys Forum<\/a>. See you tomorrow. Until then, peace.<\/p>\n<p><strong>Ed Wilson, Microsoft Scripting Guy<\/strong><span style=\"font-size:12px\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Summary: Microsoft PFE, Gary Siepser, talks more about the basics of the Windows PowerShell pipeline. Microsoft Scripting Guy, Ed Wilson, is here. Today Gary Siepser delivers his Part\u00a04 of his five part series. To follow along, you can read: Back to the Basics Part 1: Learn About the PowerShell Pipeline Back to the Basics Part [&hellip;]<\/p>\n","protected":false},"author":596,"featured_media":87096,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[317,51,56,149,3,45],"class_list":["post-1070","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-scripting","tag-gary-siepser","tag-getting-started","tag-guest-blogger","tag-pipeline","tag-scripting-guy","tag-windows-powershell"],"acf":[],"blog_post_summary":"<p>Summary: Microsoft PFE, Gary Siepser, talks more about the basics of the Windows PowerShell pipeline. Microsoft Scripting Guy, Ed Wilson, is here. Today Gary Siepser delivers his Part\u00a04 of his five part series. To follow along, you can read: Back to the Basics Part 1: Learn About the PowerShell Pipeline Back to the Basics Part [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/1070","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/users\/596"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/comments?post=1070"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/1070\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media\/87096"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media?parent=1070"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/categories?post=1070"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/tags?post=1070"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}