{"id":17941,"date":"2010-06-26T00:01:00","date_gmt":"2010-06-26T00:01:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/heyscriptingguy\/2010\/06\/26\/weekend-scripter-playing-around-with-the-windows-powershell-tokenizer\/"},"modified":"2010-06-26T00:01:00","modified_gmt":"2010-06-26T00:01:00","slug":"weekend-scripter-playing-around-with-the-windows-powershell-tokenizer","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/scripting\/weekend-scripter-playing-around-with-the-windows-powershell-tokenizer\/","title":{"rendered":"Weekend Scripter: Playing Around with the Windows PowerShell Tokenizer"},"content":{"rendered":"<p>&nbsp;<\/p>\n<p><!-- AddThis Button BEGIN --><\/p>\n<p>\n\/\/ \n<a href=\"http:\/\/www.addthis.com\/bookmark.php?v=250&amp;username=scriptingguys\" class=\"addthis_button\"><img decoding=\"async\" height=\"16\" width=\"125\" src=\"http:\/\/s7.addthis.com\/static\/btn\/v2\/lg-share-en.gif\" alt=\"Bookmark and Share\" style=\"border: 0\" \/><\/a><\/p>\n<p><!-- AddThis Button END --><\/p>\n<p>&nbsp;<\/p>\n<\/p>\n<p><span style=\"font-size: 10pt\">Microsoft Scripting Guy Ed Wilson here. Once again, the Scripting Wife and I are on the road. This weekend we are heading to Charleston, South Carolina, to hang out and to see a couple music groups. The Scripting Wife is driving, and I am playing around with Windows PowerShell. We have my Zune HD plugged into our car radio and are listening to Jimmy Buffett<\/span><span style=\"font-family: 'Calibri','sans-serif';font-size: 10pt\">&mdash;<\/span><span style=\"font-size: 10pt\">ARRH! She just interrupted my favorite song<\/span><span style=\"font-family: 'Calibri','sans-serif';font-size: 10pt\">&mdash;<\/span><span style=\"font-size: 10pt\">&ldquo;A Pirate Looks at 40&rdquo;<\/span><span style=\"font-family: 'Calibri','sans-serif';font-size: 10pt\">&mdash;<\/span><span style=\"font-size: 10pt\">to put on&hellip;Johnny Horton? Yes, I think it is Johnny Horton. <\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">Cool. I need to think cool. Before we left, I used one of my Windows PowerShell scripts to check the weather for Charleston. Tomorrow, it is going to be a high of 89 degrees with 87 percent humidity! Oh, well, it is nothing a little bit of ice tea, sunglasses, an aloha shirt, and a Panama hat cannot handle. <\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">I have been wanting to play around with the Windows PowerShell Tokenizer for a while. The Tokenizer is used to break a Windows PowerShell script into pieces of code called tokens. Using the Tokenizer, you can find commands or variables in a Windows PowerShell script. A road trip is the perfect occasion to write Windows PowerShell scripts that explore new techniques. Because it is Friday evening and the script is just for fun, there are no deadlines and no specific guidelines to which the script must adhere. <\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">The ParseScriptCommands.ps1 script is my first attempt. It is shown here. <\/span><\/p>\n<p class=\"CodeBlockScreenedHead\"><span style=\"font-size: 10pt\"><strong>ParseScriptCommands.ps1<\/strong><\/span><\/p>\n<p class=\"CodeBlockScreened\"><span style=\"font-size: x-small\"><span style=\"background-color: #f2f2f2\"><span style=\"font-family: lucida sans typewriter\">$<span style=\"font-size: small\">errors = $null <br \/>$logpath = &#8220;C:\\logs\\commandlog.txt&#8221; <br \/>$path = &#8220;C:\\data\\PSExtras&#8221; <br \/>Get-ChildItem -Path $path -Include *.ps1 -Recurse | <br \/>ForEach-Object { <br \/><span>&nbsp; <\/span>$script = $_.fullname <br \/><span>&nbsp; <\/span>$scriptText = get-content -Path $script <br \/><span>&nbsp; <\/span>[system.management.automation.psparser]::Tokenize($scriptText, [ref]$errors) | <br \/><span>&nbsp; <\/span>Foreach-object -Begin { <br \/><span>&nbsp;&nbsp;&nbsp; <\/span>&#8220;Processing $script&#8221; | Out-File -FilePath $logPath -Append } ` <br \/><span>&nbsp; <\/span>-process { if($_.type -eq &#8220;command&#8221;) <br \/><span>&nbsp;&nbsp;&nbsp; <\/span>{ &#8220;`t $($_.content)&#8221; | Out-File -FilePath $logpath -Append } } <br \/>} <br \/>notepad $logpath<\/span><\/span><\/span><\/span><\/p>\n<p class=\"CodeBlockScreened\">\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\"><span style=\"font-size: small\">T<\/span>he first thing I do in the ParseScriptCommands.ps1 script is initialize three variables. The first one is used to collect any errors by <strong>Tokenizer<\/strong>. The second variable is used for the log file, and the third one specifies the directory that contains the Windows PowerShell scripts that need to be parsed. These commands are shown here:<\/span><\/p>\n<p class=\"CodeBlockScreened\"><span style=\"font-size: x-small\"><span style=\"background-color: #f2f2f2\"><span style=\"font-family: lucida sans typewriter\">$<span style=\"font-size: small\">errors = $null <br \/>$logpath = &#8220;C:\\logs\\commandlog.txt&#8221; <br \/>$path = &#8220;C:\\data\\PSExtras&#8221;<\/span><\/span><\/span><\/span><\/p>\n<p class=\"CodeBlockScreened\">\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">The next four commands are standard Windows PowerShell cmdlets. The <strong>Get-ChildItem<\/strong> cmdlet retrieves only Windows PowerShell scripts (that have the .ps1 extension) from the script directory specified earlier. The <strong>&ndash;recurse<\/strong> parameter is required when retrieving the files from the folder. The resulting <strong>fileinfo<\/strong> objects are piped to the <strong>Foreach-Object<\/strong> cmdlet where the full path to each script is stored in the <strong>$script<\/strong> variable. Next, the <strong>Get-Content<\/strong> cmdlet reads each Windows PowerShell script and stores the content of the file in the <strong>$scriptText<\/strong> variable. This section of the script is shown here:<\/span><\/p>\n<p class=\"CodeBlockScreened\"><span style=\"font-size: x-small\"><span style=\"background-color: #f2f2f2\"><span style=\"font-family: lucida sans typewriter\">G<span style=\"font-size: small\">et-ChildItem -Path $path -Include *.ps1 -Recurse | <br \/>ForEach-Object { <br \/><span>&nbsp; <\/span>$script = $_.fullname <br \/><span>&nbsp; <\/span>$scriptText = get-content -Path $script<\/span><\/span><\/span><\/span><\/p>\n<p class=\"CodeBlockScreened\">\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">The <strong>psparser<\/strong> .NET Framework class in the <strong>system.management.automation<\/strong> namespace has the <strong>Tokenize<\/strong> static method. The first parameter is a variable containing the contents of a Windows PowerShell script, and the second parameter is a reference variable to hold the errors. The second parameter must be supplied when calling the <strong>Tokenize<\/strong> method. The tokens are then piped to the next section of the script. This command is shown here:<\/span><\/p>\n<p class=\"CodeBlockScreened\"><span style=\"font-size: x-small\"><span style=\"background-color: #f2f2f2\"><span style=\"font-family: lucida sans typewriter\">[<span style=\"font-size: small\">system.management.automation.psparser]::Tokenize($scriptText, [ref]$errors) |<\/span><\/span><\/span><\/span><\/p>\n<p class=\"CodeBlockScreened\">\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">I use the <strong>Foreach-Object<\/strong> cmdlet to process each token. The first thing I do is write the full script path to the log file. Next, I check to see if the <strong>type<\/strong> property of the <strong>token<\/strong> object is a command&mdash;if it is, I write the command to the log file as well. When I have processed all of the scripts in the folder, I display the contents of the log file. This section of the script is shown here:<\/span><\/p>\n<p class=\"CodeBlockScreened\"><span style=\"font-size: x-small\"><span style=\"background-color: #f2f2f2\"><span style=\"font-family: lucida sans typewriter\"><span>&nbsp;<span style=\"font-size: small\"> <\/span><\/span><span style=\"font-size: small\">Foreach-object -Begin { <br \/><span>&nbsp;&nbsp;&nbsp; <\/span>&#8220;Processing $script&#8221; | Out-File -FilePath $logPath -Append } ` <br \/><span>&nbsp; <\/span>-process { if($_.type -eq &#8220;command&#8221;) <br \/><span>&nbsp;&nbsp;&nbsp; <\/span>{ &#8220;`t $($_.content)&#8221; | Out-File -FilePath $logpath -Append } } <br \/>} <br \/>notepad $logpath<\/span><\/span><\/span><\/span><\/p>\n<p class=\"CodeBlockScreened\">\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">When the script runs, a text file appears that is similar to the one shown in the following image. <\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\"><a href=\"http:\/\/img.microsoft.com\/library\/media\/1033\/technet\/images\/scriptcenter\/qanda\/hsg\/2010\/june\/hey0626\/wes-06-26-10-01.jpg\"><span style=\"font-family: times new roman;color: #0000ff\"><img decoding=\"async\" style=\"border: 0px\" title=\"Image of text file generated when script is run\" border=\"0\" alt=\"Image of text file generated when script is run\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/29\/2019\/02\/8321.wes062610011_5465723A.jpg\" width=\"604\" height=\"644\" \/> <\/span><\/a><\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">Well, believe it or not, we are just about to arrive in Charleston. The Scripting Wife has abandoned Johnny Horton and is now listening to Kiss. I guess she is growing impatient. The GPS says we will be there in 6 miles, so I need to shut down my laptop and prepare to disembark. Check back tomorrow. I have some pretty cool ideas for improving my tokenizing script. I might take my laptop down to park that overlooks Fort Sumpter and work on that script tomorrow morning. Who knows what I will do really? It&rsquo;s the weekend, and tomorrow is another day. <\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-size: 10pt\">If you want to know exactly what we will be looking at tomorrow, follow us on <a href=\"http:\/\/bit.ly\/scriptingguystwitter\" target=\"_blank\"><span style=\"font-family: arial,helvetica,sans-serif\">Twitter<\/span><\/a><span style=\"font-family: arial,helvetica,sans-serif\"> <\/span>or <a href=\"http:\/\/bit.ly\/scriptingguysfacebook\"><span style=\"font-family: arial,helvetica,sans-serif\">Facebook<\/span><\/a><span style=\"font-family: arial,helvetica,sans-serif\">.<\/span> If you have any questions, send e-mail to us at <a href=\"mailto:scripter@microsoft.com\" target=\"_blank\"><span style=\"font-family: arial,helvetica,sans-serif\">scripter@microsoft.com<\/span><\/a><span style=\"font-family: arial,helvetica,sans-serif\">,<\/span> or post your questions on the <a href=\"http:\/\/bit.ly\/scriptingforum\" target=\"_blank\"><span style=\"font-family: arial,helvetica,sans-serif\">Official Scripting Guys Forum<\/span><\/a><span style=\"font-family: arial,helvetica,sans-serif\">.<\/span> See you tomorrow. Until then, peace.<\/span><\/p>\n<p class=\"MsoNormal\"><span style=\"font-family: 'Segoe UI','sans-serif';font-size: 10pt\">&nbsp;<\/span><\/p>\n<p><strong><span style=\"font-family: 'Segoe UI','sans-serif';font-size: 10pt\">Ed Wilson and Craig Liebendorfer, Scripting Guys<\/span><\/strong><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; \/\/ &nbsp; Microsoft Scripting Guy Ed Wilson here. Once again, the Scripting Wife and I are on the road. This weekend we are heading to Charleston, South Carolina, to hang out and to see a couple music groups. The Scripting Wife is driving, and I am playing around with Windows PowerShell. We have my [&hellip;]<\/p>\n","protected":false},"author":595,"featured_media":87096,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[2,3,4,61,45],"class_list":["post-17941","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-scripting","tag-running","tag-scripting-guy","tag-scripting-techniques","tag-weekend-scripter","tag-windows-powershell"],"acf":[],"blog_post_summary":"<p>&nbsp; \/\/ &nbsp; Microsoft Scripting Guy Ed Wilson here. Once again, the Scripting Wife and I are on the road. This weekend we are heading to Charleston, South Carolina, to hang out and to see a couple music groups. The Scripting Wife is driving, and I am playing around with Windows PowerShell. We have my [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/17941","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/users\/595"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/comments?post=17941"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/17941\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media\/87096"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media?parent=17941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/categories?post=17941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/tags?post=17941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}