{"id":17631,"date":"2010-07-27T00:01:00","date_gmt":"2010-07-27T00:01:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/heyscriptingguy\/2010\/07\/27\/deleting-extra-returns-and-line-feeds-from-a-text-file-using-windows-powershell\/"},"modified":"2010-07-27T00:01:00","modified_gmt":"2010-07-27T00:01:00","slug":"deleting-extra-returns-and-line-feeds-from-a-text-file-using-windows-powershell","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/scripting\/deleting-extra-returns-and-line-feeds-from-a-text-file-using-windows-powershell\/","title":{"rendered":"Deleting Extra Returns and Line Feeds from a Text File Using Windows PowerShell"},"content":{"rendered":"

 <\/p>\n

\"Hey, Hey, Scripting Guy! Is it possible to use Windows PowerShell<\/a> to groom a text file by deleting extra returns and line feeds it may contain? Here is the deal. I have a file that we need to download each day. This file contains updated prices for commodities used in our manufacturing operation. We use the updated commodity pricing to determine our base cost for profit and loss in our general ledger application (g\/l app). A back office application automatically sets our quarterly price and volume discount points based upon the information from the g\/l app. Needless to say, this file is mission critical for us. <\/p>\n

In the past, when we tried to automate the downloading and updating process, there were several \u201cincidents.\u201d As a result, our management is skittish about anything related to these applications. The problem is that right now this is a time-consuming manual process. If someone goes on vacation, we have to train a replacement. It is a lot of stress and I would love to automate it using Windows PowerShell. Help!<\/p>\n

— TB<\/p>\n

 <\/p>\n

\"Hey, Hello TB, <\/p>\n

Microsoft Scripting Guy Ed Wilson here. It is so hot and steamy outside that the crickets are hibernating. The dragonflies cannot fly because of the high humidity<\/a> that saturates their thin wings, making them hang limply like a kite left outside during a spring shower. The birds sit in the shade pleading for a breeze. The rabbits wander aimlessly around the backyard vainly seeking a single green blade of grass for a meal. <\/p>\n

Late July in Charlotte offers special opportunities and unique challenges for outdoor activities. Most people seem to shuttle from one air-conditioned spot to another with little thought or attention given to the intermediate scenery. I feel as if I would like to take a nap in the afternoon like I did when I was in Spain. The following photo is one I took in Mijas<\/a> in Southern Spain. Mijas is a really cool old town with narrow streets and lots of interesting shops. <\/p>\n

\"Photo<\/a> <\/p>\n

TB, a text file such as the one shown in the following image has a number of returns and line feeds in it. These are seen in between the paragraphs and at the end of the text file. When importing the data from the text file into a database, each line could introduce a blank record into the database and cause the problems you experienced with your application. <\/p>\n

\"Image<\/a> <\/p>\n

To remove the blank lines in the text file involves several steps.<\/p>\n

    \n
  1. Rename the text file.<\/li>\n
  2. Read the text file.<\/li>\n
  3. Find lines with text while ignoring blank lines.<\/li>\n
  4. Remove the blank lines from the text.<\/li>\n
  5. Write the revised text to a new file.<\/li>\n
  6. Save the new file with the name of the original text file.<\/li>\n<\/ol>\n

    The reason for the roundabout procedure is that, after the text file is opened, a lock will remain on the file that prevents overwriting the file with new content. This is shown in the following image.<\/p>\n

    \"Image<\/a> <\/p>\n

    To rename the text file, use the Rename-Item<\/b> cmdlet. You use the path<\/b> parameter to specify the original file, and the newname<\/b> parameter for the new name. One thing that is important to keep in mind is that the Rename-Item<\/b> cmdlet does not have the ability to move a file in the rename operation. If you need to move a file, use the Move-Item<\/b> cmdlet. Because you cannot move a file with the Rename-Item<\/b> cmdlet, the inclusion of the path in the newname<\/b> parameter is redundant. The use of the parameter names themselves is also not required. You can also use the ren<\/b> alias if you wish. The short version of the command is shown here:<\/p>\n

    PS<\/font> <\/font>C:\\><\/font> <\/font>ren<\/font> <\/font>C:\\fso\\text1.txt<\/font> <\/font>text1.bak<\/font>
    <\/font>PS<\/font> <\/font>C:\\><\/font> <\/p>\n

    The complete command with parameter names is shown here:<\/p>\n

    Rename-Item<\/font> <\/font>-Path<\/font> <\/font>C:\\fso\\Text1.txt<\/font> <\/font>-NewName<\/font> <\/font>C:\\fso\\text1.bak<\/font> <\/p>\n

    \nAfter the backup copy of the file has been created, use the Select-String<\/b> cmdlet to look for Word characters. This regular expression pattern is documented on MSDN<\/a>. The easiest way to open a text file and parse the contents is to use the Select-String<\/b> cmdlet. We talked about using the Select-String<\/b> cmdlet in yesterday\u2019s Hey, Scripting Guy! Blog<\/a>. The Select-String<\/b> cmdlet can be used to search for a wildcard character pattern or for a regular expression pattern. As shown here, the Select-String<\/b> cmdlet returns both the matching line of text, the file name, and the line number that contains the match:<\/p>\n

    PS<\/font> <\/font>C:\\><\/font> <\/font>Select-String<\/font> <\/font>-Pattern<\/font> "\\w" <\/font>-Path<\/font> <\/font>C:\\fso\\Text1.bak<\/font>
    <\/font>fso\\Text1.bak:<\/font>1<\/font>:This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font>
    <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font>
    <\/font>fso\\Text1.bak:<\/font>2<\/font>:This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font>
    <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font>
    <\/font>fso\\Text1.bak:<\/font>3<\/font>:This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font>
    <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font>
    <\/font>fso\\Text1.bak:<\/font>4<\/font>:This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font>
    <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font>
    <\/font>fso\\Text1.bak:<\/font>6<\/font>:This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font> <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font> <\/font>This<\/font>
    <\/font>is<\/font> <\/font>a<\/font> <\/font>paragraph.<\/font>
    <\/font><results<\/font> <\/font>truncated><\/font> <\/p>\n

    After the matching text is retrieved, use the ForEach-Object cmdlet<\/a> to choose the matching line of text, and pipe<\/a> the results to the new file. The complete command is shown here:<\/p>\n

    PS<\/font> <\/font>C:\\><\/font> <\/font>Select-String<\/font> <\/font>-Pattern<\/font> "\\w" <\/font>-Path<\/font> <\/font>C:\\fso\\Text1.bak<\/font> <\/font>|<\/font> <\/font>ForEach-Object<\/font> <\/font>{<\/font> <\/font>$_<\/font>.line<\/font>
    <\/font>}<\/font> <\/font>|<\/font> <\/font>Set-Content<\/font> <\/font>-Path<\/font> <\/font>C:\\fso\\Text1.txt<\/font>
    <\/font>PS<\/font> <\/font>C:\\><\/font> <\/p>\n

    After the command has run, the newly created file will contain text only. This is shown in the following image<\/p>\n

    \"Image<\/a> <\/p>\n

    TB, that is all there is to using Windows PowerShell to remove extra returns and line feeds from a text file. String Week will continue tomorrow when we will talk about selecting specific columns of data from a tab-separated file. <\/p>\n

    We would love you to follow us on Twitter<\/a> and Facebook<\/a>. If you have any questions, send email to us at scripter@microsoft.com<\/a>, or post your questions on the Official Scripting Guys Forum<\/a>. See you tomorrow. Until then, peace.<\/p>\n

     <\/p>\n

    Ed Wilson and Craig Liebendorfer, Scripting Guys<\/b><\/p>\n","protected":false},"excerpt":{"rendered":"

      Hey, Scripting Guy! Is it possible to use Windows PowerShell to groom a text file by deleting extra returns and line feeds it may contain? Here is the deal. I have a file that we need to download each day. This file contains updated prices for commodities used in our manufacturing operation. We use […]<\/p>\n","protected":false},"author":595,"featured_media":87096,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[3,4,21,45],"class_list":["post-17631","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-scripting","tag-scripting-guy","tag-scripting-techniques","tag-string-manipulation","tag-windows-powershell"],"acf":[],"blog_post_summary":"

      Hey, Scripting Guy! Is it possible to use Windows PowerShell to groom a text file by deleting extra returns and line feeds it may contain? Here is the deal. I have a file that we need to download each day. This file contains updated prices for commodities used in our manufacturing operation. We use […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/17631","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/users\/595"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/comments?post=17631"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/17631\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media\/87096"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media?parent=17631"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/categories?post=17631"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/tags?post=17631"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}