{"id":10941,"date":"2006-04-25T12:17:45","date_gmt":"2006-04-25T12:17:45","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/powershell\/2006\/04\/25\/duplicate-files\/"},"modified":"2019-02-18T13:25:07","modified_gmt":"2019-02-18T20:25:07","slug":"duplicate-files","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/powershell\/duplicate-files\/","title":{"rendered":"Duplicate Files"},"content":{"rendered":"<p>Need a way to check if two files are the same?&nbsp; Calculate a hash of the files.&nbsp; Here is one way to do it:<\/p>\n<p>&nbsp;<\/p>\n<p>## Calculates the hash of a file and returns it as a string.<\/p>\n<p>function Get-MD5([System.IO.FileInfo] $file = $(throw &#8216;Usage: Get-MD5 [System.IO.FileInfo]&#8217;))<\/p>\n<p>{<\/p>\n<p>&nbsp; $stream = $null;<\/p>\n<p>&nbsp; $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider];<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp; $hashAlgorithm = new-object $cryptoServiceProvider<\/p>\n<p>&nbsp; $stream = $file.OpenRead();<\/p>\n<p>&nbsp; $hashByteArray = $hashAlgorithm.ComputeHash($stream);<\/p>\n<p>&nbsp; $stream.Close();<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp; ## We have to be sure that we close the file stream if any exceptions are thrown.<\/p>\n<p>&nbsp; trap<\/p>\n<p>&nbsp; {<\/p>\n<p>&nbsp;&nbsp;&nbsp; if ($stream -ne $null)<\/p>\n<p>&nbsp;&nbsp;&nbsp; {<\/p>\n<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $stream.Close();<\/p>\n<p>&nbsp;&nbsp;&nbsp; }<\/p>\n<p>&nbsp;&nbsp;&nbsp; <\/p>\n<p>&nbsp;&nbsp;&nbsp; break;<\/p>\n<p>&nbsp; }<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp; return [string]$hashByteArray;<\/p>\n<p>}<\/p>\n<p>&nbsp;<\/p>\n<p>I think about the only new thing here is the trap statement.&nbsp; It\u2019ll get called if any exception is thrown, otherwise its just ignored.&nbsp; Hopefully nothing will go wrong with the function but if anything does I want to be sure to close any open streams.&nbsp; Anyway, keep this function around, we\u2019ll use it along with AddNotes and group-object to write a simple script that can search directories and tell us all the files that are duplicates.&nbsp; Now\u2026 an example of this function in use:<\/p>\n<p>&nbsp;<\/p>\n<p>MSH&gt;&#8221;foo&#8221; &gt; foo.txt<\/p>\n<p>MSH&gt;&#8221;bar&#8221; &gt; bar.txt<\/p>\n<p>MSH&gt;&#8221;foo&#8221; &gt; AlternateFoo.txt<\/p>\n<p>MSH&gt;dir *.txt | foreach { get-md5 $_ }<\/p>\n<p>33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53<\/p>\n<p>54 122 136 147 125 209 249 229 12 105 236 19 140 5 107 169<\/p>\n<p>33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53<\/p>\n<p>MSH&gt;<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p>Note that two of the files have the same hash, as expected since they have the same content.&nbsp; Of course, it is possible for two files to have the same hash and not the same content so if you are really paranoid you might want to check something else in addition to the MD5 hash.<\/p>\n<p>&nbsp;<\/p>\n<p>&#8211; Marcel<\/p>\n<p>[<i>Edit: Monad has now been renamed to Windows PowerShell.  This script or discussion may require slight adjustments before it applies directly to newer builds.<\/i>]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Need a way to check if two files are the same?&nbsp; Calculate a hash of the files.&nbsp; Here is one way to do it: &nbsp; ## Calculates the hash of a file and returns it as a string. function Get-MD5([System.IO.FileInfo] $file = $(throw &#8216;Usage: Get-MD5 [System.IO.FileInfo]&#8217;)) { &nbsp; $stream = $null; &nbsp; $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider]; [&hellip;]<\/p>\n","protected":false},"author":600,"featured_media":13641,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-10941","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-powershell"],"acf":[],"blog_post_summary":"<p>Need a way to check if two files are the same?&nbsp; Calculate a hash of the files.&nbsp; Here is one way to do it: &nbsp; ## Calculates the hash of a file and returns it as a string. function Get-MD5([System.IO.FileInfo] $file = $(throw &#8216;Usage: Get-MD5 [System.IO.FileInfo]&#8217;)) { &nbsp; $stream = $null; &nbsp; $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider]; [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/10941","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/users\/600"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/comments?post=10941"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/10941\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media\/13641"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media?parent=10941"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/categories?post=10941"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/tags?post=10941"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}