{"id":3391,"date":"2009-11-04T12:52:55","date_gmt":"2009-11-04T12:52:55","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/powershell\/2009\/11\/04\/why-is-get-childitem-so-slow\/"},"modified":"2019-02-18T13:06:07","modified_gmt":"2019-02-18T20:06:07","slug":"why-is-get-childitem-so-slow","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/powershell\/why-is-get-childitem-so-slow\/","title":{"rendered":"Why is Get-ChildItem so Slow?"},"content":{"rendered":"

We get this question fairly frequently when it comes to slow network connections.<\/p>\n

The performance of directory listings (especially on a laggy network) are limited by the .NET APIs we call to retrieve the directory information. There are two limitations to the current set of APIs:<\/p>\n

Forced Retrieval of Attributes<\/strong><\/p>\n

When we do a directory listing, we show the standard attributes of the file or directory: Mode, LastWriteTime, Length, and Name. The core Windows API is highly optimized for this basic scenario, and returns these attributes by default along with the rest of the file information. However, the .NET Framework doesn\u2019t take advantage of this data, and instead goes back to the network location to ask for all of the file attributes. This chatty behaviour adds a handful of network round trips for each file or directory, making the directory listing many times slower: hundreds or thousands of times slower in many cases. The Framework team addressed this as part of .NET 4.0<\/a>, and you\u2019ll see the benefits of this new feature as soon as we are able to adopt it.<\/p>\n

In version two, even without the benefit of the new .NET API, though, you\u2019ll see a huge improvement in wildcarded directory listings (both local and remote.)<\/p>\n

As a background, PowerShell wildcards are different than straight cmd.exe wildcards. For example, PowerShell wildcards do not match the 8.3 short file name, while the native filesystem filtering (exposed by cmd.exe wildcards) do. PowerShell\u2019s wildcards support character ranges, while the native file system filtering support does not. Because of this, PowerShell wildcard processing happens AFTER we\u2019ve retrieved all of the files. <\/p>\n

This comes at a cost, however. Native file system filtering (as exposed by the \u2013Filter<\/font> parameter) is MUCH faster, as its processing is wired into the Windows file system.<\/p>\n

In version two, we did a bunch of work to resolve this strain. When you provide a PowerShell wildcard, we convert as much of it as possible to a native filesystem filter, and then apply our wildcarding logic to the much smaller set of results. You\u2019ve probably noticed this most in tab completion, but it makes huge improvements in regular wildcarded directory listings. Especially remote ones. Since the native filtering is processed by the remote<\/em> file system, we don\u2019t need to suffer the performance penalty of accessing attributes of files that you ultimately don\u2019t care about anyways. In version one, you can work around the issue by specifying the \u2013Filter<\/font> parameter directly. If this still doesn\u2019t provide the speed you need, you can call \u201ccmd.exe \/c dir<\/font>\u201d.<\/p>\n

 <\/p>\n

Lack of Enumeration API<\/strong><\/p>\n

This issue raises itself for directory listings that contain many files. The DirectoryInfo.GetFiles() method returns an array. When creating that result list, the .NET Framework does many re-allocations (and copies) of that array, causing an exponential performance degradation:<\/p>\n

\"GetChildItemPerf\" <\/p>\n

 <\/p>\n

This, too, has been resolved in the .NET 4.0 updates, by offering an API that lets you enumerate through a directory result, rather than retrieve them all at once. If you are running into these limitations, you can again apply a wildcarding approach. If this still doesn\u2019t provide the speed you need, you can call \u201ccmd.exe \/c <command><\/font>\u201d.<\/p>\n

<\/strong><\/p>\n

Why Don\u2019t We Fix It?<\/strong><\/p>\n

Since cmd.exe isn\u2019t impacted by these issues, why don\u2019t we just do the same thing and call into the core Windows APIs directly? The reason is twofold:<\/p>\n

    \n
  1. A core tenet of PowerShell is providing access to the REAL underlying .NET objects. If we implemented the semantics ourselves, we\u2019d have to return new types of objects \u2013 something like PSFileInfo and PSDirectoryInfo. V1 scripts (or downstream cmdlets) that expect the REAL underlying .NET objects would fail to work. While we could add a new switch (-Raw<\/font>?), users would still have to change their scripts to support it. In that case, they might as well use the existing cmd \/c workaround.<\/li>\n
  2. This issue is ultimately transient. While it\u2019s annoying to drag out over a few years, it will ultimately come and go without users having to change their behaviour. One day, you\u2019ll install a build and the issues will just magically be gone.<\/li>\n<\/ol>\n

    Again, thanks for your continuing feedback. That\u2019s what ultimately helped us discover the issue and make sure the right people knew about it.<\/p>\n

     <\/p>\n

    Lee Holmes [MSFT]
    Windows PowerShell Development
    Microsoft Corporation<\/p>\n","protected":false},"excerpt":{"rendered":"

    We get this question fairly frequently when it comes to slow network connections. The performance of directory listings (especially on a laggy network) are limited by the .NET APIs we call to retrieve the directory information. There are two limitations to the current set of APIs: Forced Retrieval of Attributes When we do a directory […]<\/p>\n","protected":false},"author":600,"featured_media":13641,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3391","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-powershell"],"acf":[],"blog_post_summary":"

    We get this question fairly frequently when it comes to slow network connections. The performance of directory listings (especially on a laggy network) are limited by the .NET APIs we call to retrieve the directory information. There are two limitations to the current set of APIs: Forced Retrieval of Attributes When we do a directory […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/3391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/users\/600"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/comments?post=3391"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/3391\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media\/13641"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media?parent=3391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/categories?post=3391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/tags?post=3391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}