{"id":843,"date":"2010-06-07T13:04:00","date_gmt":"2010-06-07T13:04:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vbteam\/2010\/06\/07\/vb-xml-cookbook-recipe-7-enumerating-large-xml-files-doug-rothaus\/"},"modified":"2024-07-05T12:41:10","modified_gmt":"2024-07-05T19:41:10","slug":"vb-xml-cookbook-recipe-7-enumerating-large-xml-files-doug-rothaus","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/vbteam\/vb-xml-cookbook-recipe-7-enumerating-large-xml-files-doug-rothaus\/","title":{"rendered":"VB XML Cookbook, Recipe 7: Enumerating Large XML Files (Doug Rothaus)"},"content":{"rendered":"

VB XML Cookbook, Recipe 7: Enumerating Large XML Files (Doug Rothaus)<\/span><\/p>\n

It’s been a while since I wrote one of these XML cookbook entries. Here’s some info on a common problem: Really big XML files.<\/span><\/p>\n

I’m going to show you two things in this recipe. The first is a tip on reading very large XML files while still being able to use XML Axis Properties. The second is how to do make it available to LINQ queries by exposing it as IEnumerable<\/b>.<\/span><\/p>\n

Reading a Large XML File<\/span><\/h2>\n

If you’re new to working with XML, there’s something important that you need to know. That is, when you load an XML file into an in-memory document, the entire file gets loaded into memory. When working with small XML files, which is often the case, this is no big deal. In fact, it’s rather convenient. However, if you are working with an extremely large XML file, this is a big problem. I recently wrote some code to read through a bunch of XML files, not realizing that one of them was over half a gigabyte! My code loaded the entire file into memory using the XDocument.Load()<\/b> method—well, it tried to at least. Needless to say, when I hit the huge file, my app did not perform well.<\/span><\/p>\n

How do you read an enormous XML file then? You use the <\/span>XmlReader<\/span><\/a> class, which has been around since the first release of the .NET Framework. It reads through an XML file, but simply places a pointer on the current XML element or attribute as you go through the file. As you read through the file with the XmlReader<\/b> object, you can examine the current XML, decide if you are interested in it, process it, discard it, and move on to the next part of the file. The important thing is that you can minimize how much memory is utilized at any one time in your app. <\/span><\/p>\n

Take heed: you still need to be aware of how you are reading through the file. If you open an XmlReader<\/b>, read to the root element, then load that entire element into memory you haven’t solved anything.<\/span><\/p>\n

Now you may be saying that if you use an XmlReader<\/b> object, you don’t get all of the cool functionality of XML Axis Properties. That’s true, and that’s why there’s a ReadFrom<\/b> method that reads the XML from your XmlReader<\/b> into an XNode<\/b>, which you can then cast as an XElement<\/b> object and make use of all of the VB XML juicy goodness. Using an XmlReader<\/b> and the ReadFrom<\/b> method together ensures that you only use as much memory as the largest XML element that you load.<\/span><\/p>\n

 <\/span>Let’s look at an example. The app that I was working on was reading through XML files that contained reflection information from .NET assemblies. For each member of a particular class, there was an <api> element. Within that <api> element there was a bunch of information about that member, and my app needed to grab some of the info for use in summary counts. Here’s an abbreviated XML sample of what the data looked like.<\/span><\/span><\/p>\n

<reflection><\/p>\n

<\/span><\/p>\n

  <\/span><apis><\/p>\n

<\/span><\/p>\n

    <\/span><api id=”M:Microsoft.VisualBasic.Strings.Mid(System.String,System.Int32)”><\/p>\n

<\/span><\/p>\n

   <\/span>  <\/span> <\/span><apidata name=”Mid” group=”member” subgroup=”method” \/><\/p>\n

<\/span><\/p>\n

   <\/span>  <\/span> <\/span><containers><\/p>\n

<\/span><\/p>\n

     <\/span>  <\/span> <\/span><namespace api=”N:Microsoft.VisualBasic” \/><\/p>\n

<\/span><\/p>\n

    <\/span>  <\/span>  <\/span><type api=”T:Microsoft.VisualBasic.Strings” ref=”true” \/><\/p>\n

<\/span><\/p>\n

   <\/span>  <\/span> <\/span><\/containers><\/p>\n

<\/span><\/p>\n

 <\/span>  <\/span> <\/span><\/api>  <\/span><\/p>\n

<\/span><\/p>\n

    <\/span><api id=”M:Microsoft.VisualBasic.Strings.Mid(System.String,System.Int32,System.Int32)”><\/p>\n

<\/span><\/p>\n

  <\/span>    <\/span><apidata name=”Mid” group=”member” subgroup=”method” \/><\/p>\n

<\/span><\/p>\n

 <\/span>  <\/span>   <\/span><containers><\/p>\n

<\/span><\/p>\n

      <\/span>  <\/span><namespace api=”N:Microsoft.VisualBasic” \/><\/p>\n

<\/span><\/p>\n

      <\/span>  <\/span><type api=”T:Microsoft.VisualBasic.Strings” ref=”true” \/><\/p>\n

<\/span><\/p>\n

    <\/span>  <\/span><\/containers><\/p>\n

<\/span><\/p>\n

  <\/span>  <\/span><\/api><\/p>\n

<\/span><\/p>\n

  <\/span><\/apis><\/p>\n

<\/span><\/p>\n

<\/reflection><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Here’s  <\/span>some code to read through each <api> element, one at a time, with an XmlReader<\/b> object. Once I have loaded the <api> element into an XElement<\/b> object, I can use XML Axis properties to get values from the XML contained in the element. The most memory that I use is determined by the largest <api> element rather than the entire file.<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

<\/p>\n

        <\/span>Dim<\/span> reader = Xml.XmlReader<\/span>.Create(“….reflectionData.xml”<\/span>)<\/p>\n

<\/span><\/p>\n

        <\/span>reader.MoveToContent()<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

        <\/span>While<\/span> reader.ReadToFollowing(“api”<\/span>)<\/p>\n

<\/span><\/p>\n

            <\/span>Dim<\/span> api = TryCast<\/span>(XElement<\/span>.ReadFrom(reader), XElement<\/span>)<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

            <\/span>If<\/span> api Is<\/span> Nothing<\/span> Then<\/span> Continue While<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

            <\/span>‘ Get information from the <api> element.<\/span><\/p>\n

<\/span><\/p>\n

            <\/span>Dim<\/span> ns = api.<<\/span>containers>.<<\/span>namespace>.@<\/span>api<\/p>\n

<\/span><\/p>\n

            <\/span>Dim<\/span> containingType = api.<<\/span>containers>.<<\/span>type>.@<\/span>api<\/p>\n

<\/span><\/p>\n

        <\/span>End<\/span> While<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

        <\/span>reader.Close()<\/p>\n

<\/span><\/p>\n

 <\/p>\n

<\/span><\/span><\/p>\n

 <\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

Now this code simply reads to the first <api> element and then reads all of its sibling <api> elements. If one of the <api> elements has a child <api> element, that child element gets loaded in the call to the ReadFrom<\/b> method and the XmlReader<\/b> object’s pointer moves past it. This works fine for my app because none of the <api> elements have child <api> elements. You may have different requirements and need to adjust your code. <\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

I ran this code on a 5MB file with a little less than 30,000 <api> elements. Loading the entire file into memory consumed over 120MB. Using the XmlReader<\/b>, the code consumed less than 1MB. I gathered memory stats using the <\/span>GetTotalMemory<\/span><\/a> method.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

One last thing to note in this section is that you can also run into memory issues when writing to a file. If you create a large XDocument<\/b> in memory, and then write it to a file, you end up consuming the memory required to create the document, which is likely unnecessary. You have a couple of choices to minimize your memory footprint while writing an XML file. Similar to using the XmlReader<\/b> class and the ReadFrom<\/b> method, you can use the <\/span>XmlWriter<\/span><\/a> class and the WriteTo<\/b> method. As another option, you can use the <\/span>XStreamingElement<\/span><\/a> class to write a single element at a time from an enumerable source, such as a LINQ query. <\/span><\/p>\n

<\/span><\/p>\n

 <\/span><\/p>\n<\/p>\n

What about LINQ Queries?<\/span><\/h2>\n

In addition to having a small memory footprint, I also wanted to be able to use LINQ to query a large XML file. This can be achieved by creating a class that implements the IEnumerable<\/b> interface. By fitting the code that I would have used to loop through the XML file into a class that implements IEnumerable(Of XElement)<\/b>, I can use an instance of that class as the source of any number of LINQ queries.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

What I’ve created for this step is almost exactly the same as the class created by this walkthrough: <\/span>Walkthrough: Implementing IEnumerable(Of T) in Visual Basic<\/span><\/a> . The walkthrough shows you how to implement IEnumerable(Of String)<\/b> to expose the contents of a text file one line at a time. We’ll do the same with an XML file.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

When you implement IEnumerable<\/b>, you actually need to implement both IEnumerable<\/b> and IEnumerator<\/b>. The bulk of your code goes into the IEnumerator<\/b> implementation. You could create one class that implements both, but I like to split them into two classes.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

I’ve called the class that implements IEnumerable(Of  <\/span>XElement)<\/b> <\/span>XmlReaderEnumerable<\/span>. Following that naming convention I’ve called the class that implements IEnumerator(Of XElement) <\/b><\/span><\/span>XmlReaderEnumerator<\/span>. The behavior is the same as the earlier XmlReader<\/b> example. The <\/span>XmlReaderEnumerator<\/span> class finds the first instance of a particular element, and then finds all of its sibling elements of the same name. As a result, I’ve added a constructor that takes both the path to the XML file, and the name of the XML element to search for. Note that the name is case sensitive as XML is case sensitive. <\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

The <\/span>XmlReaderEnumerable<\/span> class doesn’t do much. All it does is return a reference to an instance of the <\/span>XmlReaderEnumerator<\/span> class. Here’s the code.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

Public<\/span> Class<\/span> XmlReaderEnumerable<\/span><\/p>\n

<\/span><\/p>\n

    <\/span>Implements<\/span> IEnumerable<\/span>(Of<\/span> XElement<\/span>)<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _filePath As<\/span> String<\/span><\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _elementName As<\/span> String<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Public<\/span> Sub<\/span> New<\/span>(ByVal<\/span> filePath As<\/span> String<\/span>, ByVal<\/span> elementName As<\/span> String<\/span>)<\/p>\n

<\/span><\/p>\n

        <\/span>_filePath = filePath<\/p>\n

<\/span><\/p>\n

        <\/span>_elementName = elementName<\/p>\n

<\/span><\/p>\n

    <\/span>End<\/span> Sub<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Public<\/span> Function<\/span> GetEnumerator() As<\/span> IEnumerator<\/span>(Of<\/span> XElement<\/span>) _<\/p>\n

<\/span><\/p>\n

        <\/span>Implements<\/span> IEnumerable<\/span>(Of<\/span> XElement<\/span>).GetEnumerator<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

        <\/span>Return<\/span> New<\/span> XmlReaderEnumerator<\/span>(_filePath, _elementName)<\/p>\n

<\/span><\/p>\n

    <\/span>End<\/span> Function<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> Function<\/span> GetEnumerator1() As<\/span> IEnumerator<\/span> _<\/p>\n

<\/span><\/p>\n

        <\/span>Implements<\/span> IEnumerable.GetEnumerator<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

        <\/span>Return<\/span> Me<\/span>.GetEnumerator()<\/p>\n

<\/span><\/p>\n

    <\/span>End<\/span> Function<\/span><\/p>\n

<\/span><\/p>\n

End<\/span> Class<\/p>\n

<\/span><\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

The <\/span>XmlReaderEnumerator<\/span> class is where the code resides to read through the XML file. In the constructor, it opens the file and moves to the start of the XML content. In the MoveNext<\/b> method, it reads to the element of the supplied name (for example, “api”). In the Dispose<\/b> method, it closes the reader. That’s it. It looks like a lot of code, but it really isn’t.<\/span><\/p>\n

\n

 <\/span><\/p>\n<\/p>\n

Public<\/span> Class<\/span> XmlReaderEnumerator<\/span><\/p>\n

<\/span><\/p>\n

    <\/span>Implements<\/span> IEnumerator<\/span>(Of<\/span> XElement<\/span>)<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _xmlReader As<\/span> Xml.XmlReader<\/span><\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _elementName As<\/span> String<\/span><\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _filePath As<\/span> String<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Public<\/span> Sub<\/span> New<\/span>(ByVal<\/span> filePath As<\/span> String<\/span>, ByVal<\/span> elementName As<\/span> String<\/span>)<\/p>\n

<\/span><\/p>\n

        <\/span>_filePath = filePath<\/p>\n

<\/span><\/p>\n

        <\/span>_elementName = elementName<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

        <\/span>_xmlReader = Xml.XmlReader<\/span>.Create(_filePath)<\/p>\n

<\/span><\/p>\n

        <\/span>_xmlReader.MoveToContent()<\/p>\n

<\/span><\/p>\n

    <\/span>End<\/span> Sub<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Private<\/span> _current As<\/span> XElement<\/span><\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

    <\/span>Public<\/span> ReadOnly<\/span> Property<\/span> Current() As<\/span> XElement<\/span><\/span><\/p>\n<\/h2>\n","protected":false},"excerpt":{"rendered":"

VB XML Cookbook, Recipe 7: Enumerating Large XML Files (Doug Rothaus) It’s been a while since I wrote one of these XML cookbook entries. Here’s some info on a common problem: Really big XML files. I’m going to show you two things in this recipe. The first is a tip on reading very large XML […]<\/p>\n","protected":false},"author":260,"featured_media":8818,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[195],"tags":[59,163,166,167,185],"class_list":["post-843","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-visual-basic","tag-doug-rothaus","tag-vb-xml-cookbook","tag-vb2008","tag-vb2010","tag-xml"],"acf":[],"blog_post_summary":"

VB XML Cookbook, Recipe 7: Enumerating Large XML Files (Doug Rothaus) It’s been a while since I wrote one of these XML cookbook entries. Here’s some info on a common problem: Really big XML files. I’m going to show you two things in this recipe. The first is a tip on reading very large XML […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts\/843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/users\/260"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/comments?post=843"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts\/843\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/media\/8818"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/media?parent=843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/categories?post=843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/tags?post=843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}