{"id":3113,"date":"2009-02-20T22:23:00","date_gmt":"2009-02-20T22:23:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vbteam\/2009\/02\/20\/web-scraping-using-visual-basics-xml-support\/"},"modified":"2024-07-05T13:33:42","modified_gmt":"2024-07-05T20:33:42","slug":"web-scraping-using-visual-basics-xml-support","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/vbteam\/web-scraping-using-visual-basics-xml-support\/","title":{"rendered":"Web-scraping using Visual Basic's XML support"},"content":{"rendered":"

We’ve just had a fascinating article<\/a> about using VB’s XML literals to produce web pages.<\/p>\n

I’ve been interested in the other side of the process: using VB’s XML support to scrape<\/em> web-pages for data. Here’s the essential bit of code, which looks for something like <div class=”content”><h2>Title<\/h2> <table><tr><td><p>Data1<\/p><\/td><\/tr><\/table><\/div>, and extracts the title and the data.<\/p>\n

Dim<\/span> fn = Fetch(url)<\/span><\/p>\n

<\/span><\/p>\n

Dim<\/span> xml = XElement.Load(fn)<\/span><\/p>\n

<\/span><\/p>\n

Dim<\/span> yui = (From<\/span> i In<\/span> xml…<<\/span>xhtml:div><\/span><\/span> Where<\/span> i.GetAttr(“class”<\/span>) = “content”<\/span>).FirstOrDefault<\/p>\n

<\/span><\/p>\n

Dim<\/span> title = yui.<<\/span>xhtml:h2><\/span><\/span>.Value.ToString.Replace(“,”<\/span>, ” “<\/span>)<\/p>\n

<\/span><\/p>\n

Dim<\/span> data = From<\/span> i In<\/span> yui…<<\/span>xhtml:td><\/span><\/span><\/p>\n

<\/span><\/span><\/p>\n

For<\/span> Each<\/span> td In<\/span> data<\/p>\n

<\/span><\/p>\n

    <\/span>Dim<\/span> text = td.<<\/span>xhtml:p><\/span><\/span>.FirstOrDefault<\/p>\n

<\/span><\/p>\n

    <\/span>…<\/p>\n

<\/span><\/p>\n

Next<\/p>\n

<\/span><\/p>\n

The only tricky bit was that XElement.Load() requires well-formed XML\/XHTML, while most web-pages are in poorly-formed HTML. So my function “Fetch” retreives the page and then turns sloppy HTML into clean XHTML. It does this through the awesome open source utility “tidy.exe<\/a>“.<\/p>\n

For more information, and complete source code, see here: http:\/\/blogs.msdn.com\/lucian\/archive\/2009\/02\/21\/web-scraping-with-vb-s-xml-support.aspx<\/a><\/p><\/p>\n","protected":false},"excerpt":{"rendered":"

We’ve just had a fascinating article about using VB’s XML literals to produce web pages. I’ve been interested in the other side of the process: using VB’s XML support to scrape web-pages for data. Here’s the essential bit of code, which looks for something like <div class=”content”><h2>Title<\/h2> <table><tr><td><p>Data1<\/p><\/td><\/tr><\/table><\/div>, and extracts the title and the data. Dim fn […]<\/p>\n","protected":false},"author":260,"featured_media":8818,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[192,195],"tags":[99],"class_list":["post-3113","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-featured","category-visual-basic","tag-lucian-wischik"],"acf":[],"blog_post_summary":"

We’ve just had a fascinating article about using VB’s XML literals to produce web pages. I’ve been interested in the other side of the process: using VB’s XML support to scrape web-pages for data. Here’s the essential bit of code, which looks for something like <div class=”content”><h2>Title<\/h2> <table><tr><td><p>Data1<\/p><\/td><\/tr><\/table><\/div>, and extracts the title and the data. Dim fn […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts\/3113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/users\/260"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/comments?post=3113"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/posts\/3113\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/media\/8818"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/media?parent=3113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/categories?post=3113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/vbteam\/wp-json\/wp\/v2\/tags?post=3113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}