Summary: Learn how to use Windows PowerShell to search web pages. Scripting Guy, Ed Wilson here. Today’s post is written by Doug Finke, a Windows PowerShell MVP, and June Blender, senior programming writer on the Windows Azure Active Directory team. Take it away, June… I was bending my brain around the cool new Windows Azure Graph REST APIs and Windows Azure Storage Services REST APIs, when I noticed that Doug Finke, author of Windows PowerShell for Developers was posting about calling Representational State Transfer (REST) APIs in Windows PowerShell. To give you a bit of background, a REST API is an interface based on the HTTP protocol. You use one of four operations—Get, Post, Put, or Delete—with a string parameter in HTTP format. The strings look like URIs, but they don’t refer to a web page. Instead, they provide the data for one of the four operations. I didn’t know how to call REST APIs in Windows PowerShell, so I asked Doug for help. Before we started, I asked Doug how to determine which websites have web services with REST interfaces. It turns out that this isn’t a very straightforward task. You have to search online for the name of your favorite website or service and “REST API.” Doug found YouTube and StackOverflow, and he pointed me to the ProgrammableWeb API search. Doug suggested that we start with the Invoke-RestMethod cmdlet, which was introduced in Windows PowerShell 3.0, and YouTube, which has a well-documented REST API. I started here: YouTube API v2.0 – API Query Parameters. This page lists the parameters that you can include in a Get or query operation. The syntax of the query string is:
https://gdata.youtube.com/feeds/api/videos? …followed by a parameter and parameter value in key=value format. For example, the instructions recommend that you include the v (version) parameter with a value of 2 to use the latest version of the REST API:
https://gdata.youtube.com/feeds/api/videos?v=2 To specify the query string (the text you are searching for), use a parameter of q (lower-case) and the search text. This query string searches for PowerShell:
https://gdata.youtube.com/feeds/api/videos?q=PowerShell To search for multiple words, concatenate the words with a plus sign (+).
https://gdata.youtube.com/feeds/api/videos?q=Windows+PowerShell To use multiple parameters, separate them with an ampersand (&). The following query string specifies v 2 and searches for PowerShell:
https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell Now, to run the query in Windows PowerShell, use the Invoke-RestMethod cmdlet. The following command searches YouTube videos with PowerShell in the title.
PS C:> Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell”
etag : W/”D0UNQH47eCp7I2A9Wh5TFU4.”
id : tag:youtube.com,2008:video:-Ya1dQ1Igkc
published : 2012-03-27T19:43:26.000Z
updated : 2013-09-30T10:01:31.000Z
category : {category, category}
title : My Final 1-Day PowerShell v2 Workshop
content : content
link : {link, link, link, link…}
author : author
accessControl : {yt:accessControl, yt:accessControl, yt:accessControl, yt:accessControl…}
comments : comments
group : group
rating : {gd:rating, yt:rating}
statistics : statistics
etag : W/”DU8DRn47eCp7I2A9WhFaGE0.”
id : tag:youtube.com,2008:video:XpHgSHQZNpU
published : 2011-09-21T15:38:13.000Z
updated : 2013-09-21T23:57:57.000Z
category : {category, category}
title : Hacking Windows Accounts with Powershell
content : content
link : {link, link, link, link…}
author : author
accessControl : {yt:accessControl, yt:accessControl, yt:accessControl, yt:accessControl…}
comments : comments
hd :
group : group
rating : {gd:rating, yt:rating}
statistics : statistics
. . . Wow, it works! I’ll select the title, author, and the URL of the video:
PS C:ps-test> Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell” | Select-Object Title, Author, Link
title author link
—– —— —-
My Final 1-Day PowerShell v2 Workshop author {link, link, link, link…}
Hacking Windows Accounts with Powershell author {link, link, link, link…}
What is PowerShell and how can it make IT … author {link, link, link, link…}
Basics of Powershell P1 author {link, link, link, link…}
… Hmm. That doesn’t quite work. The author property is in author.name and the link is in content.src. You can use a calculated property or a PSCustomObject to fix this. Let’s use a PSCustomObject.
PS C:> Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell” | foreach {[PSCustomObject]@{Title=$_.Title; Author=$_.Author.name; Link=$_.content.src}} | Format-List
Title : My Final 1-Day PowerShell v2 Workshop
Author : Don Jones
Link : https://www.youtube.com/v/-Ya1dQ1Igkc?version=3&f=videos&app=youtube_gdata
Title : Hacking Windows Accounts with Powershell
Author : David Hoelzer
Link : https://www.youtube.com/v/XpHgSHQZNpU?version=3&f=videos&app=youtube_gdata
… That’s much better! There’s lots of other useful information, including the publication date and last updated date, comments, statistics, and ratings. But before I turn you over to Doug to hear why this works, let me share one more tidbit. One of the parameters in the YouTube REST API is Max-Results. Its default value is 25, and its maximum value is 50. It doesn’t seem to work in Windows PowerShell 3.0. By default, it returns the first 13 hits. When we specified values of 5, 10, 30, and 50, we got only half of what we requested: 3, 5, 15, and 25.
PS C:> (Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell”).count
13
PS C:> 5,10,30,50 | % {
$url=https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell&max-results=$_;
(Invoke-RestMethod $url).count
}
3
5
15
25 But when we tested it in Windows PowerShell 4.0, it worked perfectly.
PS C:> (Invoke-RestMethod -Uri “https://gdata.youtube.com/feeds/api/videos?v=2&q=PowerShell”).count
25
PS C:> 5,10,30,50 | % {
$url=https://gdata.youtube.com/feeds/api/videos?v=2&q=Powershell&max-results=$_;
(Invoke-RestMethod $url).count
}
5
10
30
50 We checked with Windows PowerShell developer, Lee Holmes; and indeed, in Windows PowerShell 4.0, the team fixed a bug that returned only half of the results of Invoke-RestMethod calls. So, let me turn you over to Doug so you can learn about why all of this works… Thanks, June. That was a great write-up about how Windows PowerShell integrates with REST APIs. REST is the predominant Web API. It’s a way to serve standard programmatic access to data over HTTP. So, as June shows, you can get to a wealth of YouTube data by using a Window PowerShell one-liner. The cool part is that all REST APIs work this way. So, when you learn the fundamentals, you can identify REST sources and apply these techniques again and again.
Invoke-RestMethod
Invoke-RestMethod is great because the Windows PowerShell team has written and tested a ton of script, and I can get super productive with two lines:
PS C:>$url = “https://gdata.youtube.com/feeds/api/videos?q=PowerShell”
PS C:> (Invoke-RestMethod $url).title.’#text’
My Final 1-Day PowerShell v2 Workshop
Windows PowerShell 2.0 for Beginners Training & Overview – EPC Group
Hacking Windows Accounts with Powershell
Introduction to Windows PowerShell Part 1
What is PowerShell and how can it make IT more efficient? Jason Helmick – Interface
Windows PowerShell Remoting: Definitely NOT Just for Servers
Basics of Powershell P1
O’Reilly Webcast: Getting Started with Windows PowerShell 3.0
Advanced Automation Using Windows PowerShell 3.0
TechEd 2013: Windows PowerShell Unplugged – Jeffrey Snover
My PowerShell Scripts – Systems Administration
[Powershell] Einführung Teil 1
MCITP 70-640: PowerShell
German PowerShell Basis Video Tutorial Teil 1 von 21 Was ist PowerShell, Warum PowerShell
Bootable USB Drive Using Powershell and Windows 8
Инструменты Windows PowerShell 2.0. Часть 1
Bruce Payette – PowerShell Workflows
Powershell. Curso Hispano.
Exploring PowerShell for Beginners Dec 9th
PowerShell: Introduction and Scripting Tutorial
No Tools No Problem Building a PowerShell Botnet Christopher @obscuresec Campbell
Creating HTML Reports with PowerShell
PowerShell Scripting Basics
PowerShell for common Office 365 Operations
Windows Powershell – What are The Top 10 Cmdlets to Start Using Immediately Take note of how the title information is retrieved:
(Invoke-RestMethod $url).title.’#text’ The title data is stored in the #text property, but if you don’t enclose “#text” in quotation marks, Windows PowerShell interprets it as a comment because it begins with a # sign, and you will receive an error message. Before Windows PowerShell delivered Invoke-RestMethod, we could reach into the .NET Framework and use the System.Net.WebClient class to grab this data ourselves:
PS C:>$wc = New-Object System.Net.WebClient
PS C:>$url = “https://gdata.youtube.com/feeds/api/videos?q=PowerShell”
PS C:>$youtubedata = [xml]$wc.DownloadString($url)
PS C:>$youtubedata.feed.entry.title.’#text’ Invoke-RestMethod does a lot for us. It connects and downloads the data from the target URL, converts the returned string into XML, and extracts the feed.entry information so we can use less .NET notation to get at what we want faster. A feed is a data format for providing frequently updated content. Invoke-RestMethod leverages this standard to make our lives easier. But wait! There’s more…
Earthquake data and JSON
Invoke-RestMethod also automatically detects the type of data the URL returns. While REST is the predominate way to interact with data on the web, JavaScript Object Notation (JSON), a text-based standard for data interchange, is often used too. Great news! The designers of Invoke-RestMethod also made it easy to interact with JSON REST feeds. Here’s an Invoke-RestMethod command that gets earthquake data from Seismi.org. It returns JSON, but you don’t need to know that. Windows PowerShell returns custom objects (PSCustomObject) that you can use in your commands.
PS C:>(Invoke-RestMethod -URI “http://www.seismi.org/api/eqs”).earthquakes | Select-Object -First 10 | Format-Table –AutoSize
src eqid timedate lat lon magnitude depth region
— —- ——– — — ——— —– ——
us c000is61 2013-07-29 22:22:48 7.6413 93.6871 4.6 40.90 Nicobar Islands, India region
us c000is4s 2013-07-29 21:52:12 -57.7816 -25.3260 5.2 53.50 South Sandwich Islands region
us c000is3k 2013-07-29 21:33:34 36.6696 71.0615 4.7 234.10 Hindu Kush region, Afghanistan
us c000irvf 2013-07-29 18:27:41 -37.2993 177.2515 4.9 160.50 off the east coast of the North Island of New Zealand
us c000irpf 2013-07-29 14:53:32 24.5038 62.5255 4.5 24.30 off the coast of Pakistan
us c000irl3 2013-07-29 10:23:35 45.9104 143.0497 4.4 337.70 Hokkaido, Japan region
us c000irjj 2013-07-29 08:21:31 -6.8134 130.2114 4.6 87.80 Banda Sea
us c000irjf 2013-07-29 08:12:27 -16.9852 -177.0945 5.1 10.00 Fiji region
us c000irj9 2013-07-29 08:08:00 36.5108 70.5896 5.0 193.10 Hindu Kush region, Afghanistan
us c000irhe 2013-07-29 05:49:35 -4.6368 -104.9759 4.7 10.00 central East Pacific Rise You can use the WebClient class to build a command like this on your own. Prior to Windows PowerShell 3.0, you’d need to use a third-party .NET JSON parser or build your own. Windows PowerShell 3.0 introduced the ConvertFrom-Json cmdlet:
PS C:>$wc = New-Object System.Net.WebClient
PS C:>$data = $wc.DownloadString(“http://www.seismi.org/api/eqs”) | ConvertFrom-Json
PS C:>$data.earthquakes | select -First 10 | Format-Table -AutoSize The results are much noisier than the out-of-the-box experience of Invoke-RestMethod. Plus, Invoke-RestMethod automatically detects whether XML or JSON is being returned, and it does the right thing—it returns Windows PowerShell objects for you to easily work on.
Invoke-RestMethod under the covers
Invoke-RestMethod is one of the Windows PowerShell 3.0 core cmdlets. It is written in C# and delivered in the Microsoft.PowerShell.Commands.Utility.dll. To examine it, I used a free tool from JetBrains, called DotPeek. This tool can look into the compiled DLL and decompile it to source code so you can browse it. After reading the Invoke-RestMethod code, I asked the Windows PowerShell team to confirm my assumptions. Lee Holmes, author of the great book, Windows PowerShell Cookbook, 3rd Edition, replied, “If it looks like an RSS or ATOM feed, we process and return the elements in the feed. If the response says it returns some form of JSON (there are many), we will first try to convert it to JSON. If that fails, we will convert it to XML. If that fails, we return it as text.” Here is a simplified Windows PowerShell version I came up with that I sent along with the question:
$url = “http://www.seismi.org/api/eqs”
$request = [System.Net.WebRequest]::Create($url)
$request.Method=”Get”
$response = $request.GetResponse()
$requestStream = $response.GetResponseStream()
$readStream = New-Object System.IO.StreamReader $requestStream
$data=$readStream.ReadToEnd()
if($response.ContentType -match “application/xml”) {
$results = [xml]$data
} elseif($response.ContentType -match “application/json”) {
$results = $data | ConvertFrom-Json
} else {
try {
$results = [xml]$data
} catch {
$results = $data | ConvertFrom-Json
}
}
$results These 23 lines of code give the same results for the earthquake data as the one-line that used Invoke-RestMethod. Thank you, Windows PowerShell team! That’s a very simple overview of how YouTube and Seismi.org serve this information. The better news is that this is a web standard and many sources of data are available to you this way. Even better, Windows PowerShell makes working with this information a breeze. ~Doug Thanks, June and Doug. I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace. Ed Wilson, Microsoft Scripting Guy
0 comments