Weekend Scripter: Query the Bing Search Engine from a PowerShell Script
Summary: Microsoft MVP Sean Kearney shows how to use the Bing web services API from inside a Windows PowerShell script.
Microsoft Scripting Guy Ed Wilson here. Sean Kearney is back with us today as he continues to share with us his discoveries while playing. Read more about Sean in the introduction on this blog post. Let us see what Sean has found while playing again.
I played some more today. I decided no matter what, I would learn how to interface with Bing.com and Windows PowerShell. I had some excellent pointers from my good buddy and cohort at Tech·Ed Azure MVP Cory Fowler and found the Developers site on Bing.com to begin my quest.
With Bing, you can now pass your information directly via XML or Json. You can even access Bing with SOAP but I’ll be honest my SOAP expertise is limited to the bar kind and it is used for bathing.
XML, for those of use that use Windows PowerShell, is VERY easy to work with. The fact that Bing can return the results in XML format makes it that much better. A number of Hey, Scripting Guys! Blog posts talk about how to work with XML and Windows PowerShell.
When you sign in to www.bing.com/developers there is an excellent link called API Basics. Download it, and read it.
But here? We will cram all that information into a condensed version.
STEP 1 – GET an AppID
Your first task in working with the Bing API is to obtain what is known as an “AppID”. It is just a long series of numbers unique to your application that will do the searching. Visit www.bing.com/developers and follow the step-by-step process for obtaining an AppID. Without one, you cannot use the XML API.
You will be presented with a form to complete. See the following figure. This is just a reference information about your application (Yes, your simple Windows PowerShell script to query Bing, or your web browser is an application unto itself) and some details about you, the Developer.
(But wait I’m an ITPRO!) <— That is ok. We are not going to kick you out of the club for applying for an Application ID.
When you finish all this and accept the EULA (End-User License Agreement) you will have a number assigned to THAT application. You will see it after you have finished on this screen as shown in the following figure.
STEP 2 – Form that XML Query as a URL
Before we get to Windows PowerShell land we have to figure out what the XML query looks like. A very basic query that searches the web for the text SearchData would look like the following line of code. (Keep in mind it is a single logical line of code that is displayed on two lines due to blog formatting. No line continuation characters have been added).
http://api.bing.net/xml.aspx?AppId=XXXXXXXXXXXXXXXXXXXX&Version=2.2&Market=en-US&Query=SearchData&Sources=web
You would replace the XXXXXXXXXXXXXXXXXXXX with your numeric AppID that you obtained earlier when you requested your BING AppID. (In our case that number is 03340B083932F15ACCD89E2C1D0506D8F61E9CEB). The word SearchData is replaced with whatever you would normally search for on Bing. So let us try a simple example, and search for the word POWERSHELL. The command to search Bing using my AppID and the search word PowerShell is seen here (as before it is a single logical line).
http://api.bing.net/xml.aspx?AppId=03340B083932F15ACCD89E2C1D0506D8F61E9CEB&Version=2.2&Market=en-US&Query=POWERSHELL&Sources=web
If you were to key this into your browser that you would get a big funky XML document as response. This is seen in the following figure.
Congratulations, you have just formed an XML query. Now we have to make the XML query useful.
STEP 3 – Send it with Windows PowerShell
This is, of course, over simplifying the power of the Bing API, but today we will stick with simple tasks so that you can see how to use it with Windows PowerShell.
So, these lines will let you form a URL that you can send to the web substituting only the data we have to change for today. The $AppID variable accepts your AppID, and the $Query variable will accept the search term. The variables are added to the URL as seen here.
$AppId=”03340B083932F15ACCD89E2C1D0506D8F61E9CEB”
$Query=”Powershell”
$URL=”http://api.bing.net/xml.aspx?AppId=”+$AppId+”&Version=2.2&Market=en-US&Query=”+$Query+”&Sources=web”
Now to make this work we will connect with SYSTEM.NET.WEBCLIENT and pass the URL to the Internet. We will store the result in a boring little variable called, $Result.
We have to prepend $RESULT with [XML] to tell Windows PowerShell that the data is coming in as a text stream, but that the text will conform to XML formatting rules. So when Windows PowerShell knows this it does something beautiful. It transforms that XML gibberish stuff into something useful –R WS and COLUMNS! This portion of the code appears here.
$WEB=NEW-OBJECT SYSTEM.NET.WEBCLIENT
[XML]$RESULT=$WEB.DownloadString($URL)
STEP 4 – Parse and View Results
At this point, I am almost tempted to put on a Julia Child voice and say, “And now, we remove the pie from the oven and enjoy…” But I won’t.
Now that you have the $RESULT variable full of results that you may want to actually use it. You could run a Get-Member on it, but XML is actually very nice to work with. Just type the variable name $RESULT in your Windows PowerShell console, and it will immediately reveal the available parts of the tree.
So with $RESULT there is value within it called SearchResponse, within SearchResponse is “Web”. How did I tell that? I keyed into the Windows PowerShell Console directly. This is seen here.
$RESULT
The $Result variable reveals the following information.
|
xml |
pageview_candidate |
SearchResponse |
|
— |
—————— |
————– |
|
version=”1.0″ encoding=… |
SearchResponse |
When I query the SearchResponse property of the XML object that is stored in the $Result variable, it reveals the following information.
$RESULT.SearchResponse
|
xmlns |
Version |
Query |
Web |
|
—– |
——- |
—– |
— |
|
http://schemas.m… |
2.2 |
Query |
Web |
When I query the Web property, it reveals additional information that is seen here.
$RESULT.SearchResponse.Web
|
web |
Total |
Offset |
Results |
|
— |
—– |
—— |
——- |
|
http://schemas.m… |
73500 |
0 |
Results |
No wait, we are not done yet! When I look at the Results property, it also returns additional information. It is shown here.
$RESULT.SearchResponse.Web.Results
WebResult
———
{web:WebResult, web:WebResult, web:WebResult, web:WebResult…}
To see our results from the search we have to query the WebResult property as seen here.
$RESULT.SearchResponse.Web.Results.Webresult
After you run that you will probably want to do a Format-Table to see what columns are available. The command to do this is shown here.
$RESULT.SearchResponse.Web.Results.Webresult | FORMAT-TABLE
From this point forward, you can work with that data in whatever manner you want. Query it, export it to .csv, and pass those links to the web – whatever strikes you as useful or interesting – maybe even something as simple as returning a list of URL’s, together with the title and the description as shown here.
$RESULT.SearchResponse.Web.Results.Webresult | SELECT-OBJECT Url, Title, Description
However, here is where we make it simpler. A function called SEARCH-BING that will make it easier to search Bing from Windows PowerShell.
function global:SEARCH-BING( $Query )
{
# AppID you Acquired from www.bing.com/developers
$AppId=”03340B083932F15ACCD89E2C1D0506D8F61E9CEB”
# XML Query to pass to Bing.com
$URL=”http://api.bing.net/xml.aspx?AppId=”+$AppId+”&Version=2.2&Market=en-US&Query=”+$Query+”&Sources=web”
# Connect to the Web and download XML data from provided URL
$WEB=NEW-OBJECT SYSTEM.NET.WEBCLIENT
[XML]$RESULT=$WEB.DownloadString($URL)
# Return the goodies back home to Grandma
RETURN $RESULT
}
After you use it you can type the following command:
SEARCH-BING “The Answer to Life the Universe and EVERYTHING”
There you have it. A function that you can take with you to search Bing.com in your environment. Take note, you will have to replace THIS AppId with your own because I’ve already deleted it from Bing and don’t forget, we’ve only TOUCHED the surface of a Bing.com search. But hopefully I’ve unlocked the box and you can take it from there.
The Power of Shell (and BING) is in YOU.
Sean, thank you for sharing what you learned while playing around. I wonder what you would learn if you were really working at it?
I invite you to follow me on Twitter or Facebook. If you have any questions, send email to me at scripter@microsoft.com or post them on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.
Ed Wilson, Microsoft Scripting Guy
Light
Dark
0 comments