While I was playing with scripting the REST API for DekiWiki … I experimented with the MindTouch Dream SDK … and as a result, I’ve refactored the part of that module that actually does the HTTP work out into a little PowerShell 1.0 compatible script library. Of course, it depends on several dll’s (mindtouch.core.dll, mindtouch.dream.dll, and SgmlReaderDll.dll and log4net.dll) from the Dream SDK, which you can download from SourceForge (ok, they’ve made the Dream SDK stuff hard to find, so here’s a temporary link to my module with the DLLs in it).

Example Uses

There are basically two main functions: Invoke-Http and Receive-Http. There are also a few extra functions which are mostly just for use by these two functions, and then Set-HttpDefaultUrl and Set-HttpCredential which you can use to simplify your scripts. I intend to rewrite these as cmdlets in C# and contribute them to the PowerShell Community Extensions along with some of my other networking cmdlets (Get-DNS), so if you have any suggestions, I’d love to hear them!

Invoke-Http

Invokes HTTP verbs (like GET, POST, PUT, and DELETE) against a URL with optional parameters (eg: name = value), input, etc … and returns the output as a MindTouch.Dream.Result or (optionally) as a MindTouch.Dream.DreamMessage …

Syntax

Invoke-Http [-Verb] <String> [[-Path] <String>] [[-With] <Hashtable>] [[-Content] <Object>] [[-Type] <String>] [[-Authenticate] <String>] [-WaitForResponse]

Parameters

Verb

The HTTP verb you want to invoke. Defaults to GET, but can be anything your HTTP server supports.

Path

The URL you want to invoke against. Can be a partial URL, if you first call Set-HttpDefaultURL first. You can also specify this as an array or a hashtable. This is particularly useful if you’re specifying a partial URL, and your URL is built based on logic in your code. If you pass an array, the values in the array are joined with forward-slashes. If you pass a hashtable, the keys and values are appended joined with forward-slashes, but the values are double encoded (by UrlEncode), ie: $key/$(UrlEncode(UrlEncode($value))

With

The parameters to be passed with the verb call, as a hashtable. (these are particularly used for POST, etc). These are passed in the query as named parameters, with the values encoded.

Content

Anything you want to pass as content. In actuality, we really only handle three things: XML Documents, file paths (the file will be used as content), and everything else is passed as simple text. If you need other data types, the “DreamMessage” that the scripts depend on can handle all sorts of data types, so you’ll just need to add it around line 50 of the script.

Type

This is the type of content. We can automatically deduce this for XML or File objects, but if you want to force a type, you should use this parameter. For instance, to fake a web form submission, you need to specify the type as FORM or FORM_URLENCODED but you can also specify the type as any of: ATOM, BINARY, BMP, CSS, DREAM_EXCEPTION, FORM_URLENCODED, GIF, HTML, HTTP, JPEG, JS, JSON, MULTIPART_MIXED, NOTHING (mimetype -/-), ANY (mimetype */*), PDF, PHP, PNG, RDF, RELAXNG, SVG, ANY_TEXT (mimetype text/*), TEXT (mimetype text/plain, with the ISO-8859-1 encoding), TEXT_UTF8 (mimetype text/plain, with UTF8 encoding), TEXT_XML, TIFF, VCAL, VERSIT, XHTML, XML, XRDS.

Authenticate

You can pass either a boolean (in which case the HTTPRest scripts will use a default $global:HttpRestCredential which it will prompt for if you haven’t set it), or an actual PSCredential or NetworkCredential object. In any case, this causes the invoke to contain network credentials. Note that sometimes you may need to pre-authenticate before making calls to a web-service.

WaitForResponse

A switch parameter that causes the query to wait until the response completes downloading, and return a DreamResponse instead of returning an incomplete Result ... this is useful when you know the response is a simple XML file and you need to parse it to determine whether your call has succeeded. If you need to actually get the data out, or if you just need to know the method was invoked successfully, you shouldn’t pass this parameter.

Receive-Http

Receives the response or result from a Dream.Result or DreamMessage. as XML, XDoc, text, raw bytes, or even to file. This function is really only useful to receive the output of Invoke-Http when you need the response content.

Syntax

Receive-Http [[-Output] <string:[Xml|File|Text|Bytes]>] [[-Path] <string>] [[-InputObject] <MindTouch.Dream.Result<MindTouch.Dream.DreamMessage>>]

Receive-Http [[-Output] <string:[Xml|File|Text|Bytes]>] [[-Path] <string>] [[-InputObject] <MindTouch.Dream.DreamMessage>]

Parameters

Output

The TYPE of output you want:

• Xml will return an XmlDocument — or XmlNode(s) if you pass a Path.
• File will write to file — currently, you must pass a path, I didn’t make it guess it automatically.
• Text will output text — in the case where the actual output is XML, this returns the InnerText.
• Bytes returns an array of bytes…

• XDoc returns a MindTouch.Dream.XDoc (not the .Net native class), which has a boatload of extra web-related methods and features you might find useful such as accepting XPath operators as indexers.

Path

A string containing the path for file output or XPath selector for XML, XDoc, and Text output

InputObject

The input must be either a Result<DreamMessage> or a DreamMessage itself. Values passed on the pipeline are accepted into this parameter.

Set-HttpDefaultUrl

This takes the passed URI and sets it as the base (default) URL for the Invoke-Http function (if you set this, any path passed to Invoke-Http is treated as relative to this path).

Set-HttpCredential

This takes a PSCredential or NetworkCredential and stores it as a PSCredential for use by the Invoke-Http function. Particularly useful when you want to script a bunch of authenticated methods without being re-prompted. Note that if you don’t pass a credential, you’ll be prompted for one the usual way, so calling just Set-HttpCredential is sufficient when working on the command-line.

Miscellaneous

There are other functions in here, a few should be considered “internal” functions (when this is a module, they would not be exported: Get-DreamMessage, Get-DreamPlug, Get-HttpCredential) and a few of which came out of my utilities functions (like Encode-Twice, and Join-Url, Get-FileName, Get-UtcTime, and Get-CredentialBetter)...

Code (Updated 28 July 2010)

So I’ve taken my Huddled.Html code and worked it in together with a few extra bits … and I’m releasing it under the new name “PoshHttp” (which has the sole benefit of being short, in the case when you need to type the full command names, like: PoshHttp\Get-Web.

In lieu of providing proper documentation for this, there’s a script below which shows off some of the features of the Get-Web cmdlet … but first, you’re going to want to download PoshHttp and I can give you a basic overview.

Get-Web

Downloads a file from the web URI, optionally passing POST or GET parameters which it accepts as a Hashtable. If the resulting file is an xml or html file, it converts it to XML and outputs it as an XmlDocument object. If they are not, it saves them to file, and outputs the FileInfo object.

• Plain will prevent the conversion to xml
• Save will force saving to file (either xml or binaries)
• Path lets you specify the file name/path (normally, it selects a path based on what you’re downloading).

• Force will prevent it from asking if you want to overwrite files

ConvertFrom-Html

Converts HTML to XML. I’ll eventually tweak this so it supports passing it a file path and even saving the xml in-place on the original file … but for now you can use it on a stream of text, like this: (Get-Content file.html | ConvertFrom-Html).save( "file.xml" ). Note that it outputs an xmldocument object … you can get the xml as text to the console using Get_OuterXml().

ConvertTo-Hashtable

Converts a url-style query string or NameValueCollection into a Hashtable. It’s mainly to make it easier to use the Get-Web in cases where you’re getting the URL or POST data from elsewhere.

Here’s a script with a few example uses of Get-Web:

Edit: I made a mistake … I wrote this a few weeks ago, and made an error in the script which causes it to not work properly with most binary files … I’ve fixed the script down below now, and while I was at it, I went ahead and added a download progress report

About a year ago, I wrote a script to download files from the web using PowerShell, but it was so simple that you had to specify the url to download and the file to save to. At the time I knew there was a way to find the file name of the download in the header, but I couldn’t remember how to get to the headers, and it’s not possible using System.Net.WebClient so I just dropped it.

Then the other day I saw an old post from Script Fanatic about querying Http status codes, and it reminded me of my annoyance with my wget script. So I fixed it, and I’m letting you have it.

The new version of wget for PowerShell (or Get-WebFile) is on the PowerShell Repository and it’s been completely rewritten to use System.Net.HttpWebRequest and HttpWebResponse which gives us access to the the name of the file in those cases where the URL is something like .../scripts/?dl=417 (assuming the server puts the file name in the headers as it should).

It still doesn’t support authentication, and even though the headers include the size (and I have to do the streams myself so I could easily read a few bytes at a time and give you progress reports)

I’m probably the last person on the planet to figure out how easy it is to get a basic wget working in PowerShell … but just in case I’m not … here’s a PowerShell script which will download web files to your hard drive.

That’s all it does. This script doesn’t support authentication or anything (even though it does use your computer’s proxy settings), but I think it will be obvious how to extend it, since it’s basically just .Net Framework stuff.

That’s all there is to it. Actually, you really only need the last two lines (and the first line), as long as you understand that the default path will be your user directory rather than the current directory. (like: C:\Users\Joel OR C:\Documents and Settings\Joel\). That means that if you leave off that if(!(Split-Path -parent $path)) stuff, you’ll always have to specify the file starting with a .\ if you want it saved in the folder you’re in (otherwise it gets dumped in your user directory).

Those three lines check if the specified path has a folder in it (and if it’s a valid folder) and if not, they just save the file to the local current folder.

Here are some examples of how you’d use it, assuming you save that into a file called “wget.ps1” and put it in your path. Incidentally, just like in ‘nix shells, ~ represents your user directory … oh, and the generally more PowerShell-like name for the script would be “Get-Url.ps1” or something…

• wget http://huddledmasses.org/downloads/RunOnlyOne.exe RunOnlyOne.exe
• wget http://huddledmasses.org/downloads/RunOnlyOne.exe .\RunOne.exe

• wget http://huddledmasses.org/downloads/RunOnlyOne.exe ~\RunOnly1.exe

If anyone can tell me how to figure out what the file name should be when the URL ends with something like: .../download.php?file=361 I’ll post an amendment here that will let you skip specifying the file name.