Awhile back I wrote a Get-Web script, which I later converted to a compiled Get-Web cmdlet and module (pssnapin) ... adding support for POST forms. At the time, it seemed like GET and POST with some string variables pretty much covered what I’d ever want to do with web pages, however I found a few places where I needed to POST a file attachment, and then a couple of full-on REST APIs, which use not just GET and POST, but PUT and DELETE as well.

While I was playing with scripting the REST API for DekiWiki … I experimented with the MindTouch Dream SDK … and as a result, I’ve refactored the part of that module that actually does the HTTP work out into a little PowerShell 1.0 compatible script library. Of course, it depends on several dll’s (mindtouch.core.dll, mindtouch.dream.dll, and SgmlReaderDll.dll and log4net.dll) from the Dream SDK, which you can download from SourceForge.

Example Uses

There are basically two main functions: Invoke-Http and Receive-Http. There are also a few extra functions which are mostly just for use by these two functions, and then Set-HttpDefaultUrl and Set-HttpCredential which you can use to simplify your scripts. I intend to rewrite these as cmdlets in C# and contribute them to the PowerShell Community Extensions along with some of my other networking cmdlets (Get-DNS), so if you have any suggestions, I’d love to hear them!

Invoke-Http

Invokes HTTP verbs (like GET, POST, PUT, and DELETE) against a URL with optional parameters (eg: name = value), input, etc … and returns the output as a MindTouch.Dream.Result or (optionally) as a MindTouch.Dream.DreamMessage …

Syntax

Invoke-Http [-Verb] <String> [[-Path] <String>] [[-With] <Hashtable>] [[-Content] <Object>] [[-Type] <String>] [[-Authenticate] <String>] [-WaitForResponse]

Parameters

Verb

The HTTP verb you want to invoke. Defaults to GET, but can be anything your HTTP server supports.

Path

The URL you want to invoke against. Can be a partial URL, if you first call Set-HttpDefaultURL first. You can also specify this as an array or a hashtable. This is particularly useful if you’re specifying a partial URL, and your URL is built based on logic in your code. If you pass an array, the values in the array are joined with forward-slashes. If you pass a hashtable, the keys and values are appended joined with forward-slashes, but the values are double encoded (by UrlEncode), ie: $key/$(UrlEncode(UrlEncode($value))

With

The parameters to be passed with the verb call, as a hashtable. (these are particularly used for POST, etc). These are passed in the query as named parameters, with the values encoded.

Content

Anything you want to pass as content. In actuality, we really only handle three things: XML Documents, file paths (the file will be used as content), and everything else is passed as simple text. If you need other data types, the “DreamMessage” that the scripts depend on can handle all sorts of data types, so you’ll just need to add it around line 50 of the script.

Type

This is the type of content. We can automatically deduce this for XML or File objects, but if you want to force a type, you should use this parameter. For instance, to fake a web form submission, you need to specify the type as FORM or FORM_URLENCODED but you can also specify the type as any of: ATOM, BINARY, BMP, CSS, DREAM_EXCEPTION, FORM_URLENCODED, GIF, HTML, HTTP, JPEG, JS, JSON, MULTIPART_MIXED, NOTHING (mimetype -/-), ANY (mimetype */*), PDF, PHP, PNG, RDF, RELAXNG, SVG, ANY_TEXT (mimetype text/*), TEXT (mimetype text/plain, with the ISO-8859-1 encoding), TEXT_UTF8 (mimetype text/plain, with UTF8 encoding), TEXT_XML, TIFF, VCAL, VERSIT, XHTML, XML, XRDS.

Authenticate

You can pass either a boolean (in which case the HTTPRest scripts will use a default $global:HttpRestCredential which it will prompt for if you haven’t set it), or an actual PSCredential or NetworkCredential object. In any case, this causes the invoke to contain network credentials. Note that sometimes you may need to pre-authenticate before making calls to a web-service.

WaitForResponse

A switch parameter that causes the query to wait until the response completes downloading, and return a DreamResponse instead of returning an incomplete Result ... this is useful when you know the response is a simple XML file and you need to parse it to determine whether your call has succeeded. If you need to actually get the data out, or if you just need to know the method was invoked successfully, you shouldn’t pass this parameter.

Receive-Http

Receives the response or result from a Dream.Result or DreamMessage. as XML, XDoc, text, raw bytes, or even to file. This function is really only useful to receive the output of Invoke-Http when you need the response content.

Syntax

Receive-Http [[-Output] <string:[Xml|File|Text|Bytes]>] [[-Path] <string>] [[-InputObject] <MindTouch.Dream.Result<MindTouch.Dream.DreamMessage>>]

Receive-Http [[-Output] <string:[Xml|File|Text|Bytes]>] [[-Path] <string>] [[-InputObject] <MindTouch.Dream.DreamMessage>]

Parameters

Output

The TYPE of output you want:

• Xml will return an XmlDocument — or XmlNode(s) if you pass a Path.
• File will write to file — currently, you must pass a path, I didn’t make it guess it automatically.
• Text will output text — in the case where the actual output is XML, this returns the InnerText.
• Bytes returns an array of bytes…
• XDoc returns a MindTouch.Dream.XDoc (not the .Net native class), which has a boatload of extra web-related methods and features you might find useful such as accepting XPath operators as indexers.

Path

A string containing the path for file output or XPath selector for XML, XDoc, and Text output

InputObject

The input must be either a Result<DreamMessage> or a DreamMessage itself. Values passed on the pipeline are accepted into this parameter.

Set-HttpDefaultUrl

This takes the passed URI and sets it as the base (default) URL for the Invoke-Http function (if you set this, any path passed to Invoke-Http is treated as relative to this path).

Set-HttpCredential

This takes a PSCredential or NetworkCredential and stores it as a PSCredential for use by the Invoke-Http function. Particularly useful when you want to script a bunch of authenticated methods without being re-prompted. Note that if you don’t pass a credential, you’ll be prompted for one the usual way, so calling just Set-HttpCredential is sufficient when working on the command-line.

Miscellaneous

There are other functions in here, a few should be considered “internal” functions (when this is a module, they would not be exported: Get-DreamMessage, Get-DreamPlug, Get-HttpCredential) and a few of which came out of my utilities functions (like Encode-Twice, and Join-Url, Get-FileName, Get-UtcTime, and Get-CredentialBetter)...

Code

Hashtables are IEnumerable, but they don’t behave that way in PowerShell … this seems to cause all sorts of odd behavior and such, so I thought I’d write up all the examples I can think of in one place. That means this post is going to be a little bit rambling, so please bear with me.

PowerShell enumerates IEnumerable

It turns out that in the case of Hashtables, PowerShell does NOT enumerate them into the pipeline. Instead, it passes the entire Hashtable object. Of course, nobody realizes this … because the ouput cmdlets unwrap them (what?!).

But there are bugs caused by special treatment

The first bug is in Add-Member, which doesn’t work on Hashtables until you’ve already used the Hashtable.

NOT ONLY does Add-Member not work the first time, it’s not just a matter of calling it twice: you just have to try to access something in the hashtable before you can use Add-Member on it:

Here’s another buggy manifestation, in the Formatting cmdlets. This time, the Format-* cmdlets unroll the hashtable … to make it look like it’s being enumerated the way it should be.

Clearly, the Format-* cmdlets have magic code that unwraps hashtables. Which just leads to even more confusion: $table looks the same (in the console output) as $table.GetEnumerator() ... but it doesn’t behave the same way, EXCEPT to the format cmdlets.

PowerShell 2.0 uses Hashtables more

In PowerShell 2.0, the PowerShell team is adding another special feature based on hashtables (which appears at first to be based on IEnumerables):

Jeffrey Snover gave an example of splatting in his PDC presentation. Splatting is where a collection is unwrapped so that you can take an array of values and pass one to each parameter of a cmdlet or function. But in Jeffrey Snover’s demo, he splatted a hashtable. Basically, the hashtable keys are matched up to parameter names as though they had been specified by name. That made me wonder why splatting can’t work with custom objects, but after investigating a bit, I’m actually frustrated with the inconsistency of how hashtables are treated in Posh.

• In the splatting scenario they are unrolled as a collection of named parameters …
• If you pipe them, they’re treated as a single object instead of being unrolled …
• Even though you can access hashtable items using dotted property syntax, you can’t use them to set ValueFromPipelineByPropertyName values, because they aren’t really properties.

The new splatting feature seems to only work with simple arrays and hashtables … adding yet another scenario where the hashtable is being treated specially (even though it doesn’t need to be: if they just splatted IEnumerable, we could work with List, and any IEnumerable of DictionaryEntry objects, or KeyValuePair could be matched by name … that would make hashtables work, but it would also let you use the more powerful generic collections, etc.

You know what would be cool? If I could splat any object (like a custom PSObject that I have added members to), and have it’s property names matched to parameter names as though all the parameters had ValueFromPipelineByPropertyName set.

You know what would be really cool? If I could specify that I want pipeline objects splatted, forcing ALL parameters to be treated as ValueFromPipelineByPropertyName, without needing to use: ForEach-Object { Test-Splatting @_ } … maybe a syntax like: Get-HashTablesToSplat | Test-Splatting @@ …

I’ll write up more information later, but a couple people have asked for this in #PowerShell on irc.freenode.net, and I had it already written, so here you go … my ConvertFrom-Html cmdlet (in a Huddled.HtmlSnapin). It converts HTML to valid xml using the SGML Parser which was available on GotDotNet years ago. It only works with files (doesn’t do URL downloads yet). Use it like this:

The source code to my plugin may be considered public domain, and is included in the Huddled HTML SnapIn Zip.

However, the SgmlReader library is a Microsoft Sample which is licensed under the old MS Samples license which doesn’t allow reuse with viral open source software. I’ve seen some work being done on an HtmlAgilityPack on CodePlex (using a Creative Commons ASA license) but I have not really looked at it except to see that it has a several active issues related to entity encoding and dropping malformed tags which I haven’t encountered in SgmlReader …

The SGMLReader library has been re-released on MSDN Code Gallery under an Ms-PL license, and all is well with the world.

Well, the first alpha CTP release of PowerShell 2.0 is out, and there’s a lot of new stuff in it … but I won’t repeat the list from the PowerShell blog, because I’m sure you’ve seen it five or six times already. Instead, lets just skip straight to talking about one of the features we’ve been hearing about the longest: in PowerShell 2, you can create Cmdlets in script … bringing nearly full parity between whats possible in a C# cmdlet and what’s possible in script.

There are a few caveats still (Parameter Sets aren’t working yet, and neither is help, really), and a few surprises … there’s a few downsides to PowerShell script vs C# ... but in this particular context one thing that stands out is that in C# the BeginProcessing, ProcessRecord, and EndProcessing blocks are actually methods which can call each other, and as demonstrated in my tutorial for writing cmdlets that work in the pipeline, they can be recursive — without getting duplicate variables.

A sample Script Cmdlet

In the interests of being the first to publish an interesting script cmdlet and to continue my recent trend of talking about writing for the PowerShell pipeline, I’ve merged the logic of my script function and my pipeline cmdlet into a single sample script cmdlet for PowerShell 2.0 and it works great!

A few observations from the process, in no particular order:

• If you recursively call your cmdlet from within itself, you have to test for parameters using the new $CommandLineParameters.ContainsKey because parameter variables keep their values through recursion if you don’t explicitly pass a value.
• $CommandLineParameters.ContainsKey works differently in the Begin block where it will return $false for arguments which will get their values from the pipeline, than in the Process block where it will treat values which were passed as CommandLineParameters the same as those which were passed via the pipeline.
• If you want to see how your function behaves in a pipeline, you should make sure to test it at different points in the pipeline: at the front, in the middle, and at the end.
• Cmdlets are functions: they show up in the Function provider.
• Cmdlets are functions: they have to be dot-sourced before you can call them.
• Cmdlets are not functions: they are a single command Cmdlet which takes a name (which must have a – in it) and a couple of other parameters followed by a function script block.
• When you recurse by executing &($MyInvocation.InvocationName), that second invocation has an InvocationName of “&” ... so you can’t go any further (this might be a good thing, if you want to stop recursion at one level no matter what. If you want to go further, you need to put your commands into a string, and use Invoke-Expression.

A test case

In a continuation of what is, sadly, becoming a series on how the PowerShell Pipeline works … Karl Prosser brought to my attention that certain powershell commands which have an -InputObject parameter don’t actually work when you pass something into it … so I thought I should create a cmdlet to show you how to correctly handle the InputObject parameter with the ValueFromPipeline set so you can pass the input in either way.

To demonstrate the problem, try this:

This should expose two weirdnesses about how the Select-Object cmdlet works:

1. The -First parameter affects the input before the -Unique parameter does.
2. When you pass the input in via -InputObject, the whole array is treated as a single object, and the command basically doesn’t do anything.

The big problem with this behavior is that there’s essentially no hint that you’ve done something wrong — there’s actually no way to make Select-Object work properly except by passing the objects in via the pipeline. The bigger problem is that it would have been simple for the Microsoft team to catch this and alert you, but they didn’t — so you probably won’t even notice there’s a problem until you run it on a trivial data set like my example. The even bigger problem is that it doesn’t just affect Select-Object (try it with Where-Object, just for instance). (more…)

I have several PowerShell scripts and functions that I’ve written in the last few months which take a file or folder path as an argument, and the problem is that most of them need to validate the path in one way or another, and generally speaking this means that you have to pass in a fully qualified path, because they have no simple way to find the file if you pass in a partial, relative path, or the name of a file that’s on your system path.

Anyway, I finally worked up a function to resolve paths taking the current directory and the environment path variable into account:

It’s actually pretty simple … it uses the Test-Path method to see if you passed in a full path, and otherwise searches the current directory and all the directories in the PATH to find a matching item. You can specify if you want the -Type "Leaf" (e.g.: files) or -Type "Container" (e.g.: folders) and you can set the -All switch to return all matches in the path instead of just the first one. As you may have guessed, the path doesn’t necessarily have to be a FileSystem path, it will resolve other PSProvider paths based on the current Location. I have it search the paths in the Environment:\Path because I’m mostly concerned with file and folder paths … but aside from that, there’s no equivalent collection of paths for other providers … maybe someone should add some …

So, I’ve been working on PoshConsole for awhile now, and with the help of some of the guys in #PowerShell@irc.freenode.net have been playing with trying to find ways to enhance output, like using ANSI escape sequences, and even creating an Out-WPF cmdlet which can output objects as databound WPF controls which look really good.

We have been trying to find a way to incorporate the colored and even bold/italic formatting in a way that would be compatible with the existing format-* cmdlets, and we’ve even looked at replacing out-default with a cmdlet that would be compatible with the default PowerShell host. But today I finally locked in on the import of this post on the PowerShell blog. It’s just not possible.

The format cmdlets (Format-List, Format-Custom, Format-Table, Format-Wide) output data in the form of undocumented .Net objects which are “subject to change without notice,” and are therefore basically useless. Of course, that means that if you want to replace Out-Default you have to not only replace the output, but you have to replace the formatting cmdlets, and of course the Update-FormatData cmdlet too — in fact, you have to either parse the largely undocumented format data files, or create some replacement for them to allow users to specify formatting for types you haven’t thought of…

All of this amounts to what many of you probably already knew: it’s a lot of work to create a complete PowerShell host, but it’s at least as much work to create a replacement PowerShell formatter.

Someone asked in the #PowerShell channel on FreeNode for a way to register dll’s (you know, old COM libraries for use in scripts or apps) ... specifically, they wanted to create a script that could register a bunch of Dll files, and know which ones passed or failed.

It turned out to be an interesting problem, because even though the source code on the MSDN Library has a /C parameter to cause the application to write to stdout, the one which comes with windows doesn’t, so the only obvious way to get output is in a MessageBox. However, there is also an exit code, but the exit code doesn’t seem to end up in $LASTEXITCODE or $? at all (maybe because the app is actually not a console app? I don’t get it, really). So the real problem became: how do I capture the exit code of an application in PowerShell?

Well, the simplest thing would be to just go around that — to compile and distribute the MSDN sample with the /c console output option so I don’t need the exit code. Of course, they are very clear that we shouldn’t do that because the sample isn’t “safe,” and distributing it afterward would probably be a problem with licensing. Well, I don’t know about that, but I figured the best thing would be to just use the exit code from the existing RegSvr32.exe rather than asking people to use what they would perceive as a 3rd party RegSvr32.

And if we play around a bit with the .Net Process and ProcessStartInfo classes, it’s not to hard to get the exit code …

After a bit more experimentation with the return values, I came up with a function called RegisterDllServer which appears below. The function has a -Verbose switch and a -Debug switch which turn on some helpful messages (the Debug switch actually allows the RegSvr32 message boxes to show up, so you shouldn’t use that if you’re calling it from within a script). (more…)

A while back I switched to GeSHi for source code highlighting in my posts, and recently I started writing PowerShell scripts in my posts, and calmly sticking it in <code lang=“posh”> tags, half expecting it to just work, like all the other languages … but of course it didn’t. So after a few searches on Google and Ask, I concluded that a PowerShell syntax file doesn’t yet exist. So, I made one, feel free to grab it (it’s GPL, and I’ve even left intact the “any later version” clause from GeSHi’s license).

EDIT: 6/10/2007
I should mention that the way I do highlighting doesn’t care about “Nouns” at all: it just uses a list of Verbs, and matches anything that starts with a “Verb-” ... that means that for the sake of the highlighting, you can’t just say “Content” for Get-Content or (even though that actually works in a script). Also, I changed the version I was using so that it uses regular expression patters for command parameters, instead of a list. Because you can abbreviate parameters to the shortest distinguishable form, a list doesn’t really work.

Let’s just see in action, shall we? (more…)

Microsoft has published a new Command Line Standard which sets out the standard for writing command line applications for Microsoft platforms … While it’s clearly based on the way they’ve written PowerShell, it is a spec that is “independent of any specific implementation of a shell, set of utilities or command creation technologies” and should be used for any command line interface apps for DOS, too (especially since doing so would enable your app to function in a PowerShell pipeline as well).

Anyway, it looks really interesting, and I can’t help but think that perhaps the Linux command-line could benefit from apps implementing this standard too . In particular, I like the TCSV format option as a way of enhancing plain old CSV data with type and structure information … and the standardized naming convention. Ultimately, these five points (from the spec) should be the standard for any command-line interface (although I know many linux die-hards will note they’ve been living without number five for many years):

1. Consistent verb and parameter naming conventions.
2. Consistent command syntax
3. Common behaviors.
4. Consistent error messages and exit codes.
5. Schema-enhanced data-streams and formatting.

However, Appendix A – Standardized Verb Sets and Verb Names got my attention in particular, because it actually obsoletes a few of the Verbs that were in this list on the PowerShell blog and which they’ve actually been using in the Community Extensions … Specifically, Write is marked as obsolete in favor of Set — and it’s pair Get, is to be used instead of the obsolete Read. In addition, there’s no mention of Out and *Where* is only mentioned as an obsolete version of Resolve (which really made me wonder about “Where-Object” but I guess that can be the exception that proves the rule?). The spec actually says that it is required to use verbs from the list of standard verbs in Appendix A., so there doesn’t seem to be an option to invent your own verbs.

If you have any comments about the spec, I guess you should put them on this post on their blog, rather than on this thread in the PSCX forum.

I’ll have to write another post on this later, after I’ve had a chance to re-read the spec and digest it a bit … so far it looks really good, but it sure would throw the monkey-wrench in my old console “Hello World” apps in C++ and Java … I wonder if someone will create a wrapper to make writing compliant command line apps easier (without resorting to writing PowerShell commandlets which are practically compliant automatically). I also wonder how long it will be until a new PowerShell release comes out that actually understands TCSV...