Huddled Masses

Posts Tagged ‘Pipeline’

I just posted an article to the FAQ on the PoshCode Wiki answering this question that came up again on our IRC PowerShell user group today. It came up in the context of determining whether a function was the last function on the pipeline, because one of our users was looking for a way (other than creating ps1xml files) to output objects onto the pipeline for use in other functions, but still format those objects nicely if they were output directly.

Before I give the solution, I just want to say: don’t change your output based on where you are in a pipeline.

There are numerous scenarios where your function will be the last one on a pipeline, but still be participating in further pipelines, including formatting and output modification. For example, take our function Test-Pipeline (defined below) in these three scenarios below. In none of these scenarios would it be appropriate for the function to write formatted output instead of outputting the raw object, but in each case, the function is the last function in the pipeline.

However, if you want to determine your function’s position in the pipeline for some other reason, the answer is simple. You need to use $MyInvocation and compare the PipelineLength and PipelinePosition properties:

One of the consistent questions about PowerShell is: what’s the best way to write a script or a function to process pipeline objects and be able to take it’s parameters as a normal function?

Of scripts and functions

The first thing to know is that in PowerShell, there’s really no difference between a script (just a file with a .ps1 ending) and a function as you’ll see written below. If you take a function Get-Square, and remove the first and last lines ( Function Get-Square { … } ) you could put them in a file called “Get-Square.ps1” in your PATH, you use them exactly the way you would the function that was pre-loaded into memory by dot-sourcing or pasting it on the command line. Of course, doing that easily requires writing the function parameters on their own line using PARAM(...) syntax, which is why I recommend doing that.

Of functions and the pipeline

When your script or function is used on the pipeline its begin block is called once when the pipeline starts up, and then the process block is called repeatedly: once to process each pipeline object, and finally, the end block is called after all the objects have been processed through the whole pipeline. If you don’t understand that, you should play with this function, try calling it by passing a series of numbers through multiple instances of it:

Incidentally, if you don’t specify the begin, process, or end blocks, the body of your function is treated as the end block. This is so that you can use the special $Input variable, which collects all the things passed in on the pipeline, and thus only works in the end block. That would allow the following function to behave the same way regardless of whether it was invoked on the pipeline or by passing an $InputObject. Notice, however, the difference between this, and the function above.

Our challenge

The basic idea here is to rewrite that function such that it can be used to process a set of numbers from either the pipeline or an argument, without interfering with the processing of other parameters. We require that the function process items as they come in, rather than waiting until it’s received all input before processing them the way Test-SquareEnd does.

In PowerShell 2.0 this would be easy

In the current CTP 3 of PowerShell 2, you just specify ValueFromPipeline=$true for the parameter you want to set from the pipeline, and the function will work the same way whether you pass the numbers as a parameter or along the pipeline — you can even control which attribute of the objects on the pipeline will be used, but that’s a whole other article.

Of course, PowerShell 2.0 is still in beta status, and even after it’s released you may need to write scripts that are backwards compatible to PowerShell 1.0, and the excercise of doing so may help you to understand more about how PowerShell functions work, and particularly how they behave in the pipeline.

Our solution

Since in a PowerShell 1.0 function it’s not really supported directly, we need to do some extra work:

An explanation

Of course, this is just an example method, squaring things isn’t that exciting — but what’s special about it is that the output is almost exactly the same whether you call it with parameters Get-Square 1,2,3,4 or on the pipeline: 1,2,3,4 | Get-Square.

The trick is that it actually executes the same way in either case:

1. If you call it by passing the int (or array of ints) as an argument ($InputObject), it calls itself and passes those values on the pipeline.
2. When the integers are passed on the pipeline, the special pipeline iterator variable $_ is set, and the process block is executed.

1. When it has to (re)invoke itself, it passes any other parameters as parameters, which means you need to have default values for them.

There is one tiny difference in the processing, which in real-world use is practically never noticeable (you can see it in our example if you call it like this):

You’ll see that unlike the PowerShell 2.0 pipeline function, when you pass the numbers as a parameter, the first one of them actually gets passed through the process block before the begin block of the second function on the pipeline is called. This usually doesn’t have any effect, but it’s something to keep in the back of your head.

A few precautions:

• You have to default values for parameters, because you’ll be passing them all as named parameters, in the re-invoke step, and $null can cause problems.
• If you need to do additional processing in the begin block, you should only do so in an ELSE case: when $InputObject is null. That way, the code will only execute once each time you call the function.
• The same goes for the end block: you have to keep your code in an If(!$InputObject) block to avoid executing it twice (when you pass the values as an argument, and it re-invokes itself).

• The test cases in the process block must wrap all of your process block code, so that you don’t process the arguments twice, and you shouldn’t refer to $InputObject, but instead should use the automatic $_ variable which is the value passed when the function is (re)invoked via the pipeline.

Here’s some sample output, in case you’re wondering:

I was just having some fun with some recent blog posts…

WPF & PowerShell – Part 5 has a script for “Get-Listbox” and for “Show-Control” and Halr9000 wrote a script he called Get-PSBlogroll

I had modified the example from the WPF post to create a listbox which will “start” whatever you double click …

And I tried something like this to let me launch links from hal9000’s blogroll:

Of course, what I’d like is to see the titles, instead of the URL, but because of how Get-ListBox was written, it only accepts strings, and therefore only returns strings. I tried adding the URL as a NoteProperty on the title string, but there’s a bug in the script cmdlet implementation that strips ETS properties. So if you specify your parameter type as a [string], you loose the NoteProperty.

Idea 1: Use [PsObject] for your pipeline parameter

Of course, if they fix the bug this tip will go away, but in the meantime, you’re better off not casting the input from the PowerShell native [PsObject] type, because you loose any extended type system attributes (and if you cast XML to a string, it comes out yucky).

Idea 2: Don’t output xml nodes

Even after fixing the Get-Listbox script so it outputs what it gets in without casting it, I still have to craft PsObjects with the output from Halr9000’s script, because otherwise Get-ListBox just outputs the .ToString() representation of the xml node, which is invariably useless — I could data-bind to the xml attributes, but it requires a separate syntax in the XAML — which means you’d have to have separate code for xml vs objects. So basically, you’re better off with a | Select * at the end of your Get-PsBlogRoll.

Idea 3: Display scripts should take a property (or a property list)...

What I eventually ended up doing is rewriting Get-Listbox to take a “Properties” parameter so that I could specify which of the attributes of an object I wanted to use … that way, I can pass in the almost unmodified output from Get-PSBlogRoll into the new Get-ListBox, and run with it. In the end, it looks something like this:

Of course, because of how I wrote the DataTemplate for the properties (using a StackPanel and a loop to output multiple TextBlock objects) you could choose to show the Url as well, and all you’d have to do is change the call to Get-Listbox like this: Get-Listbox title,htmlUrl … which is pretty cool. Does anyone else have any ideas for making these even easier to compose?

A really nice trick would be if the wpf cmdlet could use Write-Output in the click handler and actually yield to allow the rest of the pipeline to process that item, but so far, I haven’t been able to find a way to do that (short of closing the WPF window, of course).

A relatively new PowerShell user came into #PowerShell on IRC.FreeNode.net this week to ask a question about scripts and the pipeline, and the conversation went so well, that I thought I’d share it with you all in case it helps clear things up for you. We’ll call him “user” since he left before I could get his permission to paste this, and I’ve cleaned up, reordered and in a couple of cases added lines to make the conversation seem a little more linear than the chat really was.

user: Can you explain the distinction between $_ and a param in a script context?
Jaykul: I sure can. And I think I can do it simply … take an example script|function … actually, we’ll just use a scriptblock
Jaykul: it can have pipeline blocks, or not (the pipeline blocks are the BEGIN{...} PROCESS{...} END{...})
user: yeah, I’m with you
Jaykul: so if you pass a value via an argument, it’s present in all three blocks, and has the same value the whole time (unless you change it in your script)
user: really
Jaykul: But when you put that script in a pipeline, your BEGIN {} block is called first, and then for EACH item in the pipeline, powershell sets $_ and calls your PROCESS block, and finally, your END{} block is called after all pipeline items have been processed
Jaykul: Here, check this out (run it so you can see) Read the rest of this entry »

I started this post a few weeks ago, but never quite got around to posting it, and then I saw today that Dmitry wrote about the memory problems with using measure-object and I thought this would be a great time to post this. Essentially, this is a continuation of my exploration of development in the PowerShell pipeline, and we’ll see some of the problems with the way that the pipeline works and what happens if you overlook the numbers of items that could be in the pipeline.

Dmitry was trying to count the number of users in Active Directory, and ran into ridiculous memory use when using Measure-Object because it appears to collect all the items in the pipeline into memory before it counts, resulting in huge amounts of memory use based on the number of objects you’re counting. Of course, for the purpose of counting, it’s a pretty obvious fix (as Dmitry explains), but for other purposes it may not be as obvious.

Select Random

When I originally started this post, I was trying to upgrade my original Select-Random script (which was published in the PowerShell Community Extensions) to use reservoir sampling in an attempt to solve the same problem. Select-Random is a script which selects a random element from a collection or from the pipeline. In the past, if the elements were passed in as an argument, it simply collected them into an array (in a really inefficient way, but never mind) and then picks a random number and selects the item by index.

I started by improving the collection method to use an ArrayList and reduced the memory usage to about 1/5th of the original method, and then rewrote the whole script to use reservoir sampling instead. Greg wrote a good explanation of reservoir sampling but basically it’s an algorithm for choosing random items from a collection of unknown size without having to traverse the collection twice (to count it the first time and then to select) — in my case, the need to count the collection was requiring me to store the pipeline input in an array, so the improvement was more about memory use than about needing to enumerate the list twice. However, there is a downside: it’s slower. Read the rest of this entry »

In response to Kirk Munro’s comment on my Writing Cmdlets for the PowerShell Pipeline post:

You know, I’ve looked at your articles about cmdlets/functions in the pipeline and I feel you’re missing something. The purpose of the InputObject parameter is to pass in a collection as a single object. This is as opposed to using the pipeline where a collection is passed along the pipeline one item at a time. There are cases where you want to pass in a collection as a collection.

Quite simply, I disagree. The documentation for these parameters says quite clearly that inputObject “Specifies an object or objects to input to the cmdlet.” This clearly means that I should be able to pass multiple objects, and have them treated as multiple objects, not as a single array object.

If you look at your example (Select -First 3 -Unique -InputObject $a), this does in fact work. It receives one object, an array. It then selects the first 3 objects, but there is only 1 so that is moot. And lastly it selects unique objects, but again there is only 1 so that is moot as well and finally the object is output using the default formatter. In this case the default formatter is showing the contents of the array.

In this example, Select-Object has no reason to take a single object as an input object, at all. The only time that it would be useful for Select-Object to take a single inputObject would be in combination with the property parameters. In fact, if you want to Select-Object from an array to get the first of last n objects, or to get a set of unique objects, you have to pass the objects in via the pipeline — there’s no other way to make it select from an array. If that was indeed the intent, it should have been written as a separate ParameterSet, and the documentation should be changed to reflect that only a single object can be passed in, and that you can’t use the inputObject parameter with the first, last, or unique parameters at all. That’s worse than useless, it’s misleading and confusing.

Kirk is absolutely right that if you assume that the InputObject argument is only allowed to take a single object, then the behavior is correct – but it’s not logical. In fact, the behavior you see in the output of this command is so useless as to be a bug – even if the documentation did not say the parameter accepts multiple objects as input:

I know that that’s the way the built-in cmdlets work.

But quite frankly, just because someone important wrote something useless is no reason to emulate the behavior. The inputObject parameter IS the same parameter which pipeline objects go into. There’s no logical explanation for us to get different results when we pass an array in via the parameter by name instead of via pipeline: the PowerShell pipeline passing the things in the pipeline into the –inputObject parameter … it’s not using some mystical variable like it does in script functions.

Of course, we all know the powershell pipeline unwraps arrays — that’s convenient, and we can work around it when we really want to pass an array in:

My point in all of this is that InputObject is actually a very useful parameter, because there are cases where you really want to pass a collection as a collection into a cmdlet and then do something with it. By making InputObject instead split the collection passed in and pipeline it through, you’re forcing users to wrap collections in an array just to get them passed in as a collection, and personally I don’t feel they should have to do that.

While it’s true that passing in an array is sometimes desirable that’s not the reason the parameter exists, and I don’t believe it should be the default behavior here. It should be just as easy for me to use the cmdlet with the inputObject parameter directly as it is to input them via the pipeline. If I put in unwrapping for the inputObject parameter, you can work around it in the same way I did in the examples above. Incidentally, I think *PowerShell* should unwrap arrays to ValueFromPipeline parameters regardless of whether they’re on the pipeline, but I recognize it’s probably too late for that.

Basically, this is my argument: If inputObject unwraps arrays, the syntax for passing an array by wrapping it in @(,$array) is simple, for those rare occasions when that’s actually what you want. But if it does not unwrap arrays, you’re forced to call it via a separate pipeline, because unwrapping the array and passing it in one at a time in a foreach loop will almost certainly not do the same thing, and this is much uglier — and not compatible with use within the pipeline, particularly if you need to pass the pipeline output into a different parameter.

I guess my final word would be to agree with Kirk that “InputObject … isn’t documented clearly enough” … in fact, it’s clearly behaving incorrectly according to the documentation, and that’s why I originally proposed to unwrap the inputObject parameter when it’s passed as a parameter: to make it work the way the documentation suggests it would, which seems to me to be a better way than the way it actually works.

Well, the first alpha CTP release of PowerShell 2.0 is out, and there’s a lot of new stuff in it … but I won’t repeat the list from the PowerShell blog, because I’m sure you’ve seen it five or six times already. Instead, lets just skip straight to talking about one of the features we’ve been hearing about the longest: in PowerShell 2, you can create Cmdlets in script … bringing nearly full parity between whats possible in a C# cmdlet and what’s possible in script.

There are a few caveats still (Parameter Sets aren’t working yet, and neither is help, really), and a few surprises … there’s a few downsides to PowerShell script vs C# ... but in this particular context one thing that stands out is that in C# the BeginProcessing, ProcessRecord, and EndProcessing blocks are actually methods which can call each other, and as demonstrated in my tutorial for writing cmdlets that work in the pipeline, they can be recursive — without getting duplicate variables.

A sample Script Cmdlet

In the interests of being the first to publish an interesting script cmdlet and to continue my recent trend of talking about writing for the PowerShell pipeline, I’ve merged the logic of my script function and my pipeline cmdlet into a single sample script cmdlet for PowerShell 2.0 and it works great!

A few observations from the process, in no particular order:

• If you recursively call your cmdlet from within itself, you have to test for parameters using the new $CommandLineParameters.ContainsKey because parameter variables keep their values through recursion if you don’t explicitly pass a value.
• $CommandLineParameters.ContainsKey works differently in the Begin block where it will return $false for arguments which will get their values from the pipeline, than in the Process block where it will treat values which were passed as CommandLineParameters the same as those which were passed via the pipeline.
• If you want to see how your function behaves in a pipeline, you should make sure to test it at different points in the pipeline: at the front, in the middle, and at the end.
• Cmdlets are functions: they show up in the Function provider.
• Cmdlets are functions: they have to be dot-sourced before you can call them.
• Cmdlets are not functions: they are a single command Cmdlet which takes a name (which must have a – in it) and a couple of other parameters followed by a function script block.

• When you recurse by executing &($MyInvocation.InvocationName), that second invocation has an InvocationName of “&” ... so you can’t go any further (this might be a good thing, if you want to stop recursion at one level no matter what. If you want to go further, you need to put your commands into a string, and use Invoke-Expression.

A test case

In a continuation of what is, sadly, becoming a series on how the PowerShell Pipeline works … Karl Prosser brought to my attention that certain powershell commands which have an -InputObject parameter don’t actually work when you pass something into it … so I thought I should create a cmdlet to show you how to correctly handle the InputObject parameter with the ValueFromPipeline set so you can pass the input in either way.

To demonstrate the problem, try this:

This should expose two weirdnesses about how the Select-Object cmdlet works:

1. The -First parameter affects the input before the -Unique parameter does.

1. When you pass the input in via -InputObject, the whole array is treated as a single object, and the command basically doesn’t do anything.

The big problem with this behavior is that there’s essentially no hint that you’ve done something wrong — there’s actually no way to make Select-Object work properly except by passing the objects in via the pipeline. The bigger problem is that it would have been simple for the Microsoft team to catch this and alert you, but they didn’t — so you probably won’t even notice there’s a problem until you run it on a trivial data set like my example. The even bigger problem is that it doesn’t just affect Select-Object (try it with Where-Object, just for instance). Read the rest of this entry »

I wrote a post last week about how to write functions for use in the PowerShell pipeline and I’ve been using the template I wrote in that post as the basis for several of my other scripts … and I’ve been gradually fleshing it out, and improving it, so I thought I’d drop it here with all of it’s inline comments. Hopefully that will be better than what you would get if I just trying to explain it in a blog post .

Incidentally, this script is also on PowerShellCentral scripts Repository where I will probably post any future modifications … Read the rest of this entry »

[new] Update, I created a better version of a pipeline function for powershell …

Every once in a while the question of how to best use the process block of a function to process pipeline objects comes up on IRC, and although I’m sure others have already written this up on the web in the past, we’ve been polishing up this example sending it back and forth to each other for a couple of weeks, and it seems to me it’s about time to publish it a little more prominently.

The basic idea was to write a function that could be used to process a set of inputs from either the pipeline or an argument, while still allowing other arguments to be passed and processed. This is something that’s very easy for a cmdlet, because you can specify that a certain parameter will receive pipeline input (and even control which attribute of the objects on the pipeline will be used), but in a script it’s not really supported. In addition, we require that the function process items as they come in, rather than waiting until it’s received all input before processing them.

Some explanation

This is just an example method, it doesn’t really do much, except that it outputs exactly the same thing whether you call it like Test-Pipeline -inputObject "Foo","Bar","Baz" "Kill Each Process" or like: ""Foo","Bar","Baz" | Test-Pipeline "Kill Each Process".

It works by calling itself with the inputObject argument passed on the pipeline when you specify it as an argument, so that code is processed exactly right. This means that the first line you see in the process block must remain the first line, even if you add additional code to the block.

We use a trick to make sure that additional unnamed parameters work normally by using a switch parameter -inputObject followed by a second parameter $io which we treat as inputObject if inputObject is set. This works because switch parameters don’t expect values, so we’re able to sort-of hijack it to emulate the cmdlet behavior. Note that if it is not set, we move the value of the $in parameter into the $args array which contains any additional parameters. This arrangement allows it to behave rather like a cmdlet would, but it’s a little delicate: if you’re going to pass input objects, you must specify the switch as the first argument, and pass the $io (inputObjects) immediately following (as though they were the value of the inputObjects parameter). All of that is necessary so that we can optionally pass additional parameters.

You can also add additional named parameters — but you would need to either always specify their names when you called them, or carefully adjust the code so that they get shifted if -inputObject isn’t specified.

Here’s some sample output, in case you’re wondering:

Credit where it’s due

I wanted to officially acknowledge that /\/\o\/\/ and Gaurhoth and Oisin and Brandon have all contributed to the polishing and testing of this script in various forms … thanks to all