Before I give the solution, I just want to say: don’t change your output based on where you are in a pipeline.

There are numerous scenarios where your function will be the last one on a pipeline, but still be participating in further pipelines, including formatting and output modification. For example, take our function Test-Pipeline (defined below) in these three scenarios below. In none of these scenarios would it be appropriate for the function to write formatted output instead of outputting the raw object, but in each case, the function is the last function in the pipeline.

However, if you want to determine your function’s position in the pipeline for some other reason, the answer is simple. You need to use $MyInvocation and compare the PipelineLength and PipelinePosition properties:

Of scripts and functions

The first thing to know is that in PowerShell, there’s really no difference between a script (just a file with a .ps1 ending) and a function as you’ll see written below. If you take a function Get-Square, and remove the first and last lines ( Function Get-Square { … } ) you could put them in a file called “Get-Square.ps1” in your PATH, you use them exactly the way you would the function that was pre-loaded into memory by dot-sourcing or pasting it on the command line. Of course, doing that easily requires writing the function parameters on their own line using PARAM(...) syntax, which is why I recommend doing that.

Of functions and the pipeline

When your script or function is used on the pipeline its begin block is called once when the pipeline starts up, and then the process block is called repeatedly: once to process each pipeline object, and finally, the end block is called after all the objects have been processed through the whole pipeline. If you don’t understand that, you should play with this function, try calling it by passing a series of numbers through multiple instances of it:

Incidentally, if you don’t specify the begin, process, or end blocks, the body of your function is treated as the end block. This is so that you can use the special $Input variable, which collects all the things passed in on the pipeline, and thus only works in the end block. That would allow the following function to behave the same way regardless of whether it was invoked on the pipeline or by passing an $InputObject. Notice, however, the difference between this, and the function above.

Our challenge

The basic idea here is to rewrite that function such that it can be used to process a set of numbers from either the pipeline or an argument, without interfering with the processing of other parameters. We require that the function process items as they come in, rather than waiting until it’s received all input before processing them the way Test-SquareEnd does.

In PowerShell 2.0 this would be easy

In the current CTP 3 of PowerShell 2, you just specify ValueFromPipeline=$true for the parameter you want to set from the pipeline, and the function will work the same way whether you pass the numbers as a parameter or along the pipeline — you can even control which attribute of the objects on the pipeline will be used, but that’s a whole other article.

Of course, PowerShell 2.0 is still in beta status, and even after it’s released you may need to write scripts that are backwards compatible to PowerShell 1.0, and the excercise of doing so may help you to understand more about how PowerShell functions work, and particularly how they behave in the pipeline.

Our solution

Since in a PowerShell 1.0 function it’s not really supported directly, we need to do some extra work:

An explanation

Of course, this is just an example method, squaring things isn’t that exciting — but what’s special about it is that the output is almost exactly the same whether you call it with parameters Get-Square 1,2,3,4 or on the pipeline: 1,2,3,4 | Get-Square.

The trick is that it actually executes the same way in either case:

1. If you call it by passing the int (or array of ints) as an argument ($InputObject), it calls itself and passes those values on the pipeline.
2. When the integers are passed on the pipeline, the special pipeline iterator variable $_ is set, and the process block is executed.

1. When it has to (re)invoke itself, it passes any other parameters as parameters, which means you need to have default values for them.

There is one tiny difference in the processing, which in real-world use is practically never noticeable (you can see it in our example if you call it like this):

You’ll see that unlike the PowerShell 2.0 pipeline function, when you pass the numbers as a parameter, the first one of them actually gets passed through the process block before the begin block of the second function on the pipeline is called. This usually doesn’t have any effect, but it’s something to keep in the back of your head.

A few precautions:

• You have to default values for parameters, because you’ll be passing them all as named parameters, in the re-invoke step, and $null can cause problems.
• If you need to do additional processing in the begin block, you should only do so in an ELSE case: when $InputObject is null. That way, the code will only execute once each time you call the function.
• The same goes for the end block: you have to keep your code in an If(!$InputObject) block to avoid executing it twice (when you pass the values as an argument, and it re-invokes itself).

• The test cases in the process block must wrap all of your process block code, so that you don’t process the arguments twice, and you shouldn’t refer to $InputObject, but instead should use the automatic $_ variable which is the value passed when the function is (re)invoked via the pipeline.

Here’s some sample output, in case you’re wondering:

WPF & PowerShell – Part 5 has a script for “Get-Listbox” and for “Show-Control” and Halr9000 wrote a script he called Get-PSBlogroll

I had modified the example from the WPF post to create a listbox which will “start” whatever you double click …

And I tried something like this to let me launch links from hal9000’s blogroll:

Of course, what I’d like is to see the titles, instead of the URL, but because of how Get-ListBox was written, it only accepts strings, and therefore only returns strings. I tried adding the URL as a NoteProperty on the title string, but there’s a bug in the script cmdlet implementation that strips ETS properties. So if you specify your parameter type as a [string], you loose the NoteProperty.

Idea 1: Use [PsObject] for your pipeline parameter

Of course, if they fix the bug this tip will go away, but in the meantime, you’re better off not casting the input from the PowerShell native [PsObject] type, because you loose any extended type system attributes (and if you cast XML to a string, it comes out yucky).

Idea 2: Don’t output xml nodes

Even after fixing the Get-Listbox script so it outputs what it gets in without casting it, I still have to craft PsObjects with the output from Halr9000’s script, because otherwise Get-ListBox just outputs the .ToString() representation of the xml node, which is invariably useless — I could data-bind to the xml attributes, but it requires a separate syntax in the XAML — which means you’d have to have separate code for xml vs objects. So basically, you’re better off with a | Select * at the end of your Get-PsBlogRoll.

Idea 3: Display scripts should take a property (or a property list)...

What I eventually ended up doing is rewriting Get-Listbox to take a “Properties” parameter so that I could specify which of the attributes of an object I wanted to use … that way, I can pass in the almost unmodified output from Get-PSBlogRoll into the new Get-ListBox, and run with it. In the end, it looks something like this:

Of course, because of how I wrote the DataTemplate for the properties (using a StackPanel and a loop to output multiple TextBlock objects) you could choose to show the Url as well, and all you’d have to do is change the call to Get-Listbox like this: Get-Listbox title,htmlUrl … which is pretty cool. Does anyone else have any ideas for making these even easier to compose?

A really nice trick would be if the wpf cmdlet could use Write-Output in the click handler and actually yield to allow the rest of the pipeline to process that item, but so far, I haven’t been able to find a way to do that (short of closing the WPF window, of course).

user: Can you explain the distinction between $_ and a param in a script context?
Jaykul: I sure can. And I think I can do it simply … take an example script|function … actually, we’ll just use a scriptblock
Jaykul: it can have pipeline blocks, or not (the pipeline blocks are the BEGIN{...} PROCESS{...} END{...})
user: yeah, I’m with you
Jaykul: so if you pass a value via an argument, it’s present in all three blocks, and has the same value the whole time (unless you change it in your script)
user: really
Jaykul: But when you put that script in a pipeline, your BEGIN {} block is called first, and then for EACH item in the pipeline, powershell sets $_ and calls your PROCESS block, and finally, your END{} block is called after all pipeline items have been processed
Jaykul: Here, check this out (run it so you can see)

user: that really helps, thank you
Jaykul: no problem. Incidentally … in a script-block like that, you can use the Params() statement, but I wanted to loop over $args
user: based on that I’m left wondering what the value of parameters is.
user: If I’m writing a script, chances are that I’m going to want to use it in a pipeline
user: I guess I can use parameters, but put it in a foreach-object block
user: $arr | %{myscript $_}
Jaykul: Well, if you use the foreach-object block and pass arguments, your script doesn’t persist through the life of the pipeline, so you can’t do things like counting, or comparing to the previous object, etc.
Jaykul: But you can only pass one thing at a time via the pipeline, so you still need arguments in a script like the one I just used to paste that script to the pastebin (which is here: http://huddledmasses.org/powershell-send-paste-script/ )
user: right. So I guess the concept is that scripts grab stuff off the pipeline themselves?
Jaykul: sort-of, yeah. I mean, it’s actually PowerShell that does it, but basically, the process block grabs each item off the pipeline without needing it to be specified as an argument.

Jaykul: In v2, you can be more explicit about this, because you can write a “cmdlet” in script which lets you specify parameters which should get the pipeline value
user: okay, I totally get it now. That’s why you’d write a cmdlet instead of a script?
Jaykul: Yeah! The particularly cool thing about cmdlet parameter handling … is you can do this:
Jaykul: [Parameter(FromPipelineByPropertyValue)] [Alias(“FullName”)] $FileName
Jaykul: a parameter like that would accept a string as a parameter, but if you did gci * | scriptcmdlet instead of getting the [System.IO.FileInfo] object you would get the STRING corresponding to that object’s FullName property
user: right. sweet!
Jaykul: so you can even do tricky stuff like getting multiple properties from it (as different parameters to your script)

user: I noticed from http://powershellcentral.com/scripts/184 that you’re testing the type of the pipeline object
Jaykul: yeah
user: FIleInfo vs. string
Jaykul: because it lets me do this: ls *.ps1 | Send-Paste
Jaykul: or this: get-history -count 5 | % { $_.CommandLine } | Send-Paste
Jaykul: or even send multiple files as a single paste, like: get-content *.ps1 | Send-Paste
Jaykul: (note that in this script, that’s VERY different than ls *.ps1 | Send-Paste …. which would create a paste for each file)
user: wow, this is good stuff, thanks for the time…

Dmitry was trying to count the number of users in Active Directory, and ran into ridiculous memory use when using Measure-Object because it appears to collect all the items in the pipeline into memory before it counts, resulting in huge amounts of memory use based on the number of objects you’re counting. Of course, for the purpose of counting, it’s a pretty obvious fix (as Dmitry explains), but for other purposes it may not be as obvious.

Select Random

When I originally started this post, I was trying to upgrade my original Select-Random script (which was published in the PowerShell Community Extensions) to use reservoir sampling in an attempt to solve the same problem. Select-Random is a script which selects a random element from a collection or from the pipeline. In the past, if the elements were passed in as an argument, it simply collected them into an array (in a really inefficient way, but never mind) and then picks a random number and selects the item by index.

I started by improving the collection method to use an ArrayList and reduced the memory usage to about 1/5th of the original method, and then rewrote the whole script to use reservoir sampling instead. Greg wrote a good explanation of reservoir sampling but basically it’s an algorithm for choosing random items from a collection of unknown size without having to traverse the collection twice (to count it the first time and then to select) — in my case, the need to count the collection was requiring me to store the pipeline input in an array, so the improvement was more about memory use than about needing to enumerate the list twice. However, there is a downside: it’s slower.

Reservoir Sampling

Reservoir sampling means that instead of generating a single random number and indexing into the collection, you sample as you iterate. You start by filling your “reservoir” with the first items in the pipeline, and then for each item remaining in the pipeline, you generate a random number with a decreasing probability of choosing the item. You’re trading memory usage for speed. Basically, if you want $count items, then the script stores $count elements (the eventual result) at all times. It continues processing elements until it reaches the end of the input.

• For each input element $n (where n is the count of the inputs so far) there is a $count/$n chance that it becomes part of the result.
• For each previously selected element, there is a $count/($n-1) chance of it remaining selected (ie: not being replaced).

• For the ones selected, there’s a ($count/$n * 1/$count = 1/$n) chance of it being replaced, so a ($n-1)/$n chance of it remaining … thus, it’s cumulative probability of being among the selected elements after the nth input is processed is $count/($n-1) * ($n-1)/$n = $count/$n, as it should be.

The bottom line is that using this method is a LOT slower than random indexing. However, it uses a LOT less memory when iterating large numbers of items that are output by cmdlets as long as they weren’t already collected into an array. As a simple example, Get-ChildItem outputs each item as it traverses the provider, rather than creating an in-memory collection. If nothing in the pipeline “collects” all of the output, then it never needs to use memory except for the text display of them … or if it’s piped through a reservoir sampling algorithm, it only uses the amount of memory required to store as many items as the size of our sample.

Performance comparison

Using as an example the output of Get-ChildItem -Recurse on my profile directory, where there are about 98 thousand items total in the subdirectories, I ran each test in it’s own clean console so that I could see the results of the memory usage cleanly. So that you could tell the difference, I changed the letters in the prompt for each test (I also use a counter in my prompt, and I change the character from > to : to make my life easier when pasting into HTML).

In case you’re not familiar with the commands used here, Measure-Command is a cmdlet that times how long a script takes to run, and if you wrap it in a string, it outputs a simple elapsed time statement, instead of more verbose output. There are other ways of accomplishing the same thing, but this is the least distracting way. Also, -f is the format operator, so putting "{0}" -f 42 is the same as @[string]::format(”{0}”,42) but in less
code.

Hopefully I didn’t make that example too complicated, and you can see not only that collecting the output into an array is faster, but that is uses a boatload more memory. You can also see that my -collect method (which now uses an ArrayList (see the code below)) uses a lot less memory than collecting the items in an array the way the first example does. I’m not 100% sure why this is, but my theory is that doing (ls -recurse) is in effect the same as using the += operator on an array — it makes a copy. To be more precise, it makes about 98 thousand copies.

You’re not getting quite the whole picture in that example, but to fill it in a little: the memory use starts at around 39 MB … JUST doing the Get-ChildItem to count the files increases the working set memory usage of the PowerShell process by about 3.5 MB. Using the reservoir method to select 5 of them increased the memory usage of the PowerShell process by about 5 MB of memory (only 2 MB more than simple counting). Using the pre-collection method requires around 151 MB. That’s a really heavy price to pay in memory, at over 1.5 KB per item in the collection. And even the -collect method uses almost 10x as much memory.

Some perspective

To make sure you have a chance to get the same perspective I have on this … let me show you the output another way. The key thing to see in these last two examples is that the memory cost is the same for both — the same 151MB of the earlier collection example. The difference in time is large and undeniable. If you compare this with example A, you can see that basically, listing the files takes 32 seconds, and picking 25 of them takes about a tenth of a second, while iterating them (which requires generating about 98 thousand random numbers) takes about 20 seconds.

Conclusion

So, the point is: there are times when you need to sacrifice speed in order to keep your memory usage low … But when you’re writing a general-purpose script or a cmdlet for the PowerShell pipeline, you need to at least understand the choice you’re making so you can make it deliberately. Sometimes you’ll choose to write it fast and memory-expensive. Sometimes you’ll choose to write it slower and memory-lean. And sometimes, like I did here, you might choose to do both .

The script …

It’s also script 118 on the PowerShell central script repository (where it has a comment header that’s giving me problems here).

You know, I’ve looked at your articles about cmdlets/functions in the pipeline and I feel you’re missing something. The purpose of the InputObject parameter is to pass in a collection as a single object. This is as opposed to using the pipeline where a collection is passed along the pipeline one item at a time. There are cases where you want to pass in a collection as a collection.

Quite simply, I disagree. The documentation for these parameters says quite clearly that inputObject “Specifies an object or objects to input to the cmdlet.” This clearly means that I should be able to pass multiple objects, and have them treated as multiple objects, not as a single array object.

If you look at your example (Select -First 3 -Unique -InputObject $a), this does in fact work. It receives one object, an array. It then selects the first 3 objects, but there is only 1 so that is moot. And lastly it selects unique objects, but again there is only 1 so that is moot as well and finally the object is output using the default formatter. In this case the default formatter is showing the contents of the array.

In this example, Select-Object has no reason to take a single object as an input object, at all. The only time that it would be useful for Select-Object to take a single inputObject would be in combination with the property parameters. In fact, if you want to Select-Object from an array to get the first of last n objects, or to get a set of unique objects, you have to pass the objects in via the pipeline — there’s no other way to make it select from an array. If that was indeed the intent, it should have been written as a separate ParameterSet, and the documentation should be changed to reflect that only a single object can be passed in, and that you can’t use the inputObject parameter with the first, last, or unique parameters at all. That’s worse than useless, it’s misleading and confusing.

Kirk is absolutely right that if you assume that the InputObject argument is only allowed to take a single object, then the behavior is correct – but it’s not logical. In fact, the behavior you see in the output of this command is so useless as to be a bug – even if the documentation did not say the parameter accepts multiple objects as input:

I know that that’s the way the built-in cmdlets work.

But quite frankly, just because someone important wrote something useless is no reason to emulate the behavior. The inputObject parameter IS the same parameter which pipeline objects go into. There’s no logical explanation for us to get different results when we pass an array in via the parameter by name instead of via pipeline: the PowerShell pipeline passing the things in the pipeline into the –inputObject parameter … it’s not using some mystical variable like it does in script functions.

Of course, we all know the powershell pipeline unwraps arrays — that’s convenient, and we can work around it when we really want to pass an array in:

My point in all of this is that InputObject is actually a very useful parameter, because there are cases where you really want to pass a collection as a collection into a cmdlet and then do something with it. By making InputObject instead split the collection passed in and pipeline it through, you’re forcing users to wrap collections in an array just to get them passed in as a collection, and personally I don’t feel they should have to do that.

While it’s true that passing in an array is sometimes desirable that’s not the reason the parameter exists, and I don’t believe it should be the default behavior here. It should be just as easy for me to use the cmdlet with the inputObject parameter directly as it is to input them via the pipeline. If I put in unwrapping for the inputObject parameter, you can work around it in the same way I did in the examples above. Incidentally, I think *PowerShell* should unwrap arrays to ValueFromPipeline parameters regardless of whether they’re on the pipeline, but I recognize it’s probably too late for that.

Basically, this is my argument: If inputObject unwraps arrays, the syntax for passing an array by wrapping it in @(,$array) is simple, for those rare occasions when that’s actually what you want. But if it does not unwrap arrays, you’re forced to call it via a separate pipeline, because unwrapping the array and passing it in one at a time in a foreach loop will almost certainly not do the same thing, and this is much uglier — and not compatible with use within the pipeline, particularly if you need to pass the pipeline output into a different parameter.

I guess my final word would be to agree with Kirk that “InputObject … isn’t documented clearly enough” … in fact, it’s clearly behaving incorrectly according to the documentation, and that’s why I originally proposed to unwrap the inputObject parameter when it’s passed as a parameter: to make it work the way the documentation suggests it would, which seems to me to be a better way than the way it actually works.

There are a few caveats still (Parameter Sets aren’t working yet, and neither is help, really), and a few surprises … there’s a few downsides to PowerShell script vs C# ... but in this particular context one thing that stands out is that in C# the BeginProcessing, ProcessRecord, and EndProcessing blocks are actually methods which can call each other, and as demonstrated in my tutorial for writing cmdlets that work in the pipeline, they can be recursive — without getting duplicate variables.

A sample Script Cmdlet

In the interests of being the first to publish an interesting script cmdlet and to continue my recent trend of talking about writing for the PowerShell pipeline, I’ve merged the logic of my script function and my pipeline cmdlet into a single sample script cmdlet for PowerShell 2.0 and it works great!

A few observations from the process, in no particular order:

• If you recursively call your cmdlet from within itself, you have to test for parameters using the new $CommandLineParameters.ContainsKey because parameter variables keep their values through recursion if you don’t explicitly pass a value.
• $CommandLineParameters.ContainsKey works differently in the Begin block where it will return $false for arguments which will get their values from the pipeline, than in the Process block where it will treat values which were passed as CommandLineParameters the same as those which were passed via the pipeline.
• If you want to see how your function behaves in a pipeline, you should make sure to test it at different points in the pipeline: at the front, in the middle, and at the end.
• Cmdlets are functions: they show up in the Function provider.
• Cmdlets are functions: they have to be dot-sourced before you can call them.
• Cmdlets are not functions: they are a single command Cmdlet which takes a name (which must have a – in it) and a couple of other parameters followed by a function script block.

• When you recurse by executing &($MyInvocation.InvocationName), that second invocation has an InvocationName of “&” ... so you can’t go any further (this might be a good thing, if you want to stop recursion at one level no matter what. If you want to go further, you need to put your commands into a string, and use Invoke-Expression.

A test case

To demonstrate the problem, try this:

This should expose two weirdnesses about how the Select-Object cmdlet works:

1. The -First parameter affects the input before the -Unique parameter does.

1. When you pass the input in via -InputObject, the whole array is treated as a single object, and the command basically doesn’t do anything.

The big problem with this behavior is that there’s essentially no hint that you’ve done something wrong — there’s actually no way to make Select-Object work properly except by passing the objects in via the pipeline. The bigger problem is that it would have been simple for the Microsoft team to catch this and alert you, but they didn’t — so you probably won’t even notice there’s a problem until you run it on a trivial data set like my example. The even bigger problem is that it doesn’t just affect Select-Object (try it with Where-Object, just for instance).

The simplest fix

When this came up in the #PowerShell IRC channel Oisin initially defended this as an unavoidable side effect of the way the cmdlet system works. However, after playing with the idea for a bit, we found it’s actually trivial to stop, although I found it hard to explain without actually demonstrating an alternative. The simplest possible alternative is just to throw an exception if the value is passed in as an argument instead of via the pipeline. That would preserve the same level of functionality you have now — but cause an error in those cases where it wouldn’t work anyway.

A better way to handle input

Of course, you can do better than that . So, I hereby present the first version of my PowerShell Pipeline Template Cmdlet. It’s pretty simple really (once you get past all the cmdlet overhead): basically, you check in the BeginProcess() method to see if the InputObject parameter has been set, and set an alternate private variable. Then, in the ProcessRecord() method, we have two alternate computation paths: the normal path, and a second path for when the collection is passed in as an argument. In that case, you recurse and call the ProcessRecord method once for each item in the collection.

I’m sure some of you will have some improvements you can make, feel free to continue the development on the PowerShell Central scripts page or by sending feedback in the form below, but for now, here’s the Test-Pipeline Cmdlet Binary and the source code.

The Code

Incidentally, this script is also on PowerShellCentral scripts Repository where I will probably post any future modifications …

Every once in a while the question of how to best use the process block of a function to process pipeline objects comes up on IRC, and although I’m sure others have already written this up on the web in the past, we’ve been polishing up this example sending it back and forth to each other for a couple of weeks, and it seems to me it’s about time to publish it a little more prominently.

The basic idea was to write a function that could be used to process a set of inputs from either the pipeline or an argument, while still allowing other arguments to be passed and processed. This is something that’s very easy for a cmdlet, because you can specify that a certain parameter will receive pipeline input (and even control which attribute of the objects on the pipeline will be used), but in a script it’s not really supported. In addition, we require that the function process items as they come in, rather than waiting until it’s received all input before processing them.

Some explanation

This is just an example method, it doesn’t really do much, except that it outputs exactly the same thing whether you call it like Test-Pipeline -inputObject "Foo","Bar","Baz" "Kill Each Process" or like: ""Foo","Bar","Baz" | Test-Pipeline "Kill Each Process".

It works by calling itself with the inputObject argument passed on the pipeline when you specify it as an argument, so that code is processed exactly right. This means that the first line you see in the process block must remain the first line, even if you add additional code to the block.

We use a trick to make sure that additional unnamed parameters work normally by using a switch parameter -inputObject followed by a second parameter $io which we treat as inputObject if inputObject is set. This works because switch parameters don’t expect values, so we’re able to sort-of hijack it to emulate the cmdlet behavior. Note that if it is not set, we move the value of the $in parameter into the $args array which contains any additional parameters. This arrangement allows it to behave rather like a cmdlet would, but it’s a little delicate: if you’re going to pass input objects, you must specify the switch as the first argument, and pass the $io (inputObjects) immediately following (as though they were the value of the inputObjects parameter). All of that is necessary so that we can optionally pass additional parameters.

You can also add additional named parameters — but you would need to either always specify their names when you called them, or carefully adjust the code so that they get shifted if -inputObject isn’t specified.

Here’s some sample output, in case you’re wondering:

Credit where it’s due

I wanted to officially acknowledge that /\/\o\/\/ and Gaurhoth and Oisin and Brandon have all contributed to the polishing and testing of this script in various forms … thanks to all