I’m here to say: it’s not so much limited, as it it cautiously restrained. A lot of people were apparently hoping for a SQL Server provider like this demo one that could fully navigate tables and list their contents etc., but I think the SQL Team has decided not to allow that, and I think they probably did the right thing.

Honestly, a provider that can navigate into SQL datatables would be extremely frightening, to the point that it might need to be banned or blocked in some instances. Just imagine a SQL Server dba who’s brand new to PowerShell, but thinks that new provider sounds interesting … so he fires up PowerShell and switches over to the customer orders table (you know, the one with 2 million records of purchases) and does:

Just think about that line for a second. Not only is it fetching every record in the table, it has to build a .net object for every single one. One of the most commonly used cmdlet’s in PowerShell is Get-Member, which we typically see used when you’re trying to figure out which properties are available on an item. Applied to the previous example, you can imagine doing something like this:

Now you’ve retrieved every record from the table … just for the sake of finding out what the columns are! You’re not even using the data. If you’re lucky, the memory use of creating 2 million System.Data.Row objects would cripple your dba’s PC before they were able to try something like this:

Anyway. Hopefully this counter example will be enough to not only make you feel better about using a cmdlet instead of a provider, but also to make some people think about how they choose to design things that can cause potentially crippling network traffic or server load. The thing is that although it’s possible to allow the -Filter parameter of Get-ChildItem (ls) to specify both a WHERE clause and a TOP count … that wouldn’t help unless you were sure your users would always filter that way instead of by passing it through | Where-Object.

As a side note: The management studio protects you from this sort of load on your system to some extent by caching data and limiting how many rows it pulls over at a time, but not as much as it could if it allowed you to reorder columns and sort them after you had retrieved the data. It also has the distinct advantage of not necessarily having to create .Net objects to pass into the pipeline for each row on the fly (although, maybe that’s how it works, I don’t know, but I would hope not).

Anyway, from what I’ve understood, the SQL powershell provider will let you navigate the schemas, and will provide some cmdlets to do queries that should replace some of the functions we’ve all been using to do queries. Maybe they’ll even provide something that can do built in paging, like: Get-SqlQuery “Select * from Northwind.Orders” -Next 10 …

I’d love to hear some thoughts on why I got the speed results I got here. At first I thought it was something about type coercion, but as you can see, I tried casting everything to no avail. There’s no difference if I don’t output anything, and in fact, the non-recursive algorithm can trivially output every Fibonacci number in the sequence without affecting it’s time (I actually altered it so it didn’t to keep the output clean).

Obviously I expected the recursive algorithms to be slow and scale badly because of the amount of work involved, but how is it that the pipeline-recursive “filter” outperforms the plain old recursive function (albeit by an infinitesimal fraction)?

P.S.: The numbers in braces with the colon (like [18]: ... ) is my prompt.

Dmitry was trying to count the number of users in Active Directory, and ran into ridiculous memory use when using Measure-Object because it appears to collect all the items in the pipeline into memory before it counts, resulting in huge amounts of memory use based on the number of objects you’re counting. Of course, for the purpose of counting, it’s a pretty obvious fix (as Dmitry explains), but for other purposes it may not be as obvious.

Select Random

When I originally started this post, I was trying to upgrade my original Select-Random script (which was published in the PowerShell Community Extensions) to use reservoir sampling in an attempt to solve the same problem. Select-Random is a script which selects a random element from a collection or from the pipeline. In the past, if the elements were passed in as an argument, it simply collected them into an array (in a really inefficient way, but never mind) and then picks a random number and selects the item by index.

I started by improving the collection method to use an ArrayList and reduced the memory usage to about 1/5th of the original method, and then rewrote the whole script to use reservoir sampling instead. Greg wrote a good explanation of reservoir sampling but basically it’s an algorithm for choosing random items from a collection of unknown size without having to traverse the collection twice (to count it the first time and then to select) — in my case, the need to count the collection was requiring me to store the pipeline input in an array, so the improvement was more about memory use than about needing to enumerate the list twice. However, there is a downside: it’s slower.

Reservoir Sampling

Reservoir sampling means that instead of generating a single random number and indexing into the collection, you sample as you iterate. You start by filling your “reservoir” with the first items in the pipeline, and then for each item remaining in the pipeline, you generate a random number with a decreasing probability of choosing the item. You’re trading memory usage for speed. Basically, if you want $count items, then the script stores $count elements (the eventual result) at all times. It continues processing elements until it reaches the end of the input.

• For each input element $n (where n is the count of the inputs so far) there is a $count/$n chance that it becomes part of the result.
• For each previously selected element, there is a $count/($n-1) chance of it remaining selected (ie: not being replaced).

• For the ones selected, there’s a ($count/$n * 1/$count = 1/$n) chance of it being replaced, so a ($n-1)/$n chance of it remaining … thus, it’s cumulative probability of being among the selected elements after the nth input is processed is $count/($n-1) * ($n-1)/$n = $count/$n, as it should be.

The bottom line is that using this method is a LOT slower than random indexing. However, it uses a LOT less memory when iterating large numbers of items that are output by cmdlets as long as they weren’t already collected into an array. As a simple example, Get-ChildItem outputs each item as it traverses the provider, rather than creating an in-memory collection. If nothing in the pipeline “collects” all of the output, then it never needs to use memory except for the text display of them … or if it’s piped through a reservoir sampling algorithm, it only uses the amount of memory required to store as many items as the size of our sample.

Performance comparison

Using as an example the output of Get-ChildItem -Recurse on my profile directory, where there are about 98 thousand items total in the subdirectories, I ran each test in it’s own clean console so that I could see the results of the memory usage cleanly. So that you could tell the difference, I changed the letters in the prompt for each test (I also use a counter in my prompt, and I change the character from > to : to make my life easier when pasting into HTML).

In case you’re not familiar with the commands used here, Measure-Command is a cmdlet that times how long a script takes to run, and if you wrap it in a string, it outputs a simple elapsed time statement, instead of more verbose output. There are other ways of accomplishing the same thing, but this is the least distracting way. Also, -f is the format operator, so putting "{0}" -f 42 is the same as @[string]::format(”{0}”,42) but in less
code.

Hopefully I didn’t make that example too complicated, and you can see not only that collecting the output into an array is faster, but that is uses a boatload more memory. You can also see that my -collect method (which now uses an ArrayList (see the code below)) uses a lot less memory than collecting the items in an array the way the first example does. I’m not 100% sure why this is, but my theory is that doing (ls -recurse) is in effect the same as using the += operator on an array — it makes a copy. To be more precise, it makes about 98 thousand copies.

You’re not getting quite the whole picture in that example, but to fill it in a little: the memory use starts at around 39 MB … JUST doing the Get-ChildItem to count the files increases the working set memory usage of the PowerShell process by about 3.5 MB. Using the reservoir method to select 5 of them increased the memory usage of the PowerShell process by about 5 MB of memory (only 2 MB more than simple counting). Using the pre-collection method requires around 151 MB. That’s a really heavy price to pay in memory, at over 1.5 KB per item in the collection. And even the -collect method uses almost 10x as much memory.

Some perspective

To make sure you have a chance to get the same perspective I have on this … let me show you the output another way. The key thing to see in these last two examples is that the memory cost is the same for both — the same 151MB of the earlier collection example. The difference in time is large and undeniable. If you compare this with example A, you can see that basically, listing the files takes 32 seconds, and picking 25 of them takes about a tenth of a second, while iterating them (which requires generating about 98 thousand random numbers) takes about 20 seconds.

Conclusion

So, the point is: there are times when you need to sacrifice speed in order to keep your memory usage low … But when you’re writing a general-purpose script or a cmdlet for the PowerShell pipeline, you need to at least understand the choice you’re making so you can make it deliberately. Sometimes you’ll choose to write it fast and memory-expensive. Sometimes you’ll choose to write it slower and memory-lean. And sometimes, like I did here, you might choose to do both .

The script …

It’s also script 118 on the PowerShell central script repository (where it has a comment header that’s giving me problems here).