How do I find an item (in an array) based on one of it’s properties?

Actually, the question was rather more complicated than that. They were importing a bunch of users from a csv file, and wanted to sort them and search them based on a specific column. There are many ways to skin this cat. Imagine that you have a CSV file, and have imported it, like so:

Now, imagine that the CSV file has thousands of users in it, and that you need to not only sort the data by first name or last name on demand, but you also need to pull users from the list (by name) on demand.

These are trivial tasks in PowerShell:

This works but will have some performance issues when the list gets too long, because your search algorithm is … well, we could say it’s O(n), or just say it’s slow, because you’re basically going through the whole list every time. And I do mean the WHOLE list. You were looking for “Bob” ... but the Where-Object cmdlet doesn’t know your data is sorted, and doesn’t know you only want one user, so it’s going to iterate the whole list, because Select-Object can’t signal it to stop.

There are several ways to solve this, but what it comes down to is that you need a (better) search algorithm than Where-Object can implement: simply put, you’d like to use a binary search, since the data is already sorted.

Binary Search

I’m not going to get into this much, but to put it simply: a binary search algorithm is pretty much the fastest search you can have, but it depends on the data being pre-sorted in order. It works by starting in the middle, and comparing the middle item to the item you are searching for. If what you’re searching for comes BEFORE that item, then it discards the second half of the items, and examines the middle item of the first half. In this way, it eliminates half the items each time, and returns the correct item (if it exists) in roughly (log n) steps.

So how do we do that in PowerShell?

Well, that’s easy: Just use the static method: [Array]::BinarySearch… The only problem is, in order for BinarySearch to work, the items have to be not only sorted, but comparable. Because we’re trying to compare PSObjects, you can’t just compare two items from the $users array and determine the correct order…

If you wanted to use [Array]::Sort you could use an overload that lets you pass in a second array (of comparable items) to serve as the keys for the non-comparable items, but this won’t work for BinarySearch:

So you need what is an “IComparer.” That is, a class which implements a “Compare” method which can be used to determine the order of the objects. Sadly, this requires a custom type — something you can’t do in pure PowerShell script, believe it or not. PowerShell does not have the ability to create custom classes (which is how you can tell it’s not a real programming language), so when you need a custom type in PowerShell you have to drop to another .Net language like C# or Python.

In PowerShell 2.0 there is a cmdlet “Add-Type” which can take a string representing C# source-code, and compile it in memory for immediate use, but in PowerShell 1.0 you need to have a script version of it, which I called New-Type, to avoid confusion because it only handles one of Add-Type’s four usage models:

Once you have Add-Type, you can create a ScriptBlock-based implementation of IComparer, by passing this as a string:

So now you have everything you need, and you can write your custom comparer, and do binary search. The one thing you need to be aware of is that when you’re doing binary search, you’re going to want to pass in the thing you’re searching for (ie: the first name) not an Object with a FirstName property. So your binary search needs a case for when the second parameter is a string:

Now, you may not think this is a big deal, so let me just share with you the results of searching a collection of about 49000 users:

What you see here is two things:

1. Searching for a single user using Where-Object takes pretty much as long as (re)sorting the whole array, because every single item must be checked.

1. Using BinarySearch rocks

BinarySearch is not always faster: If your array is initially unsorted, the requirement to pre-sort the array costs basically the same as any speedup you might get for a single search. Also, because of the call to the script, if the array is very small, like the four users in the sample above, it’s actually faster to just loop through them (20ms versus 24ms on my PC). The BinarySearch starts paying off pretty quickly as you scale up the number of searches and records, but it does take both more records (the break-even point for just the search is about 10, in this contrived example), and more searches (if you have to eat the cost of the initial sort).

In any case, New-Type certainly has many other uses, and the ScriptBlockComparer can be used for other things as well … although I will say that as far as I can tell, sorting is almost always better performed with the sort cmdlet than with [Array]::Sort and an IComparer.

I’ve fixed that, and weighted the search so that words in the title count more than words in the code by creating a relevance column by hand:

Incidentally, the FULLTEXT index means that words shorter than 4 characters don’t count (I’m going to try to get this changed, but it’s an option for MySQL, so it has to be changed in the config file) in the meantime you can search for words using the wildcard character, like: SQL* and it sort-of works. The PoshCode cmdlet actually was adding *‘s to the query (although I’ve just decided that’s not a good idea, because it means that queries from the cmdlet appear to have different results than queries on the website.

MySQL’s FULLTEXT BOOLEAN search has all sorts of features (and limitations): there is a stopword list, maximum and minimum word lengths, and all sorts of operators for setting word precedence or negating words, or weighting them negatively … to REQUIRE that a word be present, it must have a + in front, and in order to mark a word as more important, you have to put > in front, not just put it first… I’ve been thinking about trying to apply a few of those tricks myself (eg: put * on words under four characters, and put > on the first 30% of words and < on the last 30% to try to simulate weighting them …) but my original feeling was that the search is more powerful if you just know that it’s a fulltext boolean search and can write your queries accordingly.

If anyone has any ideas for how to improve search in MySQL … or opinions on whether I should try to apply boolean operators to queries which don’t already have them … please let me know.