In PowerShell, Get-Command is a command that serves two purposes: first it lets you search for commands using verb, noun, wildcards, module names etc. and then it also returns metadata about commands. In PowerShell 2, it could only search commands that were in modules (or snapins) you had already imported, or executables & scripts that were in your PATH.

So here’s the deal: Get-Command has always behaved differently when it thinks you’re searching. The only way it can tell that you’re searching is that you don’t provide a full command name. So, if you use a wildcard (e.g.: Get-Command Get-Acl* or even Get-Command Get-Ac[l]), or search using a Noun or Verb (e.g.: Get-Command -Verb Get or Get-Command -Noun Acl or even Get-Command -Verb Get -Noun Acl), then PowerShell assumes you’re searching (and won’t throw an error when no command is found).

In PowerShell 3, because modules can be loaded automatically when you try to run a command from them, Get-Command had to be modified to be able to return commands that aren’t already loaded. The problem the PowerShell team faced is that in order to get the metadata about a command, they needed to actually import the module. What they came up with is that if you’re searching … then Get-Command will not load modules which aren’t already loaded. If you specify a full command name with no wildcards, then PowerShell will load any module(s) where it finds a matching command in order to get the metadata (parameter sets, assembly info, help, etc). And of course, if you specify a full command that doesn’t exist, you’ll get an error!

Perhaps a few examples will help:

Launch PowerShell 3 using:

And then try this, noticing how much more information you get when you specify a specific full name:

But there are several problems:

Get-Command has another parameter: -Module, which allows you to specify which modules should be searched, and in PowerShell 3, it changes the behavior in weird (buggy) ways:

1. If you specify a single module, then that module is imported (to search it more thoroughly?), even if you specify a specific command that’s not in that module.
2. If you specify a single module that does not have a command that matches, then Microsoft.PowerShell.Management is loaded also. I don’t know why yet.
3. If you specify more than one module, and you’re searching, and none of them have a command that matches … it’s just as though you hadn’t specified modules, and nothing unexpected happens.
4. If you specify more than one module, and a specific command, then it gets really wierd:
   • If the command is in one (or more) of the specified modules, the first module (in PATH order, not the order you specified) which you listed that has the command is imported.
• If it’s a valid command in a different module, the first module with the command is loaded … and so is Microsoft.PowerShell.Management. I don’t know why! Oh, and you still get the error because it can’t find the command where you told it to look.

I filed a bug on Connect to cover that last scenario where the module containing the command is loaded even though you gave Get-Command a list of modules to look in, here’s another example, and notice that even though all I do here is run the same command over and over (I added some Get-Module to show you WHY you get these results, but it’s the same without them), but I get different results:

Some back-story

During development of a PowerShell module like ShowUI, I am usually not using a released version of the module, since I’m obviously working on the next version. In addition, sometimes I need to have “beta” and “stable” releases … so I might have two or three versions of the module that I need to be able to load in order to support users and actually do development.

In case you don’t know how modules work, here’s the deal: There’s an environment variable PSModulePath. It’s just like the Path environment variable, but for modules. The variable contains a semi-colon delimited list of folders paths, which PowerShell searches for modules. Some people think of a PowerShell module as basically a .dll (binary module), a .psm1 (script module), or a .psd1 (manifest module)... but it’s never just a file, it’s a *Folder* and a file. In order for PowerShell to find ShowUI when you write Import-Module ShowUI, you have to have a Folder in the PSModulePath named “ShowUI” and in it, a file (either .dll, .psm1, or .psd1) also named ShowUI.

What I would like

The Import-Module command has a -Version parameter. What I expected What I would like to be able to do is create a folder “ShowUI” and in there, create folders “1.0” and “1.1” and “1.2” and then be able to use Import-Module ShowUI -Version 1.1 to load a specific version.

Sadly, this doesn’t work. First of all, the -Version parameter on Import-Module, is a minimum only. Additionally, Import-Module requires the file and folder to match names exactly, so putting the module inside a subfolder doesn’t work even if there’s only one.

However, there are two things we can use to fix our problem:

1. Just like the PATH variable, PSModulePath works in order. PowerShell searches the listed paths one at a time and uses the first Module that meets the specified minimimum version.
2. Additionally, PowerShell allows you to specify a relative Path to a module that starts in a PSModulePath … I’ll explain more in a moment.

My actual working environment

All of my development projects are in $Home\Projects, so I put a “Modules” folder in there which is where I put modules that I’m actively working on. Then I add that folder to my PSModulePath environment variable: @$Env:PSModulePath += “;$Home\Projects” ...

Since I put it on the end, modules in there won’t be imported if there’s a module with the same name in my regular Module folder ( $Home\Documents\WindowsPowerShell\Modules ), unless I increment the version number in the psd1 metadata file and specify -Version when importing.

For example, I have two copies of WASP:

$Home\Documents\WindowsPowerShell\Modules\WASP\Wasp.psd1 (version 1.0)
$Home\Projects\Modules\WASP\Wasp.psd1 (version 2.0.0.2)

If I want to load the released version, I can just Import-Module WASP — this ensures that scripts that I have in my scripts folder generally load the released version of WASP. When I want to load the development version, I can Import-Module WASP -Version 2.0, and get the version from my Projects folder, because the released version’s version number is lower than 2.0.

More layers onion boy?

When I need to have more than one older version, this presents a problem, because I would need another folder in my path, and for the numbering scheme to work despite the fact that -Version is only a minimum value, I would need the released version (say 1.0) to come first in the path, then the beta version (say 2.0.1), then the development version (say 2.1) ... which means I’d need to add a $Home\Projects\BetaReleaseModules or something like that, and put it in my PSModulePath before the $Home\Projects\Modules folder. That would work, but you can imagine that it wouldn’t scale very well.

Here’s what I do: past the release and development versions, I start creating folders like this:

$Home\Documents\WindowsPowerShell\Modules\ShowUI1.0\ShowUI.psd1 (version 1.0)
$Home\Documents\WindowsPowerShell\Modules\ShowUI\ShowUI.psd1 (Release version: 1.1)
$Home\Documents\WindowsPowerShell\Modules\ShowUI1.1\ShowUI.psd1 (Release version: 1.1)
$Home\Documents\WindowsPowerShell\Modules\ShowUI1.2\ShowUI.psd1 (Beta version 1.2)
$Home\Projects\Modules\ShowUI\ShowUI.psd1 (Development version 1.3)

Now, if I Import-Module ShowUI, I’ll get 1.1 and if I specify Import-Module ShowUI -Version 1.2 I’ll get … version 1.3 (whoops).

To load ShowUI 1.2 with that structure I have to do this: Import-Module ShowUI1.2\ShowUI — specifying the folder name and the module name. Of course, I could use “ShowUIBeta” as the folder name instead, but usually when I get a request for help writing scripts, I can get people to give me the specific version that they’re working with, so I like the numbered folders instead.

A last important note is that I do not rename the psd1 or anything else.

I could rename the metadata file to $Home\Documents\WindowsPowerShell\Modules\ShowUI1.2\ShowUI1.2.psd1 and then I’d be able to Import-Module ShowUI1.2, but there’s a catch. If I rename it, then I’ve renamed the module. If I leave the metadata file as “ShowUI.psd1” then once I’ve imported the module I can call Get-Module ShowUI or Get-Command -Module ShowUI and they still work as they should.

In any case, this folder-naming stuff is less than ideal and will hopefully be fixed in a future release of PowerShell, but with a “Projects\Modules” folder I can have two versions, and usually a development and release version is enough. Personally, I keep old releases of major modules around in numbered folders just in case, but I rarely actually use them.

Rule #1: Never use Get-Module -ListAvailable in a module.

You need to run the command outside of your module. The simplest way to do that is to use Invoke-Command { Get-Module -List Available } … that should give you the same output, but without the nasty side effects.

There is a bug in the ListAvailable parameter, which causes PowerShell to mark other modules which are already loaded as being “nested” in your module.

It does that to all modules which have correct manifests, and the problem is that if your module gets unloaded later (using Remove-Module), it removes these nested modules as well … even though the user had loaded them separately.

Edit: 2010-08-05

I should have mentioned this originally (so I’m adding it now): one of the reasons developers were using Get-Module with -ListAvailable was to get information about their own module during the initial load (such as to check module versions, or load data from the PrivateData). If you were doing that, you can use Test-ModuleManifest instead with the path to your manifest: Test-ModuleManifest $PSScriptRoot\ModuleName.psd1

Rule #2: Do not use Import-Module in a module.

If you have a dependency on another module, you should load it by specifying it as a NestedModules in your module manifest.

There is a bug in the way that Remove-Module unloads modules which causes modules which are loaded from inside your module (whether by Import-Module or by Get-Module -ListAvailable), to be unloaded completely even if they’ve been previously loaded in the console (global) scope interactively by the user (or via their profile).

You could possibly make an exception to this rule, if the module is actually in your module’s folder (and thus, not easily loadable from outside your module), but you’re making the assumption that no one will ever use that module outside of your module.

Modules loaded via the metadata file’s NestedModules property don’t have this problem, so you should always load nested modules that way.

Almost every author of a binary module has probably had someone ask about this at some point, because there’s always someone who has their user profiles stored on a network location, and therefore installed their modules on that network path and can’t get them to load because they get a warning that .Net “Failed to grant minimum permission requests.”

Before we get into this any further let me just say: by far the simplest thing to do is to create a local folder on your local hard drive, add that to your environment PSPathModules variable, and just install your modules there.

Other than that, the solution depends on the version of .Net that you’re using (you can tell by checking the $PSVersionTable.CLRVersion

The .Net 2.0 framework (and 3.0 and 3.5 and 3.5 SP1)

The problem is not a PowerShell problem at all, it’s a .Net problem. The .Net framework 2.0 (remember that PowerShell targets 2.0, and is actually based on .Net 1.1) didn’t trust assemblies loaded from network shares. You can fix that for an individual assembly or for a whole share using the Caspol tool.

A complete discussion of that tool and it’s myriad command-line options is beyond me, but for a simple solution, you can run this command specifying the server and share you want to load from (in my example the “Modules” share on the “ProfileServer” server).

Hopefully the only thing that needs explaning there is that 1.2 is the default “Local Intranet” group, and that CasPol.exe is in your Framework Runtime directory. Once you’ve run that, you’ll be able to import any modules that are in subdirectories of that share.

Note: You must run the version of CasPol.exe which is in the lcation defined by the GetRuntimeDirectory() command (it’s important to use the same version as the runtime you want to be affected).

You can read more about importing binary modules from network shares, including how it changed in .Net 3.5 SP1 and why it’s not automatically fixed in .Net 4 over on the PoshCode wiki.

Import-Module : The module manifest ‘C:\Modules\Test\Test.psd1’ could not be processed because it is not a valid PowerShell restricted language file. Please remove the elements that are not permitted by the restricted language: The command ‘Get-ChildItem’ is not in allowed in restricted language mode or a Data section.

Import-Module : The module manifest ‘C:\Modules\Test\Test.psd1’ could not be processed because it is not a valid PowerShell restricted language file. Please remove the elements that are not permitted by the restricted language: A variable that cannot be referenced in restricted language mode or a Data section is being referenced. Variables that can be referenced include the following: $PSCulture, $PSUICulture, $true, $false, and $null.

BUT THEN I came across this in the Windows 7 built-in BitsTransfer module:

RequiredAssemblies=Join-Path $psScriptRoot “Microsoft.BackgroundIntelligentTransfer.Management.Interop.dll”

AND IT WORKS?!

Well, that’s very weird, because Join-Path is certainly not allowed in a normal “restricted language” file, and (as you can tell from above) neither is $PsScriptRoot — in fact, as far as I know, you shouldn’t have to do that at all, since RequiredAssemblies knows enough to look in your module folder… but nevermind that, why does it work?

The Rest of the Story

So I went digging, and it turns out that although they are parsed as Restricted Language (like a “data section,” see about_Data_Sections), module manifests are allowed extra cmdlets: “Import-LocalizedData”, “ConvertFrom-StringData”, “Write-Host”, “Out-Host”, “Join-Path” and even special variables that aren’t normally allowed in data sections: specifically $PsScriptRoot which is the ModuleBase (the parent folder of the psd1) and environment variables ($Env:), plus the usual $PSCulture, $PSUICulture, and of course $true, $false, and $null.

However, although you can use those variables, you can’t embed them in strings, so if you wanted to use $Env:Windir or $Env:Temp as part of a path (for instance), you need to take advantage of the availability of Join-Path.

Now, I can’t find this documented anywhere (although I did add it to the module manifest documentation on MSDN), but it’s true, nonetheless — you’ll just have to trust me Yeah, PowerShell is starting to drive me crazy again.

Related articles by Zemanta

• A question about PowerShell Module Manifests (huddledmasses.org)
• Another Module Manifest Gotcha (huddledmasses.org)

Of course, there’s an exception: if your module is dependent on a specific host. For instance, if you’ve written a module for PoshConsole which exploits the WpfHost display features or the BgHost hotkeys feature … or if you’ve written ISEPack based on the menu and script-editor features of PowerShell ISE), then it makes sense to specify the PowerShellHostName and the PowerShellHostVersion together. However, if you aren’t taking a dependency on a specific host … you should definitely never specify PowerShellHostVersion by itself.

RequiredModules

I’ve got another one that I think you should not use. PowerShell has several ways of specifying prerequisites for your modules, such as assemblies that should be loaded, modules that must be nested, etc. In every case except RequiredModules, PowerShell will actually import those things for you.

That is, if you specify that you RequiredAssemblies = “System.Windows.Forms” it will automatically load it. If you specify NestedModules = “BitsTransfer” it will automatically load that. But if you specify RequiredModules = “BitsTransfer” you’re sore out of luck.

Not only does PowerShell not load them for you, it doesn’t give you all the information you need to load them when it fails. For example, consider that you have a module called “ReallyRequired” and one called “TestModule” which requires ReallyRequired. It’s metadata file is very simple, it looks like this:

Before we try to import it, we might try to check which modules it requires, and since it doesn’t appear to require any, we’ll go ahead and import it:

Apparently, there’s just no way to tell which module(s) you need to preload (or even whether there are any) until you get an error message. You might think you could just write a script to test for that error and then run Import-Module ReallyRequired … but what you don’t know is that after you load your copy of the ReallyRequired module:

If you’re like me, right now you’re tearing out your hair wondering why the error message didn’t tell you that in the first place. Then you have to go off and try to find a newer version of your RequiredModules. And that’s not even the crazy thing! The crazy thing is, if you find the wrong one, you could still get a message about the GUID not matching next.

So yeah, RequiredModules is really unlikely to be helpful right now. You’re better off just putting an Import-Module ReallyRequired -version 2.5 -ErrorAction Stop at the top of you script module (although that won’t help you if you’re writing a binary cmdlet module).

Some good news

This is just one of the things that the new PoshCode is being designed to address, so I’ll be blogging more about this shortly, but for now, you can use this function to list RequiredModules:

Related articles by Zemanta

• A question about PowerShell Module Manifests (huddledmasses.org)

Scripts are still there, but they’re still basically done the way they are now: you just upload a single file.

Modules, however, will be treated specially. You’ll have to have an account to upload, but you’ll be able to just upload a .zip file of your module folder using your login and not fill in any other forms. Our system will take care of parsing the metadata out of the manifest in your module folder.

The problem is that I need a little data that isn’t a part of the standard module manifest format… and I can’t add it because PowerShell won’t load a module with extra fields in the manifest metadata hashtable: https://connect.microsoft.com/PowerShell/feedback/ViewFeedback.aspx?FeedbackID=421837 — My request was closed “by design” early in the beta cycle and I wasn’t able to convince them to rethink that.

At a minimum, PoshCode requires a LICENSE field, and a CATEGORY field (think of the categories on CPAN or TechNet Script Center).

So I’m having an informal poll. (you’ll have to “vote” by commenting or tweeting to me, because I want more than just “I choose A”).

Which option do you prefer, or can you think of a better way:

Require the License and Category to be part of the PRIVATEDATA field in the standard manifest

This might require authors to rewrite parts of their modules, because we’d be forcing PrivateData to be a Hashtable, and to contain keys that they don’t need in the module.

Some existing modules use an array in PrivateData, or even a simple string, rather than a Hashtable. However, it’s not a huge difference, and might be the cleanest method: it wouldn’t require you to manage a second file of data, and the additional data will be easily available to users and scripts via the standard Get-Module output.

Add a “doc comment” system for Module manifest files.

The upside of this is that PoshCode wants to create module-level documentation anyway. If we use documentation comments like those used on functions we would be able to just create our own standard for them, and add any extra fields we need. It would be guaranteed not to conflict with anything you’re already doing, and in addition to missing metadata, you could have some module-level documentation beyond just the Description field of the metadata.

The problem with this is that there’s no built-in way to parse that, and doing so isn’t trivial, so you would need to just read it on our website, or read the source of the file, or have our PoshCode cmdlets in order to make any sense of it once you had a module on your system. It doesn’t integrate with PowerShell in any meaningful way.

Embed the extra data right in the manifest hashtable, using a special comment to hide it from Get-Module

@{
author=“Joel Bennett”
description=“My latest crazy module”
<#!PoshCode
License=“Ms-PL”
Category=“WPF”,“GUI”,“Toolkit” #>
CompanyName=“Huddled Masses”
...
}


I like this because it’s fairly trivial for us to strip out the comment and just turn that into a plain-old DATA section. It also lends itself to reminding the PowerShell team that these fields are missing, and maintains the existing simple syntax of the manifest.

The problem here is, again, that the data doesn’t appear using any of the standard PowerShell tools — but getting it out is significantly easier than getting out document comments…

Require a separate “ReadMe.psd1” (or “PoshCode.pds1” or “Metadata.psd1”) metadata file

You could start with a copy of your module manifest, and then add the extra stuff that PoshCode needs. This would be nice because we would be able to add any extra fields we wanted as mandatory members, and we could include the “documentation” stuff I mentioned earlier…

But the downside is that it would be a third file (second manifest) that authors would have to maintain and keep current.

Any thoughts? Suggestions?

I’ve put some further thoughts about CPAN and the data problem on the wiki.

Improvements to the underlying web search API

You’ve always been able to pass a LIST parameter to the API, and get more results by specifying a higher number. But it never worked with the “path” notation (until now).

That is, you used to be able to do:

• http://poshcode.org/api1?q=start&list=10

• http://poshcode.org/api1?q=start&list=100

To make the API a little easier to use I’ve enhanced it just now:

1. You can now page the search results.
2. You can use the word “limit” instead of “list”
3. If you specify limit=0 (or list=0) I’ll give you everything I’ve got. Please use a little precaution about that, as it could be a LOT of data. I’d much rather you retrieve, say … 25, and then get the second page if you want more.

1. You can use path notation.

So, you can use any of these URLs:

• http://poshcode.org/api1/start/list/25/page/1
• http://poshcode.org/api1/start/limit/25/page/2
• http://poshcode.org/api1?q=start&list=25&page=3

• http://poshcode.org/api1?q=start&limit=25&page=4

There are a lot of search results for “start” ... feel free to play with enhancing the PoshCode module, or incorporating this into your apps, etc.

I should add that you don’t have to specify the limit or page number. By default you’ll get the first 10 items, which should be enough.

Once you’ve got your PowerShell all installed and have set up your first profile to auto-load … you’re going to want some scripts (well, maybe you’ll want to learn more about how to use PowerShell, but go with me on this)!

One of the best places to look for scripts is the PowerShell Code Repository, and although you can browse and search on the website, you can also do it using the PoshCode module (or the version 1 compatible script). These scripts include a Get-PoshCode cmdlet which you can use with search terms to get a list of scripts and cmdlets back, or with numeric IDs to download scripts (you’ll see what I mean later on, for now, go ahead and grab the appropriate version of that script).

I’m going to assume you put it into your AutoModules folder. If you grabbed the module, it should be saved to WindowsPowerShell\AutoModules\PoshCode\PoshCode.psm1 otherwise, to WindowsPowerShell\AutoModules\PoshCode.ps1 … but you may have run into a minor problem if you load the .ps1 version ). Both the module and the script are signed, but they are signed by my self-issued code signing certificate which your computer almost certainly doesn’t trust… You can use the signature to verify that the file hasn’t been modified since I signed it, but that’s about all (and even that’s a bit of a trick). To actually use the script (module), you’ll need to sign the script yourself (see the steps and how to get a certificate in part 2).

If you’re on CTP2, this would be a good time to get my Authenticode script module to help with signing, and to learn a little about those PoshCode cmdlets …

Using PoshCode to get a module.

Create a folder in your AutoModules folder named “Authenticode” ... and then inside it, try running Get-PoshCode Authenticode … You should get a list of 3 or more versions of my script, entitled “Get/Set Signature (CTP2)”. You want the latest version, so pick the one with the biggest number for the ID, and run Get-PoshCode 464. That should download it and save it. You’ll be able to load it using Add-Module (since Add-Module in the current CTP doesn’t care about code-signing), but first you need to create a settings file (it will tell you how, if you try to load it without setting it). change the CertificateThumbprint that’s set in the first line of code in that script — set it to the thumbprint of your personal code-signing cert (which you’ve hopefully imported into your certificate store).

Now you’ll be able to sign without specifying the certificate each time, and you’ll be able to pipe files in too, so you could, for instance, sign all the script files in a whole folder tree like:

So, here’s what I do.

First of all, I don’t actually put .ps1 scripts into the AutoModules folder, I have a separate AutoScripts folder, and I dot-source scripts from there, as well as loading ps1xml format files from it. I also specifically use a series of scripts there: Aliases.ps1, Variables.ps1, and EyeCandy.ps1. I assume that the functions of the first two are obvious, the last one is a modified version of the PSCX prompt and MOTD script to modify my prompt and startup message.

On top of the PoshCode script module and the Authenticode script module I mentioned earlier, which I use for signing and resigning, and the new CTP2 version of the PowerTab script module, I also use a source build of the upcoming PSCX 1.2 release (which needs a bunch of scripts to be useful, so I just renamed their 3-line profile.ps1 to PSCX.psm1 and stuffed the whole build with all it’s sub-folders into my WindowsPowerShell\AutoModules\PSCX). I also preload my PoshHttp module for downloading files from the web, and I a couple of functions (ellipsis and Get-PerformanceHistory).

Almost everything else I use I write into scripts (I even use the in-line cmdlet syntax to get full-power parameter parsing in a script file) which I put in directories by category inside my main WindowsPowerShell\Scripts directory (in my profile directory, and add to my path so that I can just run a script named “Get-GUID.ps1” by typing Get-GUID on the command-line.

Here’s the profile script off my laptop as an example — I basically use the same profile on all my PCs, and make changes to the “Variables” scripts on each PC to differentiate them. In fact, I have a sub-directory of my “WindowsPowerShell\Scripts” called “Work” which I even keep around at home (although I practically never use those scripts at home — they’re mostly for resetting servers and doing db queries, and only work from home when my VPN connection is up). Let me know if you have any questions.

Incidentally, I use “RemoteSigned for my Execution Policy — I wish there were a way to require signing for just my profile, because then this system would basically protect me from tampering with my auto-run scripts. However, I’m actually comfortable not worrying about it any more than I worry about viruses replacing the executable for Notepad++ on my thumbdrive or something — I do realize that this profile script could be simplified a lot if I replaced the signature checking with just using the “AllSigned” policy … but there’s two major problems with that:

1. I’m a PowerShell-ed developer — I spend a lot of time editing scripts and running them, and re-signing scripts that I’m working on (even though it’s just a matter of typing “sign” before I run a script) becomes an annoyance, such that if I were working in that environment, I’d inevitably have a “Sign-and-Run” script, which would really make me less secure, rather than more — for the average user, I do think that if you’re not writing scripts yourself on a daily basis, you should consider using AllSigned.

1. PowerShell 2 CTP2 doesn’t check signatures on modules, so even if I have it set to all-signed, it will load script modules and will let you dot-source any script using the add-module command, so I’d have to avoid modules altogether to really be 100% secure.