Cosmos, getting distracted in the Milky Way

I’ve been having a hard time expressing my disappointment with the new Cosmos. It actually upset me a bit, so I decided to blog about it as a way of getting closure. I guess I was expecting more science and less CGI. Maybe future episodes will be better, but I’m no longer excited, so I may or may not watch: the first episode of the new 2014 Cosmos seemed to be mostly CGI — including ten minutes of cheesy cell-style animations about the Roman Inquisition and an Italian friar and philosopher you’ve never heard of…

I assumed, at the beginning, that there were a few actual images from telescopes and spaceships in the “our Cosmic Address” sequence, but I have to say I think I’ve seen the same idea presented better elsewhere — in this version they did their best to keep any of the imagery from seeming real by super-imposing the bubbles-and-chrome-ship-of-our-imagination. Frankly, the ship was the corniest part of the original Cosmos, and I could have lived without it, but if we did had to have it, couldn’t we have just had a transparent bubble like the ones from Oz the great and powerful? ;)

Throughout the “Cosmic Address” and “Cosmic Calendar” pieces, anyone who’s been to a science museum in the last 30 years can be forgiven for feeling that they’ve heard this all before, and perhaps for this reason the show didn’t feel any need to get into the specific evidence, or the details of how we’ve learned any of the things they told us. The fact is, there was a lot of lip-service to science, but no actual science here, and barely a mention in passing of the evidence science has discovered. In fact, even after getting kind-of preachy about the need to have an open mind, accept only what can be proven, and question everything — they then proceeded to teach for most of an hour with little more than a hand-wave about the amount helium in the universe and the “glow of radio waves” (yes, he actually said GLOW of radio waves).

The worst part, by far? Spending ten minutes on the persecution of Giordano Bruno in cheap low-budget cell-style animation. First of all, this man was no scientist (they even said as much on the show). He was executed by the Roman Inquisition due to his pantheistic beliefs in reincarnation and more — that is to say, not because of his ideas about the universe, but because of his ideas about god(s). Now, while we all know the time of the inquisition was terrible, and many horrible things have been done in the name of the “church” ... I couldn’t understand why we weren’t talking about Galileo, or anyone who actually applied the scientific method to … something. I can’t imagine any valid reason for making the story of Giordano Bruno part of this show — and frankly, I am convinced that they deliberately twisted the story to make it seem Bruno was persecuted from one end of Europe to the other for his cosmology (which is simply not true) in order to paint Christians as dangerous anti-science bigots.

I really wanted to love a show about the amazing discoveries of science that would get my kids fired up for science and the wonders of the universe. But this one couldn’t stick to what it was good at. Right in the middle of our science lesson we got a ten minute detour through a twisted, misleading story of the inquisition that made me long for Bill Nye. I hear that some scientists are feeling like there’s a wave of general science denial, and perceive it to be led (at least in the United States) by religious extremists. Maybe they really are afraid of some sort of new inquisition, but seriously: what was that?

Configuration Files for PowerShell Scripts and Modules – Part 1

One of the common sets of questions we get in the Virtual User Group about PowerShell (particularly from authors who are just graduating from scripts to writing their first modules) are about how to store configuration data. There are actually several options in PowerShell, but most of them aren’t optimal for various reasons.

The most obvious solution is the PrivateData block in the module manifest. Added to PowerShell v2, it’s designed for use in modules to store configuration, settings, default values, etc. In your module manifest you can put pretty much anything (as long as it’s valid “RestrictedMode” PowerShell, see about_Language_Modes in the PowerShell help, although you can also use $PSScriptRoot and Join-Path).

The nice thing about this is that the syntax is native PowerShell. The PSD1 file extension is “PowerShell Data” and it is to PowerShell what JSON is to Javascript: a restricted subset of the language written in the normal syntax of the language, with simple restrictions. What makes PrivateData obvious and easy to use is that it’s already parsed for you in your module and available so settings in your PrivateData are immediately available. For example, a function like this will output it’s values:

function Read-Config {
   [CmdletBinding()]param()
   $MyInvocation.MyCommand.Module.PrivateData
}

The only downside of this is that changing the settings (permanently) requires either editing the manifest file by hand (in a text editor), regenerating the manifest (using New-ModuleManifest from inside your module — as I did in the Authenticode module), or reading in the manifest content and parsing it via regex or parser to edit and rewrite it. Any of those will work, but the real problem is that you have to manually generate a string representation of your PrivateData hashtable.

In case you wondered, even for code-signed modules there are no problems with editing or regenerating the module manifest, as long as you don’t change the rest of the manifest. PowerShell does not check the manifest signature, so as long as you keep your PrivateData valid, and don’t change the rest of it, you’re ok.

Some other options:

The most powerful round-trip solution are the Clixml cmdlets. With Import-CliXml and Export-CliXml we have almost total fidelity in serializing objects to disk, and we can easily store any hashtable of settings, and re-import it without the Restricted Mode limitations. Unfortunately, the CliXML format is nearly impossible for a human to read and edit, so using them for configuration is not ideal: you have to give up on letting people edit your settings in a text editor in favor of having them work with the hashtable or custom object inside PowerShell.

The JSON cmdlets which shipped in PowerShell 3 are an easy choice also. While they don’t write to disk, making them do so is simply a matter of using ConvertTo-Json and ConvertFrom-Json together with Get-Content and Set-Content. However, the JSON serializer falls down on simple things like serializing $Pwd (a System.Management.Automation.PathInfo object), and depending on what you store you may be very surprised at how convoluted the JSON gets. You’ll have no problem if you keep yourself to a single top-level hashtable containing other hashtables, arrays, strings and numbers that would be allowed in restricted language mode, however (and it will output human-readable formatted JSON). The only real problem with the JSON serializer is that it is a new syntax. If you ask your users to edit it in a text file, many PowerShell users won’t know the slightly different rules for JavaScript object notation (vs PowerShell Data), and will struggle with the use of colons instead of equals for assignment, square braces for arrays instead of and parentheses, and curly braces without the sign for dictionaries.

What would it take to “fix” PowerShell Data?

As I said before, the only real problem with PSD is that there’s no built-in mechanism round-tripping for it: you can’t easily modify the values in the file from code, because we can’t safely get the modified value back into our module’s manifest. We could fix that, of course, by writing a simple serializer. For the simplest case we want to be able to serialize a hashtable as our settings dictionary, and support round-trip for numbers, strings, arrays and hashtables. Something like this should work:

function ConvertTo-PSData {
   [CmdletBinding()]
   param(
      [Parameter(Mandatory=$true, ValueFromPipeline=$true, Position=0)]
      $InputObject
   )
   # $t is our indent
   begin { $t = "  " }
   process {
      if($InputObject -is [Int16] -or
         $InputObject -is [Int32] -or
         $InputObject -is [Byte]) {
         "{0:G}" -f $InputObject
      }
      elseif($InputObject -is [Double])  {
         "{0:G}" -f $InputObject -replace '^(\d+)$','.0'
      }
      elseif($InputObject -is [Int64])  {
         "{0:G}l" -f $InputObject
      }
      elseif($InputObject -is [Decimal])  {
         "{0:G}d" -f $InputObject
      }
      elseif($InputObject -is [bool])  {
         if($InputObject) { '$True' } else { '$False' }
      }      
      elseif($InputObject -is [String]) {
         "'{0}'" -f $InputObject
      }      
      elseif($InputObject -is [System.Collections.IDictionary]) {
         "@{{`n$t{0}`n}}" -f ($(
         ForEach($key in @($InputObject.Keys)) {
            if("$key" -match '^(\w+|-?\d+\.?\d*)$') {
               "$key = " + (ConvertTo-PSData $InputObject.($key))
            }
            else {
               "'$key' = " + (ConvertTo-PSData $InputObject.($key))
            }
         }) -split "`n" -join "`n$t")
      }
      elseif($InputObject -is [System.Collections.IEnumerable]) {
         "@($($(ForEach($item in @($InputObject)) { ConvertTo-PSData $item }) -join ','))"
      }
      else {
         "'{0}'" -f $InputObject
      }
   }
}

What I would recommend is that if you need to change your settings in your module, you create two functions: the Read-Config from earlier, and a new Write-Config which would take the hashtable as the parameter, and would have the values you want in your manifest hard-coded in a call to New-ModuleManifest. You can build the path for your manifest, and don’t forget to read the name, version, and GUID from the current values so you don’t make it hard to rename your module. Something like this:

function Write-Config {
   param([Hashtable]$Config)

   $Name = $ExecutionContext.SessionState.Module.Name
   $Guid = $ExecutionContext.SessionState.Module.Guid
   $Version = $ExecutionContext.SessionState.Module.Version
   $Path = Join-Path $PSSCriptRoot "${Name}.psd1"

   $Properties = @{
      ModuleVersion =  $Version
      Guid = $Guid
      ModuleToProcess = "${Name}.psm1"
      Author = 'Joel Bennett'
      Company = 'HuddledMasses.org'
      PowerShellVersion = '2.0'
      Copyright = '(c) 2014, Joel Bennett'
      Description = 'A special module for something'
      FileList = "${Name}.psm1","${Name}.psd1"
      FunctionsToExport = "*"
      CmdletsToExport = @()
      VariablesToExport = @()
      AliasesToExport = @()
      PrivateData = $Config | ConvertTo-PSData
   }
   New-ModuleManifest $Path @Properties
}

Further thoughts

I’ve got some code to take this further, which I’ll post in Part 2 of series, but in the meantime, I’m interested in hearing from any PowerShell authors who’ve worked on modules, if you’ve done any configuration, whether in PrivateData, or some other way, whether you think your way is better than this or not, and whether you think this method will work for you or not. Really, any feedback at all ;)

About_PipelineBinding

One of the most powerful features of an object-oriented shell like PowerShell is that it can accept input from the pipeline based on the properties of the objects in the pipeline…

When you’re writing an advanced function in PowerShell and using the Parameter attribute to control binding values to the parameters, you can specify that you want a specific parameter to accept the full object that was output by a previous command in the pipeline, or that you want it to accept the value of a property of the object that matches the parameter name (or alias) ... or you can mix and match both. That’s what allows you to run code like this:

Get-Module | Get-ChildItem

Because the Path parameter of Get-ChildItem is set to take the value from the pipeline based on matching property names (that is, ValueFromPipelineByPropertyName is True) as well as just a straight string from the pipeline (ValueFromPipeline is also True), any object with a Path property can be piped to Get-ChildItem.

I used to think that ValueFromPipelineByPropertyName took precedence over ValueFromPipeline. It turns out that I was wrong. In a recent email exchange, someone pointed out that in Windows PowerShell in Action 2nd Ed., Bruce Payette said that ValueFromPipeline takes precedence over ValueFromPipelineByPropertyName (see table 2.3 in section 2.5.2). However, that’s not the whole story (and I think that table is wrong).

It turns out that when an object is piped into a command, PowerShell tries to bind the object without type coercion first, and only if that fails does it try type coercion. In case you’ve never heard that phrase before, “type coercion” is when an object is changed from one type of object into another. In each case (without coercion and with), PowerShell does in fact give ValueFromPipeline precedence over ValueFromPipelineByPropertyName…

Perhaps An Example Can Clear Things Up

As an example, in this function I defaulted everything so that the ValueFromPipeline parameter should take precedence, even down to setting the “Value” parameter set to be the default. Since PowerShell can coerce pretty much any object to a string, that means you can pipe anything to this, and it will bind to the $Name parameter as the ValueFromPipeline. However, if the object has a string property named Mode

function test-binding {
   [CmdletBinding(DefaultParameterSetName='Value')]
   param (
      [Parameter(ValueFromPipelineByPropertyName=$true, ParameterSetName='PropertyName')]
      [String]$Mode,
   
      [Parameter(ValueFromPipeline=$true, ParameterSetName='Value')]
      [String]$Name
   )
   process {
      if(!($Result = $Name)){ $Result = $Mode }
      Write-Host "Got '$Result' via $($PSCmdlet.ParameterSetName)"
   }
}

# Will output the MODE (ValueFromPipelineByPropertyName)
get-item c:\windows | test-binding

If you execute that code you can see that the output seems to show that ValueFromPipelineByPropertyName took precedence. What happened is that the ValueFromPipeline would have required a cast (the item is a DirectoryInfo object, not a string), but the ValueFromPipelineByPropertyName doesn’t require a cast (because the “Mode” property is a string representation of the Attributes).

You can see what happens more clearly if you use Trace-Command to see the ParameterBinding:

$Dir = Get-Item C:\Windows
Trace-Command -Name ParameterBinding -Option All -Expression {
   $Dir | Test-Binding
} -PSHost

# I'll skip over the mandatory parameter check
# Here is the relevant output:
BIND PIPELINE object to parameters: [test-binding]
    PIPELINE object TYPE = [System.IO.DirectoryInfo]
    RESTORING pipeline parameter’s original values
    Parameter [Name] PIPELINE INPUT ValueFromPipeline NO COERCION
     BIND arg [C:\Windows] to parameter [Name]
        Executing DATA GENERATION metadata: [System.Management.Automation.ArgumentTypeConverterAttribute]
            result returned from DATA GENERATION: C:\Windows
        BIND arg [C:\Windows] to param [Name] SKIPPED
    Parameter [Mode] PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
     BIND arg [d——] to parameter [Mode]
        Executing DATA GENERATION metadata: [System.Management.Automation.ArgumentTypeConverterAttribute]
            result returned from DATA GENERATION: d——
        BIND arg [d——] to param [Mode] SUCCESSFUL

If we changed the [String]$Mode parameter to make it [DateTime]$LastAccessTime instead, you will see the same thing happening: it can’t bind the Name from the pipeline without casting, so it tries the other parameter based on the property name, and since it doesn’t require a cast, it’s value is accepted successfully.

If we changed the [String]$Mode parameter be [String]$LastAccessTime then that parameter will require casting also, and you’ll see that it gets skipped in the first pass too, and the $Name parameter will win when PowerShell does it’s second pass at binding using type coercion (casting).

In other words, by observation from the Trace-Command output, what PowerShell does when it’s binding objects from the pipeline is that it tries four different ways, in this order:

  1. PIPELINE INPUT ValueFromPipeline NO COERCION
  2. PIPELINE INPUT ValueFromPipelineByPropertyName NO COERCION
  3. PIPELINE INPUT ValueFromPipeline WITH COERCION
  4. PIPELINE INPUT ValueFromPipelineByPropertyName WITH COERCION
  1. .

P.S.: As a bonus tip: ParameterSets can only be determined by binding a unique parameter, not by how that parameter is bound, so you can’t use ValueFromPipeline vs ValueFromPipelineByPropertyName vs Positional or named parameters to distinguish parameter sets … and you can only have as many ParameterSets as you do unique parameters that go in them. However, as a workaround you might consider (in the case of pipeline binding) adding a unique throw-away parameter that maps to a property (that you don’t care about) of an object you’re expecting…

Splunk adventures with SNMP and Cisco Nexus (Part 2)

I’ve recently been working with the Splunk SNMP Modular Input and some Cisco Nexus switches to see what sort of data and information I could gather using just the SNMP collector.

It has been an interesting exercise. We were able to get access to Cisco’s product labs where I could (remotely) access some of their high-end hardware, and I was able to test the SNMP collector against the Nexus series 3000, 5000, and 7000 switches.

Lots of devices use SNMP, and the MIBs which I’ll discuss below are relatively universal among switch/router devices, so consider this a practical example of working with SNMP data in Splunk, and some lessons we learned along the way.

In Part 1, I walked through what I had to do to get the data about the switches into Splunk using the SNMP Modular Input. In Part 2 (which is this post) I’ll explain just a few of the things you can do with that data once it’s in Splunk!

Step 4. Writing SPL Queries

Now that we have SNMP data being pulled into Splunk, we can start analyzing and reporting on it. In my testing I was querying 3 switches for just 3 tables of information, and only querying every 5 minutes, since I was doing all this from my dev laptop. This came out to about 100,000 events an hour, which is obviously just a drop in the bucket compared to what you would get if you were gathering all the information available via SNMP.

The simple stuff like device status, sysName, software versions and so on are easy to query, but the tricky part of working with SNMP data in Splunk (and the motivation for writing a second part to this blog post) turned out to dealing with tables.

A lot of SNMP data is table-based: when you have a switch with 150 ethernet “interfaces,” you have the same information for each one, so you can either write a query to pull a single piece of information about a single ethernet port, or you can do a bulk get for the whole table and get all the information for all the entries at once.

The SNMP Modular Input gives you two primary choices to deal with tables: putting each field=value pair as a separate event in Splunk (which results in them being parsed automatically), or writing all the results as a single string (which means you need to write regular expressions to parse the data). Additionally (partly as a result of my frustration with these two options), it now has custom response handlers: the idea is that you can provide a custom response handler in python to convert the data to tabular csv format (or whatever makes sense to you) and get it into Splunk in customized ways. This last option was added after I’d finished my work, so I haven’t written such a handler, instead, I’ve written queries to turn the split bulk data into tabular data…

I’ve posted this to GitHub, but the basics of the queries are worth explaining. Most of them are in the macros.conf in the github project and the savedsearches.conf, although there are some in the rejected.xml view which I didn’t think I would use, but kept around just in case.

First, some transforms

When dealing with SNMP table data, the output from the SNMP modular input in Splunk looks like this:

MIBNAME::PropertyName."123456789" = "value"

When working with tables, the default SNMP Modular Input entries include the unique identifier number in quotes. In the case of the ifTable, ifXTable, dot3StatsTable and so on, the unique identifier refers to a specific “interface” (that is, it identifies a particular ethernet jack), and when you do a single snmp bulk get, there are a dozen or more properties returned for each interface, and dozens or hundreds of interfaces on each router.

Thus, in the transforms file I have this transform which is a regular expressions that defines fields for the MIB, the unique identifier, the property name, and the value, as well as ensuring that the Property=Value field is being generated.

[snmp_mib-uid]
REGEX = ([^:]+)::([^\.]+)\.("?)([^"]*)\3 = \"([^\"]*)\"(?= |\n|$)
FORMAT = MIB::$1 UID::$4 Name::$2 $2::$5 Value::$5

Given that transform, every event that’s coming from these table will have not only the usual _host and _time, but also the UID to identify which interface jack it’s referring to. For my own sake, I want all of the events that are returned from a single query to be grouped into single events per UID: that is, for the sake of displaying the data, I want one event with a bunch of fields for each ifEntry in the ifTable.

Then, some lookups

In order to be able to show the sysName associated with each device I’ve queried, I create a lookup table which I update using a scheduled saved search:

sourcetype=nexus_snmp_info | bucket _time span=60s | 
stats first(*) as * by host | 
table host, sysName, sysLocation | outputlookup snmpInfo.csv

The idea is to bucket by 60 seconds just to make sure that if any of the results have a slightly different timestamp they’re all normalized to the same time. Remember this data gets queried once an hour, but sometimes the SNMP modular input writes them with a timestamp that varies by a second or two, and it can ruin the use of the stats command. The “stats(*) as * by host” serves to group all of the events with the same host into a single event with fields for each of the fields found in the original events. It returns the first value for each field name, but we’re only concerned with a single result in any case. In other words, it turns something like this:

_host=192.168.0.71 SNMPv2-MIB::sysName="7k1"
_host=192.168.0.71 SNMPv2-MIB::sysLocation="San Jose, CA"
_host=192.168.0.71 SNMPv2-MIB::sysContact="The IT guy!"

into something like this:

_host=192.168.0.71 MIB=SNMPv2-MIB sysName=7k1 sysLocation="San Jose, CA" sysContact="The IT guy!"

Finally, some search!

When we’re ready to look at actual performance data, we’ll want to collect that data into a single event per interface, so the “by” clause of the stats command changes, and we’ll use search terms to narrow down the events we process to only the fields we’re actually interested in:

sourcetype=nexus_snmp if*Octets OR if*Speed OR ifDescr | 
bucket _time span=60s | stats first(*) as * by _time, host, UID 

That gives us events that make sense when you look at them in tables, but to actually make performance calculations, we will need to calculate deltas. For instance, here’s Cisco’s documentation of how to calculate Bandwidth Utilization Using SNMP, which shows that you need to calculate the delta of inOctets (and outOctets) and divide by the time delta. Calculating a delta in splunk can be confusing if you’ve never done it before, but there are several simple approaches. My favorite is the streamstats command, which allows you to calculate the delta between two measurements of the same value using the range function.

Given our earlier search, we’ll tack on the streamstats:

sourcetype=nexus_snmp if*Octets OR if*Speed OR ifDescr | 
bucket _time span=60s | stats first(*) as * by _time, host, UID | 
streamstats current=true window=0 global=false count(UID) as dc, range(if*Octets) as delta*, range(_time) as seconds by host, UID 

You’ll notice this time we’re doing something a little more sophisticated with our wildcards: the range function is being applied to every field which matches “ifOctets” ... and a maching output field will be created with the name “delta“ where the * is replaced with the portion of the original field name that matched the * there. In other words: the total range (delta) in value of ifInOctets becomes the new field deltaIn and ifOutOctets becomes deltaOut, and if they’re available, ifHcInOctets becomes deltaHcIn, and ifHcOutOctets becomes deltaHcOut.

Calculating Deltas

Of course, for these numbers to mean anything, they have to be divided by the calculated field “seconds” which is the time range: the amount of time which has passed. With the streamstats “window” set to zero, this will calculate an ever increasing delta (from the first event to the current event), so you’ll want to limit the search to a timespan that only returns a few results, or else you’ll be showing the average over the whole time period.

The other option is that you can change the window on streamstats to 2 so that you get a rolling average instead:

sourcetype=nexus_snmp if*InOctets OR if*OutOctets OR if*Speed | 
bucket _time span=60s | stats first(*) as * by _time,host,UID | 
streamstats current=true window=2 global=false count(UID) as dc, range(if*Octets) as delta*, range(_time) as seconds by host, UID | 
where dc=2 | eval bytesIn = coalesce(deltaHcIn, deltaIn)*8 | 
eval bytesOut = coalesce(deltaHcOut, deltaOut)*-8 | 
eval "bytesInPerSec"=floor(bytesIn/seconds) | 
eval "bytesOutPerSec"=floor(bytesOut/seconds) | bucket _time span=10m | 
chart sum(bytesInPerSec) as in, sum(bytesOutPerSec) as out over _time by sysName

Note that in this case I prefer the “hc” (high capacity) values over the non-hc fields if they’re present (on modern, faster routers), because otherwise the values sometimes hit the max for the int field. I therfore use eval with coalesce to pick the first one that’s present, and then multiply by 8 to get bytes (since these are octets), and divide by the time, as stated earlier.

Because we’re using a -8 for bytesOut, and +8 for bytesIn, when we chart these, the output shows up as negative on a chart and input shows up as positive, allowing us to have paired charts showing both input and output clearly.

Additionally in the github project I have example queries showing summary configuration information (sysName, # interfaces, location, uptime, software version, etc), and counts of ports by speed, errors and dropped packet counts, etc.

Some Final Notes

If you need to monitor hundreds or even thousands of switches, you should know that although I did all of this on one box for my demo (and queried the status of the switches only every 5 minutes), you can obviously scale this infinitely by deploying the SNMP collector to multiple Splunk forwarders and distributing the work of collecting the data.

I mentioned the problems I had dealing with the data format and the changes that have been made to the SNMP modular input to accomodate future users — I also had some issues with connectivity where one of the switches was not configured properly for my SNMP queries, and found that the error handling in the SNMP collector didn’t tell me which of my input stanzas was the one timing out — that’s also been fixed, and the current release shows the input stanza name and “destination” string whenever it has errors.

I also included in the github project a report which shows all the errors since the last time the modular input was restarted — that is: since the last time we logged the message about snmp.py being a new scheduled exec process:

index=_internal snmp.py source="*\\var\\log\\splunk\\splunkd.log" | 
eval restart=if(match(message,"New scheduled exec process: python .*snmp\.py.*"),1,0) | 
head (restart=0)

Splunk adventures with SNMP and Cisco Nexus (Part 1)

I’ve recently been working with the Splunk SNMP Modular Input and some Cisco Nexus switches to see what sort of data and information I could gather using just the SNMP collector.

It has been an interesting exercise. We were able to get access to Cisco’s product labs where I could (remotely) access some of their high-end hardware, and I was able to test the SNMP collector against the Nexus series 3000, 5000, and 7000 switches.

Lots of devices use SNMP, and the MIBs which I’ll discuss below are relatively universal among switch/router devices, so consider this a practical example of working with SNMP data in Splunk, and some lessons we learned along the way.

In Part 1 (which is this post), I’ll walk through what I had to do to get the data about the switches into Splunk using the SNMP Modular Input. In Part 2 I’ll explain just a few of the things you can do with that data once it’s in Splunk!

Just to be clear: the Cisco Nexus switches have syslog forwarding capabilities and even support Netflow (IPFIX), so there are plenty of ways to get information about what’s happening on them into Splunk — the one piece we hadn’t experimented with was getting configuration information. In fact, we considered gathering data via Netconf instead of SNMP, but determined that since our goals were read-only, and SNMP is everywhere (not to mention that the modular input was already written), we would go that route.

Step 1: enable SNMP on the Cisco switches.

This isn’t so much about turning on SNMP, as it is about making sure that you know how to query it. You have two choices: you can set a “community” string (which lets anyone query the device if they know the string), or you can set up SNMP v3 usernames and passwords. The configuration is fully documented (with examples) in the configuration guide (this is the Nexus 7000 one), including how to use v3 users with passwords and groups for authorization.

When I started out, we didn’t have SNMP v3 support in the Modular Input, so I went the “community” authorization route with SNMP v2C. Of course, while I was writing this, v3 support arrived in the latest release of the Splunk SNMP Modular Input, so all you have to do is get the latest version, and then download the appropriate pyCrypto package for your Splunk server to enable it.

To find an existing SNMP community name, log into your Nexus switch remotely (via SSH); you can review all the snmp configuration at once using the command “show snmp” or get just the community strings by running “show snmp community” ... any snmp community string will do, since we’ll only need read access for now. If there are none configured, you’ll need to use SNMP v3 or create a community. Since I haven’t had a chance to try this the v3 way, here’s how to create a read-only SNMP community (let’s call it “nexus_stats”) in your ssh session:

config
snmp-server community nexus_stats ro

While you’re in there, you might want to make sure that the server’s contact and location information are correct:

snmp-server contact The IT guy!
snmp-server location San Jose, CA

After exiting config, you can verify that it’s all correct by running the “show snmp” command again.

Step 2: get Cisco MIB definitions for pySNMP.

The SNMP modular input uses pySNMP, which requires that the MIB definition files be converted to python. It ships with the core SNMP MIBs pre-defined, of course, but in order to get custom CISCO information, you’ll need their MIB definitions. You can download the Cisco MIBs from their SNMP Object Navigator and compile them yourself using commands like this:

build-pysnmp-mib -o ./py/CISCO-CALLHOME-MIB.py CISCO-CALLHOME-MIB.my

If you like, you can download my converted copies.

NOTE: The pySNMP project is working on allowing the .my MIB definitions to be dropped in directly without first converting them, so this requirement should go away eventually. Also, please note that I did this conversion in November 2013 — the older these get, the more likely it is that you should update them from Cisco’s source. Regardless of where you get them, you can either compile them into an egg for your particular platform, or just drop the loose .py files into the snmp_ta/bin/mibs folder.

Step 3: Configure Input Stanzas.

You can configure SNMP inputs directly in the Modular Input’s management pages, or you can write config stanzas. In my case, because I hadn’t worked with this Modular Input before, I configured one via the UI, and then copied it and edited it to configure all the other devices I needed to monitor.

Here is the configuration I used for each device: two stanzas, one for the config data which was collected every hour, and one for the performance statistics which was collected more frequently (in the examples below, every 5 minutes). Note that you have to list the MIBs that will be loaded for parsing the data, and the community string, and give each stanza a name that will help you identify it when you see it in the logs.

As I worked on what information I needed to query, this list grew gradually. I needed names and software versions, so I started with “system” (from SNMPv2-MIB). When I needed interface performance statistics, I had to search around the internet and Cisco’s Object Navigator before hitting on the interfaces.ifTable and ifMIB.ifMIBObjects.ifXTable etc.

One thing that ended up being very helful was a full snmpwalk of the devices (dumping it to file), and then looking through the log to see where the interesting infromation was. All told, as a developer, determining which OIDs to query for the information you need is something I never quite felt I had a handle on, and it’s clearly a steep learning curve (as you’ll see below, I still have several more mib_names listed than I’m actually querying in the object_names).

[snmp://nexus 7k1 – info]
destination = 192.168.0.71
communitystring = nexus_stats
mib_names = SNMPv2-MIB
object_names = iso.org.dod.internet.mgmt.mib-2.system
do_bulk_get = 1
split_bulk_output = 1
ipv6 = 0
listen_traps = 0
snmp_version = 2C
snmpinterval = 3600
interval = 3600
index = nexus
sourcetype = nexus_snmp_info

[snmp://nexus 7k1 – ifStats]
destination = 192.168.0.71
communitystring = nexus_stats
mib_names = IF-MIB, EtherLike-MIB
object_names = 1.3.6.1.2.1.31.1.1.1, 1.3.6.1.2.1.2.2.1, 1.3.6.1.2.1.10.7.2.1
do_bulk_get = 1
split_bulk_output = 1
ipv6 = 0
listen_traps = 0
snmp_version = 2C
snmpinterval = 600
interval = 600
index = nexus
sourcetype = nexus_snmp

There are lots of choices you can make when collecting SNMP data. In the examples above I am doing bulk get queries of tables using SNMP v2C, and splitting the output. This results in a raw event in splunk for each field of data, which (as you’ll see) meant I had to pull the events back together in my queries.

I’ve configured the index and sourcetypes, and I’m going to use that source_type in all my queries. In the listing above I’ve shortened the list of MIB names and shown the object names as OID values (for the sake of formatting on the blog — if you check out the github project you’ll find them spelled out as text, but either way works.

I didn’t get very far in examining the Cisco MIB data, apart from the ifTable, and ifXTable to support reporting multi-gigabit speeds. There’s a lot more information available, including temperatures (which are a good way of detecting potential problems before they become catastrophes) but this is just an example, and the data I’m querying here should work with any switch or router that works with SNMP.

In Part 2 I explain a few of the things you can do with that data once it’s in Splunk…

Ruminations about a federal shield law

When you read, over the next few weeks (or months), about the “Shield Law” (aka the “Free Flow of Information Act”) you will undoubtedly find people claiming that the First Amendment already protects journalists1, and we don’t need anything more. They’re wrong.

First of all, the first amendment protects the freedom of speech and of the press from congress… but the courts have ruled several times that it does not protect anyone from having to testify in court when they’re issued a subpoena. Not even journalists or reporters testifying against their sources. I haven’t made up my own mind completely about this particular law, but I can say this: given the rulings from the federal courts, we need a federal shield law to protect the press’ ability to have anonymous sources.

This is what Shield Laws do: “at their most basic, shield laws protect journalists from being compelled to reveal the names of confidential sources. Beyond that, they may cover any pertinent information related to the production of a story, such as notes, outtakes, drafts, or audio and video recordings. These protections — also known as reporters’ privilege — are essential to safeguarding press freedom2.” Many people assert that this is the “freedom of the press” that the first amendment was meant to protect. However, in the federal courts, that freedom has been eroded by court rulings dating as far back as 1971’s Branzburg v. Hayes.

All states except Wyoming now have shield laws or protections derived from court rulings3. But the federal government, backed by the Supreme Court, still insists that reporters can be forced to testify against their sources in a federal trial4. That is why congress (the Senate, specifically) is reconsidering a federal shield law: not to abridge the freedom of speech or of the press, but to protect it.

Having said all of that, there are major problems with shield laws (and in particular, a federal shield law).

The first problem is that a law must define what it’s going to protect, and if the definition isn’t right, a law could limit the definition of the press. This law is a little better than it was (now that Charles Schumer’s amendment has been adopted), and it’s been left open-ended, so that it will protect anyone that a federal judge deems appropriate… but it will still not protect organizations like WikiLeaks5. The problem can be summed up by quoting Schumer: “there are people who write and do real journalism, in different ways than we’re used to. They should not be excluded from this bill.” He’s right: the shield law should protect all people who are “engaged in journalism” and not just people who’ve worked for “media outlets.” This law tries to do that, but I’m not sure if it’s succeeded or not, because it still has this definition of a journalist:

Someone employed by or in contract with a media outlet for at least one year within the last 20 years or three months within the last five years; someone with a substantial track record of freelancing in the last five years; or a student journalist.

The second problem is that this law explicitly states situations in which journalists will be compelled to reveal their sources. The good news is that this will clear up when journalists can offer their sources anonymity and when they can’t. The bad news is that it means that certain types of reporting are only going to happen via websites like wikileaks where the media outlet is outside the jurisdiction of American courts. The biggest case for that is national security: the bottom line is that if the government says national security is at stake, journalists would have no secrets. That’s no surprise given our current surveillance state, but it may surprise you to know that there are some journalist groups supporting the new law as well as groups who are rather opposed to the current version.

1 Forbes: ... we already have the first amendment

2 FreePress: The Journalism Shield Law: How We Got Here

3 Society of Professional Journalists: Federal shield would protect public’s right to know

4 NYTimes: Court Tells Reporter to Testify in Case of Leaked C.I.A. Data

5 TechDirt: Shield Law… defines journalism to leave out wikileaks…

A glimpse into my working/thought process

Last week we were talking about code-golf in the PowerShell Virtual User Group (in #PowerShell on Freenode). If you’ve never heard of it, code golf is basically a recreational computer programming competition where you try to find the shortest possible code that implements a certain algorithm correctly. That is: presented with a (usually simple) problem, you have to write a program to solve it in as few bytes as possible.

Anyway, after I got done explaining the concept, everyone was surfing around codegolf.stackexchange.com and someone noticed that even though there was a Scripting Games event with the same goal a few years ago, there were no PowerShell entries on the Christmas Tree of height N challenge. Several of us set out to fix that, and I kept notes on how I arrived at my eventual submission… since I’ve got them, I thought I’d share the steps here, in case anyone finds them interesting:

Given a number N, print out a Christmas tree of height N. N is assumed constrained to a min val of 3, and a max val of 30 (you need not test for that).
N is given as the one and only command line argument to your program or script.

A Christmas tree is generated with asterisks *, such that N defines the height of the branches not including a one line trunk:

N = 3:

   *
  ***
 *****
   *

N = 4:

    *
   ***
  *****
 *******
    *

N = 5:

     *
    ***
   *****
  *******
 *********
     *

Given that criteria, the first thing I did was figure out how to solve the problem at all. Since we need to go N high, we need N rows of stars, and each one has to be indented one less than the one above it, that means we want to…

  1. Start by printing N spaces followed by a single *
  2. Then print N-1 spaces, followed by 3 *s (that is: 1 + 2 *s)
  3. then N-2 spaces, followed by 1+(2*2) *s
  4. Loop until N-N, taking off one leading space and adding a pair of * each time
  5. Print one EXTRA line: print N spaces, and one * (just like on the first row)

As a side note: this actually pads the whole tree by one space on the left, but that doesn’t actually matter. Given that, here’s my first simple working script.


param($n)
foreach($i in 0..($n-1)){
   (" " * ($n-$i)) + "*" + ("*" * ($i*2))
}
" "*$n + "*"
 

Two things worth pointing out, even if you know PowerShell. First, this is a script and it has to go in a file called get-tree.ps1 so I don’t need the “function” decorator. Second, in the first iteration $i is zero, so we only get one star from all of this: "*" + ("*" * ($i*2)).

Since the goal is to produce the shortest script, I always trim it all down and stick it on one line when possible (although if you need “;” you might as well have a new-line character). However, this post asks us to leave white space and newlines for readability, so I’ll leave extra newlines in the following examples, although I’ll still take the rest out so I can count the significant characters (69 characters, not counting the unnecessary newlines):

param($n)
foreach($i in 0..($n-1)){
" "*($n-$i)+"*"+"*"*$i*2
}
" "*$n+"*"

That’s obviously harder to read, but this is code-golf, and it really doesn’t matter if you can read it. It’s the same code, minus some unnecessary parenthesis and spaces, of course. The first optimization I tried was to switch to using ForEach-Object instead of the foreach(){} language feature, because there’s a one-character “%” alias for the cmdlet. Of course, that requires us to pipe in the input

param($n)
0..($n-1)|%{" "*($n-$_)+"*"+"*"*$_*2}
" "*$n+"*"

That gets us down to 57 characters, counting that last newline which is actually required now. I tried switching to use $args instead of the param block, but it came out the same 57 characters:

0..($n=$args[0]-1)|%{" "*($n-$_)+"*"+"*"*$_*2}
" "*$n+"*"

However, after staring at that for a bit, I thought that maybe I could use a while(){} loop instead of the ForEach-Object. To do that, I’d have to count down from $n to zero (at zero, the while loop will terminate). I tried several different things, and then hit on the idea of using a new variable to also count up. I tried a few iterations of that which were all longer than what I already had, until I hit on this:

while($n=$args[0]--){
" "*$n+"*"+"*"*($i++)*2
}
" "*$i+"*"

Once again, there are a couple of tricks here which you may not be completely aware of. First, remember how the decrement operator works: if you write $n-- it returns the current value and then decrements, if you write --$n it decrements first and then returns the new value. PowerShell (like many languages) allows you to assign a variable at the same time as you use the result of the assignment, thus while($n=$args[0]--) actually assigns $n to the current value of $args[0] and then decrements $args[0]. The while loop will still exit when $n hits zero, so we are down to 55 characters, and all of our newlines are unnecessary again.

After staring at it for a while, I realized I could just write "**" instead of "*"*2:

while($n=$args[0]--){
" "*$n+"*"+"**"*($i++)
}
" "*$i+"*"

And almost as soon as I ran that, I realized that the parentheses around $i-- were not necessary, so we’re now down to 52 characters (not counting the unnecessary newlines):

while($n=$args[0]--){
" "*$n+"*"+"**"*$i++
}
" "*$i+"*"

After I submitted that, I almost quit, but I kept thinking there might be a way to shorten the generation of the number list for iteration. One thing I tried was to use the param block to generate the array. However, since you have to accept the input, you end up needing two parameters (one for $n, one to calculate, based on $n), so it doesn’t work out shorter (and in fact was 2 characters longer). I’m noting it here just because it’s a significantly different approach:

param($n,$o=$n..1)
$o|%{" "*$_+"*"+"**"*$i++}
" "*$i+"*"

I was looking over some other submissions for inspiration when I realized that I could use a for loop instead of a while loop, if I put the logic in the conditional statement and left the assignment empty. Did you know you can do that in PowerShell? This way I can subtract and test at the same time, like I did in the while clause, but with 1 less character.

for(;$n=$args[0]--){
" "*$n+"*"+"**"*$i++
}
" "*$i+"*"

Now we were down to 51 characters. However, the Perl solution which was at the top of the votes had only 50 characters, and it was driving me crazy that I was just one character larger … I tried variations on the param() block, including using $o=$n..1,$n, but couldn’t get it to work correctly.

Eventually, as I looked over my existing solutions looking for a clue to something I hadn’t considered, Isuddenly realized that now that I was counting down I could go back to the first solution using ForEach-Object:

$args[0]..1|%{" "*$_+"*"+"**"*$i++}
" "*$i+"*"

As before, the newline character is necessary now, but even counting it, we’re down to only 46 characters!

I haven’t been able to shorten it any further, but I did come up with another variation which is the same 46 characters, using the short -ov alias for the -OutVariable parameter to capture the output so we can re-use the first line:

$args[0]..1|%{" "*$_+"*"+"**"*$i++}-ov b
$b[0]

In any case, that’s where it stands now. Hopefully you’ve picked up a trick or two, and I didn’t waste my time writing all the explanations around this ;-)

If you can think of a trick that I missed, and you can get it down to 45 characters or less, please do submit it on the StackExchange Code Golf challenge page, but also drop me a line here and let me know!

Creating PowerShell Modules, the easy way.

Today I’m going to show you a new way to build a PowerShell module: I think it’s the fastest, easiest way to turn a few PowerShell script files into a reusable, shareable module.

Here’s the “too long; didn’t read” version: I’ve written a function that will allow you to specify a list of scripts and collect them into a module. Get the script from the end of this post, and you can run this command to build a module named “MyUtilities” within the current directory, moving the ps1 scripts into the new sub-directory:

New-ScriptModule .\MyUtilities *.ps1 -move

Or you can run this command to create the “MyUtilities” module in your regular modules folder from all the ps1 or psm1 scripts in this directory or any sub-directories, overwriting any existing module:

New-ScriptModule MyUtilities *.ps1, *.psm1 -recurse -force

If you want greater precision, you can pipe the files in. This command will let you collect only the ps1 and psm1 files in the current directory, but recursively collect ALL files from sub-directories (like language data files, or nested modules), and generate the module containing them.

dir *.ps1, *.psm1, *\* | dir -s | New-ScriptModule MyUtility

Once you’ve created a script module using New-ScriptModule, you can run this command to update it with the files that are currently in it’s folder tree, incrementing the version number:

New-ScriptModule MyUtilities -Upgrade

Note: none of these commands move the scripts, they just copy them. However, you can add the -Move switch to move them instead.

It’s important to note that the scripts you put into a module need to define functions (they can’t be pure scripts that you run by file name, or else they’ll execute when you import the module).

What follows is an explanation of what this function does (and steps for doing it by hand). The first step, of course, is to create a “MyUtilities” folder in your PSModulePath, and copy all of the script files into it. Then we can get started with a simple Root Module.

The Root Module (ModuleToProcess)

The root module is the main script or assembly that’s going to be executed/loaded when the module is imported. In our example case, it’s the “MyUtilities.psm1” file (the name matches the folder and ends in psm1). Basically, copy the line below into a new file and save it as “MyUtilities.psm1”:

Get-ChildItem $PSScriptRoot -Recurse -Filter *.ps1 | Import-Module

That simple version just imports every ps1 file in the module folder into the module itself (it’s just like dot-sourcing them, but cleaner to write). In the end, I decided that I wanted to have the list of scripts that would be imported hard-coded in the root module (which can be signed), to make the whole thing harder to tamper with, so in the script below you’ll see I’m building a slightly more complicated root module.

The Module Manifest

I want you to end up with a good reusable and redistributable module, so you should definitely create a module manifest. You don’t really have to set any properties in the manifest except the RootModule and the version (but that defaults to 1.0, so you don’t have to worry about it at first), but I recommend you also pass at least:

  • The Author parameter (it defaults to your $Env:UserName which is usually not a full name).
  • The CompanyName (otherwise it comes out “Unknown”, you could at least set it to your blog url).
  • The FileList (which will matter for packaging someday).
  • The Description.

Here’s an example of how to use New-ModuleManifest to create one:

$Module = "~\Documents\WindowsPowerShell\Modules\MyUtilities"

$Files = Get-ChildItem $Module\*.* -recurse | Resolve-Path -Relative

New-ModuleManifest -Path "$Module\MyUtilities.psd1" -RootModule "MyUtilities.psm1" -FileList $Files -Author "Joel Bennett" -Functions "*-*" -Aliases "*" -Cmdlets $null -Variables "MyUtilities*" -Types "*Types.ps1xml" -Format "*Formats.ps1xml"

That’s it. We’re done. You can try now try: Import-Module MyUtilities and Get-Command -Module MyUtilities to see the results!

Some additional information, and the script.

I want to add some detail here about the conventions I follow which are enforced or encouraged by this function and the modules and manifests that it produces, but as I’ve gone on long enough, I will save that for another post.

I do want to state one caveat: although the function below produces much more complete modules than the few lines above, it’s still basically a beta — it’s only been tested by me (and a couple of people from IRC, thanks guys), so it’s possible that everything is not perfect. That said:

  • It supports piping files in to generate the module from, as well as specifying a pattern to match files (and even the -recurse switch).
  • It handles nested psm1 modules as well as ps1 scripts, and treats *.Types.ps1xml and *.Formats.ps1xml files appropriately.
  • It sets reasonable defaults for the export filters of functions, aliases and variables.
  • It supports upgrading modules, updating the file list and incrementing the version number, but won’t overwrite your customizations.

The current version generates a default module which lists specifically the files to import, and then pushes location to the module folder before importing the scripts, one at a time, writing progress messages and error messages as appropriate.

This function is part of the larger PoshCode Packaging module that I am working on. Hopefully you’ll find it an easy way to put a bunch of scripts into a module. It should work in PowerShell 2-4 (although I haven’t been able to test on 2 yet), but let me know if you have any problems (get it from poshcode).

A new way to zip and unzip in PowerShell 3 and .Net 4.5

I’ve got an article coming about WPF 4.5 and what’s new there for ShowUI, but in the meantime I thought I’d share this little nugget I noticed today: System.IO.Compression.FileSystem.ZipFile. You can use that class and it’s buddy ZipArchive to do all sorts of zipping and unzipping tasks.

In order to use these new classes, you have to use Add-Type to import the System.IO.Compression.FileSystem assembly. If you don’t understand that, don’t worry about it, just copy the first line of the examples below. You only have to do it once, so it might be worth sticking it in your profile (or in a module file with a couple of scripts) if you think you’ll do a lot of this sort of thing.

Add-Type -As System.IO.Compression.FileSystem
[IO.Compression.ZipFile]::CreateFromDirectory( (Split-Path $Profile), "WindowsPowerShell.zip", "Optimal", $true )
 

That’s basically a one-liner to zip up a folder. Of course, you should have a one liner to unzip it too:

Add-Type -As System.IO.Compression.FileSystem

$ZipFile = Get-Item ~\Downloads\Wasp.zip
$ModulePath = $Env:PSModulePath -split ";" -like "$Home*" | Select -first 1

[IO.Compression.ZipFile]::ExtractToDirectory( $ZipFile, $ModulePath )
 

In order for the second example to work the way you want it to, you need a zip file where all the files are in a single folder — which is why in the first example I used the overload for CreateFromDirectory which takes a boolean for whether or not to include the root directory in the zip as opposed to just it’s contents. Otherwise, you would need to create the “WASP” folder, first, and then extract to that directory. Of course, if you do that when there is a directory in the zip (as there was, in this case), you’ll end up with a Modules\WASP\WASP ... which in PS3 will work (although it shouldn’t), but is rather frustrating.

So, to avoid ending up with folders inside folders, we can use the ZipArchive class.

The easiest way to get an actual ZipArchive is to use the “Open” method on the ZipFile class. Once you’ve done that you can easily check all the files in it: $archive.Entries | Format-Table and you can extract a single entry using ExtractToFile or the whole archive using ExtractToDirectory

So for a final example, I’ll use ZipArchive to create a script that will always unzip to a new folder (unless there’s just one single file in the zip), and will create an “archive name” folder if there isn’t already a single folder root inside the archive. In fact, as you’ll see below, I went further and forced the resulting folder to always end up named after the archive.


Add-Type -As System.IO.Compression.FileSystem

function Expand-ZipFile {
  #.Synopsis
  #  Expand a zip file, ensuring it's contents go to a single folder ...
  [CmdletBinding()]
  param(
    # The path of the zip file that needs to be extracted
    [Parameter(ValueFromPipelineByPropertyName=$true, Position=0, Mandatory=$true)]
    [Alias("PSPath")]
    $FilePath,

    # The path where we want the output folder to end up
    $FolderPath = $Pwd,

    # Make sure the resulting folder is always named the same as the archive
    $Force
  )
  process {
    $ZipFile = Get-Item $FilePath
    $Archive = [System.IO.Compression.ZipFile]::Open( $ZipFile, "Read" )
    # The place where we expect it to end up
    $Destination = join-path $FolderPath $ZipFile.BaseName
    # The root folder of the first entry ...
    $ArchiveRoot = $Archive.Entries[0].FullName.split("/",2)[0]

    # If any of the files are not in the same root folder ...
    if($Archive.Entries.FullName | Where-Object { $_.split("/",2)[0] -ne $ArchiveRoot }) {
      # extract it into a new folder:
      New-Item $Destination -Type Directory -Force
      [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory( $Archive, $Destination )
    } else {
      # otherwise, extract it to the FolderPath
      [System.IO.Compression.ZipFileExtensions]::ExtractToDirectory( $Archive, $FolderPath )

      # If there was only a single file in the archive, then we'll just output that file...
      if($Archive.Entries.Count -eq 1) {
        Get-Item (Join-Path $FolderPath $Archive.Entries[0].FullName)
      } elseif($Force) {
        # Otherwise let's make sure that we move it to where we expect it to go, in case the zip's been renamed
        if($ArchiveRoot -ne $ZipFile.BaseName) {
          Move-Item (join-path $FolderPath $ArchiveRoot) $Destination
          Get-Item $Destination
        }
      } else {
        Get-Item (Join-Path $FolderPath $ArchiveRoot)
      }
    }

    $Archive.Dispose()
  }
}
 

Update and continuation

I just posted a new version of this to PoshCode including a New-ZipFile function, just because I can’t stop myself, and I’ve always wanted a zip function that behaved exactly the way I wanted it to. If anyone want’s to add to that ZipFile module, that would be the place to do it, for now. Here’s the current version:

Long-running background tasks in ShowUI GUIs from PowerShell

A lot of the time when you’re writing ShowUI user interfaces in PowerShell, you’re just asking users for inputs, prompting them for choices, or showing them the results of some calculations (whether that be dashboards, graphics, etc).

However, sometimes you need to prompt the user for input and then do some work, and you want to present the users with output as the work progresses. With any graphical user interface, when you want to do any significant amount of work, you need to do it on a background thread. The ShowUI way to do that is to use the Invoke-Background command, which gives you an event-driven way to run a script block (or command) and capture all of it’s various outputs.

I’ll show you an example here which writes output and progress events, and show you how to handle updating the user interface with both (hopefully you’ll be able to figure out the other output options, including the errors).


Import-Module ShowUI -Req 1.4

New-Grid -Columns "*","5","70" -Rows "21","Auto","5","*" -Margin 5 {
  ProgressBar -Name ProgressBar -ColumnSpan 3 -Maximum 100 -Margin "0,0,0,5"

  TextBox -Name count -Text 12 -MinWidth 150 -Row 1

  TextBox -Name output -MinHeight 100 -ColumnSpan 3 -IsReadOnly -Row 3

  Button "Start" -Name StartButton -Column 2 -Row 1 -On_Click {
    # If you need to pass values from your form to the background function
    # They need to be serializable, and go through the -Parameter hashtable:
    Invoke-Background -Parameter @{ work = [int]$count.Text } {
      param($work=10)
      # You would do real work here, but I'll just fake it
      foreach($item in 1..$work) {
        # Anything that goes to Output triggers the OutputChanged
        Write-Output "Processing item $item of $work"
        Start-Sleep -milli 500
        # And of course, progress triggers the ProgressChanged
        $progress = (([double]$item / [double]$work) * 100)
        Write-Progress "Processing" -PercentComplete $progress
      }
    } -On_OutputChanged {
      # The actual event is on the DataContext object
      $output.Text = $args[0].DataContext.Output -join "`n" | Out-String
    } -On_ProgressChanged {
      $ProgressBar.Value = $args[0].DataContext.LastProgress.PercentComplete
    }
  }
} -Show
 

The first key, of course, is knowing that Invoke-Background sets a DataContext on the object (either the object specified to the -Control parameter, or the parent who’s event handler calls it). That DataContext is a ShowUI.PowerShellDataSource (it’s code is in the C# folder in the ShowUI module), and it has events and properties for each of the streams: Output, Error, Warning, Verbose, Debug, and Progress. It also has a property for the “Last” item from each of those streams, and a TimeStampedOutput property which collects all of the output, in order, with a NoteProperty for the Stream and the TimeStamp. More importantly, it has events which fire on every output.

The second key is knowing that the properties like “Output” contain all the output, and properties like “LastOutput” contain only the very last thing output. Since in the case of the progress we only care about the very last value, we’re able to use the “LastProgress” property. Since we want to show all of the output, rather than try to collect the “last” one each time, we can just overwrite the textbox’s text with the output array.

It’s also very important to remember that the properties are arrays, and in particular to remember that the Output is an array of PSObject, and the others are arrays of ProgressRecords, DebugRecords, ErrorRecords, etc… so we use Out-String to make sure that we convert them to a string before we set the Text of our output pane.

Hopefully this function will help you out — I’ll try to improve it’s help and examples in the next release of ShowUI.

You can do more than breathe for free…