Validating Self-Signed Certificates From .Net and PowerShell

In this article I'm going to present a module that helps you deal with one of the common problems for Windows PowerShell users (and even .Net developers) who are trying to interact from command-line applications with web interfaces (especially those that are hosted internally): Self-signed certificates, and how to ignore the errors that come when you try to validate them. If you don't care about why I wrote it the way I wrote it, you can just check out the module on GitHub or install it with PoshCode or PowerShellGet by running:

Install-Module TunableSSLValidator

The module includes commands for importing certificates from files, loading them from the web server response of an http url, importing them to the Windows certificate store (to be trusted), and temporarily trusting them for a single PowerShell session. It also includes proxy function command wrappers for Invoke-WebRequest and Invoke-RestMethod to add an -Insecure switch which allows single queries to ignore invalid SSL certificates.

One caveat is that the way that it works is based on implementing the ServerCertificateValidationCallback, and the results of that callback are cached by your system, so in testing I've found that if you allow a certain URL, that URL is going to be valid for several seconds until the cache expires, so if you make several repeated calls to the same URL, and only the first one is flagged -insecure, they may all succeed. I can't remember what the documentation says about the timing or flushing the cache right now, but although it's obviously not a security issue, I am going to look for a way to flush that if it's possible.

Some Background

The root of the problem is that the web request classes in .Net automatically validate SSL certificates, and until version 4.5 there was not an easy way to ignore SSL errors or skip the validation on an individual request. That means that when you try to get a web page from a server like the computer science house at RIT, you get an error like:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

For most linux command line apps like curl you have a parameter like --insecure which allows you to just ignore invalid SSL certificates for a single web request, but for .Net and PowerShell there is no such a flag, resulting in a lot of questions on StackOverflow and other such websites about the best way to deal with them.

Beginning with version 3.0, PowerShell ships with Invoke-WebRequest and Invoke-RestMethod cmdlets, but these cmdlets don't have an -insecure switch, and so you can't use them unless you disable validation. The module I'm going to present here will fix that, and I want to show you how I did it so you can do the same thing for other similar cmdlets (like the Export-ODataEndpointProxy cmdlet that showed up in the July preview of PowerShell 5, and the cmdlets it generates).

Removing SSL Validation

There are a couple of common approaches to this problem in PowerShell (and .Net too, actually). The first is to just add the SSL certificate to your trusted store. The second is to set the ServicePointManager's ServerCertificateValidationCallback to a function that just returns TRUE -- thus ignoring all certificate errors.

That first solution is probably the right one, and I actually wrote a couple of function in this module to make it easier (one to fetch a cert from a web server, and another to add a certificate to your trusted store). Of course, it would be even better to set up an actual trusted root certificate authority (for instance, using Active Directory), but the bottom line is that either of these require permanently altering the trust of the computer in order to make a web request. The problem is that if you're writing scripts, cmdlets or .Net applications that may be used by others, it basically amounts to telling your potential users they have to deal with the problem themselves (which, for the record, is exactly what Microsoft did when they wrote their web cmdlets for PowerShell 3).

So if you want to proactively avoid SSL errors, you have to set the ServerCertificateValidationCallback. For certain situations, it's enough to just do this in PowerShell:

$url = "https://csh.rit.edu"
$web = New-Object Net.WebClient
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
$output = $web.DownloadString($url)
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null

Note, that I set the ServerCertificateValidationCallback back to $null when I was done -- this restores normal validation, so I have, in effect, only disabled the validation for that one call (since PowerShell isn't generally multi-threaded, I don't normally have to worry about other threads being affeected). However, setting the ServerCertificateValidationCallback to a scriptblock won't work for an asynchronous callback (one that happens on a task thread), because the other thread won't have a runspace to execute the script on. So, if you try to use the same technique with Invoke-WebRequest or Invoke-RestMethod instead of calling the .Net WebClient methods directly, you'll just get a different error:

The underlying connection was closed: An unexpected error occurred on a send.

If you dig in a little bit, you'll find that the InnerException of the error was:

There is no Runspace available to run scripts in this thread. You can provide one in the DefaultRunspace property
of the System.Management.Automation.Runspaces.Runspace type. The script block you attempted to invoke was:  $true

Obviously this approach doesn't work at that point, and the Invoke-WebRequest cmdlet is not the only place which will call web APIs on background threads, so to make those cmdlets and APIs work, we need to write it in C# (and do so in a way that's flexible enough that we can control it from PowerShell).

Additionally, simply returning true will disable all validation, and that's not really a safe practice -- it's certainly not what you want to do all of the time. If you're on .Net 4.5 or later, you can set the callback on the a raw HttpWebRequest and only affect that one request, but obviously that only works if you write your code at that low level, don't use the PowerShell cmdlets, and are on the latest version of .Net -- plus, you still have to write the logic that determines when that should happen.

The bottom line is that what we probably want is something like an insecure switch on the PowerShell cmdlets, so that we can make a specific request be insecure, and then when we were writing functions, we could just pass that parameter up to our functions.

Custom SSL Validator

There are several example validators on the Using Trusted Roots Respectfully page from mono project, but I'm going to try to blend a few of those validators to give us the option of tuning the SSL validation to our specific needs.

The point is that I don't want to weaken all validation, I just want to trust a specific cert for a specific domain, or perhaps just ignore problems on one domain, or make one specific request regardless of whether the SSL certificate is valid or not.

Let's look at the callback and see the information we have to work with:

bool TunableValidationCallback(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
  1. The sender is usually going to be the WebRequest that was calling an https domain that failed validation.
  2. The certificate, of course, is the one the actually failed ...
  3. The chain is the series of certificates that issued the original one, back to the root certificate authority, along with the trust information about them.
  4. The sslPolicyErrors tells us what went wrong: Was there no cert? Was the cert for the wrong domain? Was the root CA not trusted?

So, what I've written is first a check for the three main SSL errors, and a way to pre-emptively ignore them once, or post-humously trust a certificate that failed the first time, as well as some better error messages (which have to be output using Console.Error.WriteLine rather than Write-Error because they might be running on a background thread).

For now, that's enough of an explanation, I've posted the tunable SSL validator code to github, and this blog post as the ReadMe, where I'll update it with more details if need be.

A Fresh Start

A few weeks ago, someone somehow injected a shell onto my WordPress powered blog site. They managed to get multiple hacking tools installed, delete files required for hosting, and who knows what else. Needless to say I was frustrated and upset. In fact, I was so frustrated that to get rid of it I just tore down the whole site.

I apologize for all the hundreds of broken links that are pointed at my domain right now. I have a database backup, but I want to try a few other blogging tools out, and I am not sure I can trust the downloads or even images on the old site -- I don't want to restore all the posts and then find out the downloads are infected or the images have been replaced. Furthermore, when I stood up a new WordPress site, it did not perform well on my DreamHost server, and I started to wonder if I needed new hosting or new blogging tools.

A state of flux

In the end, to try switching to a static site generator instead of a server-side blogging tool, so I went to StaticGen and I've started working my way through some of the many interesting options. I'll probably be switching back and forth between them over the coming days, so the site may change quite a bit (not just visually, but in permalinks, etc.) until I settle on one.

I also decided that I will not restore the database backup and deal with cleaning all the downloads that have accumulated in the last ... almost 20 years of blogging (isn't that crazy?). Instead, I will start over. I can still restore individual articles from my old blog by hand (at least the ones that people really are still looking for), and that will give me the chance to update the content and to convert them to markdown (yeah, I think textile lost that battle).

Please let me know when you hit a missing page that you want back, and I'll move that up my priority list.

Missing Content

Since I've blown away all my old blog content, there are a lot of pages missing that were really useful. I'm going to bring the best of them back, and I have plans to actually rewrite some of the older ones to update them (part of the reason I let the old ones go is that some of them were so out-of-date that they might have been misleading).

If there is one or more of my old articles in particular that you'd like to see come back, please post a comment below!

About Huddled Masses

This is web site is dedicated to the musings of Joel Bennett (aka Jaykul) about technology, software, software development, the web, and the world.

Any resemblance of the views expressed and the views of my employer, my terminal, or the view out my window are purely coincidental. The resemblance between them and my own views is non-deterministic. The question of the existence of views in the absence of anyone to hold them is left as an exercise for the reader.

The Man of Many Hats

Not like the giant companies of great fame,
With componentized factories cranking out clones;
Here on our white-washed pages shall stand
A solitary man with a keyboard whose products
Are the collected sparks, and his name:
The DevOps King.

—Joel Bennett

With apologies to Emma Lazarus:

The New Colossus

Not like the brazen giant of Greek fame,
With conquering limbs astride from land to land;
Here at our sea-washed, sunset gates shall stand
A mighty woman with a torch, whose flame
Is the imprisoned lightning, and her name
Mother of Exiles. From her beacon-hand
Glows world-wide welcome; her mild eyes command
The air-bridged harbor that twin cities frame.
"Keep ancient lands, your storied pomp!" cries she
With silent lips. "Give me your tired, your poor,
Your huddled masses yearning to breathe free,
The wretched refuse of your teeming shore.
Send these, the homeless, tempest-tost to me,
I lift my lamp beside the golden door!"

—Emma Lazarus

Please Note: I occasionally link to things I think are great. When I do, I occasionally find a "referral code" so I can make a little cash. I promise that I don't link to anything just because of the cash (I wouldn't cross the street for the amount of cash those links bring in, never mind write a whole blog post), and I promise I won't let anyone buy links in my blog content ... but I do not promise that things I link to will stay great as time passes, nor that you will agree with me about their greatness!