Native Commands in PowerShell – A New Approach

Jim Truher

In this two part blog post I’m going to investigate how PowerShell can take better advantage of native executables. In the first post, I’m going to discuss a few of the ways that PowerShell can better incorporate native executables into our object oriented world and how we can use these tools to better fit into our model of more discrete operations. In the second post, I’ll be exploring some of the work to more easily convert the use of native tools into the cmdlet/object output model of PowerShell.

PowerShell provides a number of benefits to its users

• consistent parameter naming for similar uses
• a single parameter parser so errors about mis-parameter use are consistent across all commands
• output consisting of objects (no text parsing)
• common way to get help
• cross platform consistency
• easy execution of existing tools

Some of these are not unique to PowerShell, and many tools also provide some of these behaviors. Every shell has the capability of executing utilities, you could argue that this is the main use for a shell and PowerShell is no different. Additionally, UNIX systems have man pages and the mostly ubiquitous --help for getting assistance directly from the command. Microsoft Windows may provide /?, but the point is the same. It’s a way to get help from the application.

PowerShell spent much of its first decade in building up a large number of tools for system management, but does not have cmdlets for all aspects of administration on all platforms. There are a number of very excellent and vital standalone tools that target the scenarios for administration and management very well. Some of these tools have existed for many years and have grown in functionality and complexity including their own ‘mini-language’. Examples of these include:

• Package managers such as apt, yum, brew
• Source control applications such as git
• The utility for Docker management docker
• The utility for Kubernetes management kubectl
• netsh.exe which is the command for automating Microsoft Windows network configuration
• The Microsoft Windows utility net.exe for creating and managing local user accounts on Microsoft Windows

The authors of these tools have their own specific approach to solving the problems that are uniquely theirs. In some cases, utilities such as awk and sed, an entire new mini-languages were created. Mostly these are glue utilities, they were created to manipulate the outputted data into a form to be more easily filtered, or changed to fit a specific need.

Solution Options

If I want to have the familiar, comfortable PowerShell experience, there are only a few options that are available to me.

• You can re-implement the tool in managed code or script
• You can call web based APIs. For example, SWAGGER provides a very easy way to do this
  • https://github.com/Azure/autorest is one example
• You can wrap the native application in a PowerShell script
• I can just use the tool as is without change

Reimplementation

There a many benefits in a complete rewrite of a command:

• The expression of behavior can be made more “natural” to the new environment
• Performance issues can be addressed
• New code means that new technologies can be used advantageously

Issues with Reimplementations

The biggest issue with reimplementation is probably the amount of work that is needed to achieve behavior expressed in the original. This is especially the case if the reimplementor is not intimately familiar with the workings of the tool. Another issue with reimplementation is that you need to continue to track changes in the original code. This can be a challenge as depending on the activity and updates in the tool, wholescale changes can occur that then need to be reimplemented, or the reimplementation will be out of date. Worse, if the the command is the client side of a client/server app, changes in the server may negatively effect the reimplementation.

API wrapping

Many native apps use a REST endpoint to retrieve data. These can be used to interact with the data endpoint, retrieve data from it and then present it to the user. For example, the following shows how you could present the data about kubernetes pods by interacting with the REST endpoint and display the data. In comparison is the output from the native command.

kubectl get pods; get-pods.ps1|ft

NAME                     READY   STATUS      RESTARTS   AGE
hello-1589924940-rv2v2   0/1     Completed   0          3m5s
hello-1589925000-gs5n7   0/1     Completed   0          2m5s
hello-1589925060-j4bjc   0/1     Completed   0          65s
hello-1589925120-jvxtd   0/1     Completed   0          4s

Name                   Ready Status    Restarts Age
----                   ----- ------    -------- ---
hello-1589925000-gs5n7 0/1   Completed        0 00:02:05.2602110
hello-1589925060-j4bjc 0/1   Completed        0 00:01:05.2607090
hello-1589925120-jvxtd 0/1   Completed        0 00:00:04.2612030

Issues with API wrapping

The most impactful issues with this approach are about authentication and complexity. Also, simple API wrapping generally results in a command that is developer rather than administrator focused. There is quite a bit of logic wrapped up in a command to avoid just calling the API. The script that produced the output can be used to illustrate some of the problems with this approach

# retrieve data from REST endpoint
$baseUrl = "http://127.0.0.1:8001"
$urlPathBase = "api/v1/namespaces/default"
$urlResourceName = "pods"
$url = "${baseUrl}/${urlPathBase}/${urlResourceName}"
$data = (invoke-webrequest ${url}).Content | ConvertFrom-Json

# manipulate data for output
foreach ( $item in $data.Items ) {
    $replicaCount = $item.status.containerstatuses.count
    $replicaReadyCount = ($item.status.conditions | Where-Object {$_.Ready -eq "True"}).Count
    $Age = [datetime]::now.touniversaltime() - ([datetime]$item.status.conditions.lastTransitionTime[-1])
    [pscustomobject]@{
        Name     = $item.metadata.name
        Ready    = "{0}/{1}" -f $replicaReadyCount, $replicaCount
        Status   = @($item.status.containerstatuses.state.terminated.reason)[-1]
        Restarts = $item.status.containerstatuses.restartcount
        Age      = $age
        }
}

In the above example, the work of the script is broken into 2 sections.

• a section that gets the data from the REST endpoint
• a section that converts the JSON data into an object that has the specific properties I want to see

There are a few shortcuts in the first section:

• I’m not providing a parameter to retrieve different resources
• I’m not using any authentication
• I’m using what I already know with regard to the actual url to retrieve data

The second section that alters the data to a form I need by converting the JSON to a view I’m more comfortable with. With regard to the output, I made a decision to handle the presentation of elapsed time in a way that most cmdlets do.

This approach casts the problem in the light of the developer again. Much like re-implementation, there is a certain amount of code that is required just to get the data, and with this example I’m showing the absolute simplest case since I’m not doing any authentication and I _know what the endpoint to which I’m connecting. The part that is familiar is the second part of the script that creates an object that I can use with our other filters. This can be done in many different ways, I could have written this code using Select-Object as follows:

$data.Items | Select-Object -Property @{ N = "Name"; E = {$_.metadata.Name}},
     @{ N = "Ready"; E = { "{0}/{1}" -f ($_.item.status.conditions|Where-Object {$_.Ready -eq "True"}).Count, $_.status.containerstatuses.count}},
     @{ N = "Status"; E = { @($_.status.containerstatuses.state.terminated.reason)[-1]}},
     @{ N = "Restarts"; E = { $_.status.containerstatuses.restartcount}},
     @{ N = "Age"; E = { [DateTime]::now.touniversaltime() - [datetime]($_.status.conditions.lastTransitionTime[-1])}}

Regardless of how it’s written, I believe that the second section is well understood by most PowerShell scripters, but the first section is less known and needs knowledge about the service and how to authenticate.

However, I think the biggest issue with this approach is that for anything complicated (or anything more complicated than simple “gets”) is that the REST APIs are developer constructs made for developers. This means that if you want to use these REST APIs, you need to put on a developer hat and produce a solution that has a different set of problems. This is what the developer did initially. They took the available APIs (REST or otherwise) and built up the administrative experience in the application, sheltering the admin from the programming problems. In the kubernetes example above, if I needed to query the REST endpoint to see what types of resources were available, that means more calls back and forth from the service.

Native Application Wrapping

Because it is possible to call native applications easily from within PowerShell, it is possible to write a script that provides a more PowerShell-like experience. It can provide parameter handling such as prompting for mandatory parameters and tab-completion for parameter values. It can take the text output from the application and parse it into objects. This allows you to take advantage of all the post processing tools such as Sort-Object, Where-Object, etc.

The previous example can be greatly simplified and written as follows:

$data = kubectl get pods -o json | ConvertFrom-Json
$data.Items | Select-Object -Property @{ N = "Name"; E = {$_.metadata.Name}},
     @{ N = "Ready";    E = { "{0}/{1}" -f ($_.item.status.conditions|Where-Object {$_.Ready -eq "True"}).Count, $_.status.containerstatuses.count}},
     @{ N = "Status";   E = { @($_.status.containerstatuses.state.terminated.reason)[-1]}},
     @{ N = "Restarts"; E = { $_.status.containerstatuses.restartcount}},
     @{ N = "Age";      E = { [DateTime]::now.touniversaltime() - [datetime]($_.status.conditions.lastTransitionTime[-1])}}

PowerShell has tools that make it easy to convert structured text. In this case, kubectl has an option to output JSON. The ConvertFrom-Json cmdlet converts that to an object. Then we can use the same code to present data.

This approach has some advantages:

• We avoid the entire problem of how to authenticate to access the data
  • We are protected from changes in the service and API endpoint
• changes in the tool shouldn’t affect the wrapper or can be easily managed by simple changes to the script
• If the application supports uniform cross-platform execution, the wrapper can be run easily on whatever platform is needed

One of my first experiences with this was a very simple processes of getting information about PDF files with the tool pdfinfo.exe. I needed to retrieve information from a very large set of set of PDF files (1000s). I wrapped both the parameters and the output to have it behave much like a regular cmdlet. Of course, I could have just used the native app, but I wanted a command I could pipe files to and filter the results:

$a = get-childitem -rec -filt *.pdf | Get-PdfInfo | Where-Object { $_.subject -like "sibelius" }
$sa | ft file,title,subject,pagesize

File           Title                        Subject            Pagesize
----           -----                        -------            --------
SIB08.pdf      Sibelius - Finlandia, Op. 26 Trumpet            720x936 pts
SIB08.pdf      Sibelius - Finlandia, Op. 26 Viola              720x936 pts
...
SIB08.pdf      Sibelius - Finlandia, Op. 26 Cello              720x936 pts
SIB08.pdf      Sibelius - Finlandia, Op. 26 Bassoon            720x936 pts

The point of all this was that I wanted a native PowerShell experience rather than the experience provided by the standalone application.

Issues with application wrapping

The issues are roughly the same as above. There is a certain amount of programming needed to call the application. There is some programming needed to convert the text output to objects so they can participate in the PowerShell pipelines. The significant difference is that unlike the REST approach is that I don’t have extra work determining how to invoke the app. I can just invoke it. Further, it seems a more natural use of the tool. I’m familiar with the workings of the tool. I’m just parsing the output into objects. It’s important to note that if the tool emits JSON, XML, or other structured data, a lot less effort is needed to create the objects that I want.

Use the tool as is

PowerShell is a great shell, it can execute any executable, the same way that any good shell can do. No change is needed, just run kubectl and you’re done!

Issues with simple execution

If I want the native application just running the executable is not the only criteria for fully participating in the PowerShell environment. Tab-completion for parameters and auto-generated values for parameters is something that PowerShell users have come to expect. For most native executables, there are tab-completers for most shells (bash/zsh) More importantly, is the object pipeline which is one of the places that PowerShell brings so much, well, power. The ability to not have to use cut, sed, awk, etc. but to treat the results logically is extremely compelling.

In the next post, I’ll go through a couple possible approaches for taking advantage of all these native tools. And I’ll investigate whether there’s a way to automatically use the help for the native utility to generate cmdlets and convert output to objects that participate in the PowerShell pipeline.You can take a peek at my experiments by downloading this module: https://www.powershellgallery.com/packages/Microsoft.PowerShell.KubeCtl

James Truher Software Engineer PowerShell Team

Jim Truher Senior Software Engineer, PowerShell

Follow