September 26th, 2019

Integrate Linux Commands into Windows with PowerShell and the Windows Subsystem for Linux

Mike Battista
Senior Program Manager

A common question Windows developers have is “why doesn’t Windows have <INSERT FAVORITE LINUX COMMAND HERE> yet?”. Whether longing for a powerful pager like less or wanting to use familiar commands like grep or sed, Windows developers desire easy access to these commands as part of their core workflow.

The Windows Subsystem for Linux (WSL) was a huge step forward here, enabling developers to call through to Linux commands from Windows by proxying them through wsl.exe (e.g. wsl ls). While a significant improvement, the experience is lacking in several ways:

  • Prefixing commands with wsl is tedious and unnatural
  • Windows paths passed as arguments don’t often resolve due to backslashes being interpreted as escape characters rather than directory separators
  • Windows paths passed as arguments don’t often resolve due to not being translated to the appropriate mount point within WSL
  • Default parameters defined in WSL login profiles with aliases and environment variables aren’t honored
  • Linux path completion is not supported
  • Command completion is not supported
  • Argument completion is not supported

The result of these shortcomings is that Linux commands feel like second-class citizens to Windows and are harder to use than they should be. For a command to feel like a native Windows command, we’ll need to address these issues.

PowerShell Function Wrappers

We can remove the need to prefix commands with wsl, handle the translation of Windows paths to WSL paths, and support command completion with PowerShell function wrappers. The basic requirements of the wrappers are:

  • There should be one function wrapper per Linux command with the same name as the command
  • The wrapper should recognize Windows paths passed as arguments and translate them to WSL paths
  • The wrapper should invoke wsl with the corresponding Linux command, piping in any pipeline input and passing on any command line arguments passed to the function

Since this template can be applied to any command, we can abstract the definition of these wrappers and generate them dynamically from a list of commands to import.

# The commands to import.
$commands = "awk", "emacs", "grep", "head", "less", "ls", "man", "sed", "seq", "ssh", "tail", "vim"

# Register a function for each command.
$commands | ForEach-Object { Invoke-Expression @"
Remove-Alias $_ -Force -ErrorAction Ignore
function global:$_() {
    for (`$i = 0; `$i -lt `$args.Count; `$i++) {
        # If a path is absolute with a qualifier (e.g. C:), run it through wslpath to map it to the appropriate mount point.
        if (Split-Path `$args[`$i] -IsAbsolute -ErrorAction Ignore) {
            `$args[`$i] = Format-WslArgument (wsl.exe wslpath (`$args[`$i] -replace "\\", "/"))
        # If a path is relative, the current working directory will be translated to an appropriate mount point, so just format it.
        } elseif (Test-Path `$args[`$i] -ErrorAction Ignore) {
            `$args[`$i] = Format-WslArgument (`$args[`$i] -replace "\\", "/")
        }
    }

    if (`$input.MoveNext()) {
        `$input.Reset()
        `$input | wsl.exe $_ (`$args -split ' ')
    } else {
        wsl.exe $_ (`$args -split ' ')
    }
}
"@
}

The $command list defines the commands to import. Then we dynamically generate the function wrapper for each using the Invoke-Expression command (first removing any aliases that would conflict with the function).

The function loops through the command line arguments, identifies Windows paths using the Split-Path and Test-Path commands, then converts those paths to WSL paths. We run the paths through a helper function we’ll define later called Format-WslArgument that escapes special characters like spaces and parentheses that would otherwise be misinterpreted.

Finally, we pass on pipeline input and any command line arguments through to wsl.

With these function wrappers in place, we can now call our favorite Linux commands in a more natural way without having to prefix them with wsl or worry about how Windows paths are translated to WSL paths:

  • man bash
  • less -i $profile.CurrentUserAllHosts
  • ls -Al C:\Windows\ | less
  • grep -Ein error *.log
  • tail -f *.log

A starter set of commands is shown here, but you can generate a wrapper for any Linux command simply by adding it to the list. If you add this code to your PowerShell profile, these commands will be available to you in every PowerShell session just like native commands!

Default Parameters

It is common in Linux to define aliases and/or environment variables within login profiles to set default parameters for commands you use frequently (e.g. alias ls=ls -AFh or export LESS=-i). One of the drawbacks of proxying through a non-interactive shell via wsl.exe is that login profiles are not loaded, so these default parameters are not available (i.e. ls within WSL and wsl ls would behave differently with the alias defined above).

PowerShell provides $PSDefaultParameterValues, a standard mechanism to define default parameter values, but only for cmdlets and advanced functions. Turning our function wrappers into advanced functions is possible but introduces complications (e.g. PowerShell matches partial parameter names (like matching -a for -ArgumentList) which will conflict with Linux commands that accept the partial names as arguments), and the syntax for defining default values would be less than ideal for this scenario (requiring the name of a parameter in the key for defining the default arguments as opposed to just the command name).

With a small change to our function wrappers, we can introduce a model similar to $PSDefaultParameterValues and enable default parameters for Linux commands!

function global:$_() {
    …

    `$defaultArgs = ((`$WslDefaultParameterValues.$_ -split ' '), "")[`$WslDefaultParameterValues.Disabled -eq `$true]
    if (`$input.MoveNext()) {
        `$input.Reset()
        `$input | wsl.exe $_ `$defaultArgs (`$args -split ' ')
    } else {
        wsl.exe $_ `$defaultArgs (`$args -split ' ')
    }
}

By passing $WslDefaultParameterValues down into the command line we send through wsl.exe, you can now add statements like below to your PowerShell profile to configure default parameters!

$WslDefaultParameterValues["grep"] = "-E"
$WslDefaultParameterValues["less"] = "-i"
$WslDefaultParameterValues["ls"] = "-AFh --group-directories-first" 

Since this is modeled after $PSDefaultParameterValues, you can temporarily disable them easily by setting the "Disabled" key to $true. A separate hash table has the additional benefit of being able to disable $WslDefaultParameterValues separately from $PSDefaultParameterValues.

Argument Completion

PowerShell allows you to register argument completers with the Register-ArgumentCompleter command. Bash has powerful programmable completion facilities. WSL lets you call into bash from PowerShell. If we can register argument completers for our PowerShell function wrappers and call through to bash to generate the completions, we can get rich argument completion with the same fidelity as within bash itself!

# Register an ArgumentCompleter that shims bash's programmable completion.
Register-ArgumentCompleter -CommandName $commands -ScriptBlock {
    param($wordToComplete, $commandAst, $cursorPosition)

    # Map the command to the appropriate bash completion function.
    $F = switch ($commandAst.CommandElements[0].Value) {
        {$_ -in "awk", "grep", "head", "less", "ls", "sed", "seq", "tail"} {
            "_longopt"
            break
        }

        "man" {
            "_man"
            break
        }

        "ssh" {
            "_ssh"
            break
        }

        Default {
            "_minimal"
            break
        }
    }

    # Populate bash programmable completion variables.
    $COMP_LINE = "`"$commandAst`""
    $COMP_WORDS = "('$($commandAst.CommandElements.Extent.Text -join "' '")')" -replace "''", "'"
    for ($i = 1; $i -lt $commandAst.CommandElements.Count; $i++) {
        $extent = $commandAst.CommandElements[$i].Extent
        if ($cursorPosition -lt $extent.EndColumnNumber) {
            # The cursor is in the middle of a word to complete.
            $previousWord = $commandAst.CommandElements[$i - 1].Extent.Text
            $COMP_CWORD = $i
            break
        } elseif ($cursorPosition -eq $extent.EndColumnNumber) {
            # The cursor is immediately after the current word.
            $previousWord = $extent.Text
            $COMP_CWORD = $i + 1
            break
        } elseif ($cursorPosition -lt $extent.StartColumnNumber) {
            # The cursor is within whitespace between the previous and current words.
            $previousWord = $commandAst.CommandElements[$i - 1].Extent.Text
            $COMP_CWORD = $i
            break
        } elseif ($i -eq $commandAst.CommandElements.Count - 1 -and $cursorPosition -gt $extent.EndColumnNumber) {
            # The cursor is within whitespace at the end of the line.
            $previousWord = $extent.Text
            $COMP_CWORD = $i + 1
            break
        }
    }

    # Repopulate bash programmable completion variables for scenarios like '/mnt/c/Program Files'/<TAB> where <TAB> should continue completing the quoted path.
    $currentExtent = $commandAst.CommandElements[$COMP_CWORD].Extent
    $previousExtent = $commandAst.CommandElements[$COMP_CWORD - 1].Extent
    if ($currentExtent.Text -like "/*" -and $currentExtent.StartColumnNumber -eq $previousExtent.EndColumnNumber) {
        $COMP_LINE = $COMP_LINE -replace "$($previousExtent.Text)$($currentExtent.Text)", $wordToComplete
        $COMP_WORDS = $COMP_WORDS -replace "$($previousExtent.Text) '$($currentExtent.Text)'", $wordToComplete
        $previousWord = $commandAst.CommandElements[$COMP_CWORD - 2].Extent.Text
        $COMP_CWORD -= 1
    }

    # Build the command to pass to WSL.
    $command = $commandAst.CommandElements[0].Value
    $bashCompletion = ". /usr/share/bash-completion/bash_completion 2> /dev/null"
    $commandCompletion = ". /usr/share/bash-completion/completions/$command 2> /dev/null"
    $COMPINPUT = "COMP_LINE=$COMP_LINE; COMP_WORDS=$COMP_WORDS; COMP_CWORD=$COMP_CWORD; COMP_POINT=$cursorPosition"
    $COMPGEN = "bind `"set completion-ignore-case on`" 2> /dev/null; $F `"$command`" `"$wordToComplete`" `"$previousWord`" 2> /dev/null"
    $COMPREPLY = "IFS=`$'\n'; echo `"`${COMPREPLY[*]}`""
    $commandLine = "$bashCompletion; $commandCompletion; $COMPINPUT; $COMPGEN; $COMPREPLY" -split ' '

    # Invoke bash completion and return CompletionResults.
    $previousCompletionText = ""
    (wsl.exe $commandLine) -split '\n' |
    Sort-Object -Unique -CaseSensitive |
    ForEach-Object {
        if ($wordToComplete -match "(.*=).*") {
            $completionText = Format-WslArgument ($Matches[1] + $_) $true
            $listItemText = $_
        } else {
            $completionText = Format-WslArgument $_ $true
            $listItemText = $completionText
        }

        if ($completionText -eq $previousCompletionText) {
            # Differentiate completions that differ only by case otherwise PowerShell will view them as duplicate.
            $listItemText += ' '
        }

        $previousCompletionText = $completionText
        [System.Management.Automation.CompletionResult]::new($completionText, $listItemText, 'ParameterName', $completionText)
    }
}

# Helper function to escape characters in arguments passed to WSL that would otherwise be misinterpreted.
function global:Format-WslArgument([string]$arg, [bool]$interactive) {
    if ($interactive -and $arg.Contains(" ")) {
        return "'$arg'"
    } else {
        return ($arg -replace " ", "\ ") -replace "([()|])", ('\$1', '`$1')[$interactive]
    }
}

The code is a bit dense without an understanding of some bash internals, but basically:

  • We register the argument completer for all of our function wrappers by passing the $commands list to the -CommandName parameter of Register-ArgumentCompleter
  • We map each command to the shell function bash uses to complete for it ($F which is named after complete -F <FUNCTION> used to define completion specs in bash)
  • We convert PowerShell’s $wordToComplete, $commandAst, and $cursorPosition arguments into the format expected by bash completion functions per the bash programmable completion spec
  • We build a command line that we can pass to wsl.exe that ensures the completion environment is set up correctly, invokes the appropriate completion function, then outputs a string containing the completion results separated by new lines
  • We then invoke wsl with the command line, split the output string on the new line separator, then generate CompletionResults for each, sorting them, and escaping characters like spaces and parentheses that would otherwise be misinterpreted

The end result of this is now our Linux command wrappers will use the exact same completion that bash uses! For example:

  • ssh -c <TAB> -J <TAB> -m <TAB> -O <TAB> -o <TAB> -Q <TAB> -w <TAB> -b <TAB>

Each completion will provide values specific to the argument before it, reading in configuration data like known hosts from within WSL!

<TAB> will cycle through options. <Ctrl + Space> will show all available options.

Additionally, since bash completion is now in charge, you can resolve Linux paths directly within PowerShell!

  • less /etc/<TAB>
  • ls /usr/share/<TAB>
  • vim ~/.bash<TAB>

In cases where bash completion doesn’t return any results, PowerShell falls back to its default completion which will resolve Windows paths, effectively enabling you to resolve both Linux paths and Windows paths at will.

Conclusion

With PowerShell and WSL, we can integrate Linux commands into Windows just as if they were native applications. No need to hunt around for Win32 builds of Linux utilities or be forced to interrupt your workflow to drop into a Linux shell. Just install WSL, set up your PowerShell profile, and list the commands you want to import! The rich argument completion shown here of both command options and Linux and Windows file paths is an experience even native Windows commands don’t provide today.

The complete source code described above as well as additional guidance for incorporating it into your workflow is available at https://github.com/mikebattista/PowerShell-WSL-Interop.

Which Linux commands do you find most useful? What other parts of your developer workflow do you find lacking on Windows?

Let us know in the comments below or over on GitHub!

Author

Mike Battista
Senior Program Manager

13 comments

Discussion is closed. Login to edit/delete existing comments.

  • Sasha Firsov

    Powershell functions are not only options anymore. wsl-call allows to create symlinks to each linux in wsl command and if placed in PATH run as usual linux cli without wsl.exe upfront.
    https://github.com/sashafirsov/wsl-call

  • Théo FORAY

    It doesn’t seem to remember which commands I add to my PowerShell every time I close it :/

    • Mike BattistaMicrosoft employee Author

      Have you added the Import-WslCommand call to your PowerShell profile?

  • Julian Knight

    Nice! Just discovered this after coming back to WSL and the new Terminal.

    Maybe I’m greedy but it would be fantastic to get colour output, especially on an ls command 🙂

    • Mike BattistaMicrosoft employee Author

      Does the –color parameter work for you? I add this as a default parameter for ls which works for me.

      • David MoncadaMicrosoft employee

        Follow-up: is there a way to tweak the colors scheme used for the colorized output? I tried with:

        $WslEnvironmentVariables["LS_COLORS"] = "colors here..."

        But no luck =(

      • Mike BattistaMicrosoft employee Author

        Yes. From my quick tests, when setting LS_COLORS you need to ensure you set the colors on the right file types for Windows files which isn't exactly obvious.

        For example, changing di colors will not have any effect on Windows directories, but changing ow colors will.

        The best guide I could find on LS_COLORS was https://askubuntu.com/questions/466198/how-do-i-change-the-color-for-directories-with-ls-in-the-console.

        Once you know the foreground/background codes, you can narrow down which file type to modify based on the default colors you see...

        Read more
  • David Driscoll

    Is there a nice way to get completion with command lines that use python. It would be great to suppport the azure-cli items using az!

    • Mike BattistaMicrosoft employee Author

      As long as you install the azure-cli tools inside WSL, and it registers a bash completion spec (looks like it does), you can just add Import-WslCommand "az" to your PowerShell profile and it will allow you to call az from PowerShell with argument completion. Just tried it myself and it works great as long as you’re ok with az running within WSL. Make sure you take the latest code from the GitHub repo since it...

      Read more
  • ryan wolfe

    I don’t get it when you could just use Linux. Seems like extra plumbing in an already clogged toilet to me.

    • Peter da Silva

      Or at least go the other way and run whatever occasional Windows commands you need from bash under wsl, since the UNIX "execl" lets bash securely and correctly expand arbitrary file names and pass them to commands without using musical quotes (I notice the script concatenates file names separated by spaces, yuck!).

      The best thing that Windows could do is adopt the UNIX fork/exec/wait argument passing and process hierarchy, and use the "jam everything into a...

      Read more
  • Meteorhead

    This is mighty cool and an awesome post!