January 23rd, 2023

PowerShell/OpenSSH Team Investments for 2023

For 2022, the team had set out lots of objectives and I’m happy to say we accomplished most of them!

This year, we’re continuing on improving quality and also adding new features across our portfolio based on feedback from the community, partners, and customers.

PowerShell 7 in Windows

PowerShell 7 shipped in Windows continues to be a common question asked by the community and we’re continuing to work on making it easier to install PS7 on Windows. As a reminder, PowerShell 7 is built on the latest .NET which does not have the same support lifecycle as Windows. We can’t ship something as a feature in Windows that does not align with Windows’ support lifecycle. While we continue to explore many options, we do plan on proceeding with a Install-PowerShell7 command that could be shipped as part of Windows. Please add your feedback to that RFC particularly if it’ll help you or your organization (for both Windows client and server SKUs).

PowerShell 7.4

At the end of 2022, we were able to ship Preview 1 of PowerShell 7.4. It contains fixes, features, and enhancements that were merged, but not absolutely needed (or perhaps too risky) for the 7.3 release.

Once .NET 8 previews are available, we’ll be moving to those releases for 7.4 and continue to have monthly preview releases to get feedback on changes from the team and the community.

Much thanks to our community for continuing to identify issues as well as submitting pull requests to add features or fix bugs!

Special thanks to Ilya who is our long term community maintainer for PS7 and continues to be highly engaged in the project!

We will continue to address issues and fix bugs as they are reported in our GitHub repo, however, there are a few specific areas where we are focusing efforts for the 7.4 release:

FeedbackProvider

As part of the 7.4-preview.1 release, we added a new subsystem plugin for FeedbackProvider to replace the legacy SuggestionFramework. Many folks may not realize it, but they have already been using the SuggestionFramework which was already in Windows PowerShell. If you tried to run a command or script in the current working directory without the “./” prefix, one of the suggestions built into the SuggestionFramework was to suggest adding the “./” prefix to properly execute the command.

The PSCommandNotFoundSuggestion Experimental Feature was built on top of the SuggestionFramework which exposed some limitations that prevented further work on enhancing and expanding that feature.

The PSFeedbackProvider Experimental Feature addresses these issues and will allow for plugins to be delivered via PowerShellGallery.

$env:PSModulePath

We’ve had discussions on some of the problems with $env:PSModulePath and would like to move it out of the Documents folder. However, many tools (including PowerShellGet) have built-in knowledge on where to find modules in the Documents folder. A necessary step is to expose an API to allow tools to discover the location of the user, system, and shared module paths. After that, we can then figure out how to move the user part of $env:PSModulePath out of the Documents folder.

More Linux distro/version support

Our longer term goal is to support all Linux distros, versions, and architectures that are supported by .NET. However, one of our biggest challenges is regular validation across all supported platforms as even 1 or 2 test failures across multiple platforms can result in lots of investigation time. We continue to make incremental progress towards stabilizing the tests as well as making it easier for our engineering team to onboard and validate against new platforms. We also plan to add support for some popular requested platforms for 7.4 and continue to improve coverage over time.

Enhanced Native Commands

An area that we’ve been focusing on for the past few releases is improving the experience for using native commands (i.e. executables) within PowerShell. We’ve already made many improvements so that native commands work more naturally in PowerShell and also similar to how they work in other shells.

We also recognize that although cmdlets provide a more consistent experience across platforms, there are still many native commands that are used widely that are not available as cmdlets (even with tooling like Crescendo) being available.

This is an experimental area that is unlikely to be ready for production by the time 7.4 GA releases. Conceptually, native commands that emit JSON (instead of just text) can provide a more consistent experience within PowerShell by enabling discovery, filtering, formatting, tab completion, etc… with structured data instead of freeform text. Although JSON is emitted, the user would work with output as if they were deserialized objects using their existing knowledge of PowerShell and cmdlets.

A detailed RFC will be published to the PowerShell RFC repo detailing the design early in 2023.

JSON adaptor

Enhancing native commands requires making changes to existing code. We recognize that some tools cannot be modified to emit JSON so we’re also exploring enabling JSON adapters so that someone else can write a command that can convert the text output of a native command to JSON. There are existing projects like JC which already has a bunch of Python scripts that can convert the output of many popular commands to JSON.

An initial RFC has been published to get feedback for this feature.

Native pipe

PowerShell (via .NET) converts output from native commands to text. This can break scenarios where you want to pipe binary data from one command into another. We expect to have a new experimental feature to enable the pipeline to correctly handle binary output from native commands.

Desired State Configuration v3

I published a RFC awhile back on a new direction for DSC particularly how it integrates with PowerShell 7. The high level goals at the time:

  • Cross platform
  • Bring your own agent (meaning DSC would simply be a platform for invoking resources, orchestration would be handled by other existing tools like Automanage Machine Configuration, Chef, Puppet, Ansible, etc…)
  • Separation of DSC from PowerShell engine so the two can iterate independently
  • Deprecate use of MOF

We published a beta of PSDesiredStateConfiguration module which implements some of these goals. Since that time, we’ve gotten feedback from the DSC Community and Microsoft partners and shifting our strategy a bit.

In addition to the above goals, we are adding some new ones:

  • Enable authoring of DSC Resources in any language
  • Simplified authoring experience of DSC Resources in PowerShell script
  • Move from MOF to JSON

We’re already actively working with the WinGet team to enable their configuration scenarios building on top of DSC v3.

Expect some detailed RFCs to be published on this within the first quarter of 2023 where you can provide feedback.

OpenSSH 9.1

At the end 2022, we released the Windows port of OpenSSH 9.1 to our Github repo. We continue to validate the port of OpenSSH 9.1 for Windows along with addressing user feedback in preparation of shipping it in Windows 11 as an updated Feature-on-Demand. We appreciate the bug reports and feedback and will continue to work on addressing reported issues within that repo.

We’re also planning on shipping an update to the Feature-on-Demand for older versions of Windows.

SSHDConfig

As part of the DSC v3 work described previously, we’re working on a cross-platform DSC resource for managing SSHD configuration. This will make it easier to audit and apply configuration across thousands of Windows and Linux systems in a programmatic and declarative way. This DSC resource is being written in Rust and will be Open Source.

SSHArc

We announced almost a year ago the public preview of SSHArc which enables secure (multi-factor auth via Azure) and easy access (AzCLI and AzPowerShell) to your Arc-enabled servers without exposing a public ip address or having a firewall port opened for SSH.

We will continue to address feedback and quality towards General Availability this year.

PowerShell Gallery

The original architecture of PowerShellGallery.com has hit limitations in terms of scale for our current customer needs. The number of modules being published and downloaded has exceeded the scale goals of the original code base. We’ve made targeted fixes to unblock partners and customers, but need a new architecture to satisfy the expected growth for the next 10 years.

REST API support for publishing/unlisting

For the near term, we’ve had requests from both partners and customers to enable publishing resources to PowerShell initially as unlisted. They can then be tested end-to-end and then have an API they can call to list the new version and make it public. For this we’ll need to create a new REST API and expose the capability through PowerShellGet v3.

Private Galleries

As noted above, scale and reliability have been challenges with the current architecture so we are partnering with the Azure Container Registry team to leverage ACR as a backend store for PowerShell artifacts.

Initially, we’re looking at using this as a way for customers to have private galleries that can guaranteee a higher level of availability. Publishing, searching, and installing from a private gallery will still be handled with PowerShellGet v3.

PowerShellGet v3

This project has turned out to be more challenging than we originally anticipated. We recently shipped our 18th preview release and continue to focus on quality towards a Release Candidate and eventual GA release this year.

Concurrent installations for v3.1

A very common request is to improve performance particularly for installing modules. This is particularly needed for automation and continuous integration use cases where you want the latest version of a set of modules. For example, the Azure PowerShell module can take quite a bit of time to install because it’s actually a large family of modules.

Install-PSResource already supports a hashtable of modules to install via the -RequiredResource parameter. However, it currently installs them all serially. Instead, we need to build a dependency graph removing and enable downloading and installing to occur concurrently.

Nuget.org dependencies for v3.1

Modules and scripts commonly use .NET APIs from assemblies that aren’t shipped with PowerShell. Currently, each module needs to redistribute the assemblies with their module and sometimes multiple copies for different platforms/architectures. Instead, the module manifest should declare the assembly and version required and have PowerShellGet retrieve the dependency from Nuget.org, expand the nupkg, and extract the assemblies appropriate for the current system runtime.

FeedbackProvider for PowerShellGet

With the new FeedbackProvider subsystem in PS7.4, we can now enable suggestions for installing a module from PowerShellGallery if the user attempts to execute a cmdlet from a module that is not installed.

PowerShell VSCode-Extension/EditorServices

We continue to have frequent preview releases leading towards new stable releases addressing user reported issues. There’s a few areas we’re focusing our efforts:

  • Semantic syntax highlighting fixes
  • Syncing debugger breakpoints
  • Support WDAC environments
  • Popular user requests

PlatyPS v2

PlatyPS is for module authors to generate updateable help from markdown. It’s used by teams at Microsoft that publish PowerShell modules. Although preview 1 was published awhile ago, we will continue working on this towards making 2.0 generally available.

Crescendo

Since the release of Crescendo, the number of modules built using Crescendo has been steadily growing.

Based on user reported requests and issues, we released Crescendo 1.1 preview 1 at end of 2022. We expect general availability of Crescendo 1.1 in the first half of 2023.

PSReadLine

We continue to address issues and also add small features to improve our prediction experience. As more predictors get published, we’re looking at ways to improve how results from multiple predictors get presented.

Other modules

Continue to address issues and review community contributions for our other modules such as PSScriptAnalyzer, ConsoleGuiTools, SecretManagement, and other modules as we find them.

Conclusion

As you can see, we have lots of work planned! Some work has already started while some other will come later this year. Please continue to provide us feedback on each of our projects on how we can make them more productive for you!

Author

2 comments

Discussion is closed. Login to edit/delete existing comments.

  • Justin Grote · Edited

    Excellent update Steve, these are great commitments that have a broad impact on the larger PowerShell community.

    I'm very happy to see the renewed interest in making the installed PSModulePath more pliable.

    For PowerShell Gallery, could you expand upon the decision to use Azure Container Registry? This will require the NuGet packages to also be wrapped in an OCI artifact, and this all seems very heavyweight compared to what nuget.org does, which is publish packages to a...

    Read more
    • Steve LeeMicrosoft employee

      Module authors just use `PowerShellGet` and it’ll abstract whether you’re publishing to nuget or a container registry. More details of why and how with ACR will come in the future.