PowerShell Team 2021 Investments

Steve Lee

Steve

PowerShell Team 2021 Investments

The PowerShell Team plans engineering work per semester (6 months at a time). However, PowerShell itself follows the .NET schedule of release every year. In this blog post, we’ll cover the year long plan for PowerShell 7.2 as well as tooling and modules work we have planned for the next semester.

PowerShell 7.2

PowerShell 7.2 is the next major release of PowerShell 7, built on top of .NET 6 and targeting a Generally Available supported release towards the end of 2021 (following the .NET 6 schedule). Like .NET 6, PowerShell 7.2 will be a long-term servicing (LTS) release, meaning it will be supported for ~3 years under the terms of .NET 6’s support lifecycle.

As such, PowerShell 7.0 will go out of support ~3 months after the GA of PowerShell 7.2 (roughly February 2022). This is aligned with .NET Core 3.1 support lifecycle.

However, much of the PowerShell Team’s work exists outside of PowerShell repo; we plan to actively invest in the following themes and features over the next 6-12 months:

Process Improvements and 7.2 Issues/PRs

We’re continuing to operationalize PowerShell Working Groups as a way to scale our development to meet the increasing number of issues and pull requests in the PowerShell project as community contributions continue to grow.

Working Groups are sub-committees empowered to triage and make decisions on issues and pull requests within their areas of expertise.

In the future, we’ll be reaching out to particularly active community contributors to invite them to participate as Working Group members.

As first steps in implementing Working Groups, they will be focused primarily on new incoming issues (which are now automatically marked with Needs-Triage), as well as issues and pull requests that are being potentially considered for 7.2.

Not all issues considered for 7.2 will be addressed in time, but can be addressed in a subsequent release. Pull requests marked as Backport-7.2.x-Consider are ones we have identified, particularly from the community, as ones we will spend effort reviewing and providing feedback to help get them into a quality state that can be merged into a preview release generally as an Experimental Feature.

As we experiment with the new Working Group processes, we will continue to revise them based on feedback from team members and the community. You can comment on the current Working Group proposal within the respective pull requests within the PowerShell and PowerShell-RFC repositories.

DSC for PowerShell 7

Desired State Configuration (DSC) enables a PowerShell experience for writing configuration as code.

This platform was originally built on top of WMI which meant it was only available for Windows.

In PowerShell 7.1, we started making DSC cross-platform by enabling Invoke-DscResource to directly use a DSC resource without going through the Local Configuration Manager (LCM).

We are continuing on this journey to make DSC a cross-platform technology during 7.2 timeframe:

  • Separate out the DSC parts in the PowerShell engine and moved them as a subsystem into the PSDesiredStateConfiguration module
  • Remove PSDesiredStateConfiguration module from the PowerShell 7 package. This allows the PSDesiredStateConfiguration module to be developed independently of PowerShell and users can mix and match versions of PowerShell and PSDesiredStateConfiguration for their environment.
  • Removing the dependency on MOF:
    • Initially, only support DSC Resources written as PowerShell classes. This includes tooling to convert existing script based DSC Resources to be wrapped as PowerShell classes.

Additional work that we are considering, but may not make it as part of the initial release:

  • Change generated DSC configuration files from MOF to using JSON
  • Enabling integration of DSC with existing agents (no LCM support)
  • Open sourcing the PSDesiredStateConfiguration module

Module Isolation

As PowerShell 7’s popularity continues to grow and more community members and partners author modules, they are more likely to encounter a conflict with dependencies on different versions of .NET assemblies. By default, all .NET assemblies are loaded into the same pwsh process. Different versions of the same assembly are not allowed to be loaded into the same process. This means that if PowerShell or a module loads a version of a .NET assembly, then anything else that depends on that assembly must use the same version and sometimes that can be incompatible.

For example, the PowerShell web cmdlets currently depend on Newtonsoft.Json.dll which is shipped with PowerShell. However, a community module may depend on features available in a newer version of Newtonsoft.Json.dll. This means you can’t use both the web cmdlets and cmdlets from this other module within the same PowerShell session.

Rob, from the PowerShell team, has a detailed blog post on how to handle this situation within your module. However, this is complicated and requires every module author to have similar code.

Instead, we want to expose a capability that modules authors can opt into so that the module loading code within the PowerShell engine handles the isolation of .NET assemblies for that module. More details of the design and limitations will be available in an RFC design document.

Enhanced ANSI Support

In PowerShell 7.0, we leveraged the use of ANSI escape sequences to provide more color to output. More specifically, we added color to errors and detailed error information via Get-Error to help visually separate metadata and data, making it easier to understand the information.

With Windows Terminal every operating system that supports PowerShell can now render ANSI escape sequences. Now, we can leverage more of them and also make it easier for script authors to use them.

We are introducing a new automatic variable called $PSStyle to make it easy to decorate text and for users to configure decoration settings. Now, scripters and even C# developers don’t need to understand raw ANSI escape sequences to take advantage of color and special formatting!

This also allows end users to choose if they want to never have ANSI rendering or only if it’s written to the console host. This is helpful if, for instance, you don’t want to pass ANSI escape sequences to CLI utilities in a pipeline.

Embracing ANSI also allows us to update how progress is rendered by having a consistent experience across both Windows and Linux. In particular, we can reduce time spent on rendering to the console minimal so execution time is less impacted, and use less screen buffer space.

To learn more about $PSStyle, check out the original issue #13071, as well as other issues around enhancing $PSStyle.

Improve native command experience

One of the goals of PowerShell 7.1 was to improve the shell experience. This includes using native commands within PowerShell more intuitively like other shells.

For PowerShell 7.2 we will continue to improve the shell experience, but specifically focusing on three areas:

  • Ideally, users should be able to “copy and paste” command lines from help documentation and tutorials into PowerShell and have it work as expected. This is complicated due to how Unix and Windows handle passing arguments to commands differently. Fixing this behavior would be a breaking change for users who may already be implementing workarounds. Please comment on this issue on our proposed new behavior.
  • PowerShell currently treats all output from native commands as text. This means that binary output from commands is converted to text, resulting in data loss, even if the intent is to redirect that output to a file or another command that accepts the binary output.
  • Unix systems rely on signals to communicate between processes which is currently not handled by PowerShell. This can cause some native Unix commands to crash or not behave as expected.

Pluggable remoting transports

PowerShell currently supports many remoting transports: named pipes, WS-Man, SSH, and VMBus (aka PowerShellDirect). However, most remoting users only rely on one transport, or would like to use some other remoting transport. We are working to enable pluggable remoting transports that would be installed as modules from the PowerShell Gallery. Users would be able to install or not install transports they use and also allow the community to enable new transports for new scenarios. In addition, it would mean that improvements to WS-Man on Linux, for example, can be improved by the community and not tied to a specific version of PowerShell.

Microsoft Update

One of our top customer asks is to ship PowerShell 7 as part of Windows. While this is still something we’re actively investigating, the problem is that the Windows support lifecycle (10 years) is much longer than the 3 year long-term servicing (LTS) lifecycle of the underlying .NET runtime that we use.

This currently prevents us from shipping PowerShell 7 into Windows while complying with the existing Windows support structure. In parallel, we’re working to make PowerShell 7 easier to install and update on Windows machines. For instance, Powershell 7.1 was the version of PowerShell that ships in the Microsoft Store on Windows 10 machines.

In 2021, we’re working to publish PowerShell 7 to the Microsoft Download Catalog and to Microsoft Update. On the Download Catalog, you’ll be able to use application deployment tools like SCCM, WSUS, and Intune to easily install and update PS7. And with Microsoft Update, you’ll receive updates to PS7 right alongside Windows Update, even if you had installed PowerShell 7 originally via MSI.

These changes will make it easier than ever to make sure that you’re running the latest version of PS7 on as many Windows machines as possible.

SecretManagement and SecretStore 1.0

Securely managing secrets and credentials is hard. With SecretManagement, we’re making it easier than ever to store and access your secrets in PowerShell, whether you need to save credentials locally on your dev machine, or pull secrets from a remote vault within your cloud automation. This enables sharing scripts that rely on secrets without sharing the actual secrets within the script.

As an extensible abstraction layer for secrets, you get to decide which vault extensions you want to use in conjunction with SecretManagement. You can use the SecretStore module for cross-platform, local management of secrets. Or you can leverage vault technologies from different products and vendors (like Azure KeyVault) without having to learn a new set of cmdlets or update your scripts. Already, you can find a number of community extension vaults on the PowerShell Gallery by searching for the SecretManagement tag.

We plan on releasing one more Release Candidate (RC) of both SecretManagement and SecretStore around March. Barring any significant feedback on basic usage, this will be followed by a Generally Available (GA) release around April.

Community feedback for preview versions of both SecretManagement and SecretStore modules have proven that we can achieve higher quality and appropriate functionality before we declare the design complete. Your contributions have helped us feel confident about the architecture and utility of both of these modules.

Crescendo 1.0

Crescendo simplifies the creation of PowerShell cmdlets that leverage non-PowerShell CLI utilities by providing a set of tools for describing their interface and generating PowerShell modules. For instance, by writing a simple document to describe the input parameters and output formatting of ipconfig.exe, you can generate cmdlets that support tab completion and object-based output, enabling easy interactive usage and powerful PowerShell pipelines. (Check out this scenario as an example in this great blog post about the first Crescendo preview.)

One of the great features of Crescendo is that it generates modules that can be published on the PowerShell Gallery and leveraged on Windows PowerShell 5.1 in addition to PowerShell 7. We’re optimistic that the PowerShell community will begin collaborating on modules which support some of the most popular CLI utilties and publishing them to the Gallery so that all PowerShell users can take advantage of them. (Note: Crescendo itself requires PS7 to generate these modules in the first place.) We plan to continued developing Crescendo over the next few months based on your feedback and the development of user-created Crescendo modules in the Gallery. As we feel more confident in the long-term interface established by Cresendo, we’ll follow up with at least one Release Candidate (RC) and eventually a supported Generally Available (GA) release.

PSReadLine 2.2

As part of improving the interactive experience with PowerShell, we’ve continued to invest in PSReadLine. In PowerShell 7.1, we added the ability to have prediction plugins and visualization of the predictions were rendered by PSReadLine 2.1. We will continue this work with presenting predictions as a list instead of on the same line showing multiple predictions you can choose from. In addition, a new dynamic help feature is being developed in PSReadLine 2.2 that will make it easier to get help on your pipeline as you are authoring it at the console: Get specific help on where you are in the pipeline, for example, details on the parameter you typed or parameter values. Allow using the secondary screen buffer to display full help information without the need to open a another terminal session and return exactly to where you were in the console.

PowerShellGet 3.0

The original goals of PowerShellGet 3.0 are to have a rewrite that addressed some major issues:

  • Improve the user experience by making it more intuitive. This was a breaking change that necessitated a major version update.
  • Remove the dependency on nuget.exe and instead call Nuget APIs using their published .NET assemblies.
  • Remove dependency on PackageManagement which we are no longer supporting. This will simplify the code base making it more maintainable, easier to debug, and easier to extend.

Our first preview of 3.0 was published almost a year ago and we still have lots of work to do to complete the 3.0 release addressing not only the major issues above, but also make it possible to replace use of PowerShellGet 2.0. More details of this roadmap will be published as a separate blog post.

VSCode Extension/PSEditorServices

PowerShell Editor Services and the PowerShell extension for Visual Studio Code serve as the foundation for PowerShell 7’s authoring and development experience in Visual Studio Code. While we’ve had a bit of slowdown in development on these projects over the holiday months, we’re more committed than ever to addressing some of the performance and stability issues that folks occasionally see with larger modules and scripts. In fact, just like week, we announced and released new stable versions of both the stable and preview PowerShell extensions for Visual Studio Code!

We encourage you to create issues (or even better, contribute code!) in the PowerShell extension repo with your suggestions or problems.

Updateable help and PlatyPS 2.0

Last year, we began a project within Microsoft to standardize updateable help publishing for all first-party modules. This ensures that all customers can use Update-Help to download and access up-to-date documentation for Microsoft modules without leaving their terminal. We’ve already onboarded teams like SCCM to this new pipeline, and will be prioritizing teams like Windows and Office going forward. As part of this effort, we also plan to update platyPS, our module for generating and managing both Markdown and XML-based documentation that conforms with PowerShell’s help system.

PlatyPS 2.0 will include a new schema and other fixes to esnure that XML documentation is being generated appropriately, as well as many other proposed fixes. (Note that comment-based help is still currently out of scope for platyPS, as it is not something we can use to deliver updateable help. Going forward, we may investigate improvements to comment-based help or even native Markdown parsing for the help system.)

Feel free to jump into the platyPS repo and weigh in on planned issues, or post your own if you feel that we’ve got a blind spot in our priorities.

OpenSSH 8.x

In addition to PowerShell investments, our team also owns the Win32 port of OpenSSH that ships as part of Windows 10 and Windows Server. In the 21H1 release of Windows 10, we will be updating Win32-OpenSSH to version 8.1 which includes support for Kerberos-based authentication (among a slew of smaller fixes). We had originally planned to deliver these updates in the 20H2 update. Due to some internal challenges, this was delayed to 21H1, but we plan to service many older versions of Windows 10 and Windows Server with Win32-OpenSSH 8.1 in order to reach as many customers as possible.

The backported fix for older versions will ship roughly around late April / early May and will be delivered via Windows Update.

Over the course of the next year, we also plan to update Win32-OpenSSH to the latest official version of OpenSSH 8.4.

Engineering Improvements

What is not apparent based on visible work in our GitHub repos is the amount of work not in the public required for publishing a quality signed release. Additional validation and static analysis is run against our signed release packages before we deem them ready for customer use. As PowerShell now has overlapping LTS and non-LTS servicing releases, the team can be working on 3 concurrent releases around the same time. For example, we worked on 7.0.5, 7.1.2, and 7.2 preview 3 releases simultaneously to ensure our customers get necessary security fixes aligned with .NET servicing releases. Although the team has already spent significant effort to automate much of the release pipeline, there are still manual steps which can slow down publishing time, require an engineer to execute tasks, or introduce problems due to human error.

In addition, we follow strict compliance requirements set by Microsoft and Azure. Sometimes, these requirements change, requiring us to update our release pipeline to ensure we are compliant. Finally, as we continue to do more of our work on individually published modules outside of the PowerShell repository, each of those repos require their own publishing pipeline and the need to meet compliance requirements. We have identified many work items we still need to complete to get to a goal of having high quality, consistent, and timely releases that we will work on for this semester.

At some point, we expect to have sufficient automation so that we can refocus our efforts back to improving the product. Finally, since our PowerShell Gallery outage last year, we continue to spend effort and time improving our service to be more robust and detect potential issues much more quickly.

Closing

As you can see, the team has lots of work planned during the PowerShell 7.2 timeframe. There is specific work to continue to improve PowerShell itself while a significant portion of the team is focused on modules and tooling that supports the PowerShell ecosystem. As noted in the SecretManagement section above, early feedback from users based on our preview releases can have significant impact to ensure that our releases meets customer needs.

We really appreciate the positive and constructive feedback particularly when based on real world usage as that can help us prioritize features and bug fixes as there is always more that we want to do than what we can do.

 

Steve Lee and Joey Aiello

PowerShell Team

5 comments

Leave a comment

    • Steve Lee
      Steve LeeMicrosoft employee

      We have some Microsoft partners building on our DSC for PS7 work. However, this is the start of re-investing in DSC and we won’t have a complete solution for end users until later, so you should view the current work as providing a platform. If someone is already using a CM solution (Chef, Puppet, etc…), they should continue to do that. With the platform we are building, it means you can author DSC resources as PowerShell classes and have them work cross-platform with existing CM solutions.

  • Avatar
    Isabella Simonsen

    Hello,

    In the past Powershell 2.0 and 3.0 have also been released to Microsoft Update.

    Then they were pulled back in a hurry, because people started to download and install them as a part of Tuesday patch day. As a consequence, Microsoft applications relying on a specific version of Powershell (like Exchange and Sharepoint) started to experience critical failures on a worldwide scale.

    Lets’ hope you’ll find a way to not repeat these mistakes.