{"id":19562,"date":"2022-06-27T13:14:39","date_gmt":"2022-06-27T21:14:39","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/powershell\/?p=19562"},"modified":"2022-06-27T13:14:39","modified_gmt":"2022-06-27T21:14:39","slug":"hosting-powershell-in-a-python-script","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/powershell\/hosting-powershell-in-a-python-script\/","title":{"rendered":"Hosting PowerShell in a Python script"},"content":{"rendered":"

Yes Virginia, languages other than PowerShell do<\/em> exist.<\/p>\n

I was working with a partner group here at Microsoft and they explained that they wanted to parse PowerShell scripts from Python.\nTheir natural approach was to invoke the PowerShell executable and construct a command-line that did what they needed.\nI thought there might be a better way as creating a new PowerShell process each time is expensive, so I started doing a bit of research to see something could be done.\nI’ve been aware of IronPython<\/a> (Python that tightly integrates .NET) for a long time, and\nwe met with Jim Hugunin shortly after he arrived at Microsoft and PowerShell was just getting underway,\nbut the group is using cPython<\/a> so I went hunting for Python modules that host .NET and found the pythonnet<\/code> module.<\/p>\n

The pythonnet<\/a> package gives Python developers extremely easy access to the dotnet runtime from Python.\nI thought this package might be the key for accessing PowerShell,\nafter some investigation I found that it has exactly what I needed to host PowerShell in a Python script.<\/p>\n

The guts<\/h2>\n

I needed to figure out a way to load the PowerShell engine.\nFirst, there are a couple of requirements to make this all work.\nDotnet has to be available, as does PowerShell and pythonnet<\/code> provides a way to specify where to look for dotnet.\nSetting the environment variable DOTNET_ROOT<\/code> to the install location,\nenables pythonnet<\/code> a way find the assemblies and other support files to host .NET.<\/p>\n

import os\r\nos.environ[\"DOTNET_ROOT\"] = \"\/root\/.dotnet\"<\/code><\/pre>\n
Now that we know where dotnet is, we need to load up the CLR and set up the runtime configuration.\nThe runtime configuration describes various aspects of how we’ll run.\nWe can create a very<\/em> simple pspython.runtimeconfig.json<\/code><\/p>\n
{\r\n  \"runtimeOptions\": {\r\n    \"tfm\": \"net6.0\",\r\n    \"framework\": {\r\n      \"name\": \"Microsoft.NETCore.App\",\r\n      \"version\": \"6.0.0\"\r\n    }\r\n  }\r\n}<\/code><\/pre>\n
The combination of the DOTNET_ROOT<\/code> and the runtime configuration enables\nloading the CLR with the get_coreclr<\/code> and set_runtime<\/code> functions.<\/p>\n
# load up the clr\r\nfrom clr_loader import get_coreclr\r\nfrom pythonnet import set_runtime\r\nrt = get_coreclr(\"\/root\/pspython.runtimeconfig.json\")\r\nset_runtime(rt)<\/code><\/pre>\n
Now that we have the CLR loaded, we need to load the PowerShell engine.\nThis was a little non-obvious.\nInitially, I just attempted to load System.Management.Automation.dll<\/code> but that failed\ndue to a strong name validation error.\nHowever, If I loaded Microsoft.Management.Infrastructure.dll<\/code> first, I can avoid that error.\nI’m not yet sure about why<\/em> I need to load this assembly first, that’s still something\nI need to determine.<\/p>\n
import clr\r\nimport sys\r\nimport System\r\nfrom System import Environment\r\nfrom System import Reflection\r\n\r\npsHome = r'\/opt\/microsoft\/powershell\/7\/'\r\n\r\nmmi = psHome + r'Microsoft.Management.Infrastructure.dll'\r\nclr.AddReference(mmi)\r\nfrom Microsoft.Management.Infrastructure import *\r\n\r\nfull_filename = psHome + r'System.Management.Automation.dll'\r\nclr.AddReference(full_filename)\r\nfrom System.Management.Automation import *\r\nfrom System.Management.Automation.Language import Parser\r\n<\/code><\/pre>\n
Eventually I would like to make the locations of dotnet and PSHOME<\/code> configurable,\nbut for the moment, I have what I need.<\/p>\n
Now that the PowerShell engine is available to me,\nI created a couple of helper functions to make handling the results easier from Python.\nI also created a PowerShell object (PowerShell.Create()<\/code>) that I will use in some of my functions.<\/p>\n
ps = PowerShell.Create()\r\ndef PsRunScript(script):\r\n    ps.Commands.Clear()\r\n    ps.Commands.AddScript(script)\r\n    result = ps.Invoke()\r\n    rlist = []\r\n    for r in result:\r\n        rlist.append(r)\r\n    return rlist\r\n\r\nclass ParseResult:\r\n    def __init__(self, scriptDefinition, tupleResult):\r\n        self.ScriptDefinition = scriptDefinition\r\n        self.Ast = tupleResult[0]\r\n        self.Tokens = tupleResult[1]\r\n        self.Errors = tupleResult[2]\r\n\r\n    def PrintAst(self):\r\n        print(self.ast.Extent.Text)\r\n\r\n    def PrintErrors(self):\r\n        for e in self.Errors:\r\n            print(str(e))\r\n\r\n    def PrintTokens(self):\r\n        for t in self.Tokens:\r\n            print(str(t))\r\n\r\n    def FindAst(self, astname):\r\n        Func = getattr(System, \"Func`2\")\r\n        func = Func[System.Management.Automation.Language.Ast, bool](lambda a : type(a).__name__ == astname)\r\n        asts = self.Ast.FindAll(func, True)\r\n        return asts\r\n\r\ndef ParseScript(scriptDefinition):\r\n    token = None\r\n    error = None\r\n    # this returns a tuple of ast, tokens, and errors rather than the c# out parameter\r\n    ast = Parser.ParseInput(scriptDefinition, token, error)\r\n    # ParseResult will bundle the 3 parts into something more easily consumed.\r\n    pr = ParseResult(scriptDefinition, ast)\r\n    return pr\r\n\r\ndef ParseFile(filePath):\r\n    token = None\r\n    error = None\r\n    # this returns a tuple of ast, tokens, and errors rather than the c# out parameter\r\n    ast = Parser.ParseFile(filePath, token, error)\r\n    # ParseResult will bundle the 3 parts into something more easily consumed.\r\n    pr = ParseResult(filePath, ast)\r\n    return pr\r\n\r\ndef PrintResults(result):\r\n    for r in result:\r\n        print(r)<\/code><\/pre>\n
I really wanted to mimic the PowerShell AST methods with some more friendly Python functions.\nTo create the FindAst() function, I needed to combine the delegate in c# with the lambda feature in Python.\nNormally, in PowerShell, this would look like:<\/p>\n
$ast.FindAll({$args[0] -is [System.Management.Automation.Language.CommandAst]}, $true)<\/code><\/pre>\n
But I thought from a Python script, it would easier to use the name of the type.\nYou still need to know the name of the type,\nbut bing<\/a> is great for that sort of thing.\nAs I said, I don’t really know the Python language,\nso I expect there are better ways to handle the Collection[PSObject]<\/code> that Invoke()<\/code> returns.\nI found that I had to iterate over the result no matter what, so I built it into the convenience function.\nAnyone with suggestions is more than welcome to improve this.<\/p>\n
The glory<\/h2>\n
Now that we have the base module together, we can write some pretty simple Python to\nexecute our PowerShell scripts.\nInvoking a PowerShell script is now as easy as:<\/p>\n
#!\/usr\/bin\/python3\r\n\r\nfrom pspython import *\r\n\r\nscriptDefinition = 'Get-ChildItem'\r\nprint(f\"Run the script: '{scriptDefinition}\")\r\nresult = PsRunScript(scriptDefinition)\r\nPrintResults(result)<\/code><\/pre>\n
\/root\/__pycache__\r\n\/root\/dotnet-install.sh\r\n\/root\/get-pip.py\r\n\/root\/grr.py\r\n\/root\/hosted.runtimeconfig.json\r\n\/root\/pspar.py\r\n\/root\/pspython.py\r\n\/root\/psrun.py<\/code><\/pre>\n
You’ll notice that the output is not formatted by PowerShell.\nThis is because Python is just taking the .NET objects and (essentially) calling ToString()<\/code> on them.<\/p>\n
It’s also possible to retrieve objects and then<\/em> manage formatting via PowerShell.\nThis example retrieves objects via Get-ChildItem<\/code>,\nselects those files that start with “ps” in Python,\nand then creates a string result in table format.<\/p>\n
scriptDefinition = 'Get-ChildItem'\r\nresult = list(filter(lambda r: r.BaseObject.Name.startswith('ps'), PsRunScript(scriptDefinition)))\r\nps.Commands.Clear()\r\nps.Commands.AddCommand(\"Out-String\").AddParameter(\"Stream\", True).AddParameter(\"InputObject\", result)\r\nstrResult = ps.Invoke()\r\n# print results\r\nPrintResults(strResult)<\/code><\/pre>\n
    Directory: \/root\r\n\r\nUnixMode   User             Group                 LastWriteTime           Size Name\r\n--------   ----             -----                 -------------           ---- ----\r\n-rwxr-xr-x root             dialout             6\/17\/2022 01:30           1117 pspar.py\r\n-rwxr-xr-x root             dialout             6\/16\/2022 18:55           2474 pspython.py\r\n-rwxr-xr-x root             dialout             6\/16\/2022 21:43            684 psrun.py<\/code><\/pre>\n
But that’s not all<\/h2>\n
We can also call static methods on PowerShell types.\nThose of you that noticed in my module there are a couple of language related functions.\nThe ParseScript<\/code> and ParseFile<\/code> functions allow us to call the PowerShell language parser\nenabling some very interesting scenarios.<\/p>\n
Imagine I wanted to determine what commands a script is calling.\nThe PowerShell AST makes that a breeze, but first we have to use the parser.\nIn PowerShell, that would be done like this:<\/p>\n
$tokens = $errors = $null\r\n$AST = [System.Management.Automation.Language.Parser]::ParseFile(\"myscript.ps1\", [ref]$tokens, [ref]$errors)<\/code><\/pre>\n
The resulting AST is stored in $AST<\/code>, the tokens in $tokens<\/code>, and the errors in $errors<\/code>.\nWith this Python module, I encapsulate that into the Python function ParseFile<\/code>,\nwhich returns an object containing all three of those results in a single element.\nI also created a couple of helper functions to print the tokens and errors more easily.\nAdditionally, I created a function that allows me to look for any type of AST (or sub AST)\nin any arbitrary AST.<\/p>\n
parseResult = ParseFile(scriptFile)\r\ncommandAst = parseResult.FindAst(\"CommandAst\")\r\ncommands = set()\r\nfor c in commandAst:\r\n    commandName = c.GetCommandName()\r\n    # sometimes CommandName is null, don't include those\r\n    if commandName != None:\r\n       commands.add(c.GetCommandName().lower())\r\nPrintResults(sorted(commands))<\/code><\/pre>\n
Note that there is a check for commandName<\/strong> not being null.\nThis is because when & $commandName<\/code> is used, the command name cannot be\ndetermined via static analysis since the command name is determined at run-time.<\/p>\n
…a few, uh, provisos, uh, a couple of quid pro quo<\/h2>\n
First, you have to have dotnet installed (via the install-dotnet),\nas well as a full installation of PowerShell.\npythonnet<\/code> doesn’t run on all versions of Python,\nI’ve tested it only on Python 3.8 and Python 3.9 on Ubuntu20.04.\nAs of the time I wrote this, I couldn’t get it to run on Python 3.10.\nThere’s more info on pythonnet at the pythonnet web-site<\/a>.\nAlso, this is a hosted<\/em> instance of PowerShell.\nSome things, like progress, and verbose, and errors may act a bit differently than you\nwould see from pwsh.exe<\/code>.\nOver time, I will probably add additional helper functions to retrieve more runtime information\nfrom the engine instance.\nIf you would like to pitch in, I’m happy to take Pull Requests or to simply understand your use cases integrating PowerShell and Python.<\/p>\n
Take it out for a spin<\/h2>\n
I’ve wrapped all of this up and added a Dockerfile (running on Ubuntu 20.04) on\ngithub<\/a>.\nTo create the docker image, just run\nDocker build --tag pspython:demo .<\/code>\nfrom the root of the repository.<\/p>\n","protected":false},"excerpt":{"rendered":"
How to use Python to call the PowerShell engine without running the PowerShell executable<\/p>\n","protected":false},"author":2413,"featured_media":13641,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[248,3183],"class_list":["post-19562","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-powershell","tag-powershell","tag-python"],"acf":[],"blog_post_summary":"
How to use Python to call the PowerShell engine without running the PowerShell executable<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/19562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/users\/2413"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/comments?post=19562"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/posts\/19562\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media\/13641"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/media?parent=19562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/categories?post=19562"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powershell\/wp-json\/wp\/v2\/tags?post=19562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}