Using CPython’s Embeddable Zip File

Steve Dower

On the download page for CPython 3.5.1, you’ll see a wide range of options. Not all of these are well explained, especially for Windows users who have seven (seven!) choices.

Let me restructure the Windows items into a more feature-focused table:

As is fairly common with installers these days, you have the choice to download everything in advance (the “executable installer”), or a downloader that will let you minimize the download size (the “web installer”). The latter allows you to select options before downloading the components, which can reduce the download size to around 8MB. (For those of us with fast, reliable internet access, this sounds irrelevant – for those of us tethering through a 3G mobile phone connection in the middle of nowhere, it’s a really huge saving!)

But what is the third option – the “embeddable zip file”? It looks like a reasonable download size and it doesn’t have any installer, so it seems quite attractive. However, the embeddable zip file is not actually a regular Python installation. It has a specific purpose and a narrow audience: developers who embed Python in their own native applications.

Why embed Python?

For many users, “Python” is the interactive shell that lets you type code and see immediate results. For others, it is an executable that can run .py files. While these are both true, in reality Python is itself a library that is used to interpreter code. Let’s look at the complete source code for python.exe on Windows:

#include "Python.h"
#include 

int
wmain(int argc, wchar_t **argv)
{
    return Py_Main(argc, argv);
}

That’s it! The entire purpose of python.exe is to call a function from python35.dll. Which means it is really easy to create a different executable that will run exactly what you want:

#include "Python.h"

int
wmain(int argc, wchar_t **argv)
{
    wchar_t *myargs[3] = { argv[0], L"-m", L"myscript" };
    return Py_Main(3, myargs);
}

This version will ignore any command line arguments that are passed in, replacing them with an option to always start a particular script. If you give this executable its own name and icon, nobody ever has to know that you used Python at all!

But Python has a much more complete API than this. The official docs are the canonical source of information, but let’s look at a couple of example programs that you may find useful.

Executing a simple Python string

The short program above lets you substitute a different command line, but if you have a string constant you can also execute that. This example is based on the one provided in the docs.

#include "Python.h"

int wmain(int argc, wchar_t *argv[]) {
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    PyRun_SimpleString("from time import time, ctimen"
                       "print('Today is', ctime(time()))n");
    Py_Finalize();
    return 0;
}

Executing Python directly

Running a string that is predefined or dynamically generated may be useful enough, but the real power of hosting Python comes when you directly interact with the objects. However, this is also when code becomes incredibly complicated.

In almost every situation where it is possible to use Cython or CFFI to generate code for wrapping native objects and values, you should probably use them. However, while they’re great for embedding native code in Python, they aren’t as helpful (at time of writing) for embedding Python into your native code. If you want to allow users to automate your application with a Python script, you’ll need some way of importing the user’s script, and to provide Python functions to call back into your native code.

As an example of hosting Python directly, the program below replicates the one from above but uses direct calls into the Python interpreter rather than a script. (Note that there is no error checking in this sample, and you need a lot of error checking here.)

#include "Python.h"

int wmain(int argc, wchar_t *argv[]) {
    PyObject *time_module, *time_func, *ctime_func;
    PyObject *time_value, *ctime_value, *text_value;
    wchar_t *text;
    Py_ssize_t cch;
    
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    
    // NOTE: Practically every line needs an error check.
    time_module = PyImport_ImportModule("time");
    time_func = PyObject_GetAttrString(time_module, "time");
    ctime_func = PyObject_GetAttrString(time_module, "ctime");
    
    time_value = PyObject_CallFunctionObjArgs(time_func, NULL);
    ctime_value = PyObject_CallFunctionObjArgs(ctime_func, time_value, NULL);
    
    text_value = PyUnicode_FromFormat("Today is %S", ctime_value);
    
    text = PyUnicode_AsWideCharString(text_value, &cch);
    wprintf(L"%sn", text);
    PyMem_Free(text);
    
    Py_DECREF(text_value);
    Py_DECREF(ctime_value);
    Py_DECREF(time_value);
    Py_DECREF(ctime_func);
    Py_DECREF(time_func);
    Py_DECREF(time_module);
    
    Py_Finalize();
    return 0;
}

In a larger application, you’d probably call Py_Initialize as part of your startup and Py_Finalize when exiting, and then have occasional calls into the Python engine wherever it made sense. This way, you can write parts of your application in Python and interact with them directly, or allow your users to extend it by providing their own Python scripts.

How does the embeddable zip file help?

Where does the embeddable zip file come into play? While you need a full Python install to compile these programs, when you install them onto a user’s computer, you only need the contents of the embeddable zip, as well as any (pre-built) packages you need. Header files, documentation, tests and shortcuts are not necessary,

Tools like pynsist will help produce installers for pure Python programs like this, using the embeddable zip file so that you don’t have to worry about whether your users already have Python or not.

Why wouldn’t you just run the regular Python installer as part of your application? Let’s play the “what if two programs did this?” game: program X runs the 3.5.0 installer and then program Y runs the 3.5.1 installer. What version does program X now have? If it ran the installer with a custom install directory, it probably has nothing left at all, but at best it now has 3.5.1.

The regular installer is designed for users, not applications. Programs that are not Python, but use Python, need to handle their own installation to make sure they end up with the correct version in the correct location with all the correct files. The embeddable zip file contains the minimum Python runtime for an application to install by itself.

What about other packages?

The embeddable zip file does not include pip. So how do you install packages? If you didn’t read the last sentence of the previous section, here it is again: the embeddable zip file contains the minimum Python runtime for an application to install by itself.

Using the embeddable zip file implies that you want the minimum required files to run your application, and you have your own installer. So if you need extra files at runtime – such as a Python package – you’ll need to install them with your installer. As mentioned above, for developing an application you should have a full Python installation, that does include pip and can install packages locally. But when distributing your application, you need to take responsibility.

While this seems like more work (and it is more work!), the value is worth it. Do you want your installer to fail because it can’t connect to the internet? Do you want your application to fail because a different version of a library was installed? When you provide a bundle for your users, include everything that it needs (tools like pynsist will help do this automatically).

Where else can I get help?

Though I’m writing about the embeddable distribution on a Microsoft blog, this is actually a CPython feature. The doc page is part of the official Python documentation, and bugs or issues should be filed at the CPython bug tracker.

Steve Dower Software Engineer, Python

Follow