{"id":109126,"date":"2023-12-11T07:00:00","date_gmt":"2023-12-11T15:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=109126"},"modified":"2023-12-11T09:20:16","modified_gmt":"2023-12-11T17:20:16","slug":"20231211-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20231211-00\/?p=109126","title":{"rendered":"The mysterious second parameter to the x86 ENTER<\/CODE> instruction"},"content":{"rendered":"

The x86 instruction set has an ENTER<\/code> instruction which builds a stack frame. It is almost always used with a zero as the second parameter.<\/p>\n

        enter   n, 0\r\n<\/pre>\n
This is functionally equivalent to<\/p>\n
        push    ebp\r\n        mov     ebp, esp\r\n        sub     esp, n\r\n<\/pre>\n
But what happens if you increase that second parameter beyond zero?<\/p>\n
Values greater than zero for the second parameter are intended for languages like Pascal which support nested functions that can access the local variables of their lexical parents.  We learned about these functions a short time ago<\/a>. But the designers of the x86 instruction set had a different design in mind for how a function can access the variables of its lexical parent: Instead of receiving a pointer to the start of a linked list of lexical parent frames, they receive an array<\/i> of pointers to lexical parent frames.<\/p>\n
In its full generality, the<\/p>\n
    enter n, k + 1\r\n<\/pre>\n
instruction goes like this:<\/p>\n
\u00a0<\/div>\n
    push    ebp\r\n    mov     internal_register, esp\r\n    sub     ebp, 4 <\/span> \u23b1 k<\/var> times<\/span>\r\n    push    [ebp]  <\/span> \u23b0\r\n    push    internal_register\r\n    mov     ebp, internal_register\r\n    sub     esp, n\r\n<\/pre>\n
If you ignore the order of operations and worry just about the final state, then you can reinterpret it like this, which I think captures the essence of the instruction better:<\/p>\n
    push    ebp\r\n    push    [ebp-4]   <\/span>\r\n    push    [ebp-8]   <\/span> k<\/var> pushes<\/span>\r\n    :                 <\/span>\r\n    push    [ebp-4*k<\/var>] <\/span>\r\n    lea     ebp, [esp + 4*k<\/var>]    ; where we pushed the previous ebp\r\n    push    ebp\t\t        ; add our own frame to the array\r\n    sub     esp, n\r\n<\/pre>\n
Let’s look at our example function again.<\/p>\n
function Outer(n: integer) : integer;\r\n    var i: integer;\r\n\r\n    procedure Update(j: integer);\r\n    begin\r\n        i := i + j\r\n    end;\r\n\r\n    procedure Inner(m: integer);\r\n\r\n        procedure MoreInner;\r\n        begin\r\n            Update(m)\r\n        end;\r\n\r\n    (* Inner body begins here *)\r\n    begin\r\n        MoreInner\r\n    end;\r\n\r\n(* Outer body begins here *)\r\nbegin\r\n    i := 0;\r\n    Inner(n);\r\n    Outer := i\r\nend;\r\n<\/pre>\n
On entry to Outer<\/code>, the stack looks like this:<\/p>\n\n\n\n\n
n<\/code> parameter<\/td>\n <\/td>\n<\/tr>\n
return address<\/td>\n\u2190 esp<\/var><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The Outer<\/code> function establishes its stack frame by performing an enter 4, 1<\/code>. The extra 1<\/code> at the end means that this is the outermost of a chain of nested functions. In our cookbook, k<\/var> is zero, so the functional equivalent is<\/p>\n
    push    ebp\r\n                                ; no pointers copied from parent\r\n    lea     ebp, [esp+0]        ; equivalently, \"mov ebp, esp\"\r\n    push    ebp                 ; pointer to our own frame\r\n    sub     esp, 4\r\n<\/pre>\n
and we wind up with this stack frame for Outer<\/code>:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
 <\/td>\n <\/td>\nOuter<\/code> frame<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nn<\/code> parameter<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nreturn address<\/td>\n <\/td>\n<\/tr>\n
\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n\u2190 ebp<\/var><\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\ni<\/code><\/td>\n\u2190 esp<\/var><\/td>\n<\/tr>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
That extra ,1<\/code> caused us to push the address of where we saved the previous ebp<\/code>, which I’ve called the Outer<\/code> frame pointer. That value isn’t really useful to us right now, since we already have that value in the ebp<\/var> register. But it comes in handy when we call Inner<\/code>.<\/p>\n
On entry to Inner<\/code>, the stack looks like this:<\/p>\n\n\n\n\n\n
 <\/td>\n <\/td>\nm<\/code> parameter<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nreturn address<\/td>\n\u2190 esp<\/var><\/td>\n<\/tr>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The Inner<\/code> function performs an enter 0, 2<\/code>. The 0 means that Inner<\/code> has no local variables, and the 2<\/code> means that we are now the second level in a chain of nested functions.<\/p>\n
The functional equivalent now has one extra memory push before we push a pointer to our own frame:<\/p>\n
    push    ebp\r\n    push    [ebp-4]  <\/span>           ; one pointer copied from parent\r\n    lea     ebp, [esp+4]\r\n    push    ebp                 ; pointer to our own frame\r\n    sub     esp, 4\r\n<\/pre>\n
Before pushing the address of its own frame, the enter<\/code> instruction also copies one pointer from the parent’s frame, namely the Outer<\/code> frame pointer.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
 <\/td>\n <\/td>\nInner<\/code> frame<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nm<\/code> parameter<\/td>\n <\/td>\n <\/td>\n <\/td>\nOuter<\/code> frame<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nreturn address<\/td>\n <\/td>\n <\/td>\n <\/td>\nn<\/code> parameter<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n\u2190 ebp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nreturn address<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nInner<\/code> frame pointer<\/td>\n\u2190 esp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\ni<\/code><\/td>\n<\/tr>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Now things are interesting.<\/p>\n
The Inner<\/code> function has access to its own frame, via the ebp<\/var> register (and redundantly via the Inner<\/code> frame pointer on its stack). It also has access to the Outer<\/code> frame through its local copy of the Outer<\/code> frame pointer.<\/p>\n
The next thing that happens is that Inner<\/code> calls More\u00adInner<\/code> with no parameters. This time More\u00adInner<\/code> uses enter 0, 3<\/code> where the 0 means that More\u00adInner<\/code> has no local variables, and the 3<\/code> means that it is a nested function three levels deep, so it should copy two<\/i> frame pointers from its parent.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
 <\/td>\n <\/td>\nMoreInner<\/code> frame<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nreturn address<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nInner<\/code> frame<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nm<\/code> parameter<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nreturn address<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nn<\/code> parameter<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nInner<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nreturn address<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nMoreInner<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nInner<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\ni<\/code><\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The frame for MoreInner<\/code> contains its own parameters and local variables (nothing), plus pointers to both parent frames, plus a pointer to its own frame (which More\u00adInner<\/code> doesn’t use, but which is ready for any nested function to use).<\/p>\n
The code generation for MoreInner<\/code> therefore reads the value of m<\/code> by following the Inner<\/code> frame pointer and then reading the m<\/code> parameter from the Inner<\/code> frame’s parameter space.<\/p>\n
After More\u00adInner<\/code> calls Update<\/code>, the Update<\/code> function starts with an enter 0, 2<\/code> because it is a level-2 nested function. This copies only the Outer<\/code> frame pointer to Update<\/code>‘s frame, resulting in this:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
 <\/td>\n <\/td>\nUpdate<\/code> frame<\/td>\n <\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nj<\/code> parameter<\/td>\n <\/td>\n <\/td>\n <\/td>\nOuter<\/code> frame<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\nreturn address<\/td>\n <\/td>\n <\/td>\n <\/td>\nn<\/code> parameter<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n\u2190 ebp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nreturn address<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n\u25b6\ufe0e<\/td>\nprevious ebp<\/var><\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\nUpdate<\/code> frame pointer<\/td>\n\u2190 esp<\/var><\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nOuter<\/code> frame pointer<\/td>\n<\/tr>\n
\u00a0<\/td>\n<\/tr>\n
 <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\ni<\/code><\/td>\n<\/tr>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
I didn’t draw it, but the “previous ebp<\/var>” in the Update<\/code> frame points to the More\u00adInner<\/code> frame.<\/p>\n
The Update<\/code> function reads j<\/code> from its own parameter space and uses to update the i<\/code> variable in Outer<\/code>‘s frame by following the Outer<\/code> frame pointer.<\/p>\n
The result is the same as the  System V Application Binary Interface static chain pointer<\/a>, but it’s done in a different way. Instead of passing the head of a linked list of frames, the enter<\/code> instruction copies an entire array of pointers to frames. This reduces the number of instructions required in order to access faraway frames, but it increases the cost of a function call due to the extra copying.<\/p>\n
I wonder if anybody uses the Intel design for nested functions. I suspect it’s silicon on the CPU that is completely wasted.<\/p>\n
\n