We start our survey of historical stack limit checking functions on Windows with the 80386 family of processors. This function has actually changed form over the years, so we’ll start with the “original flavor”.

Originally, the _chkstk function was called by putting the desired number of bytes in the eax register and calling the _chkstk function. The function touched each page of the stack, adjusted the stack pointer, and then returned with the adjusted stack pointer. This is an unusual calling convention since it is neither caller clean, nor is it callee clean. It’s callee-dirty! The function returns with more stack than it started.

_chkstk:
    push    ecx             ; remember desired allocation size

    ; calculate the stack pointer of the caller
    mov     ecx, esp
    add     ecx, 8          ; 4 bytes were auto-pushed for the return address,
                            ; we pushed 4 bytes for the ecx

touch:
    cmp     eax, PAGE_SIZE  ; less than a page to go?
    jb      finalpage       ; do the last page and finish
    sub     ecx, PAGE_SIZE  ; allocate a page from our pretend stack pointer
    or      dword ptr [ecx], 0 ; touch the memory
    sub     eax, PAGE_SIZE  ; did a page
    jmp     touch           ; go back and do some more

finalpage:
    sub     ecx, eax        ; allocate the leftovers from our pretend stack pointer
    or      dword ptr [ecx], 0 ; touch the memory
    mov     eax, esp        ; remember original stack pointer
    mov     esp, ecx        ; move the real stack to match our pretend stack
    mov     ecx, [eax]      ; recover original ecx
    mov     eax, 4[eax]     ; recover return address
    jmp     eax             ; "return" to caller

A function with a large stack frame would go something like

function:
    push    ebp         ; link into frame chain
    mov     ebp, esp
    push    ebx         ; save non-volatile register
    push    esi
    push    edi
    mov     ecx, 17320  ; large stack frame
    call    _chkstk     ; allocate it from our stack safely
                        ; behaves like "sub esp, ecx"

This goes into the competition for “wackiest x86-32 calling convention.”¹

Next time, we’ll look at how stack probing happens on MIPS, which has its own quirks, but nothing as crazy as this.

Bonus chatter: The strange calling convention dates back to the 16-bit 8086. And back then, there were two versions of the chkstk function, depending on whether you were calling it far or near.

; frame size in ax

chkstk:
#if NEAR
    pop     bx          ; pop 16-bit return address
#else // FAR
    pop     bx          ; pop 32-bit return address
    pop     dx
#endif

    inc     ax
    and     al, 0xFE    ; round up to even

    sub     ax, sp      ; check for stack overflow
    jae     overflow    ; Y: overflow
    neg     ax          ; ax = new stack pointer

    cmp     ax, ss:[pStackTop]
    ja      overflow    ; stack mysteriously too high

    cmp     ax, ss:[pStackMin] ; new stack limit?
    jbe     nochange
    mov     ss:[pStackMin], ax ; update stack limit
nochange:

    mov     sp, ax      ; update the stack pointer

#if NEAR
    jmp     bx          ; "return" to caller
#else // FAR
    push    dx          ; restore return address
    push    bx
    retf                ; return to caller
#endif