March 19th, 2026
0 reactions

Windows stack limit checking retrospective: amd64, also known as x86-64

Raymond Chen
Raymond Chen

Our survey of stack limit checking reaches the modern day with amd64, also known as x86-64. This time, there are two versions of the function, one for user mode and one for kernel mode. We’ll look at the user mode version.

Actually, there are two user mode versions. One is in msvcrt, the legacy runtime.

; on entry, rax is the number of bytes to allocate
; on exit, stack has been validated (but not adjusted)

chkstk:
    sub     rsp, 16
    mov     [rsp], r10          ; save temporary register
    mov     [rsp][8], r11       ; save temporary register

    xor     r11, r11            ; r11 = 0
    lea     r10, [rsp][16][8]   ; r10 = caller's rsp
    sub     r10, rax            ; r10 = desired new stack pointer
    cmovb   r10, r11            ; clamp underflow to zero

    mov     r11, gs:[StackLimit]; user mode stack limit

    cmp     r10, r11            ; are we inside the limit?
    jae     done                ; Y: nothing to do

    and     r10w, #-PAGE_SIZE   ; round down to page start

probe:
    lea     r11, [r11][-PAGE_SIZE]  ; move to previous page
    test    [r11], r11b         ; probe it
    cmp     r10, r11            ; finished probing?
    jb      probe               ; N: keep going

done:
    mov     r10, [rsp]          ; restore temporary register
    mov     r11, [rsp][8]       ; restore temporary register
    add     rsp, 16             ; clean up stack
    ret

Bonus reading: Windows is not a Microsoft Visual C/C++ Run-Time delivery channel.

The other is in ucrtbase, the so-called universal runtime. That one is identical except that the probing is done by writing rather than reading.

    mov     byte ptr [r11], 0   ; probe it

In both cases, the function ensures that the stack has expanded the necessary amount but leaves it the caller’s responsibility to adjust the stack after the call returns. This design preserves compliance with shadow stacks (which Intel calls Control-Flow Enforcement Technology, or CET).

A typical usage might go like this:

    mov     eax, #17328         ; desired stack frame size (zero-extended)
    call    chkstk              ; validate that there is enough stack
    sub     rsp, rax            ; allocate it

Next time, we’ll wrap up the series with a look at AArch64, also known as arm64.

Category
Old New Thing
Topics
Code

Author

Raymond Chen

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

2 comments

Sort by :
  • Joshua Hudson

    Bonus: wine amd64 has the simplest version of chkstk:

    __chkstk:
    ret

    where it’s actually written in inline assembly at block scope.

    This works because there are no guard pages; all stack is already reserved at time of thread start. There seems to be some implicit assumption that no program actually causes a megabyte-sized stackoverflow, which isn’t a completely unreasonable assumption given the program had to work to get shipped.

    (It’s actually quite easy to write programs that work on Wine that don’t have a ghost of a chance of running on Windows; nobody seems to care.)

  • Csaba Varga

    I’m wondering why the probe loop does “lea r11, [r11][-PAGE_SIZE]” rather than “sub r11, PAGE_SIZE”. It can’t be for preserving the flags, the next TEST will wipe them anyway. Even in the ucrtbase version, the CMP after the probe will wipe the flags.

    I doubt it matters performance-wise, but it’s still bugging me.

Read next