Windows XP SP 2 introduced Data Execution Prevention (DEP), which takes advantage of a then-new feature of x86-class processors that allowed you to deny execution from data pages. The new feature was Physical Address Extensions (PAE) which also allowed those 32-bit processors to access physical RAM above the 4 GB boundary. Although you could turn on Data Execution Prevention on all systems, only server products would use the memory above 4 GB.
A reader asked, “What was the real reason client editions were prevented from using more than 4 GB of RAM?”
The use of the word “real” in the question implies that the reader believed that the official reason was a lie, and there was some nefarious evil reason for the limitation. It’s unclear what this nefarious reason would be. Maybe the reader thought the “real” reason was “To force users to buy copies of Windows Server, which is far more lucrative”, though that doesn’t make sense. The cheapest version of Windows Server 2003 32-bit edition that supported more than 4 GB of RAM was Enterprise Edition, which sold for $3,999.¹ This is an outrageous price for a consumer operating system.
The reason why consumer products don’t use RAM above 4 GB is explained in the documentation that accompanied the introduction of the feature under “Driver issues”.
Typically, device drivers must be modified in a number of small ways. Although the actual code changes may be small, they can be difficult. This is because when not using PAE memory addressing, it is possible for a device driver to assume that physical addresses and 32-bit virtual address limits are identical. PAE memory makes this assumption untrue.
…
[M]any device drivers designed for these systems may not have been tested on system configurations with PAE enabled. In order to limit the impact to device driver compatibility, changes to the hardware abstraction layer (HAL) were made to Windows XP SP2 and Windows Server 2003 SP1 Standard Edition to limit physical address space to 4 GB.
As explained above, memory above 4 GB was not enabled for compatibility reasons. Many drivers inadvertently assume that all physical address fit in 32 bits. (DMA drivers for example.) Those drivers would corrupt memory if memory above 4 GB were made available.
Memory above 4 GB is enabled on server because if you are a server administrator, you don’t install random drivers for that hand-held scanner you bought at Best Buy from the bargain bin for $10. Server administrators typically run only the plain vanilla drivers that come with Windows. (They don’t even install manufacturer video drivers.) All the drivers that come with Windows have been tested for addresses above 4 GB. That 2001 driver for the $10 handheld scanner has not, and there’s a good chance that it will truncate addresses above 4 GB and corrupt memory as a result.
The consumer market and the server market are very different in terms of usage pattern. Consumers will install practically anything. Server administrators install as little as possible. Consumers have no technical expertise. Server administrators have access to highly-skilled staff.
Of course, this is all now a historical oddity. Systems with only 4 GB of RAM are vanishingly rare, and Windows began discouraging the production of systems using 32-bit processors in 2020, finally ending the production of 32-bit editions entirely with Windows 11.
¹ The only other version that supported more than 4 GB of RAM was Datacenter Edition, and on the pricing sheet I found, they didn’t even bother listing the price. If you have to ask, you can’t afford it.
I used to run Server 2003 Standard as my desktop OS back in those days, and when I bought my first Athlon64 (3500+), I also got 4 GB RAM. Installed Server 2003, which saw all the RAM (as long as the BIOS settings were correct, otherwise that board only mapped 3200 MB), but ran into a problem with NVidia graphic driver – as soon as I installed it, screen wouldn't update properly: login screen showed, but didn't seem to react to keypresses, and the mouse pointer didn't move. I somehow figured out that the machine wasn't frozen, just the screen...
> if you are a server administrator, you don’t install random drivers for that hand-held scanner you bought at Best Buy from the bargain bin for $10.
Ah, I see you have not worked in smaller IT shops 🙂
I wouldn’t install random hand-held scanner drivers… but I would install the video drivers that were supplied with the hardware; it’s been some time since I’ve been happy with 800×600.
Edit: I since see that the default drivers now do 1024×768 but that still feels small these days unless you have a small monitor (particularly as you tend to want to run LCD monitors at optimal resolution).
Interestingly, XP SP2 with DEP/PAE had a 4GB limit including hardware reservations, while XP SP1 with PAE had a 4GB limit excluding hardware reservations.
Yes, I vaguely recall this was the case. SP1 license limited physical RAM, but SP2 license limited physical address space.
Sorry, but how the arguments about usage patterns of the consumer and the server markets explain the limitation of Windows Server 2003 Standard Edition?
It’s a server, right? Its administrators install as little as possible and have access to highly-skilled staff, right?
It still sounds like “if you paid a ton of money for top-tier hardware with some unfathomable for early 2000s amount of RAM, you’re probably a rich corporation doing serious business, so you might as well fork out for the premium OS”.
Most drivers at the time would immediately fail even with the /3GB boot option (e.g.: the Intel GMA video driver), so 4GB was just a nice distant concept. (Yes, I am aware that these are different features, but it all boils down to the drivers choosing to just ignore them all).
I believe that part of the “conspiracy theory” comes from the fact that this limit is enforced from the license system. Someone with the correct keys could genenerate a windows license that doesn’t have this limitation.
I used to do this on Windows 7. I had 6GB of RAM.
Reasons:
1) Between 6 and 8 (or possibly 10) GB RAM, 32 bit with PAE was *better* than 64 bit without PAE. This is due to pointer-based software needing almost twice as much RAM because pointers are twice the size.
2) SQL Server 2000 did not like x64 in the slightest. (Yeah sure, now the answer is VM that, but with 6GB, ha!)
3) Almost all software I had to run was 32 bit, so loading two copies of everything (one for Explorer, one for the actual software)...
Also I hear that a certain someone said “4GB ought to be enough for anyone” so they figured why try to disprove important people.
More seriously, could real Win11 even boot on 4GB without spending 90%+ of its time in swap? I suspect not, given how poorly it behaves on my one machine that regrettably only has 8gb.
The Snapdragon Developer Kit for Windows on ARM is a little desktop device with 4 GB RAM, it shipped with Windows 10 on ARM but surprisingly its processor made the list for Windows 11 support, so it could be upgraded to Windows 11 on ARM. It's a miserably slow experience. We're fortune the Windows DevKit 2023 shipped with 32 GB RAM instead, showing that Windows on ARM ran like any other edition of Windows and had just been hamstrung by devices without enough RAM. It turns out that when you give Windows 11 plenty of RAM, it's just noticeably sluggish...
That’s quite interesting – so on one hand there’s ABIs for 32-bit addresses with 64-bit instructions, and on the other hand there’s 32-bit Windows versions that can make use of more than 4 GB RAM. The transition from 32-bit to 64-bit never ceases to amaze.