Why does the Windows calculator generate tiny errors when calculating the square root of a perfect square?

Raymond Chen

Raymond

Try this: Fire up the Windows calculator and ask it to compute √4 − 2. The answer is not zero. It’s −1.068281969439142e−19. “I thought the Windows calulator used an arbitrary precision arithmetic library. How come the answer isn’t exactly zero?”

If you read closely in that article, you’ll see that it says that the Windows calculator uses an arbitrary precision arithmetic library for basic operations but not for advanced operations, and square-root is called out explicitly as an advanced operation. Specifically, it uses an arbitrary precision arithmetic library for rational operations: addition, subtraction, multiplication, division, and raising to a positive integer power. Other operations use an extended precision library that produces 32 digits of precision.

Even if the value you are taking the square root of happens to be a perfect square, the calculator doesn’t realize that. It doesn’t try to calculate an integer square root first, and then fall back to the floating point version if the integer square root fails to produce an exact result. It just goes for the square root via the extended precision library.

What you’re seeing, therefore, is the usual vagaries of floating point arithmetic, compounded by the fact that the extended precision library does not have a custom algorithm dedicated to high-precision square roots. Instead, the extended precision library calculates the square root via the following identity:

{
error if x < 0
x =if x = 0
exp(½ ln x)if x > 0

The value of 1.99999999999999999989317180305609 is good to 20 digits of precision. Not as good as the 32 digits that a custom-written square root algorithm would have produced, but not too shabby.

(An error of one part in 10²⁰ lets you express the distance from earth to Alpha Centauri with an error of less than a millimeter.)

Update: The Windows Calculator no longer generates tiny errors when calculating the square root of a perfect square.

Raymond Chen
Raymond Chen

Follow Raymond   

0 comments

Comments are closed.