# Integer overflow in the new[] operator

Raymond

Integer overflows are becoming a new security attack vector. Mike Howard’s article discusses some of the ways you can protect yourself against integer overflow attacks.

One attack vector he neglects to mention is integer overflow in the new[] operator. This operator performs an implicit multiplication that is unchecked:

```int *allocate_integers(int howmany)
{
return new int[howmany];
}
```

If you study the code generation for this, it comes out to

```  mov  eax, [esp+4] ; eax = howmany
shl  eax, 2       ; eax = howmany * sizeof(int)
push eax
call operator new ; allocate that many bytes
pop  ecx
retd 4
```

Notice that the multiplication by sizeof(int) is not checked for overflow. Somebody can trick you into under-allocating memory by passing a value like howmany = 0x40000001. For larger structures, multiplication overflow happens sooner.

Let’s look at a slightly longer example:

```class MyClass {
public:
MyClass(); // constructor
int stuff[256];
};
MyClass *allocate_myclass(int howmany)
{
return new MyClass[howmany];
}
```

This class also contains a constructor, so allocating an array of them involves two steps: allocate the memory, then construct each object. The allocate_myclass function compiles to this:

```  mov  eax, [esp+4] ; howmany
shl  eax, 10      ; howmany * sizeof(MyClass)
push esi
push eax
call operator new ; allocate that many bytes
mov  esi, eax
test esi, esi
pop  ecx
je   fail
push OFFSET MyClass::MyClass
push [esp+12]     ; howmany
push 1024         ; sizeof(MyClass)
push esi          ; memory block
call `vector constructor iterator`
mov  eax, esi
jmp  loop
fail:
xor  eax, eax
done:
pop  esi
retd 4
```

This function does an unchecked multiplication of the size, then tries to allocate that many bytes, then tells the vector constructor iterator to call the constructor (MyClass::MyClass) that many times.

If somebody tricks you into calling allocate_myclass(0x200001), the multiplication overflows and only 1024 bytes are allocated. This allocation succeeds, and then the vector constructor tries to initialize 0x200001 of those items, even though in reality only one of them got allocated. So you walk off the end of the memory block and start corrupting memory.

To protect against this, you can wrap an integer overflow check around the array allocation.

```template<typename T>
T* NewArray(size_t n)
{
if (n <= (size_t)-1 / sizeof(T))
return new T[n];
// n is too large - act as if we
// ran out of memory
return NULL;
}
```

Note: If you use a throwing “new”, then replace the “return NULL” with an appropriate throw.

You can now use this template to allocate arrays in an overflow-safe manner.

```MyClass *allocate_myclass(int howmany)
{
return NewArray<MyClass>(howmany);
}
```

This generates the following code:

```  push edi
mov  edi, [esp+8] ; howmany
cmp  edi, 4194303 ; overflow?
ja   overflow
mov  eax, edi
shl  eax, 10
push esi
push eax
call operator new
mov  esi, eax
test esi, esi
pop  ecx
je   failed
push OFFSET MyClass::MyClass
push edi
push 1024
push esi
call
call `vector constructor iterator`
mov  eax, esi
jmp  done
failed:
xor  eax, eax
done:
pop  esi
jmp  exit
overflow:
xor     eax, eax
exit:
pop  edi
retd 4
```

Notice the new code that checks for a possible integer multiplication overflow.

But how could you get tricked into an overflow situation?

The most common way of doing this is by reading the value out of a file or some other storage location. For example, if your code is parsing a file that has a section whose format is “length followed by data”, somebody could intentionally put an overflow-inducing value into the “length” field, then get somebody else to try to load the file.

This is particularly dangerous if the filetype is something that is generally considered “not dangerous”, like a JPG.