Quite a few folks have reported longer build times and noticed that the build details view has a lot more output in it, and as a result of the extra output being logged, the builds are slower.  Aaron has written a post about a hotfix to speed up your builds in the SP1 version of Team Build.  This hotfix, like many others for TFS, is available on the MSDN Code Gallery site.

KB958845: TFSBuild TargetsNotLogged property fails to cut down noisy builds