{"id":4273,"date":"2009-09-10T12:12:00","date_gmt":"2009-09-10T12:12:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vcblog\/2009\/09\/10\/linker-throughput\/"},"modified":"2019-02-18T18:45:47","modified_gmt":"2019-02-18T18:45:47","slug":"linker-throughput","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/linker-throughput\/","title":{"rendered":"Linker throughput"},"content":{"rendered":"

Hello, my name is Chandler Shen, a developer from the Visual C++ Shanghai team.<\/font><\/p>\n

We have made some changes in the upcoming Visual C++ 2010 release to improve the performance of linker. I would like to first give a brief overview of the linker and how we analyze the bottlenecks of current implementation. Later, I will describe the changes we made and the impact on linker performance.<\/font><\/p>\n

Our Focus<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

<\/font> <\/p>\n

We were targeting the linker throughput of large scale projects full build scenario because this scenario matters most in linker throughput scalability. Incremental linking and smaller projects will not benefit from the work I describe in this blog.<\/font><\/p>\n

Brief Overview of Linker<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

<\/font> <\/p>\n

Traditionally, what’s done by linker can be split into two phases: <\/font><\/p>\n

1.<\/font>       <\/span><\/span><\/span>Pass1: collecting definitions of symbols (from both object files and libraries)<\/font><\/p>\n

2.<\/font>       <\/span><\/span><\/span>Pass2: fixing up references to symbols with final address (actually Relative Virtual Address) and writing out the final image.<\/font><\/p>\n

Link Time Code Generation (LTCG)<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

If <\/font>\/GL (Whole Program Optimization)<\/font><\/a> is specified when compiling, the compiler will generate a special format of object file containing intermediate language. When linker encounters such object files, Pass1 becomes a 2-phase procedure. From these object files, the linker first calls into compiler to collect definitions of all public symbols to build a complete public symbol table. Then the linker supplies this symbol table to the compiler which generates the final machine instructions (or code generation). <\/font><\/p>\n

Debug Information<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

During Pass2, in addition to writing the final image, linker will also write debug information into a PDB (Program Database) file if user specifies <\/font>\/DEBUG (Generate Debug Info)<\/font><\/a>. Some of this debug information, such as address of symbols, is not decided until linking.<\/font><\/p>\n

Bottlenecks<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

In this section, I will show how we analyze some test cases to figure out bottlenecks of performance.<\/font><\/p>\n

Test Cases<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

To get an objective conclusion, four real world projects (whose names are omitted) differ in scale, including proj1, proj2, proj3 and proj4, were chosen as test cases.<\/font><\/p>\n

<\/a>Table <\/font><\/strong>1 Measurements of test cases<\/font><\/strong><\/p>\n\n\n\n\n\n\n\n
\n

 <\/p>\n

<\/font><\/font><\/strong><\/p>\n<\/td>\n

\n

Proj1<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

Proj2<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

Proj3<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

Proj4<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n<\/tr>\n

\n

Files<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

Total<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

55<\/font><\/p>\n<\/td>\n

\n

27<\/font><\/p>\n<\/td>\n

\n

168<\/font><\/p>\n<\/td>\n

\n

1066<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

.obj<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

4<\/font><\/p>\n<\/td>\n

\n

6<\/font><\/p>\n<\/td>\n

\n

7<\/font><\/p>\n<\/td>\n

\n

882<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

.lib<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

51<\/font><\/p>\n<\/td>\n

\n

21<\/font><\/p>\n<\/td>\n

\n

161<\/font><\/p>\n<\/td>\n

\n

184<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

Symbols<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

6026<\/font><\/p>\n<\/td>\n

\n

22436<\/font><\/p>\n<\/td>\n

\n

69570<\/font><\/p>\n<\/td>\n

\n

110262<\/font><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In Table 1, the number of “symbols” is the number of entries of the symbol table which is internally used by linker to store the information of all external symbols. It is noticeable that “proj4” is much bigger than others. <\/font><\/p>\n

Test Environment<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

Following is the configuration of the test machine<\/font><\/p>\n

·<\/font>         <\/span><\/span><\/span>Hardware<\/font><\/p>\n

o<\/font>   <\/span><\/span><\/span>CPU       Intel Xeon CPU 3.20GHz, 4 cores<\/font><\/p>\n

o<\/font>   <\/span><\/span><\/span>RAM      2G<\/font><\/p>\n

·<\/font>         <\/span><\/span><\/span>Software             Windows Vista 32-bit<\/font><\/p>\n

Results<\/p>\n

<\/font><\/font><\/font><\/span><\/h2>\n

To minimize the effect of environment, all cases were run for five times. And the unit of time is in seconds.<\/font><\/p>\n

In Table 2 and Table 3, it showed that for each test case, there is always one (usually the first, marked in red<\/span>) run which takes much longer than others.  While one run (marked in Green<\/span>) may take a much shorter run. This is because following two reasons<\/font><\/p>\n

l<\/font>  <\/span><\/span><\/span>OS will cache a file’s content in memory for next read (called prefetch on Windows XP, and <\/font>SuperFetch<\/font><\/a> on Windows Vista)<\/font><\/p>\n

l<\/font>  <\/span><\/span><\/span>Most of modern hard disks will cache a file’s content for next read<\/font><\/p>\n

\n

 <\/font><\/p>\n<\/p>\n

Comparing Table 2 with Table 3, we can notice that if \/debug is off, the time of Pass2 is much shorter. So it indicates that the majority of Pass2 is writing PDB files<\/font><\/p>\n

<\/a>Table <\/font><\/strong>2 Test result of Non-LTCG with \/Debug On<\/font><\/strong><\/p>\n\n\n\n\n\n\n\n\n\n
<\/font><\/strong><\/td>\n\n

Pass1<\/p>\n

<\/font><\/font><\/strong><\/p>\n<\/td>\n

\n

Pass2<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

Total<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n<\/tr>\n<\/thead>\n

\n

Proj1<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

1<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

4.437<\/p>\n

<\/font><\/font><\/span><\/b><\/p>\n<\/td>\n

\n

2.328<\/p>\n

<\/font><\/font><\/span><\/b><\/p>\n<\/td>\n

\n

6.765<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

2<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

0.266<\/font><\/p>\n<\/td>\n

\n

1.218<\/font><\/p>\n<\/td>\n

\n

1.484<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

3<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

0.265<\/font><\/p>\n<\/td>\n

\n

1.188<\/font><\/p>\n<\/td>\n

\n

1.453<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

4<\/p>\n

<\/font><\/font><\/b><\/p>\n<\/td>\n

\n

0.265<\/font><\/p>\n<\/td>\n

\n

1.219<\/font><\/p>\n<\/td>\n

\n

1.484<\/font><\/p>\n<\/td>\n<\/tr>\n

\n

\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n","protected":false},"excerpt":{"rendered":"

Hello, my name is Chandler Shen, a developer from the Visual C++ Shanghai team. We have made some changes in the upcoming Visual C++ 2010 release to improve the performance of linker. I would like to first give a brief overview of the linker and how we analyze the bottlenecks of current implementation. Later, I […]<\/p>\n","protected":false},"author":289,"featured_media":35994,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[19,25],"class_list":["post-4273","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cplusplus","tag-link-time-code-generation","tag-linker"],"acf":[],"blog_post_summary":"

Hello, my name is Chandler Shen, a developer from the Visual C++ Shanghai team. We have made some changes in the upcoming Visual C++ 2010 release to improve the performance of linker. I would like to first give a brief overview of the linker and how we analyze the bottlenecks of current implementation. Later, I […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/4273","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/289"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=4273"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/4273\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/35994"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=4273"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=4273"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=4273"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}